Most days, I lead a pretty happy existence. I write code. I meet with folks at HubSpot. I talk to fellow entrepreneurs and generally try to spread goodwill and love. Today, however, it sucks to be me.
Earlier this afternoon, some malicious user was able to compromise the security of Twitter Grader (one of our most popular Grader.com applications). The result was that this malicious user was able to post tweets
impersonating Twitter Grader users that had authorized our application. So, tweets went out from this malicious user, looking as if we had posted it using our user’s account information. Unsurprisingly, this caused a bunch of confusion as people wondered why the hell HubSpot would be posting spammy tweets from our application and breach our user’s trust. I spent much of the afternoon responding to people’s tweets, letting them know about the problem and that we were working on it. Everybody’s been super-understanding and patient.
There are three things I want to highlight in this whole fiasco:
#1. It was my fault. I developed Twitter Grader — and I’m the one that developed this particular feature that ended up getting hacked. I should have known better. I was an idiot.
#2. HubSpot is being super-paranoid about how we deal with the issue. We’re shutting down several of the grader.com applications (not just Twitter Grader) and will be reactivating them on completely new servers with increased security. This level of caution is likely overkill (and expensive), but it's the least we can do.
#3. OAuth is a very good thing. For those of you that don’t know what OAuth is, it’s what allows users to grant access to specific applications without revealing their username/password. Twitter supports OAuth. As such, Twitter Grader allowed users to “authorize” access. This is much better than asking users for their user name and password. Because of OAuth, although the malicious user was able to post to people’s twitter accounts, they never had access to the user’s account credentials. Given that many people use the same username/password on multiple websites, this could have been very dangerous. But, OAuth ensured that the problem was much more contained.
I’ve been working with the Twitter team — who have been just awesome. They detected the problem too, and helped shut down the application and contain the problem. I’ve had multiple emails from folks on the Twitter team today as we figure out appropriate next steps.
The application and associated keys were disabled as soon as we discovered there was a problem and as it stands, no additional action is needed for users. Your username and password were NOT compromised -- but it's never a bad idea to change your password periodically. Like today.
We are working on a permanent resolution which will allow Twitter Grader to be available publicly again. Until this work is complete, neither Twitter Grader nor the Twitter Grader API will be available. We expect this work to take a couple days - for updates, see the @Grader Twitter page.
By design, the HubSpot software applications are on different servers and systems from our free Grader.com tools. This attack did NOT affect the HubSpot software used by our 2,100 customers. Again, there is no impact on our paid product or customers.
My sincere apologies to all the users that were harmed by this security breach. This one really bothered me because all of you work hard to build trust, reputation and community on Twitter. These malicious tweets went out to your followers and compromised that trust. I really hate that I was responsible for that.
Thanks so much for your patience. We’ll continue to work hard to deserve your trust and goodwill.
And, to whoever it was that hacked in and sent out those tweets: That was not cool.
Ed 5:51 PM on February 11, 2010
Spend the resources to
hunt down the louse...
DJ Waldow 6:08 PM on February 11, 2010
Dharmesh -
Very sorry that happened to you guys today. Sucks. Big time. However, how you handled it so tremendously commendable and should be a lesson (case study?) for others. I actually saw many of the malicious tweets in my stream today. I then saw an "oops it is on us" type of message from @Hubspot which was then RT'd by my friend (and kick ass HS'er, Ellie M).
This post, especially #1 (It was my fault. I developed Twitter Grader — and I’m the one that developed this particular feature that ended up getting hacked. I should have known better. I was an idiot.) is why you guys rock.
Well played. Well played.
DJ Waldow
Director of Community, Blue Sky Factory
@djwaldow
Dave Doolin 6:17 PM on February 11, 2010
Not a good day. My hosting provider has been botnet ddos since at least 6am.
I'm totally out of business at least temporarily.
Hitzel 6:17 PM on February 11, 2010
How do we find out if we have been affected?
Josh McHugh 6:19 PM on February 11, 2010
Agree with DJ here: this post is a model for the early stage of dealing with compromised systems.
Now - get back in the game.
Barbara Nixon 6:19 PM on February 11, 2010
As a PR professor, I have learned to see through corporate "apologies" pretty well -- and I am delighted to see how you handled this one. I'll use it as an example of how to apologize & make things right in my Social Media for PR class for Georgia Southern University this semester.
Dharmesh Shah 6:21 PM on February 11, 2010
Thanks for the kind words, everyone. It means a lot to me.
@hitzel: The easiest way to see if a malicious tweet was posted on your behalf is to look at your twitter account and click on the "tweets" link (that shows tweets you've posted).
octopusgrabbus 6:24 PM on February 11, 2010
You folks would never make it in Congress. :)
1) You're clear.
2) You take responsibility.
3) You try to be helpful
4) You value your customers
If I didn't have a job, I'd be trying to come work for you, seriously.
Hitzel 6:29 PM on February 11, 2010
Many thanks, checked and no harmed is being done so far.
If any good to hear, I Belive on this great tool, and I think that it was great to create it!!
George 6:34 PM on February 11, 2010
I blogged about this incident here: http://blog.yapb.net/post/2010/02/11/Coming-In-Through-The-Back-Door.aspx .
Does the Twitter API allow an application to 'revoke itself' after it has done its deed? In Twitter Grader's case, once you send out the initial tweet and grade the user, there's no reason to retain access to the user's account. I believe that would have also solved this particular issue (again, if this feature exists).
J.R. Atkins 6:51 PM on February 11, 2010
Great response. Keep up the good work. I trust you and the company to do the right thing and you have.
Andrew Bleakley 6:57 PM on February 11, 2010
Thanks for the honesty - really appreciated, I wish more vendors had the same approach to issues
Josh McHugh 7:05 PM on February 11, 2010
Question: if there's a lawyer in the house, I'd love to hear you weigh in on whether this enlightened approach to addressing security slip-ups puts the company on slippery legal footing.
Unfortunately, it seems like this might be an area where decency and transparency are in conflict with legal prudence.
Don 7:35 PM on February 11, 2010
@Josh Admitting fault works very well in practice:
New York Times
Josh McHugh 7:56 PM on February 11, 2010
@Don - fantastic reference! Thanks so much for that link. A truly eye-opening story, and leads me to think that if transparency is a rising legal strategy in the medical malpractice world, it should have success in the business sphere, even beyond customer service cases.
JP 8:07 PM on February 11, 2010
Way to take responsibility and be totally honest. So impressed with this post. And I think that most people will be! Let the perfect cast stones, I for one have definitely had those "sucks to be me" days. We all screw up. This won't stop me from using your grader apps in the future! @paige1media
Mike Cane 8:27 PM on February 11, 2010
I was one person who had a spammy tweet sent out under his account. I surmised it was Twitter Grader within minutes, Revoked Access, spread the word, and also changed my password. I monitored propagation via Search Twitter and saw it was clamped down within 30 minutes. All in all, this was Not Fun.
Robbie Coleman 9:15 PM on February 11, 2010
Dharmesh,
Thank you for your honesty and quick action. I have not lost faith or trust with you or your products.
James 10:16 PM on February 11, 2010
Hey No Worries. It happens.
Just because everybody moves in one direction doesn't mean it's the right one.
Interesting how a OAuth could be seen as the weak link in the chain.
Don't work too hard or you will burn out. I can imagine how hard the shock was to find a breach so effecting.
Chris 10:40 PM on February 11, 2010
Very well said. If you lead like you blog I'd like to come work for HubSpot!
1day1brand 11:17 PM on February 11, 2010
I guess I wasn't affected because I'm on a Mac. (I kid!)
Thanks for letting us know what is happening and for being an outstanding CTO.
-- Axle
Alex Covic 7:47 AM on February 12, 2010
Way to handle a potential PR fail, I actually learned something today. Thank you Dharmesh and hope you don't get more opportunities to write blogposts like this in the future.
Sharon Mostyn 8:15 AM on February 12, 2010
Dharmesh, it is a pleasure to see a company handle this awful situation in the best possible manner. You set an example that other companies should follow:
1) take steps to resolve the problem
2) communicate those steps to your audience
3) develop future plans for process improvement to keep it from happening again
Nicely done. Now don't ever let it happen again ;)
Sharon Mostyn
@sharonmostyn
Amy Herndon 12:59 PM on February 12, 2010
#1 It's all good! Being able to temporarily shut down OAuth Connections is what OAuth is all about. If you didn't care about our safety you wouldn't have used it.
I quit twitter 2 weeks ago when this started happening to me because they didn't have humility and honesty with this like you did.
Please let us and twitter know if you think this could be a common flaw in other apps.
Gareth Molyneux 3:42 PM on February 12, 2010
I'll be honest - never heard or seen your app before, but I follow @safety on twitter, who posted the link to your blog.
Just wanted to right to commend your honesty and open-ness with this. You've not tried to mitigate it, you've not tried to hide it. You've held your hands up, and said "It was me" - which there is no-one in this world who can blame you.
If you were based in the UK, I'd love to buy you a beer!!
Lionel 1:36 PM on February 18, 2010
Well said, well done and good luck. Wish our world leaders where like this!
tibor 2:03 AM on February 22, 2010
Well, it's nice to apologize, but sincerely hearing a coder say "I'm an idiot, I should have known better" is really really scary. I wanted to subscribe but I'll go somewhere else - where coders know better. Sorry.
Reputation Management 2:58 PM on March 03, 2010
It's easy to cut you guys slack because you are always willing to give first before you give.
-Glen Woodfin