COMMENTS
Spend the resources to
hunt down the louse...
Dharmesh -
Very sorry that happened to you guys today. Sucks. Big time. However, how you handled it so tremendously commendable and should be a lesson (case study?) for others. I actually saw many of the malicious tweets in my stream today. I then saw an "oops it is on us" type of message from @Hubspot which was then RT'd by my friend (and kick ass HS'er, Ellie M).
This post,
especially #1
(It was my fault. I developed Twitter Grader — and I’m the one that developed this particular feature that ended up getting hacked. I should have known better. I was an idiot.) is why you guys rock.
Well played. Well played.
DJ Waldow
Director of Community, Blue Sky Factory
@djwaldow
Not a good day. My hosting provider has been botnet ddos since at least 6am.
I'm totally out of business at least temporarily.
How do we find out if we have been affected?
Agree with DJ here: this post is a model for the early stage of dealing with compromised systems.
Now - get back in the game.
As a PR professor, I have learned to see through corporate "apologies" pretty well -- and I am delighted to see how you handled this one. I'll use it as an example of how to apologize & make things right in my Social Media for PR class for Georgia Southern University this semester.
Thanks for the kind words, everyone. It means a lot to me.
@hitzel: The easiest way to see if a malicious tweet was posted on your behalf is to look at your twitter account and click on the "tweets" link (that shows tweets you've posted).
You folks would never make it in Congress. :)
1) You're clear.
2) You take responsibility.
3) You try to be helpful
4) You value your customers
If I didn't have a job, I'd be trying to come work for you, seriously.
Many thanks, checked and no harmed is being done so far.
If any good to hear, I Belive on this great tool, and I think that it was great to create it!!
I blogged about this incident here: http://blog.yapb.net/post/2010/02/11/Coming-In-Through-The-Back-Door.aspx .
Does the Twitter API allow an application to 'revoke itself' after it has done its deed? In Twitter Grader's case, once you send out the initial tweet and grade the user, there's no reason to retain access to the user's account. I believe that would have also solved this particular issue (again, if this feature exists).
Great response. Keep up the good work. I trust you and the company to do the right thing and you have.
Thanks for the honesty - really appreciated, I wish more vendors had the same approach to issues
Question: if there's a lawyer in the house, I'd love to hear you weigh in on whether this enlightened approach to addressing security slip-ups puts the company on slippery legal footing.
Unfortunately, it seems like this might be an area where decency and transparency are in conflict with legal prudence.
@Don - fantastic reference! Thanks so much for that link. A truly eye-opening story, and leads me to think that if transparency is a rising legal strategy in the medical malpractice world, it should have success in the business sphere, even beyond customer service cases.
Way to take responsibility and be totally honest. So impressed with this post. And I think that most people will be! Let the perfect cast stones, I for one have definitely had those "sucks to be me" days. We all screw up. This won't stop me from using your grader apps in the future! @paige1media
I was one person who had a spammy tweet sent out under his account. I surmised it was Twitter Grader within minutes, Revoked Access, spread the word, and also changed my password. I monitored propagation via Search Twitter and saw it was clamped down within 30 minutes. All in all, this was Not Fun.
Dharmesh,
Thank you for your honesty and quick action. I have not lost faith or trust with you or your products.
Hey No Worries. It happens.
Just because everybody moves in one direction doesn't mean it's the right one.
Interesting how a OAuth could be seen as the weak link in the chain.
Don't work too hard or you will burn out. I can imagine how hard the shock was to find a breach so effecting.
Very well said. If you lead like you blog I'd like to come work for HubSpot!
I guess I wasn't affected because I'm on a Mac. (I kid!)
Thanks for letting us know what is happening and for being an outstanding CTO.
-- Axle
Way to handle a potential PR fail, I actually learned something today. Thank you Dharmesh and hope you don't get more opportunities to write blogposts like this in the future.
Dharmesh, it is a pleasure to see a company handle this awful situation in the best possible manner. You set an example that other companies should follow:
1) take steps to resolve the problem
2) communicate those steps to your audience
3) develop future plans for process improvement to keep it from happening again
Nicely done. Now don't ever let it happen again ;)
Sharon Mostyn
@sharonmostyn
#1 It's all good! Being able to temporarily shut down OAuth Connections is what OAuth is all about. If you didn't care about our safety you wouldn't have used it.
I quit twitter 2 weeks ago when this started happening to me because they didn't have humility and honesty with this like you did.
Please let us and twitter know if you think this could be a common flaw in other apps.
I'll be honest - never heard or seen your app before, but I follow @safety on twitter, who posted the link to your blog.
Just wanted to right to commend your honesty and open-ness with this. You've not tried to mitigate it, you've not tried to hide it. You've held your hands up, and said "It was me" - which there is no-one in this world who can blame you.
If you were based in the UK, I'd love to buy you a beer!!
Well said, well done and good luck. Wish our world leaders where like this!
Well, it's nice to apologize, but sincerely hearing a coder say "I'm an idiot, I should have known better" is really really scary. I wanted to subscribe but I'll go somewhere else - where coders know better. Sorry.
It's easy to cut you guys slack because you are always willing to give first before you give.
-Glen Woodfin