Rackspace Outage
November 13th, 2007
I was pretty surprised when I saw the lights go off on my Rackspace servers in the DFW data center, in Dallas.
The outage was covered by Laughing Squid, and made it onto a lot of big tech news sites such as TechCrunch, GigaOm, Valleywag, and O’Reilly Radar. 37 Signals and other well known web companies got wiped off the face of the earth.
It was embarrassing for me, since I just handed over a new web app to the customer for testing, which relied on a web service running on one of my DFW servers.
I signed up with Rackspace a couple of months ago, and was impressed by the confidence with which they spoke of their 100% guaranteed uptime. “Not 99.99999999999?” I asked. 100%, they assured me.
Down for three hours? That puts my uptime to date at roughly at 99.791%. Sorry Rackspace, but my Nintendo Wii has a better uptime than that. If you want to continue touting your “fanatical support”, you will have to do better.
And as for the apology:
We cannot promise that hardware won’t break, that software won’t fail or that we will always be perfect. What we can promise is that if something goes wrong we will rise to the occasion, take action, resolve the issue and accept responsibility. If you are a Rackspace customer and don’t think we’ve lived up to this promise at anytime during the outage, please let your Account Manager know.
You forfeit the right to this excuse when you promised 100% uptime. Why do you think everyone else offers a bunch of 9’s? If it hadn’t been a lorry crashing into some transformer, it would have been a giant meteorite. This is God’s way of telling you to listen to your sys admin, and not your marketing guy.
p.s., I will accept a free iPod touch as a gesture of good will.
Update: Got the phone call from Rackspace ~1 hour after writing this. That is fanatical support, since whoever read this post had to find out who I was and get in contact with my account manager in that time. I’ll just clarify that, having been there myself in the past as a sys admin, and also working for a broadband provider, I know full well that these things happen. I am aware that the real test is the response when something like this does happen, and it looks like Rackspace did well to get everything sorted quickly. My issue is with the (now mathematically impossible) 100% uptime claim, which no right-minded service provider should give, and Rackspace now no longer have the right to maintain.
p.p.s., the iPod touch was just a joke
del.icio.us
digg it
November 13th, 2007 at 14:23 pm
Their apology is backhanded and awkward, obviously written by a PR muppet, but I think people should bear in mind that this must have been a very rare occurence indeed. I’ve never heard of RackSpace going down before.
You must be paying though the nose for that James? You should be hosting in Cork boy! *koff*
adam
November 13th, 2007 at 15:15 pm
Ya, Rackspace is expensive, but you’re paying for the uptime and great support (albeit slightly OTT at times). I have a nice server from Hetzner too and it is dirt cheap… but I wouldn’t like to host any critical apps on it because I’d be getting a call at 3am from an angry American about 50% packet loss.
I’d love to have some rack space in Cork - I would be in CIX in a flash if we had any need to be based in Ireland. Everything we do at the moment is for companies abroad. There is a server room in the WebWorks building where my office is too… would be nice to have some equipment in there, but that probably costs a bomb and I don’t know what the connection or redundancy is like.
November 13th, 2007 at 15:24 pm
Did you get a deal on WebWorks? I looked at that last year and nearly wet myself laughing when I found out their pricing per square foot.
November 13th, 2007 at 15:30 pm
Just bit the bullet and squeezed into the smallest office I could get in the place
It is expensive (did you see the price of a car parking spot?) but is a cool building in a central location, flexible lease, 24/7 access, etc. There’s a good chance you could get a deal, since the building is only about half full.
November 14th, 2007 at 17:38 pm
I’ll be doing 12hr shifts in CIX in the next couple of months - we’re eating our own dogfood at the outset, and running the NOC on shift - so I don’t think it’d be very good value for money no matter what they offer me.
Swap you a tour of CIX for a tour of Webworks?
November 14th, 2007 at 17:51 pm
Are ye going to have someone on site 24/7? Of course, there is always an open door down at the Webworks (except when it’s locked… which is most of the time… but it can be swiftly opened). I have been trying to work out of we have LOS to the CIX mast… I’m fairly sure you can see it from the roof, but I think Connolly hall or the City hall is blocking it from my office.
November 14th, 2007 at 17:57 pm
Yeah, us for the first couple months, and other people when we hit our first target. I’ve no idea about LOS tbh, we can’t see the entire city bowl afaik.