S3 Failure Raises Questions About Cloud Design

I watched with interest this weekend as Amazon S3 went down yet again, and I thought to myself, "there but for the grace of God go I."

My company is currently developing a cloud based data service called KloudShare. And though KloudShare is basically unrelated to what S3 does, and probably has more similarity to Google's Big Table or Amazon Simple DB,  they are all still data services, and so it got me thinking about how one might architect systems to avoid such messes. Because while people might forgive Amazon, I don't think a tiny startup like mine is going to have the same latitude Amazon does.

First, looking at Amazon's computing service called the Elastic Computing Cloud (EC2) is probably instructive as we have been using Amazon EC2 and have had an instance running for well more than six months (probably approaching a year) without a failure. And so what is clear is that there are ways to design really massive systems that do not have a single choke point. Amazon gets it right with EC2 and less so with S3.

The big question is why S3 is structured in such a way that so many problems they have seem to bring the entire system down. Invariably, things fail. You cannot avoid it no matter how smart you think you are. But what you can often do is limit the collateral damage of failure by compartmentalizing your design. Clearly EC2 instances are quite compartmentalized.

While the data in S3 is clearly stored across separate distinct systems, I would imagine that security and access rules, and perhaps other elements are centralized, though I really have no idea for sure what their internal architecture is like. What is clear though is that as all of us in the cloud computing business think about our designs, sharding or federation of all services within the cloud into separate operational silos is critical. As best we can, we must avoid allowing one failure somewhere to bring down the whole system. Strategies to keep failure localized are critical.

One of the keys in our design has been replication, and eschewing what is known in the database world as normalization. In a normalized database design you are very careful not to store data in more than one place. You want to reference data in its existing place rather than replicate it everywhere. We avoid the principles of normalization because it is impossible to provide massively scalable systems that are normalized. But what we had never considered is that our "anti-normalization" design principle also relates to stability of design.

I think what the S3 issue is demonstrating is that distributed design is critical not just for performance but for reducing the impact of failure. Of course I am not saying that we have figured all of this out yet, and without more thought I suspect we too still have vulnerabilities in our design of the type that brought Amazon down. And so this is not an attack on Amazon but as I see it a teachable moment for all of us working on how to bring the real vision of the cloud to the world. And while it is, of course, impossible to avoid all centralized services in a cloud architecture, clearly Amazon is demonstrating the critical importance of limiting your dependence on them.