Monday, May 12, 2008

Blaine Cook Writes First Blog Post Since Twitter... About Scaling

Today, Blaine Cook, formerly Twitter’s chief architect, writes his first blog post since leaving Twitter. It certainly seems like he has some stuff to get off his chest.

The gist of the piece is that languages don’t scale, architectures do. It seems clear why Blaine might want to say something about scaling, since it is clearly the number one issue at Twitter. And he did take some heat for the problems there. And in some corners of the Web, Ruby, the language that Twitter runs on, has taken even more heat.

Blaine's arugment: Scaling is fundamentally about the ability of a system to easily support many servers. So something is scalable if you can easily start with one server and go easily to 100, 1000, or 10,000 servers and get performance improvement commensurate with the increase in resources.

When people talk about languages scaling, this is silly, because it is really the architecture that determines the scalability. One language may be slower than another, but this will not affect the ability of the system to add more servers.

Typically one language could be two or three, or even ten times slower. But all this would mean in a highly scalable system is that you would need two or three or ten times the number of servers to handle a given load. Servers aren't free (just ask Facebook), but a well-capitalized company can certainly afford them.

The problem comes when your architecture is such that you can’t just add more servers. While Blaine does not discuss this, the primary reason things don’t scale has to do more with the cost of data access. Databases are almost always your bottleneck, because all your data typically needs to be stored in some central repository.

So how you architect your data storage and access will determine your scalability. For example, do you use RAM based caching like memcached to improve performance and limit the need to read the database? If so, is your caching architecture good enough to limit most reads from the database, or just a few? These are the kinds of architectural decisions that will determine system performance.

In Twitter's case, there is zero chance that the problems there are in any way related to their language. It is likely that there are architectural challenges which come from the fact that it is very hard to cache a Twitter data request since no two people ever get the same data. And even for a given user, the data requests change quickly since users are always receiving tweets. This is a hard, though not unsolvable problem that requires a very specialized caching architecture. Eran Hammer-Lahav, has done some interesting work in this area and talks about it in an extensive blog post.

The bottom line is languages don’t kill scaling, programmers do. As such, Blaine's piece, while sounding a bit defiant, might really be read more like a mea culpa. Though, to be fair, despite all the chatter and criticism, scaling Twitter is indeed a non-trivial problem.

13 comments:

  1. Seriously? Blaine would have been smarter to have blamed Rails than the architecture. WHO BUILT THE ARCHITECTURE? Blaming the framework puts the blame elsewhere, but blaming the architecture admits you didn't have a clue, either in developing it if you designed it, or in fixing it if you didn't. If, as an employer, you saw the issues with Twitter and then that blog post, would you hire him?

    ReplyDelete
  2. Yes, scalability is more important than speed, but speed is also very important because with a fast language, you shouldn't need more servers, and thus you don't need to worry about scaling as much. The difference between using Rails and Python is huge, because Python would require less servers and thus less cost. Of course, you wouldn't want to code your web application in C just because that would be extremely difficult to maintain and scale, but it's important to pick a language that is somewhere between easy to maintain and fast. Python would win this for me, because it's an incredibly clean language while also being really fast. This is also why Rails is taking a lot of crap right now, because it's SLOW, which forces the owner to buy more servers.

    Again, I'm not saying speed is more important than scalability, I'm just saying that it's more important than you and Blaine are giving it credit for. If you are even a slightly decent programmer, you will be able to scale in any high-level language, so why not pick one that's also fast?

    ReplyDelete
  3. @sasha one reason: because you have an installed base of tons of code that runs your site. Rewriting projects in another language is usually doomed to failure.

    ReplyDelete
  4. Well if you already have a ton of code, then you're right. But I was more referring to starters picking out a language. In that case, you should take speed into account.

    ReplyDelete
  5. Actually, a lot of code is rewritten.

    No architecture scales arbitrarily. Moreover, if you design for 100k processors, you're likely to get lose to some mope who designed for 100. Sure, you may get to come in and re-architect his system when it grows sufficiently, but maybe it doesn't get that big. Either way, he wins.

    ReplyDelete
  6. Throw more hardware at crappy software? Yeah... Props to Sasha.

    ReplyDelete
  7. I love my career - The capacity of data storage is always the bottleneck in every system development. I agree with Mr. Blaine - "The bottom line is language don’t kill scaling, programmers do". If the storage is uncapable of rapid data accessing, it's either the database or the design is wrong.

    ReplyDelete
  8. I am so fascinated with the entire content of your blog, from the material to the theme. I really had a great time reading this post.
    door furniture

    ReplyDelete
  9. I agree with fourlittlebees Blaming the framework puts the blame elsewhere.

    Joan

    ReplyDelete
  10. Quite agree with this article, I find it reliable regarding scalability and speed. Though we all have different views regarding these area, I find it amazing how much I learned from the content of this site and to all the comments I read. Not a bad idea to make this site as one my favorites. Thanks for sharing. ultrasound technician

    ReplyDelete
  11. I just finished my site at Building site and for me, everything is suck when your girlfriend dumped you where you tend to be serious already.

    ReplyDelete
  12. Thanks for a marvelous posting! I really enjoyed reading it, you can be a great author.I will be sure to bookmark your blog and will come back down the road. I want to encourage you to definitely continue your great writing, have a nice day ahead! lose weight fast

    ReplyDelete
  13. Everything is suck when you found out that your backlinks is broken.

    ReplyDelete