zlacker

>>evanwe+(OP)
This isn't shocking - Twitter was notorious for being held together with Scotch tape technically.

Honestly this hands-on approach is an impressive example of doing things that don't scale.

>>gringo+Hw
I found it amusing that Twitter was Rails' biggest advertisement. Everyone wanted to use Rails but Twitter turned into a franken app with different stacks to keep it running

>>gianca+1x
From 2008:

Scaling is fundamentally about the ability of a system to easily support many servers. So something is scalable if you can easily start with one server and go easily to 100, 1000, or 10,000 servers and get performance improvement commensurate with the increase in resources.

When people talk about languages scaling, this is silly, because it is really the architecture that determines the scalability. One language may be slower than another, but this will not affect the ability of the system to add more servers.

Typically one language could be two or three, or even ten times slower. But all this would mean in a highly scalable system is that you would need two or three or ten times the number of servers to handle a given load. Servers aren't free (just ask Facebook), but a well-capitalized company can certainly afford them.

http://www.businessinsider.com/2008/5/why-can-t-twitter-scal...

>>scarfa+0L
Yes, well, that's a nice idea in theory. In practice, you could get over 10x (sometimes 100x) the rps off a box running the new, JVM-based services vs their Rails-equivalents. Orders of magnitude probably matter a little less when you're well-funded and have hundreds of servers, but when you are thinking about trying to go public and your bottom line is being scrutinized and you have 10's of thousands of servers, it starts to matter.

>>sulam+VO
That's exactly what he said.

but a well-capitalized company can certainly afford them

But these days when you don't have to buy servers and make a long term capital commitment and you can use something like AWS, if you have a scalable but not efficient architecture and you have the faith of the investors, you can get enough servers to get you over the hump temporarily, slowly start replacing the most performance sensitive part of your architecture and then scale down.

Look at what HN darling Dropbox did, they bootstrapped on AWS, got big and then when the time was right, they moved to a cheaper architecture - off of AWS and built their own infrastructure.

>>scarfa+5R
And this is exactly what Twitter did, and how Twitter replaced Ruby and Rails with the JVM.

>>sulam+D72
I doubt they just "replaced" Ruby on Rails with the JVM without making any architectural changes based on the lessons they learned from thier first implementations.

>>scarfa+Ya2
Did I say that? We (I spent 4.5 years there, starting with the writing of the first service extracted from the monorail and left shortly before the 'twitter' repo was deleted) absolutely went through a huge architectural transition, arguably multiple transitions. The biggest was the breakup of the monolithic Rails-based application into microservices running on the JVM. These services generally scaled ten to one hundred times better than the code they replaced. (By "scaled" here I specifically mean that RPS was 10-100X higher per physical machine for services on the JVM as compared to the RPS the Rails stack could handle before falling over).

>>sulam+WX2
I was replying to this:

And this is exactly what Twitter did, and how Twitter replaced Ruby and Rails with the JVM

In the context of my original post where the contention was that languages don't scale that architectures do. Your post was that it was exactly what you did - replaced Ruby with Java. Not that you replaced Ruby with Java and rearchitected the entire stack - exactly what my original post said waa the problem with Twitter - the architecture.

>>scarfa+Z23
Well, that's true to a degree, but only to a degree. If I wrote my database in Ruby, it would be slow compared to the same database written in C++ (assuming equivalent competency in the developers and equivalent architecture). Even a database written in Java benchmarks slower than the same database with the same architecture written in C/C++. Of course architectural changes can make further improvements.

To the point of Twitter, what we _didn't_ do, despite a lot of Ruby expertise on the team, is write a lot of microservices in Ruby. The reason for that is that I don't think you can get the same RPS out of a Ruby service that you can out of a JVM service, all else being equal. In fact HTTP benchmarks for various platforms show this, if you bother to look.

>>sulam+W83
I'm not disagreeing with you. Looking on the outside, Twitter had two issues.

1. Twitter wasn't built on a scalable architecture

2. Ruby didn't use resources efficiently -- it was slower than other stacks.

If Twitter had been scalable, even if it were 10x slower than Java, you could throw 10x the number of servers at it until you optimized the stack then reduce the number of servers needed and the customers would have been none the wiser. Of course the investors wouldn't have been happy. Well at least in today's world. I don't know what the state of cloud services were in 2008. Then you could focus on efficiency.

But since Twitter wasn't scalable, you had to fix the stack while the customers were effected. I'm almost sure even in 2008, with the growth of Twitter they could have gotten the capital to invest in more servers if they needed them.

It's not completely analogous but Dropbox is a good counterexample. Dropbox was hosted on AWS at first. Dropbox never had to worry about running out of storage space no matter how big it grew (it had a scalable architecture) but for their use case, they weren't as efficient (ie cost not computer resources). Their customers were never affected by their lack of efficiency because they could operate at scale. They had breathing room to re-architect a more efficient solution.

>>scarfa+yd3
These are totally different problems, though. Dropbox is a trivial scaling exercise compared to Twitter. (Some Dropbox engineers are going to be all up on me now, but it's simpler by far to shard than Twitter was -- and yes some functionality in Dropbox is probably harder to shard, but the core use case of Twitter generated hot spots by definition).

FWIW, Twitter did what you're describing, we had 4 or 5 thousands hosts running the Ruby stack at its peak. Unicorns and Rainbows, oh my. Then it started shrinking until it shrank to nothing. That period was actual the relatively stable period. The crazy period, the one that I wasn't there for, was probably impossible to architect your way out of because it was simply a crazy amount of growth in a really short amount of time, and it had a number of ways in which unpredictable events could bring it to its knees. You needed the existing architecture to stay functional for more than a week at a time for a solid 6 months to be able to start taking load off the system and putting it onto more scalable software.

Any startup would be making a mistake to architect for Twitter scale. Some startups have "embarrassingly parallel" problems -- Salesforce had one of these, although they had growing pains that customers mostly didn't notice in 2004 timeframe. Dropbox is another one. If you're lucky enough to be able to horizontally scale forever, then great, throw money at the problem. Twitter, at certain points in its evolution (remember AWS was not a thing) was literally out of room/power. That happened twice with two different providers.