Cloud Server Load Testing
Posted by Max Dunn Sat, 07 Jan 2012 20:10:00 GMT
I have been working on the RealMealz site and am now looking for a suitable host server. Currently, RealMealz is in beta and has just light use, but we expect that once released the use will go up dramatically, so we want to be able to easily scale. I looked at Amazon EC2, Heroku and RackSpace Cloud Servers and the table below summarizes my findings:
Test: Login, hit Discover page 5 times
Use a 5 second ramp up
Average Discover page load time (Apache + Passenger unless otherwise indicated)
|
Server |
1 user |
5 user |
10 user |
Cost/Mo |
|
EC2, Micro |
2s |
20s |
40s |
$9 |
|
EC2, Micro, Thin |
2s |
23s |
42s |
$9 |
|
EC2, 3-Micros, balanced. Apache/Passenger |
2s |
6s |
9s-18s (40s max) |
$44 |
|
EC2, hi-cpu medium |
2s |
4s |
6s |
$75 |
|
EC2, hi-cpu medium, Thin |
2s |
5s |
10s |
$75 |
|
BlueBox |
2s |
2s |
5s |
$79 |
|
Heroku, 1 dyno, Thin |
2s |
3s |
6s |
$15 |
|
Heroku, 2 dynos, Thin |
2s |
3s |
4s |
$50 |
|
Heroku, 3 dynos, Thin |
2s |
2s |
6s (20 users!) |
$86 |
|
Rackspace |
2s |
7s (max 66s) |
40s+ |
$22 |
|
Rackspace, Thin |
2s |
4s |
9s |
$22 |
|
Rackspace, Unicorn |
2s |
4s |
8s |
$22 |
Notes.
1. Users are concurrent users
2. I selected the Discover page since this is our most CPU intensive page. This test is not that realistic in actual use, but serves as a worse case example.
3. The Rackspace server had only 512Mb of memory and for the 10 user case for Apache it started swapping out. If I was using their next size up with 1Gb of memory, the cost would have been $44 per month but wouldn’t have suffered this performance issues as drastically.
4. The Amazon Micro instances provide bursts up to 2 ECUs, but if used at a high level for more than a few seconds, get throttled back greatly for about 15 seconds. This is why the 10 user case was so high, even for the 3 instance, load balanced case.
4a. The Amazon servers are a little underpowered. Even the hi-cpu instance that provides 5 ECUs is not all that impressive. My BlueBox server which has 4 VPSes on the server performed better.
5. It wasn’t clear how the Rackspace Cloud Servers could be easily scaled since RackSpace doesn’t seem to provide a load balancer for them.
6. Amazon’s EC2 instances are a bit of a pain to scale. You can’t add an extra instance to an existing load balancer, so you need to delete the current one then create a new one. There is an autoscaling feature, but this only works from the command line and I didn’t test it.
7. Heroku offered pretty amazing performance and was super easy to scale by just typing “heroku scale web=3”. The prices include $15/month for the larger database, although under 5Mbs is free.
8. With 3 dynos, Heroku was able to handle 20 concurrent users where the other tests were only with 10.
9. A disadvantage of Heroku is that it doesn’t offer shell access and there is no way to access the database natively. We do have some table manipulation built into our program, but if there are major changes or corruptions, we will need to export the data, fix it, and then re-import.
In the end, we are going with Heroku because it has great performance, easy to scale and is low cost.