I decided to find out about Symfony Framework (note, that title says "High Performance Framework") performance by benchmarking my own site built with Symfony.
I ran the benchmark with ab. I was mainly interested about how much time server spends processing the request, i.e. code execution time, so I took the "processing" numbers from ab output. I also wrote a simple Python benchmarking tool called pymjolnir to run the benchmark on multiple concurrent users and output the results as csv data.
1st benchmark: Page with one database query
Symfony is frequently coupled with Doctrine ORM, so my first interest was to gauge their combined performance. I have a site called 13in12.com, that I use to share some web application ideas. I ran the test on the main page, where project ideas are listed.
Note, that these numbers do not include connection times. Total response time per request is connection time (not shown) + processing time below.
| Concurrency | Mean processing time (ms) | Standard deviation (ms) |
|---|---|---|
| 8.00 | 83.00 | 58.00 |
| 16.00 | 78.00 | 17.30 |
| 32.00 | 126.00 | 24.30 |
| 64.00 | 296.00 | 94.10 |
| 128.00 | 600.00 | 339.30 |
| 256.00 | 3066.00 | 6340.70 |
For this server, processing times with 32 users are still fairly quick, on average 126ms. After 64 concurrent users they increase sharply.
2nd benchmark: Static page
I was also interested how much latency Doctrine adds to processing times, so I ran the benchmark again on a page with no database queries.
| Concurrency | Mean processing time (ms) | Standard deviation (ms) |
|---|---|---|
| 8.00 | 80.00 | 53.50 |
| 16.00 | 72.00 | 58.10 |
| 32.00 | 98.00 | 23.30 |
| 64.00 | 209.00 | 80.70 |
| 128.00 | 608.00 | 366.00 |
| 256.00 | 2824.00 | 5204.10 |
Doctrine queries seem to add a bit to processing times, but not as much as I expected.
Conclusion
My anecdotal experience has been, that with proper caching Symfony can deliver pages with fairly reasonable processing times. Average processing times with <10 concurrent users is ~80ms. This is in sharp contrast to Web Framework Benchmarks at http://www.techempower.com/benchmarks/, though the latency number reported there is just a single number, with no indication of concurrency. I suppose they're just averaging the numbers with 8-256 concurrent users, which is kind of silly.
Processing times in both benchmarks seem to increase sharply after 128 concurrent users. There was also the possibility, that I was saturating something on my side. For comparison, I also ran the same benchmark against google.fi:
| Concurrency | Mean processing time (ms) | Standard deviation (ms) |
|---|---|---|
| 8.00 | 47.00 | 8.70 |
| 128.00 | 252.00 | 152.80 |
| 256.00 | 742.00 | 569.00 |
I doubt Google will mind serving 256 concurrent requests with average processing time of 742ms, so it seems obvious I was maxing out something here. What exactly it was, I don't know. I have a fairly quick LAN connection, and amount of data transferred was under 1Mb/s. CPU load was also under 1.
Server & configuration details
Server specifications:
- CPU: Intel(R) Core(TM) i5-2415M CPU @ 2.30GHz
- Memory: 8Gb
Environment:
- OS: Debian Wheezy
- Web server: nginx/1.4.4 + php-fpm
- Reverse proxy: None
- PHP: 5.4.24, /w APC