Monday, 27 April 2015

TechEmpower FrameworkBenchmarks round 10: The Python results

TechEmpower FrameworkBenchmarks round 10 results are available.

What is TechEmpower FrameworkBenchmarks ?

As explained on their website:
This is a performance comparison of many web application frameworks executing fundamental tasks such as JSON serialization, database access, and server-side template composition. Each framework is operating in a realistic production configuration. Results are captured on Amazon EC2 and on physical hardware. The test implementations are largely community-contributed and all source is available at the GitHub repository.
You can read the technical description of each test on their website.

Goal of this blogpost

I'm concentrated on the results for Python ecosystem, and especially for AsyncIO+aiohttp.web+API-Hour, because it's the first time that an independent organization benchmarks publicly AsyncIO on several use cases with a macro-benchmark that implies a complete system stack (Network+Linux+Databases), closer to a production environment.

/!\ /!\ /!\ Warning - Warning - Warning /!\ /!\ /!\

As explained several times, do not follow any benchmarks results to immediately decide on the architecture of your applications, especially micro-benchmarks:
  1. You must understand a minimum what is tested to be sure you are in the same use case as the benchmark.
  2. Benchmark by yourself with your needs: It will help you to better understand the interactions between layers and maybe to discover that your bias about efficient architecture are false.
  3. Benchmark your complete stack: Sometimes, you can have bigger bottlenecks effects in your complete stack that in your Python code: you can optimize a big loop with Cython, if your database schema contains a lot of data and sucks, you'll wait more your database than in your loop.

The good results

Note: Click on the images to see directly the results on the FrameworkBenchmarks website.

IMPORTANT: FrameworkBenchmarks isn't only for Python, you can compare Web frameworks in other programming languages. Check the main website:

With no surprises, databases benchmarks are good for AsyncIO+aiohttp.web+API-Hour:

Data updates


Multiple queries

The average result

With less interactions with database, API-Hour results are in the average of others Web frameworks.

Single query

The bad results

The benchmarks without databases interactions are bad.

JSON serialization


Why theses benchmarks are less good that them provided on my blog ?

The difference of results between my DB benchmarks and DB benchmarks in FrameworkBenchmarks can be explained because, in my benchmarks, I use a bigger payload and a SQL query more complex that in FrameworkBenchmarks.
I did this because, as explained above, it is important to me to be as close as a production scenario when benchmarking. A realistic SQL query takes more time than a trivial SELECT, and the framework therefore has a serious advantage if it can manage several other things while waiting for the query result. Hence AsyncIO can better show its power there.
For round 11, TechEmpower should add a more realistic test, with a bigger payload and more DB interactions.


Nevertheless, for a first time participation, I'm really happy of the results. We have now one month to improve results for round 11, if you want to help, send me an e-mail.

My goal in theses benchmarks isn't only to find bottlenecks in the Python stack I’m using everyday, but also to make sure that this Python stack continues to be efficient over the time.
Most stacks become bloatwares over the time.

Future improvements

To increase a bit performance in the slowest test, the HTTP pipelining in aiohttp should be improved, but as it stands today, this might have positive effects in some cases (such as multiplexed HTTP requests), while bringing negative side effect in some other cases (such as old-fashioned HTTP  requests or keep-alive requests).

But even with a better HTTP pipelining support, this test won’t probably give very good results, as pure JSON provider.

I'll reorganize tests to include more changes like several event loops and more databases.

For the next round, several improvements are on the road:
  1. aiohttp new release has several speed improvements.
  2. An alternative event loop like aiouv might help, even if it isn't really the case for now, some improvements are necessary.
  3. A potential AsyncIO booster written in C is discussed on Python mailing-list
  4. AsyncIO support in the next Cypthon release
  5. During Python language submit 2015, several presentations had some parts to improve efficiency of Python:
    1. Making Python 3 more attractive
    2. Python 3 adoption
    3. Atomicity of operations
    4. PyParallel
  6. PyPy3: with a patched version of AsyncIO and Py3.3 branch of PyPy3, it's possible to execute some FrameworkBenchmarks tests. For now, PyPy3 is slower than CPython, nevertheless, you must understand that the goal of PyPy developers is to have a working version of PyPy3 that support Python 3.3, not to improve efficiency. You can donate to the PyPy3 project.
However, don't expect a magic change in the results for the next round, the efficiency problematic in Python ecosystem isn't new.
Nevertheless, to my knowledge, I’ve never seen as many solutions to boost code in other interpreted languages, except in Python, I hope that more solutions will emerge in the future.

Special thanks

Even if I was pleased to get some attention for these benchmarks, and even if sometimes I had some “viril exchanges”/ping-pong "fights" (another blogpost should be published) on mailing-lists or during PyCONs, I won’t forget that theses results aren't possible without Python community.

Thanks a lot for everybody for your help, especially:
  1. Ukrainian (+Russian ?) Dream Team: Sorry to reduce you to your nationality/language, nevertheless, you're over-represented in AsyncIO community, especially in aiohttp ;-): aiohttp/aio-libs, they helped a lot to boost performances and I received numerous excellent advices from Andrew Svetlov, Nikolay Kim, Alexander Shorin, Alexey Popravka, Yury Selivanov and all others.
  2. Victor Stinner for AsyncIO improvements, help, and benchmarks tests.
  3. Antoine Pitrou for CPython optimizations like --with-computed-gotos and to have accepted AsyncIO PEP.
  4. Inada Naoki for all tips and challenged me in my benchmarks.
  5. Benoit Chesneau for his help to integrate Gunicorn in API-Hour.
  6. Stéphane Wirtel and Fabien Benetou to help me to promote Python in Belgium and for their support.
  7. All people I've forgotten in my list, especially AsyncIO libraries maintainers like Saúl Ibarra Corretgé (aiodns, aiouv), Jonathan Slenders (asyncio_redis), Gael Pasgrimaud (Panoramisk, irc3) Ron Frederick (asyncssh) and Tin Tvrtković (aiofiles, pytest-asyncio).
  8. Guido Van Rossum, because to maintain a community having the size of Python’s ecosystem, political skills are as important as technical skills. And also to have implemented and pushed AsyncIO on the Python scene.
  9. My Tech team at ALLOcloud/Eyepea, especially Nicolas Stein and Sylvain Francis, to challenge me everyday and to be the right place for technical innovations in Europe: Almost no management, no “reserved area” in our Tech team (Everybody must have some skills of others: Dev team must have the hands in sysadmin infrastructure, Telco team must be capable to edit source code from Dev team) and the Tech team's trust in me: I've changed their toolbox for AsyncIO, and even if we are a small team, in retrospect, they were very open to change their habits and improve the toolbox, instead of blocking innovations just to stay in their comfort zone. A working environment like that is precious and should be more mainstream.
  10. Last by not least, my family for their unconditional support.

I'm proud to be an atom inside Python community ;-)

And don't forget

IT world has as bias that Python is slow because you (as member of the Python community) believe that it is slow: With absolute values on microbenchmarks, it might be true. But, with some tips & tricks, it can easily be more than fast enough for most production use cases in companies AND with Python, you keep the simplicity of coding.
Be honest with yourself: does it make sense to try to compare yourself with the very large player such as Google or Facebook ? Most IT companies are far from having the same constraints as them, and the same human resources to address them. The main thing everyone has in common with them is “time to market”. Therefore, the simplicity of coding your business logic has much more weight in your case.
Developer speed is really more important than programs performance, but unfortunately, there are no benchmarks to measure this ;-).