Using Django’s ORM from aiohttp

When perfectionists with deadlines need better performance

Aristóbulo Meneses

4 min readSep 9, 2019

Assumptions

In this post, I will assume that you:

Already know your ways with Django
Are familiar with asyncio (at least you have heard of it)

What is this all about?

Django is a great framework to develop web applications and REST APIs but like everything in this world, it comes with pros and cons.

A common pitfall of using Django is its performance, so lets say you have spent time and effort profiling and debugging your application to get rid of all the slow pieces that are making it slower than expected but you can’t still get significant improvements, what to do then?

It’s not easy just to throw it away and start from scratch with another framework or programming language, but what if you don’t have to?

With the introduction of asyncio new web frameworks have emerged, delivering asynchronous request handling and great performance, Django itself will support ASGI in the future, but until then there is one interesting alternative to be explored: using aiohttp.

As stated before, you might not need to get rid of Django at all, on the contrary, you can benefit from the good stuff from Django (the admin interface, ORM, migrations engine, …) and gain aiohttp’s great performance.

This means that you can still serve your Django application as usual and move your API endpoints to aiohttp.

Ready, set, go!

For this post I will use an example project called mymoviedb, with a small app called movies, the idea is to be able to store movie information and have a small API to consume this data.

In this example, Django will take care of providing an administration panel to manage all movie data and aiohttp will handle our small REST API.

For our API we will need to declare and start a new aiohttp app, that will run only the async parts of mymoviedb.

How I met your ORM

Let’s review what we just did there, we are declaring a coroutine to configure Django using themymoviedb.settings module and then telling aiohttp server to execute it when starting up.

What we are achieving here is to have Django’s ORM ready to be used from our newly added async views.

When adding our views is when we will start seeing the benefits of keeping Django.

A simple movies list view to handle GET requests

Now is the time when we actually mixed both frameworks, we use our existing model in a coroutine by calling a QuerySet that fetches all movies.

So you might have noticed database_sync_to_async wrapping our QuerySet, by doing this, we allow our view to call database methods in a safe, synchronous context.

The Django ORM is a synchronous piece of code, and so if you want to access it from asynchronous code you need to do special handling to make sure its connections are closed properly.

We are borrowing this piece of code from channels

Thank you Django channels

Performance

But does it really improve performance? Let’s do some basic benchmarking using wrk.

The application will be running in a VPS 2018 SSD 1 hosted at OVH’s Graveline datacenter (France), it runs 1vCore at 2Ghz and 2GB of RAM.

To deploy the whole project, I’m using docker-compose with the compose file included in the repository, that means:

Django is served with Gunicorn with 3 workers.
aiohttp is running the standalone server.
Our movie sample was loaded to the database, thus, each request will return 10 movies on each response.
To be able to compare Django’s sync performance against Django’s ORM + aiohttp async performance, we need to add a Django view and create a /movies/ endpoint:

The Django view will do exactly the same as the aiohttp view but synchronously

Results:

Gunicorn + Django (sync):

$ wrk -t12 -c400 -d30s http://---.ovh.net/movies/ 
Running 30s test @ http://---.ovh.net/movies/
 12 threads and 400 connections
 Thread Stats Avg   Stdev    Max    +/- Stdev
 Latency      1.29s 192.55ms 1.99s  85.49%
 Req/Sec      13.30 11.19    100.00 79.92%
 3097 requests in 30.08s, 6.99MB read
 Socket errors: connect 0, read 0, write 0, timeout 760
 Non-2xx or 3xx responses: 56
Requests/sec: 102.96
Transfer/sec: 238.02KB

aiohttp + Django ORM (async)

$ wrk -t12 -c400 -d30s http://---.ovh.net/api/movies 
Running 30s test @ http://---.ovh.net/api/movies
 12 threads and 400 connections
 Thread Stats Avg   Stdev    Max    +/- Stdev
 Latency      1.04s 400.03ms 2.00s  68.65%
 Req/Sec      35.44 31.58    240.00 80.94%
 7855 requests in 30.06s, 17.91MB read
 Socket errors: connect 0, read 7, write 0, timeout 1315
Requests/sec: 261.29
Transfer/sec: 610.11KB

And just like that, without much tweaking, we are able to handle more than twice (2,55x) the number of requests per second.

Note: Please keep in mind this is a very basic test, further improvements could be achieved for both by doing more advanced configuration tweaking.

Final words

Django is a battle-tested framework, well documented and stable but it’s normal to find yourself trapped in the “but it’s slow” discussion, in this small experiment we can see that it is easy to get a boost in performance by using it alongside with something as fast as aiohttp.

Let me know in the comments if you want me to keep exploring this approach.

The complete code of this example can be found in this Github repository.