Speeding up Django tests using a RAM-bound MySQL server

A while ago, Django’s testing framework got transaction-based rollback, which obviously did wonders in terms of test performance. One thing that still bothered me though was the slow, initial table setup. For example, in a modestly sized project of mine with about 40 tables, this would take up to almost a minute. In particular when writing new tests, which is going to be an iterative process, that’s really not acceptable.

Now, one obvious things to do is using an in-memory SQLite database for testing purposes. I’ve tried that at times, but ultimately, various MySQL-specific stuff and raw SQL queries always made this an unsatisfying experience.

I’ve now finally realized that there is an easy solution, and I’m perplexed it didn’t occur to me earlier (maybe Linux, to which I’ve recently switched, just puts these kinds of options closer to one’s grasp). And it really is pretty straightforward: Mount a tmpfs, run a second MySQL instance on a different socket/port using this mount as a data dir, and tell Django to use it.

I’ve put shell script that I’m using on github.

You might want to customize the location of the data directory or the bind options, then simply do:

sudo ./mysqld-ram.sh

and when you’re done, shutdown with Ctrl+C.

The tables which previously took a minute to setup, now only need two and a half seconds. It even cuts the runtime of the actual tests, which were already using transaction-rollback before, in half. Not surprisingly, I notice that my motivation to actually write tests and keep them up-to-date has noticeably improved.

One thought on “Speeding up Django tests using a RAM-bound MySQL server

  1. This is *really* useful, thanks so much. There were a couple of extra pieces I needed. To set up a user that can create databases, I added these lines before the ‘wait’ line:

    sleep 5
    SQL=”CREATE DATABASE IF NOT EXISTS djangotests; GRANT USAGE ON *.* TO tester@localhost IDENTIFIED BY ‘password’; GRANT ALL PRIVILEGES ON djangotests.* TO tester@localhost; FLUSH PRIVILEGES;”
    mysql –port=3307 –host=127.0.0.1 -e “$SQL”

    https://github.com/spookylukey/linuxutils/commit/a6e65908fc3594e3df26923c64c1c4c9a05ad17d

    Also, to access this MySQL server automatically when doing ‘./manage.py test’, I added this to my settings file:

    (requires psutils to detect the running server)

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s