running all tests post django 1.6

Question

In django 1.5 and earlier, running python manage.py test would, by default, run all tests in a project (including all those in django.contrib). Subsequent to version 1.6, the default behaviour is to run all the tests in the current directory.

What is the best way (v 1.6) to run all tests, either with or without the django.contrib tests?

Answer 1

Django 1.6 changed the default test runner to:

TEST_RUNNER = 'django.test.runner.DiscoverRunner'

You can get the old behaviour back by adding to your settings.py:

TEST_RUNNER = 'django.test.simple.DjangoTestSuiteRunner'

As explained in the release notes:

The previous runner (django.test.simple.DjangoTestSuiteRunner) found tests only in the models.py and tests.py modules of a Python package in INSTALLED_APPS.

The new runner (django.test.runner.DiscoverRunner) uses the test discovery features built into unittest2 (the version of unittest in the Python 2.7+ standard library, and bundled with Django). With test discovery, tests can be located in any module whose name matches the pattern test*.py.

The new runner expects a list of dotted path of modules where tests shall be discovered, so you can also run the tests from django contrib this way:

python manage.py test myproject django.contrib path.to.someotherapp

This will not run all tests from apps in INSTALLED_APPS automatically though. For a more complex solution, you could write your own runner, taking from both the old and new runner.

Also note that it usually shouldn't be necessary to run tests from django.contrib, as these are not testing your application, but rather the Django distribution. Django ships with even more tests, which are not run by either runner.

Answer 2

It's a shame that Django decided to ignore custom apps in the INSTALLED_APPS that are not in the project tree. See this post https://groups.google.com/forum/#!topic/django-users/gGfVhfrfE10 for their reasoning.

My real-world case does of course does not fall in the three mentioned use cases (big suprise!): We have a site that we deploy with a different collection of tightly coupled custom apps for client / group of clients. We don't want these apps nested under the project tree, because each is in its own git repo. In the past, we've used git submodules, fake submodules and subtrees; all had their issues in our setup. On the other hand, having each app as its own package on the same level as the site satisfies most of our requirements.

Of course each app has its own tests, but I would like to be able to run the full test suite (including the site and all custom apps) for each specific differently constituted site.

Our workaround is the following:

In settings/test.py I have:

ATA_BLACKLIST = ['scary_mod1', 'scary_mod2']
ADD_TEST_APPS = [i for i in INSTALLED_APPS
                 if '.' not in i and i not in ATA_BLACKLIST]
ATA_STR = " ".join(ADD_TEST_APPS)

I have a run_tests.sh script at top-level which looks more or less like this:

#!/bin/bash

MYDIR=`dirname $0`
cd $MYDIR
DJANGO_SETTINGS_MODULE=kmxng.settings.test
ATA_STR=`python -c "from django.conf import settings; print settings.ATA_STR"`
coverage run --omit "*lib/python*" ./manage.py test . $ATA_STR
coverage report

In short, in my test settings I generate a list of extra modules that should be added to testing. I invoke the django test command with that list in addition to the default ..

(I did have a look at overriding DiscoverRunner -- it has a relatively long build_suite method that could be overridden. The work-around above is a less-invasive option.)