Teamcity rerun specific failed unstable tests

Question

I have Teamcity 7.1 and around 1000 tests. Many tests are unstable and fail randomly. Even a single test fails the whole build fails and to run a new build takes 1 hour.

So I would like to be able to configure Teamcity to rerun failed tests within the same build a specific number of time. Any success for a test should be considered as success, not a failure. Is it possible?

Also now is tests in some module fail Teamcity does not proceed to the next module. How to fix it?

Answer 1

With respect, I think you might have this problem by the wrong end. A test that randomly fails is not providing you any value as a metric of deterministic behavior. Either fix the randomness (through use of mocks, etc.) or ignore the tests.

Answer 2

If you absolutely have to I'd put loops round some of your test code and catch say 5 failures before throwing the exception as a 'genuine' failure. Something like this C# example would do...

    public static void TestSomething()
    {
        var counter = 0;
        while (true)
        {
            try
            {
                // add test code here...
                return;
            }
            catch (Exception) // catch more specific exception(s)...
            {
                if (counter == 4)
                {
                    throw;
                }

                counter++;
            }
        }
    }

While I appreciate the problems that can arise with testing asych code, I'm with @JohnHoerr on this one, you really need to fix the tests.

Answer 3

Rerun failed tests feature is part of Maven Surefire Plugin, if you execute mvn -Dsurefire.rerunFailingTestsCount=2 test then tests will be run until they pass or the number of reruns has been exhausted.

Of course, -Dsurefire.rerunFailingTestsCount can be used in TeamCity or any other CI Server.

See: http://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html