Test results for ordered test are grouped in Azure Devops pipeline

Question

I am running a few ordered tests using Azure DevOps pipelines. The result for the tests is grouped into a single ordered test. It is becoming very difficult to interpret the result. Is it possible to obtain a separate pass/fail result?

In the screenshot below, the ordered test contains about 75 tests and only one of them fails. But that results in an overall 1 Test failed result. Is it possible to show 74 passed, 1 failed result in the Tests Results?

Answer 1

According to your description, since you are using ordered test. It's an expect behavior. The tests will be grouped and be reported in a summarized view.

You could take a look at the statement in our official doc here-- View summarized test results

During test execution, a test might spawn multiple instances or tests that contribute to the overall outcome. Some examples are, tests that are rerun, tests composed of an ordered combination of other tests (ordered tests) or tests having different instances based on an input parameter (data driven tests).

As these tests are related, they must be reported together with the overall outcome derived from the individual instances or tests. These test results are reported as a summarized test result in the Tests tab:

• Rerun failed tests: .......
• Data driven tests: ...... The summarized view is also available for ordered tests (.orderedtest in Visual Studio).

It's not possible to show 74 passed, 1 failed result in the Tests Results.

To see test results of individual tests in the ordered test suite you may need to download the *.trx file and open it in Visual Studio.

Answer 2

This is what I did to get the individual results-

All the tests I wanted to run were in a single ordered test. I changed the build definition to run them individually and not as a part of a single ordered test. The only drawback of following this approach is we cannot rerun the failed tests.

Result: