How to write 1000 system tests for a process that takes 20 minutes but is parallelizable in .NET C#

Question

Goal

To write a set of system tests for a single process with the following details:

• Each run of the process typically takes 20 minutes.
• Process runs can be in parallel so over 2000 process runs can complete in an hour on our test system.
• Each process run (1) reads input data (2) processes it and (3) makes it available at an API.
• There are around 1000 combinations of input data to test.

Proposed test of a single process run

Steps:

1. Call API to create process run, and get its process run id.

2. Copy process input files to location where process run automatically picks up files and processes them.

3. Poll API until expected data is there.

   • Steps 1 and 2 pass data to step 3.
   • Steps 1 and 2 take a few seconds but step 3 can be 20 minutes. 
   • Key Part: I would like to execute all step 1s and 2s before any step 3s execute, and execute all step 3s in parallel so that 1000 waits of 20min can all happen at the same time and make the full set of system tests finish in a manageable time.

What I've tried

• I am using MsTest to run the tests and have looked at ordering methods by name and parallelisation but neither seem to meet this use case as methods are only ordered within a class.
• I have looked at NUnit, which does support ordering of test methods within a class, or ordering of test classes within a namespace. Using these, I could split steps 1, 2 and 3 into separate classes and pass data between them using a static variable (perhaps a dictionary keyed by test name) - however this feel like a hack so wanted to see if there was something cleaner.

Questions

• Are there any other features in the MsTest library which can help fulfill this use case?
• Are there any other features in the NUnit library which can help fulfill this use case?
• Do you know any other .NET test libraries which support this use case?

Answer 1

There is a lot I do not completely understand concernig the details and rationale of what you are doing and why you are doing it. Generally what you describe is what is done with SetUpFixture.

Answer 2

Parallel MSTest In your runsettings file you can set the parallel workers. 0 is max, which depends on how many cores you are running on.

<RunSettings>
  <TestRunParameters>
    <Parameter name="Environment" value="Dev" />
  </TestRunParameters>
  <MSTest>
    <Parallelize>
      <Workers>0</Workers>
      <Scope>MethodLevel</Scope>
    </Parallelize>
  </MSTest>
</RunSettings>

https://learn.microsoft.com/en-us/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file?view=vs-2022

Parallel Code Here I would suggest a main wrapper test that calls into other methods to execute the different stages of your test. You could use Threads or Parallel ForEach().