Tracking testng test performance over multiple runs

Question

I'm looking into expanding my existing CI, which uses testng as a testing framework. I have a suite of tests whose performance (time taken for each test) I'd like to track performance of each test and each tests' changes over time. For example, after 5 runs of the suite, I could see whether my test is performing better or worse.

I know testng by default outputs xml reports which contain the time attribute. At the moment I'm not seeing anything in testng's documentation to specifically address this.

At the moment, I'm thinking the only way will be to extract the relevant info (method/class name and time) and compile it into a separate file to be reported on later. Any better built-in solutions you know of?

<testsuite name="PerformanceTests" tests="1" failures="0" timestamp="26 Apr 2018 17:05:51 GMT" **time="356.232"** errors="0">
  <testcase name="testModifyFail" time="2.127" classname="PerformanceTests"/>
<testsuite/>

Answer 1

1. TestNG has possibility to output test results in JUnit format
2. There is Performance Plugin for Jenkins which can build performance stats and trends charts and add them to build dashboard. Theoretically you can even mark build as unstable or failed if some metric exceeds acceptable threshold.

So you can configure the performance plugin to read results from test-output/junitreports and it should do the trick for you.

Check out How to Use the Jenkins Performance Plugin to learn more about the concept of adding performance tests results to CI pipelines.