I was asked to enhance the performance of our automation framework. Currently, we're utilizing the TestNG framework for our test automation. We execute our test suite using Jenkins, which runs on a single cloud instance but concurrently across 12 slave agents. We organize our test classes within a customized TestNG XML file named 'run.xml', where they're placed under various 'test' tags as bellow:This is just the basic structure. we have test cases distributed among each of the 12 slave agents.
run.xml file
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Regression Test" verbose="1">
<listeners>
<listener class-name="org.uncommons.reportng.HTMLReporter" />
<listener class-name="org.uncommons.reportng.JUnitXMLReporter" />
</listeners>
<test name="Defaults" enabled="true">
<classes>
<class name="com.pragmatic.orange.test.LoginTest" />
</classes>
</test>
<test name="LeaveModuleSlave1" enabled="true" time-out="900000">
<classes>
<class name="com.pragmatic.orange.test.Leave_Module.LeaveSlave1TestClass1" />
<class name="com.pragmatic.orange.test.Leave_Module.LeaveSlave1TestClass2" />
</classes>
</test>
<test name="LeaveModuleSlave2" enabled="true" time-out="900000">
<classes>
<class name="com.pragmatic.orange.test.Leave_Module.LeaveSlave2TestClass1" />
<class name="com.pragmatic.orange.test.Leave_Module.LeaveSlave2TestClass2" />
</classes>
</test>
<test name="PerformanceModuleSlave1" enabled="true">
<classes>
<class name="com.pragmatic.orange.test.Leave_Module.PerformanceModuleSlave1TestClass1" />
<class name="com.pragmatic.orange.test.Leave_Module.PerformanceModuleSlave1TestClass2" />
</classes>
</test>
<test name="PerformanceModuleSlave2" enabled="true">
<classes>
<class name="com.pragmatic.orange.test.Leave_Module.PerformanceModuleSlave2TestClass1" />
<class name="com.pragmatic.orange.test.Leave_Module.PerformanceModuleSlave2TestClass2" />
</classes>
</test>
</suite>
We have a Groovy script file that defines how to host the environment, set up the project, and execute the automation. On the Jenkins side, we've configured certain parameters. These parameters allow us to specify which modules need to be executed. Below, you can find some important details in the Groovy file.
pipeline {
agent none
parameters {
booleanParam(name: 'Leave_Module', defaultValue: true, description: 'This category contains all the test cases related to leave module only')
booleanParam(name: 'Performance_Module', defaultValue: true, description: 'This category contains all the test cases related to the Performance module only')
string(defaultValue: '', description: 'Choose the Automation codebase repository', name: 'automation_codebase_repo')
string(name: 'REPO_SUFFIX', defaultValue: '', description: 'Repository Suffix')
string(name: 'BRANCH', defaultValue: 'main', description: 'Branch')
}
stages {
stage ('Hosting in slave1') {
agent {
label 'parallel-slave-node-1'
}
environment {
}
steps {
install_application()
}
}
stage('Hosting Instances') {
parallel {
stage ('Hosting in slave2') {
agent {
label 'parallel-slave-node-2'
}
when {
expression { params.Leave_Module || params.Training_Module || params.Performance_Module}
}
steps {
install_application()
}
}
}
}
stage ('Test Suite Execution') {
parallel {
stage('Test Case Execution on Slave-1') {
agent {
label 'parallel-slave-node-1'
}
when {
expression { params.Leave_Module || params.Performance_Module }
}
environment {
}
steps {
script {
dir('results') {
deleteDir()
}
runAutomationCodebaseSetup(1)
testcasenames = 'Defaults'
if(params.Leave_Module){
testcasenames = "${testcasenames},LeaveModuleSlave1"
}
if(params.Performance_Module){
testcasenames = "${testcasenames},PerformanceModuleSlave1"
}
try {
sh "cd ${WORKSPACE}/automationcodebase/CodeBase && mvn test -DtestNamesToExecute=${testcasenames} -Dexecutiontype=parallel -DscopetoExecuteLegacy=Regression -DscopetoExecuteLeave=Regression > reports/report_of_slave_1.txt"
} catch (err) {
currentBuild.result = 'UNSTABLE'
}
report1 = findlines()
}
}
post {
always {
script {
try {
sh 'cp -r automationcodebase/CodeBase/reports/report_of_slave_1.txt results/Slave1'
sh 'cp -r automationcodebase/CodeBase/screenshot/* results/Slave1/screenshot'
sh 'cp -r automationcodebase/CodeBase/target/surefire-reports/emailable-report.html results/Slave1'
sh 'cp -r automationcodebase/CodeBase/reports/index.html results/Slave1/index_slave1.html'
}catch (err) {
sh 'echo "Something Went Wrong while copying!!"'
} finally {
archiveArtifacts(artifacts: 'results/Slave1/**/*',excludes:'screenshot/**' ,followSymlinks: false)
zip zipFile: 'slave1.zip', archive: false, dir: 'results/Slave1/'
archiveArtifacts(artifacts: 'slave1.zip', followSymlinks: false)
}
}
}
}
}
Based on the provided code and explanation, if we set "Leave module" and "Performance module" as parameters, in the test execution of Slave 1, it will take the test classes under the tag and the tag and execute them on Slave 1. This process is the same for the other slaves as well. Each slave will execute the test classes specified for its corresponding module
The issue arises from the extended time taken for parallel execution. To address this, there's a plan to further parallelize test execution within each Jenkins slave node. we try to achieved this by leveraging threads.
However, since each slave has only one database, allowing each thread to access the same database may lead to test failures due to concurrency issues. Is there any way to set up separate databases for each thread in the Groovy file or any resources or guidance to resolve this problem effectively.