How to retry on failed Android instrumented tests?

Question

I'm searching for a way to retry run failed Android instrumented tests or called integration tests/ui tests. I think there could have 3 ways of doing this.

• Retry with Jenkins stage script -> retrieve all failed tests and perform them again with run gradle command ./gradlew clean app:connectedVariantNameAndroidTest -> I didn't try this approach yet.

• Plugins: I tried Gordon, however, its minimum Java support version is 17, which is not aligned with our CI machine (Java 11)

• Implementation in code as some helper class: Tried this post -> The retry didn't happen when the instrumented test failed.

I think retrying on instrumented failed tests is commonly used in test automation since instrumented tests are flaky and easy to fail. Please help if you have any ways to accomplish this.

Answer 1

You can restart declarative pipeline from any stage. It will works if you keep build workspace and if errors rised outside pipeline execution, not in code or test or dependencies. In that case you need to update your sources. For some cases you may need to use gradle(dotnet mvn etc ...) clean You can place it in post{always {}} step if yours code must be recompiled.

---

Examples

Jenkins side

1. simple example:

pipeline {
    agent {label 'linux'}

    stages {
        stage('Random exit code') {
            steps {
                script{

// random result implementation based on exit code
                    def status = sh(
                                    returnStdout: true,
                                    script:'''
                                            bash -c 'echo $(( $RANDOM % 2 ))'
                                            '''
                                    ).toInteger()
                    print("script status: $status")
                    if (status != 0) {
                      currentBuild.result = 'FAILED'
                    }
                }
            }
        }
    }
    post{
        success{
            echo 'ok'
        }
        failure{
            echo 'fail'
// rebuild same pipeline
            build  wait: false, job: '/dev-env/simple_example'
        }
            
    }
}

jenkins post states

1. Advanced example with declarative pipeline stage restart:

Pipeline:

pipeline {
    agent {label 'linux'}
    stages {
        stage('build') {
            steps {
                sh 'touch testfile'
            }
        }
        stage('parralel tests') {
            parallel{
                stage('unit test'){
                    steps{
                        sh "echo '${STAGE_NAME} stage' >> testfile"
                        sh 'cat testfile'
                    }
                }
                stage('smoke test'){
                    steps{
                        sh "echo '${STAGE_NAME} stage' >> testfile"
                        sh 'cat testfile'
                    }
                }
                stage('postman api test'){
                    steps{
                        sh "echo '${STAGE_NAME} stage' >> testfile"
                        sh 'cat testfile'
                    }
                }
            }
        }
        stage('integration tests') {
            steps {
                sh "echo '${STAGE_NAME} stage' >> testfile"
                sh 'cat testfile'
            }
        }
    }
    
    post{
        success{
            // cleanWs() 
            // cleanWs()  commented becouse this build always success.
            // in real world you need to cleanup your workspace if run is OK  
            sh 'cat testfile'
        }
        failure{
            sh 'cat testfile'
            echo 'something is failed'
        }
            
    }
}

output:

11:58:49  + echo integration tests stage
11:58:49  [Pipeline] sh
11:58:49  + cat testfile
11:58:49  unit test stage
11:58:49  smoke test stage
11:58:49  postman api test stage
11:58:49  integration tests stage
11:58:49  [Pipeline] }
11:58:49  [Pipeline] // stage
11:58:49  [Pipeline] stage
11:58:49  [Pipeline] { (Declarative: Post Actions)
11:58:49  [Pipeline] sh
11:58:49  + cat testfile
11:58:49  unit test stage
11:58:49  smoke test stage
11:58:49  postman api test stage
11:58:49  integration tests stage <<<<
11:58:49  [Pipeline] }
11:58:49  [Pipeline] // stage
11:58:49  [Pipeline] }
11:58:49  [Pipeline] // node
11:58:49  [Pipeline] End of Pipeline
11:58:50  Notified JIRA that a build has completed.
11:58:50  Finished: SUCCESS

Replay stage: Replay output:

12:02:32  + echo integration tests stage
12:02:32  [Pipeline] sh
12:02:32  + cat testfile
12:02:32  unit test stage
12:02:32  smoke test stage
12:02:32  postman api test stage
12:02:32  integration tests stage
12:02:32  integration tests stage
12:02:32  [Pipeline] }
12:02:32  [Pipeline] // stage
12:02:32  [Pipeline] stage
12:02:32  [Pipeline] { (Declarative: Post Actions)
12:02:32  [Pipeline] sh
12:02:33  + cat testfile
12:02:33  unit test stage
12:02:33  smoke test stage
12:02:33  postman api test stage
12:02:33  integration tests stage
12:02:33  integration tests stage
12:02:33  [Pipeline] }
12:02:33  [Pipeline] // stage
12:02:33  [Pipeline] }
12:02:33  [Pipeline] // node
12:02:33  [Pipeline] End of Pipeline
12:02:33  Notified JIRA that a build has completed.
12:02:33  Finished: SUCCESS

Explanation:

Look in post stage comments You can see that integration test stage runs in same env. That meens you dont need to restart build stage for binary compilation

12:02:33  + cat testfile
12:02:33  unit test stage
12:02:33  smoke test stage
12:02:33  postman api test stage
12:02:33  integration tests stage
12:02:33  integration tests stage <<<<<<<

You can compile simple and advanced example if you wish. But i think it will be overkill

Gradle side

You may add some logic in your pre&post wrappers to check test status and replay cases at some conditions. Unfortunately, the gradle test retry plugin does not support Android Instrumentation tests.