How to achieve Appium testing automation for an Android app in an Azure DevOps pipeline?

Question

We have an issue performing automation testing with Appium (Android app) through a pipeline in Azure Devops. We are trying to create an emulator to run our apk but its behaviour seems to be inconsistent.

• Sometimes the app starts but the interface is not responding: Screenshot

• Sometimes the app doesn’t start and the interface is not responding:
 Screenshot

• Sometimes the emulator doesn’t seem to be ready (in more than 25 minutes) according to the execution of $ANDROID_HOME/platform-tools/adb devices.

It might be worth mentioning that the emulation and testing does work correctly when I run the pipeline using my own Mac machine as a private agent. That suggests that there might be an issue with the macOS agents provided by microsoft?

We tried using macOS-11, 12 and 13 with a number of different system-images. Also changing some emulator configurations that we thought could be causing the issue (insufficient RAM or heap size) but the inconsistency remains.

Could you please take a look at our .yml file? This is my first time creating a pipeline so any advice or tricks are greatly appreciated. Thank you.

trigger:
  - master
  - feature/*

pool:
  vmImage: "macOS-12"

steps:
  - task: NodeTool@0
    condition: always()
    inputs:
      versionSpec: "16.x"
    displayName: Install Node Js

  - task: Bash@3
    inputs:
      targetType: "inline"
      script: |
        #!/usr/bin/env bash

        # Install AVD files
        # AVD FOR MAC OS 12 AGENT (x86_64)
        echo "y" | $ANDROID_HOME/cmdline-tools/latest/bin/sdkmanager --install 'system-images;android-29;default;x86_64'
        # AVD FOR MY PRIVATE AGENT (ARM64-V8A)
        # echo "y" | $ANDROID_HOME/cmdline-tools/latest/bin/sdkmanager --install 'system-images;android-29;google_apis;arm64-v8a'

        # Create emulator
        # EMULATOR FOR MAC OS 12 AGENT (x86_64)
        echo "no" | $ANDROID_HOME/cmdline-tools/latest/bin/avdmanager create avd -n xamarin_android_emulator -k 'system-images;android-29;default;x86_64' --force
        # EMULATOR FOR MY PRIVATE AGENT (ARM64-V8A)
        # echo "no" | $ANDROID_HOME/cmdline-tools/latest/bin/avdmanager create avd -n xamarin_android_emulator -k 'system-images;android-29;google_apis;arm64-v8a' --force

        # List of AVDs
        echo "List of avds:"
        $ANDROID_HOME/emulator/emulator -list-avds
        echo "End of avds list"

        # Accessing the emulator config file
        # emulator_config=~/.android/avd/xamarin_android_emulator.avd/config.ini
        emulator_config=~/.android/avd/xamarin_android_emulator.avd/config.ini

        # Reading the initial config file:

        echo "INITIAL Emulator settings ($emulator_config)"
        cat "$emulator_config"

        # Replace existing config
        sed -i .bak 's/hw.lcd.density=.*/hw.lcd.density=420/' "$emulator_config"
        sed -i .bak 's/hw.lcd.height=.*/hw.lcd.height=1920/' "$emulator_config"
        sed -i .bak 's/hw.lcd.width=.*/hw.lcd.width=1080/' "$emulator_config"
        sed -i .bak 's/hw.ramSize=.*/hw.ramSize=16384/' "$emulator_config"
        sed -i .bak 's/vm.heapSize=64M/vm.heapSize=256M/' "$emulator_config"

        # Reading the config file after changes
        echo "FINAL Emulator settings ($emulator_config)"
        cat "$emulator_config"

        echo "Starting emulator"

        # Start emulator in background

        nohup $ANDROID_HOME/emulator/emulator -avd xamarin_android_emulator -change-locale fr-FR -no-snapshot > /dev/null 2>&1 &
        $ANDROID_HOME/platform-tools/adb wait-for-device shell 'while [[ -z $(getprop sys.boot_completed | tr -d '\r') ]]; do sleep 1; done; input keyevent 82'

        $ANDROID_HOME/platform-tools/adb devices

        echo "Emulator started"

  - task: Npm@1
    displayName: "npm install dependencies"
    inputs:
      command: "install"
      workingDir: "MadAutomation"

  - task: Bash@3
    displayName: "Run Appium in background port 4723"
    continueOnError: false
    enabled: true
    inputs:
      targetType: "inline"
      script: |
        #!/bin/bash 
         nohup appium -a 127.0.0.1 -p 4723 --relaxed-security --chromedriver-executable /Applications/chromedriver_mac_arm64/mobile/91.0.4472.101/chromedriver > $(agent.buildDirectory) &

  - task: npm@1
    inputs:
      command: "custom"
      workingDir: "MadAutomation"
      customCommand: "run wdio"
    displayName: "npm run wdio"

  - task: PublishTestResults@2
    displayName: "Publish Test Results **/*.png"
    inputs:
      testResultsFiles: |
        **/*.png
      mergeTestResults: true
    continueOnError: true
    condition: succeededOrFailed()

  - task: CopyFiles@2
    displayName: "Copy Files to: $(build.artifactstagingdirectory)"
    inputs:
      SourceFolder: MadAutomation
      Contents: |
        **/*.png
      TargetFolder: "$(build.artifactstagingdirectory)"

  - task: PublishBuildArtifacts@1
    displayName: "Publish Artifact: drop"
# - script: |
#       npm install
#       npx wdio run ./wdio.conf.js -- --yes