Troubleshooting TestCafe End-to-End Tests Failing on Azure DevOps Pipeline

Question

I'm encountering issues with my TestCafe end-to-end tests when running them on Azure DevOps pipelines. Locally, the tests run smoothly without any failures, but on the pipeline, they occasionally fail with errors indicating that selectors were not found in the DOM or that the .exists assertion failed, even though I can confirm that the elements exist in the DOM.

I've already ensured that the selectors are correctly defined and that the elements are present in the DOM during the test execution. Despite this, the failures seem to be intermittent and somewhat random.

I suspect there might be some environmental differences between my local setup and the Azure DevOps pipeline environment that could be causing these issues and causing timeouts. However, I'm unsure how to identify and address these differences effectively. I've also turned on screenshots in my config file on fails to try and understand what is happening but it's saving them to somewhere in the server that I don't think I can access.

Could someone provide some troubleshooting tips on resolving such issues with TestCafe tests running on Azure DevOps pipelines? Additionally, are there any specific configurations or settings I should consider adjusting to ensure more reliable test execution on the pipeline?

Any insights or suggestions would be greatly appreciated. Thank you!

My .testcaferc.js:

module.exports = 
{
        browsers: ["chrome"],
        assertionTimeout: 50000,
        pageLoadTimeout: 50000,
        selectorTimeout: 50000,
        pageRequestTimeout: 50000,
        browserInitTimeout: 180000,
        screenshots: {
            takeOnFails: true
        },
        quarantineMode: true,
        skipJsErrors: true,
        baseUrl: www.mywebsite.com 
}

screenshots of fails:

Answer 1

You can add your local machine as a self hosted agent in Azure DevOps pipeline and then test the code with testcafe as your local environment will be used and the screenshots will be saved locally. Refer 1 and 2. Now, You are using Microsoft hosted agents to run your Devops pipeline to access the specific folders in Microsoft hosted agents, Refer this MS Document. If your test is created and running successfully in Windows machine locally, You can use windows-latest as an agent. So Azure devops will use windows-latest OS based self hosted agent.

My test.js:-

Reference here.

import { Selector } from 'testcafe';

fixture('TestCafe Example Page Test')
    .page('https://devexpress.github.io/testcafe/example/');

test('Test Page Title', async t => {
    await t.expect(Selector('title').innerText).eql('TestCafe Example Page');
});

test('Test Text Typing and Submit', async t => {
    await t.typeText('#developer-name', 'John Doe')
           .click('#submit-button');
    
    const articleHeader = Selector('#article-header').innerText;
    await t.expect(articleHeader).contains('John Doe');
});

My Azure Devops yaml code:-

trigger:
  - main  # Adjust branch name as per your repository

pool:
  vmImage: 'windows-latest'  # Use an appropriate VM image

steps:
- task: NodeTool@0
  inputs:
    versionSpec: '14.x'
  displayName: 'Install Node.js'

- script: |
    npm install -g testcafe
    npm install  # Install project dependencies
  displayName: 'Install TestCafe and Dependencies'

- script: |
    testcafe chrome test.js  # Adjust browser and test file name as needed
  displayName: 'Run TestCafe Tests'

Output:-