Automating tests using Xray, Robot Framework, and GitLab: How to get started?

Question

I'm looking to automate tests for my project using Xray, Robot Framework With python , and GitLab, but I'm unsure of how to proceed. Could someone provide guidance on how to set up the automation workflow?

Know how to automate test using this tools

Answer 1

The overall flow would be something like this, assuming you're adopting ATDD:

If you're using Xray on Jira Datacenter, you can see here a detailed tutorial showcasing some automated tests implemented in Robot Framework and then the integration with Xray; there's a GibHub repo with the sample code used.

So, and focusing on the essential, you would:

• write your Robot Framework tests
• execute your tests as usual in RF, (e.g., robot ...), which will produce and output.xml report
• submit that report to Xray using Xray's REST API; there are specific endpoints to handle RF output.xml type of reports; if you're using Xray on Jira Datacenter please see this; if you're using Xray on Jira Cloud please see this instead.

As an example, to submit the RF report to Xray on Jira Datacenter, you can make a POST request using something like this (assuming you have curl tool):

curl -H "Content-Type: multipart/form-data" -u someuser:somepass -F "[email protected]" "http://192.168.56.102/rest/raven/2.0/import/execution/robot?projectKey=CALC

If you're using Xray on Jira Cloud, there's a somehow similar tutorial.

To setup this on Gitlab, you should create a .gitlab-ci.yml file in your repository.

# Official language image. Look for the different tagged releases at:
# https://hub.docker.com/r/library/python/tags/
image: python:3.12.2

# Change pip's cache directory to be inside the project directory since we can
# only cache local items.
variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

# https://pip.pypa.io/en/stable/topics/caching/
cache:
  paths:
    - .cache/pip

stages:
  - execute_automated_tests
  - upload_test_results

before_script:
  - python --version ; pip --version  # For debugging
  - pip install virtualenv
  - virtualenv venv
  - source venv/bin/activate
  - pip install -r requirements.txt
  - apt-get update



test:
  stage: execute_automated_tests
  before_script: | 
    set -e
    apt-get install -yqq unzip curl
    # Install Chrome & chromedriver
    curl -sS -o - https://dl.google.com/linux/linux_signing_key.pub | apt-key add -
    echo "deb https://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list
    apt update && apt install google-chrome-stable -y
  http://chromedriver.storage.googleapis.com/`curl -sS chromedriver.storage.googleapis.com/LATEST_RELEASE`/chromedriver_linux64.zip
    wget -O /tmp/chromedriver.zip https://storage.googleapis.com/chrome-for-testing-public/121.0.6167.85/linux64/chromedriver-linux64.zip
    ls -la /tmp/chromedriver.zip
    unzip -j /tmp/chromedriver.zip chromedriver-linux64/chromedriver -d /usr/local/bin/
    nohup python demoapp/server.py &
  script: |
    google-chrome --version && \
    chromedriver -v && \
    pip install -r requirements.txt && \
    robot -x junit.xml -o output.xml login_tests || true
  allow_failure: true
  artifacts:
    paths:
      - output.xml
    when: always

  
upload_results_to_xray:
  stage: upload_test_results
  script:
    - echo "uploading results to Xray..."
    - 'curl -H "Content-Type: multipart/form-data" -u $XRAY_USERNAME:$XRAY_PASSWORD -F "[email protected]" "$XRAY_SERVER_URL/rest/raven/2.0/import/execution/robot?projectKey=$PROJECT_KEY"'
    - echo "done"
  dependencies:
  - test

Don't forget to add the environment variables on your GitLab CI/CD configuration, under the Pipeline area in Gitlab's UI.

When you run this, either manually or because of a commit, the workflow will be run on Gitlab, and there you can see the information on the Test Execution issue created on Xray; in the screenshot it shows "CALC-408".

If you go to your Jira/Xray instance, you'll be able to see the new Test Execution along with the results.

Note: Xray may support RF up to version 6.x; not the latest RF 7.0.

============ ============ ============ ============

If you were using CircleCI as CI/CD tool, the process would be quite similar; I'll share it as I started by implementing it using CircleCI initially to prepare the answer for you.

To setup this on CircleCI, you should create a .circleci/config.yml file in your repository, and add something like the following (dont forget to create some environment variables in your CircleCI project configuration):

# Use the latest 2.1 version of CircleCI pipeline process engine.
# See: https://circleci.com/docs/configuration-reference

# For a detailed guide to building and testing with Python, read the docs:
# https://circleci.com/docs/language-python/ for more details
version: 2.1

# Orbs are reusable packages of CircleCI configuration that you may share across projects, enabling you to create encapsulated, parameterized commands, jobs, and executors that can be used across multiple projects.
# See: https://circleci.com/docs/orb-intro/
orbs:
  # See the Python orb documentation here: https://circleci.com/developer/orbs/orb/circleci/python
  python: circleci/[email protected]
  browser-tools: circleci/[email protected]
  
# Define a job to be invoked later in a workflow.
# See: https://circleci.com/docs/jobs-steps/#jobs-overview & https://circleci.com/docs/configuration-reference/#jobs
jobs:
  build-and-test:
    # Specify the execution environment. You can specify an image from Docker Hub or use one of our convenience images from CircleCI's Developer Hub.
    # See: https://circleci.com/docs/executor-intro/ & https://circleci.com/docs/configuration-reference/#executor-job
    docker:
      # Specify the version you desire here
      # See:https://circleci.com/developer/images/image/cimg/python
      - image: cimg/python:3.12-browsers

    # Add steps to the job
    # See: https://circleci.com/docs/jobs-steps/#steps-overview & https://circleci.com/docs/configuration-reference/#steps
    steps:
      # Checkout the code as the first step.
      - checkout
      - python/install-packages:
          pkg-manager: pip
          # app-dir: ~/project/package-directory/  # If your requirements.txt isn't in the root directory.
          # pip-dependency-file: test-requirements.txt  # if you have a different name for your requirements file, maybe one that combines your runtime and test requirements.
      # get server up and running in the background
      - run:
          name: Run webserver to be target by tests
          command: python demoapp/server.py
          background: true
      - run:
          name: Run tests
          # This assumes Robot Framework is installed via the install-package step above
          command: robot -x junit.xml -o output.xml login_tests || true
      - run:
          name: Upload results to Xray DC
          command: |
            echo uploading RF output.xml, if available, to Xray...
            [ -f output.xml ] && curl -H "Content-Type: multipart/form-data" -u $XRAY_USERNAME:$XRAY_PASSWORD -F "[email protected]" "$XRAY_SERVER_URL/rest/raven/2.0/import/execution/robot?projectKey=$PROJECT_KEY"
      - store_test_results:
          path: junit.xml
          when: always

# Orchestrate jobs using workflows
# See: https://circleci.com/docs/workflows/ & https://circleci.com/docs/configuration-reference/#workflows
workflows:
  sample: # This is the name of the workflow, feel free to change it to better match your workflow.
    # Inside the workflow, you define the jobs you want to run.
    jobs:
      - build-and-test

When you run this, either manually or because of a commit, the workflow will be run on CircleCI, and there you can see the information on the Test Execution issue created on Xray; in the screenshot it shows "CALC-395", even though CircleCI redacted the project key (i.e. "CALC").

If you go to your Jira/Xray instance, you'll be able to see the new Test Execution along with the results.