Gitlab-ci stage order ignored when rules includes changes

Question

consider the following gitlab-ci.yaml, for a mono repo with multiple microfrontends

stages:
 - build
 - deploy

build:app1:
  stage: build
  script: 
    - sleep 30
    - mkdir dist1
    - touch dist1/output1.html
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
    changes:
       - app1/src/*
  artifacts:
    paths:
      - dist1
 

build:app2:
  stage: build
  script: 
    - sleep 30
    - mkdir dist2
    - touch dist2/output2.html
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
    changes:
       - app2/src/*
  artifacts:
    paths:
      - dist2

deploy:all:
   stage: deploy
   script:
      - mkdir dist
      - cp dist1/* dist
      - cp dist2/* dist
      - deploy.sh ./dist
   artifacts:
     paths:
       - dist

when ran the order defined in stages is ignored and both the build and deploy jobs run simultaneously causing a failure for the "deploy:all" job (since its still "building")

if i remove the condition for the changes the stage order is respected and build runs before deploy

how can i both only act on changes and enforce the defined build order?

in the real monorepo there are 10's of micro frontends not just 2 ...

Answer 1

When I run your gitlab_ci.yml in Gitlab CI, it does not run the build and deploy jobs simultaneously. It runs a merge request pipeline (with the build jobs) and a branch pipeline (with the deploy job). Since the artifacts from the build jobs are created in the merge request pipeline, they aren't available to the branch pipeline, and so the deploy job fails.

It's difficult to say how to fix this without knowing what your intention is, but you need to run the build and deploy jobs in the same pipeline, so either have the deploy job run with

rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'

or have the build jobs run in a branch pipeline.

Additionally it's not clear to me how you're expecting the deploy job to always work, as if you only run the build jobs when there are changes sometimes not every build job will have an artifact and your deploy job will fail.

Answer 2

In your case, you can try to use needs keyword in order to force relationships between jobs: https://docs.gitlab.com/ee/ci/yaml/#needs
You also nee to add optional: true since you have rules conditions in your build jobs.
You can add it in your deploy:all job like this:

deploy:all:
  needs:
    - job: build:app1
      optional: true
    - job: build:app2
      optional: true
  stage: deploy
  script:
    - mkdir dist
    - cp dist1/* dist
    - cp dist2/* dist
    - deploy.sh ./dist
  artifacts:
    paths:
      - dist

Answer 3

Your deploy stage might need the condition for changes as well. Unless maybe you need the deploy:all stage to always run no matter if there is changes or not.

Also indicating that the deploy stage needs artifacts is always a good idea in your case because your deploy stage is using the artifacts anyways

needs:
  - job: build:app1
    artifacts: true
  - job: build:app2
    artifacts: true