In Concourse, is it possible AT ALL to select a git resource branch interactively in web UI or fly CLI?

Question

F.e. normally you use "develop", but today you want to use "patchXYZ"... Without changing the pipeline, is there a way to prompt the user for the branch name? If not, what is the best practice with the Concourse for this flow?

Answer 1

You cannot change the branch name from the web UI and I think that it is actually a good idea, a pipeline should represent always the same branch.

What I suggest is to have two types of pipelines for a given project:

1. The main pipeline. This has a canonical name, such as projectA-main, and it follows the main branch of the repo (main, master, develop,...). it is long-lived.
2. A pipeline per feature branch. This is short-lived. Following your example, it could have a name like projectA-patchXYZ.

We now need a way to avoid the pain (and mistakes) to manually maintain the different pipelines. This is actually easy to do, using Concourse ((vars)) (see documentation).

You would still have only one pipeline file, but templatized using these ((vars)). For your example with the git branch name:

resources:
  - name: repo
    type: git
    icon: git
    source:
      uri: https://github.com/marco-m/concourse-pipelines.git
      branch: ((branch))

Then you would set the pipeline with fly, expanding that variable:

$ BRANCH=$(git branch --show-current) \
  fly -t ci set-pipeline \
    -p projectA-$BRANCH -c ... \
    -y branch=$BRANCH

(see the fly documentation for -y).

Once done, you should remember to destroy that branch pipeline, otherwise they will build up, eating resources.

Or you could keep resetting the same branch pipeline, effectively obtaining what you wanted. In this case the pipeline name should stay the same, something like

$ BRANCH=$(git branch --show-current) \
  fly -t ci set-pipeline \
    -p projectA-branch -c ... \
    -y branch=$BRANCH

Last note: when having more than one pipeline for a given project, one must also think about side effects: effects that are visible outside the pipeline.

Classic example is pushing an artifact to S3. The two pipelines (main and branch) will put stuff in the same place, which might not be what you want.

In this case, the same ((branch)) trick can be used in the regexp for the S3 resource.

You can have a look at my sample pipelines at marco-m/concourse-pipelines and you might find useful my marco-m/concourse-in-a-box, an all-in-one Concourse CI/CD system based on Docker Compose, with Minio S3-compatible storage and HashiCorp Vault secret manager. This enables to learn Concourse pipelines from scratch in a simple and complete environment.