How can I reduce the number of jobs in a Buildkite pipeline upload?

Question

I am working on a Buildkite CICD pipeline that occasionally has too many jobs to run and results in the following error:

Failed to upload and process pipeline: Pipeline upload rejected: The number of jobs in this upload exceeds your organization limit of 500. Please break the upload into batches below this limit, or contact support to discuss an increase.

Is there a way to batch upload the pipeline steps for a single pipeline in order to avoid this error?

Answer 1

I was able to resolve this error by refactoring the pipeline to use 'Build Matrices': https://buildkite.com/docs/pipelines/build-matrix and 'Group steps' in the configuration: https://buildkite.com/docs/pipelines/group-step.