In this post, would like to share my experience on creating a common CI pipeline for multiple projects which has similar build requirements but they would want to customize few parameters as per their needs.

Problem Statement

A CI pipeline to build container images is created & tested for a particular application
Similar approach has to be cascaded to other applications and projects (which are hosted as separate groups & projects underneath them)
Now, the problem statement to grapple with is how to let other applications & projects re-use pipeline code without added burden of sharing code with every team, future bug fixes roll-out, avoid developers tampering with pipeline code itself etc.,

In order to achieve this, there were 2 options,

Option # 1 - Share pipeline code (.gitlab-ci.yml) which I created for existing project with other project & ask them to deploy and run in their group / project
Option # 2 - Look for a possible generic approach where teams can uplink their project git repo to a common code, and re-use this code with some level of configuration changes based on project needs.

In order to achieve Option # 2 - there were couple of options GitLab provides like "parent-child pipelines" or "multi-project pipelines". However, they didn't fit my requirement for few reasons,

I don't want a single group or project to initiate build project for others, every project should be able to trigger their CI at their own need
Project teams should be able to pass configuration parameters of their choice & pipeline code should react according to passed params
Most importantly, once CI pipeline is part of project git repo itself, then anyone can modify build code or tamper with, which should be avoided

Reference to Downstream pipeline can be found here - https://docs.gitlab.com/ee/ci/pipelines/downstream_pipelines.html

Solution

After some research, found an easier configurable solution for my need in GitLab documentation in below link, https://docs.gitlab.com/ee/ci/pipelines/settings.html#custom-cicd-configuration-file-examples

This feature of GitLab gave me an option to create a central group in GitLab & application wise projects within the group to maintain pipeline code.

Here is the design,

How to link downstream group or projects to lookup central CI pipeline code?

Under Project Settings > CI/CD > General Pipelines > modify "CI/CD configuration file" field to link with upstream central pipeline configuration file.
For example, my central CI pipeline code group structure is as below,

    central_pipeline_group
        |__ application_a
            |__ v1.0
                |__ src
                    |__ .app_a_gitlab-ci.yml
        |__ application_b
            |__ v1.0
                |__ src
                    |__ .app__gitlab-ci.yml

Downstream project A should refer upstream pipeline as,

v1.0/src/.app_a_gitlab-ci.yml@central_pipeline_group/application_a:master

and Downstream project B should refer upstream pipeline as,

v1.0/src/.app_b_gitlab-ci.yml@central_pipeline_group/application_b:master

Conclusion

Above setup helped in achieving below requirements,

CI pipeline code is maintained centrally & doesn't need to copied at project level
Access controlled & not modifiable at project level
Downstream projects can trigger their pipelines based on their need
Any bug fix in central pipeline is automatically propagated to all downstream projects due to runtime reference of central code & not static at the project level
Additionally, central pipeline code gives the option to work on version upgrades of central pipeline code in isolation, which can be released to projects once upgraded & tested. When released, project teams have to just change their CICD configuration file location at project level settings to trigger new CICD pipeline code version.

In next post, I shall share details on how to build on top of this design to "pass project specific configuration parameters to central pipeline code".

References

https://docs.gitlab.com/ee/ci/pipelines/settings.html#custom-cicd-configuration-file-examples

Jagadish Prasad Sanaka