When I started my internship, our Jenkins server had a lot of Jenkins pipelines, and some of them shared the same scripts. The jobs that shared the same script were named differently so that people could identify what platform and spec the build just ran on. These pipelines would continuously evolve with revisions coming to the pipeline scripts that are held on GitHub. When these pipelines change, normally none of the configurations need to be changed. However, sometimes a parameter is added, giving the Jenkins job more functionality. A parameter may be added to allow the user to add a custom build description when they launch a job or to add a slack channel to notify if a build fails. If a parameter changes, it would need to be added to each individual job, which is a very repetitive task. There are 100s of jobs that would need to be manually updated which is prone to human error. If a new platform or release is supported, over 20 new jobs would need to be generated. I was tasked with a way to automate this.
When I started to look into the automated process of doing this task, I investigated how to update the jobs via code. I found that the Job DSL plugin would be an appropriate plugin for the task at hand. I noticed that this plugin is already being used at AdoptOpenJDK for their pipeline scripts to achieve the same end goal. One of the requirements for this was that user would no longer have to actively generate the pipeline jobs. The best place for the job to generate was from within the pipeline itself. This could be done because of the way the current pipelines work.
Our pipelines have a top level job that, with given parameters, would then start the appropriate downstream jobs. This way, if a parameter changed for a downstream job, it would be automatically generated with the new configuration. With the new change, it would create downstream jobs automatically and will then be launched. Another benefit of this automatic generation is that if a new version of java is released, or if we are supporting a new system, we just need to worry about the first pipeline job.
To summarize, our Jenkins pipeline consists of many different jobs, and many of them are similar or relate to each other. One starts the top level job which, depending on the parameters, would launch different downstream jobs. Now, whenever a downstream job is launched, the configuration would be generated so that it is always the most up the date current job.