Using Artifactory for Jenkins build artifact storage

Every build that we launch on Jenkins needs to temporarily archive artifacts, so that it can be tested, debugged, or both. In addition, every test that fails also needs to stash artifacts so that it can be debugged.

When I first started my internship, the artifacts were being saved on the Jenkins master for a certain period of time or until a build limit was reached. The problem with hosting the artifacts on Jenkins is that there is a limited amount of space, and we could not keep the artifacts for as long as we would like. In some cases, the artifacts would be deleted before someone even had the opportunity to debug them. In order for us to keep the artifacts for an extended period of time, we needed to look into alternate storage options. One of our available machines had a large partition of available extra storage, so we decided to use this machine. We just needed to find some software to act as the host with the machine. Our code already supports uploading to Artifactory, so it could act as familiar and reusable option (and they also have a version for open source). We tested out the free version on our available machine, to ensure that it performed satisfactorily. We testing the amount of time that it took to upload and download artifacts and it performed better than the Jenkins master.  We now have a working server that can store our artifacts.

Having solved the storage issue, we hit a new problem. Within the new Artifactory server, the jobs are not deleted automatically because the free version does not support build retention. My solution to this problem was to have a separate job that utilized the Artifactory REST API to find the builds that need to be deleted. The new script gets called from within the pipeline scripts to make sure that there are at most X amount of builds (X being a tunable parameter). The script gets called every weekend to delete any build that has been stored for an extended period of time; it makes sure that we do not go over our storage capacity.

To summarize, we had a storage issue with our current setup, and we needed to find a new way to store our artifacts. I ended up installing an Artifactory server that has a higher storage capacity, and better transfer speeds, to fix the problem. I also wrote cleanup scripts to ensure that the new solution would not be overfilled with artifacts and cause more problems in the future.

Leave a Reply