NOTE: If you intend on using techniques such as these and allowing such wide open functionality in Jenkins, I recommend that you run your entire Jenkins build system without outbound internet access by default. Allow access from the build system network segment only to approved endpoints while dropping the rest of the traffic. This will allow you to use potentially dangerous, but extremely powerful scripts while maintaining a high level of security in the system.

API Calls

Making HTTP calls in a Jenkinsfile can prove tricky as everything has to remain serializable else Jenkins cannot keep track of the state when needing to restart a pipeline. If you’ve ever received Jenkins’ java.io.NotSerializableException error in your console out, you know what I mean. There are not too many clear-cut examples of remote API calls from a Jenkinsfile so I’ve created an example Jenkinsfile that will talk to the Docker Hub API using GET and POST to perform authenticated API calls.

Explicit Versioning

The Docker image build process leaves a good amount to be desired when it comes to versioning. Most workflows depend on the :latest tag which is very ambiguous and can lead to problems being swallowed within your build system. In order to maintain a higher level of determinism and auditability, you may consider creating your own versioning scheme for Docker images.

For instance, a version of 2017.2.1929 #<year>-<week>-<build #> can express much more information than a simple latest . Having this information available for audits or tracking down when a failure was introduced can be invaluable, but there is no built-in way to do Docker versioning in Jenkins. One must rely on an external system (such as Docker Hub or their internal registry) to keep track of versions and use this system of record when promoting builds.

This versioning scheme we are using is not based on Semver, but it does encode within it the information we need to keep versions in lock and also will always increase in value. Even if the build number is reset, the date + week will keep the versions from ever being lower that the day previously. Version your artifacts however works for your release, but please make sure of these two things:

The version string never duplicates

The version number never decreases

Interacting with the Docker Hub API in a Jenkinsfile

For this example we are going to connect to the Docker Hub REST API in order to retrieve some tags and promote a build to RC. This type of workflow would be implemented in a release job in which a previously built Docker image is being promoted to a release candidate. The steps we take in the Jenkinsfile are:

Provision a node Stage 1 Make an HTTP POST request to the Docker Hub to get an auth token Use the token to fetch the list of tags on an image Filter through those tags to find a tag for the given build # Stage 2 Promote (pull, tag, and push) the tag found previously as ${version}-rc Push that tag to latest to make it generally available



This is a fairly complex looking Jenkinsfile as it stands, but these functions can be pulled out into a shared library to simplify the Jenkinsfile. We’ll talk about that in another post.

Jenkinsfile

# ! groovy /* NOTE: This Jenkinsfile has the following pre-requisites: - SECRET (id: docker-hub-user-pass): Username / Password secret containing your Docker Hub username and password. - ENVIRONMENT: Docker commands should work meaning DOCKER_HOST is set or there is access to the socket. */ import groovy.json.JsonSlurperClassic ; // Required for parseJSON() // These vars would most likely be set as parameters imageName = "technolog/serviceone" build = "103" // Begin our Scripted Pipeline definition by provisioning a node node () { // First stage sets up version info stage ( 'Get Docker Tag from Build Number' ) { // Expose our user/pass credential as vars withCredentials ([ usernamePassword ( credentialsId: 'docker-hub-user-pass' , passwordVariable: 'pass' , usernameVariable: 'user' )]) { // Generate our auth token token = getAuthTokenDockerHub ( user , pass ) } // Use our auth token to get the tag tag = getTagFromDockerHub ( imageName , build , token ) } // Example second stage tags version as -release and pushes to latest stage ( 'Promote build to RC' ) { // Enclose in try/catch for cleanup try { // Define our versions def versionImg = "${imageName}:${tag}" def latestImg = "${imageName}:latest" // Login with our Docker credentials withCredentials ([ usernamePassword ( credentialsId: 'docker-hub-user-pass' , passwordVariable: 'pass' , usernameVariable: 'user' )]) { sh "docker login -u${user} -p${pass}" } // Pull, tag, + push the RC sh "docker pull ${versionImg}" sh "docker tag ${versionImg} ${versionImg}-rc" sh "docker push ${versionImg}-rc" // Push the RC to latest as well sh "docker tag ${versionImg} ${latestImg}" sh "docker push ${latestImg}" } catch ( err ) { // Display errors and set status to failure echo "FAILURE: Caught error: ${err}" currentBuild . result = "FAILURE" } finally { // Finally perform cleanup sh 'docker system prune -af' } } } // NOTE: Everything below here could be put into a shared library // GET Example // Get a tag from Docker Hub for a given build number def getTagFromDockerHub ( imgName , build , authToken ) { // Generate our URL. Auth is required for private repos def url = new URL ( "https://hub.docker.com/v2/repositories/${imgName}/tags" ) def parsedJSON = parseJSON ( url . getText ( requestProperties: [ "Authorization" : "JWT ${authToken}" ])) // We want to find the tag associated with a build // EX: 2017.2.103 or 2016.33.23945 def regexp = "^\\d{4}.\\d{1,2}.${build}\$" // Iterate over the tags and return the one we want for ( result in parsedJSON . results ) { if ( result . name . findAll ( regexp )) { return result . name } } } // POST Example // Get an Authentication token from Docker Hub def getAuthTokenDockerHub ( user , pass ) { // Define our URL and make the connection def url = new URL ( "https://hub.docker.com/v2/users/login/" ) def conn = url . openConnection () // Set the connection verb and headers conn . setRequestMethod ( "POST" ) conn . setRequestProperty ( "Content-Type" , "application/json" ) // Required to send the request body of our POST conn . doOutput = true // Create our JSON Authentication string def authString = "{\"username\": \"${user}\", \"password\": \"${pass}\"}" // Send our request def writer = new OutputStreamWriter ( conn . outputStream ) writer . write ( authString ) writer . flush () writer . close () conn . connect () // Parse and return the token def result = parseJSON ( conn . content . text ) return result . token } // Contain our JsonSlurper in a function to maintain CPS def parseJSON ( json ) { return new groovy . json . JsonSlurperClassic (). parseText ( json ) }

Script Security

Due to the nature of this type of script, there is definitely a lot of trust assumed when allowing something like this to run. If you follow the process we are doing in Modern Jenkins nothing is getting into the build system without peer review and nobody but administrators have access to run scripts like this. With the environment locked down, it can be safe to use something of this nature.

Jenkins has two ways in which Jenkinsfiles (and Groovy in general) can be run: sandboxed or un-sandboxed. After reading Do not disable the Groovy Sandbox by rtyler (@agentdero on Twitter), I will never disable sandbox again. What we are going to do instead is whitelist all of the required signatures automatically with Groovy. The script we are going to use is adapted from my friend Brandon Fryslie and will basically pre-authorize all of the required methods that the pipeline will use to make the API calls.

Pre-authorizing Jenkins Signatures with Groovy

URL: http://localhost:8080/script

import org.jenkinsci.plugins.scriptsecurity.scripts.ScriptApproval println ( "INFO: Whitelisting requirements for Jenkinsfile API Calls" ) // Create a list of the required signatures def requiredSigs = [ 'method groovy.json.JsonSlurperClassic parseText java.lang.String' , 'method java.io.Flushable flush' , 'method java.io.Writer write java.lang.String' , 'method java.lang.AutoCloseable close' , 'method java.net.HttpURLConnection setRequestMethod java.lang.String' , 'method java.net.URL openConnection' , 'method java.net.URLConnection connect' , 'method java.net.URLConnection getContent' , 'method java.net.URLConnection getOutputStream' , 'method java.net.URLConnection setDoOutput boolean' , 'method java.net.URLConnection setRequestProperty java.lang.String java.lang.String' , 'new groovy.json.JsonSlurperClassic' , 'new java.io.OutputStreamWriter java.io.OutputStream' , 'staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods findAll java.lang.String java.lang.String' , 'staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods getText java.io.InputStream' , 'staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods getText java.net.URL java.util.Map' , // Signatures already approved which may have introduced a security vulnerability (recommend clearing): 'method java.net.URL openConnection' , ] // Get a handle on our approval object approver = ScriptApproval . get () // Aprove each of them requiredSigs . each { approver . approveSignature ( it ) } println ( "INFO: Jenkinsfile API calls signatures approved" )