A Consumer-Driven Contract Workflow on GitLab

Previously, I built a React front-end application backed by a Spring Boot Java service to demonstrate consumer-driven contract testing with Pact. This was all running locally, of course, so now it's time to take this one step further and get it working on CI/CD.

My CI/CD provider is choice is Gitlab, although I expect the instructions will roughly translate to others. Regardless, the workflow is the same.

I initially wanted to focus on just setting up the basics in Gitlab, but quickly realised that demonstrating the full Pact-flow is far more interesting. So, this post details:

• Building the consumer and provider applications into deployables (Docker images)
• Running all automated tests
• Sharing consumer contracts to the Pact Broker
• Provider verification, using these shared contracts
• Tracking which version of an application (consumer or provider) is being deployed to the production environment
• Performing a safe-to-deploy check prior to deployment

📖 Click here for a fully-worked example of the code in this blog post

The Pact Broker and Workflow

Contracts on the consumer side need to be shared with the service provider. Copy-and-pasting isn't sufficient nor scalable. One could utilise git-enabled service workers to automatically commit updated contracts to a provider's repository. However, I personally find that using the Pact Broker service is far more scalable and brings additional benefits.

The Pact Broker is a service (hosted, or run your own) for storing and sharing contracts. It provides both a REST API and a command-line utility to publish contracts from a consumer build, retrieve contracts in a provider test run and then share these verification results. As a central store of contracts, it builds a matrix of compatible consumer-provider versions. Combined with tracking information about what versions are deployed to an environment, it can then provide deploy-time safety checks. It's the spider, at the middle of a web of services, keeping track of compatibility.

As mentioned, it's possible to host the Pact Broker yourself. For simplicity's sake, I signed up to the free tier on Pactflow.io, which takes care of hosting the Broker and supports up to 5 contracts.

The Pact Broker provides you with a read-only developer authentication token, and another read-write token for CI purposes. I extracted these to environment variables for both the consumer and provider CI/CD builds in Gitlab. I opted to make PACT_BROKER_TOKEN a masked variable, preventing the plain-text value being printed to logs. Neither variable is marked as a protected variable, as I want these to be available to all branches. In Gitlab terminology, a protected branch is one with additional constraints to prevent forced pushes and deletions.

Consumer

The pipeline has four stages:

1. Test - Run the automated test suite, including Pact consumer-driven contract tests
2. Imaging - Generate a Docker image of the frontend app from a production build
3. Publish Contracts - Push contracts to the Pact Broker
4. Deployment - "Deploy" to a production environment

The test stage is simple, performing an install then kicking off Jest tests:

test:
  stage: test
  image: node:10
  variables:
    CI: "true"
  script:
    - npm ci
    - npm test

The imaging step (and this is common between both consumer and provider pipelines) utilises Google's Kaniko builder, to build Docker images from the repository's Dockerfile then push to Gitlab's project container registry. Kaniko is an excellent resource for daemonless Docker builds. And as Gitlab Runners are effectively Docker containers themselves running in Kubernetes, this is an ideal solution for Gitlab. The alternative is to use docker-in-docker.

Getting this to work requires a little jiggery-pokery in Gitlab. As each stage of a pipeline runs in a Docker container (with you specifying the image), any image that contains an ENTRYPOINT runs into trouble with Gitlab's script stage. To fix this I cleared the entrypoint by setting it to [""].

Although the mechanics of the Kaniko build aren't relevant here, in a nutshell it copies some Gitlab environment variables to a config file, so that Kaniko can authenticate with the project's container registry. It builds from a Dockerfile and pushes the image with two tags - one being the commit hash, another a sanitised branch name:

imaging:
  stage: imaging
  image:
      name: gcr.io/kaniko-project/executor:debug
      entrypoint: [""]
  script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG

Once the image is built, I know I've got a potentially deployable unit of code. Now is the time to share any consumer contracts with the Broker, so that compatibility tests can be ran at deploy-time.

Pact's CLI tools are built into a Docker image, which is ideal for Gitlab. However, it also needs the entrypoint fix I used for Kaniko. A particular pecularity of this service! Using some pre-defined environment variables, I specified the consumer Pact version to be the Git commit hash and tag it with the branch it was built from. The Broker URL and Token are passed in too, from the variables I set up earlier.

pact-publish:
  stage: pact-publish
  image:
    name: pactfoundation/pact-cli:latest
    entrypoint: [""]
  script:
    - "pact publish pact/pacts 
        --consumer-app-version=$CI_COMMIT_SHORT_SHA 
        --tag=$CI_COMMIT_REF_NAME 
        --broker-base-url=$PACT_BROKER_BASE_URL 
        --broker-token=$PACT_BROKER_TOKEN"

So far, running this pipeline builds a deployable and publishes the contract to the Pact Broker. The contract is visible in the Broker UI, tagged and versioned appropriately. But notice it's listed as "unverified". Before I can think about deployment, I need to move over to the provider side pipeline.

Provider

The pipeline has three stages:

1. Imaging - Same as the consumer side
2. Pact Provider Verification - Verify the mainline consumer contract against this build
3. Deployment - Same as the consumer side

Imaging is performed with Kaniko, exactly the same as on the consumer pipeline.

The Pact Provider Verification step looks remarkably similar to the docker-compose solution from the previous post, except now that it's querying the Pact Broker, there's some more options passed in. For a general build of the provider, I want to verify the latest consumer contract available. To do that, I look for the master tag. Additionally, passing in --publish-verification-results will post back the result of this verification to the Pact Broker.

pact-provider-verification:
  stage: pact-provider-verification
  image:
    name: pactfoundation/pact-cli:latest
    entrypoint: [""]
  services:
    - name: ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHORT_SHA}
      alias: app
  script:
    - "pact verify 
      --consumer-version-tag=master 
      --provider=pact-provider 
      --provider-app-version=$CI_COMMIT_SHORT_SHA 
      --provider-version-tag=$CI_COMMIT_REF_NAME 
      --provider_base_url=http://app:8081 
      --provider-states-setup-url=http://app:8081/provider-states 
      --pact-broker-base-url=$PACT_BROKER_BASE_URL 
      --broker-token=$PACT_BROKER_TOKEN 
      --wait=10 
      --publish-verification-results"

Now for my simple contract file, running this pipeline produces a verified Pact in the Broker:

From here, it's possible to view the Matrix of pacts in the Broker. The Broker uses a combination of version numbers, tags and file hashes to understand when a contract has actually changed, and conversely when it's unchanged. This is useful because the contracts aren't going to change every single commit. This prevents a n^2 explosion of provider verification tests!

The Matrix view is essentially the visualisation of verified, unverified and incompatible contracts, which can be queried to perform the "can I deploy" check when it's time to deploy a service.

Before moving onto deployment, there's one remaining puzzle to solve...

Webhooks for Provider Verification

What happens when a consumer contract is generated (or updated) and published to the Pact Broker? Does verification of this have to wait until the next provider CI build? Nope! What should happen is that a special provider verification build is triggered on the provider-side. It doesn't need to rebuild the entire application - but it does need to verify the new contract and publish the results.

The Pact Broker can use webhooks to trigger notifications to other systems when a particular event occurs. Gitlab pipelines can also be triggered over HTTP. Putting the two together - a webhook in the Pact Broker triggers a pipeline build in the provider repository. First, I generated an API token from the provider's CI/CD Settings page:

The API token is a secret (no, you can't have it) and I don't want this to be stored in plain-text anywhere. In the Pact Broker, I add this as a secret value with the name ProviderVerificationToken:

When creating the webhook to trigger provider verification, I can reference this secret with ${user.ProviderVerificationToken}. This won't be printed to the logs!

I configured the broker to fire the URL supplied by Gitlab when creating the trigger, but with the ref master. This is because I always want provider verification to be performed against the latest mainline branch of the provider application. Additionally, it's possible to pass additional variables to a Gitlab pipeline via the URL. I passed the consumer tag from the updated consumer contract, so that the triggered pipeline knows what contract to verify:

The full URL takes the form:

https://gitlab.com/api/v4/projects/<project-id>/ref/master/trigger/pipeline?token=${user.ProviderVerificationToken}&variables[PACT_CONSUMER_TAG]=${pactbroker.consumerVersionTags}

A full set of Webhooks and available variables are listed here, opening up a wide range of possible integrations with your own toolchain.

The job to trigger is similar, but different, to the standard provider verification job. The only differences are that the provider app uses the latest image built on master, and the consumer tag is supplied by the incoming variable. Gitlab exposes this as the PACT_CONSUMER_TAG variable. Provider verification is configured to run against the latest mainline image, which has the tag master. To mark it as a job that should execute only from the trigger and not part of a regular CI build, I used the only expression:

pact-new-contract-verification:
  stage: pact-provider-verification
  image:
    name: pactfoundation/pact-cli:latest
    entrypoint: [""]
  services:
    - name: ${CI_REGISTRY_IMAGE}:master
      alias: app
  script:
    - "pact verify 
      --consumer-version-tag=$PACT_CONSUMER_TAG 
      --provider=pact-provider 
      --provider-app-version=$CI_COMMIT_SHORT_SHA 
      --provider-version-tag=master 
      --provider_base_url=http://app:8081 
      --provider-states-setup-url=http://app:8081/provider-states 
      --pact-broker-base-url=$PACT_BROKER_BASE_URL 
      --broker-token=$PACT_BROKER_TOKEN 
      --wait=10 
      --publish-verification-results"
  only:
    - triggers

Conversely, to prevent the trigger performing every other job in the pipeline, I added, where necessary:

except:
    - triggers

To test this, I made a breaking change to the contract on the consumer-side in a new branch, by expecting a different response to /ping:

provider.addInteraction({
  state: "ping",
  uponReceiving: 'a request',      
  withRequest: {
    method: 'GET',
    path: '/ping',
  },
  willRespondWith: {
    status: 200,
    body: "pong!!!!",
  },
})

Pushing this shares a new contract with the broker, and then initiates the provider-verification trigger. Naturally, the build fails, now that the contract expects pong!!! instead of pong.

This provider verification failure is also visible in the Pact Broker. If it ever came to pass that this consumer version would attempted to be deployed, the deploy-time safety check will (correctly) fail.

Ok, now onto those deployment checks!

Deployment

The deployment steps in both the consumer and provider pipelines are exactly the same. Conceptually, they have three sub-steps:

1. Perform a deploy-time safety check? This is provided by Pact-Broker's can-i-deploy functionality.
2. Do the deployment. For simplicity's sake, this is a simple echo of the commit hash.
3. Tag the deployment as successful in the Pact Broker, with create-version-tag.

These all execute within the pact-cli image, because my deployment is simply an echo. This may need to be split into three separate jobs in a more complex setup.

Post-deployment, the contract version is marked as being in production by adding a new tag: production. This is used as part of the can-i-deploy check. Different tags can be used for different environments, such as development and testing.

deploy:
  stage: deploy
  image:
    name: pactfoundation/pact-cli:latest
    entrypoint: [""]
  when: manual
  environment:
    name: production
  only:
    refs:
      - master
  except:
    - triggers
  script:
    - "pact broker can-i-deploy 
      --pacticipant=pact-consumer 
      --version=$CI_COMMIT_SHORT_SHA 
      --to=production 
      --broker-base-url=$PACT_BROKER_BASE_URL 
      --broker-token=$PACT_BROKER_TOKEN"
    - echo "Deploying ${CI_COMMIT_SHORT_SHA} to production!"
    - "pact broker create-version-tag 
      --pacticipant=pact-consumer
      --version=$CI_COMMIT_SHORT_SHA 
      --tag=production 
      --broker-base-url=$PACT_BROKER_BASE_URL 
      --broker-token=$PACT_BROKER_TOKEN"

Next Steps

Returning to the task list from the last post, I've covered off quite a bit here:

• ✔️ Get this working on Gitlab CI
• Create a more complex interaction and utilise provider states
• ✔️ Share Pactfiles between the consumer and provider in a better way than copy/paste
• ✔️ Demonstrate how contracts can be used to perform deploy-time safety checks

So now both applications have a working CI/CD integration with the Pact Broker, it's time tofinish off this series by looking at how to write good Pactfiles and make the most of contract testing. I'll also look into how provider states can be used to isolate behaviour between interactions. Stay tuned!