All the hard work in setting up consumer and provider apps with a fully-integrated CI/CD workflow is now done. Hooray! Now though, it's time to turn our eyes towards good practice when it comes to writing contract tests for realsies.

đź“– Click here for a fully-worked example of the code in this blog post

The contract that I generated for the provider's /ping endpoint - that it returns a hard-coded piece of text - is a bit lacking to put it politely. For demonstration purposes of setting up a workflow, it's sufficient. As a reminder, this is what it looks like:

{
  "consumer": {
    "name": "pact-consumer"
  },
  "provider": {
    "name": "pact-provider"
  },
  "interactions": [
    {
      "description": "a request",
      "providerState": "ping",
      "request": {
        "method": "GET",
        "path": "/ping"
      },
      "response": {
        "status": 200,
        "headers": {
        },
        "body": "pong"
      }
    }
  ],
  "metadata": {
    "pactSpecification": {
      "version": "2.0.0"
    }
  }
}

Pretend you didn't see that, as we discuss how to make the most from contract testing.

What Should Contracts Test?

Contract testing and functional testing are distinct activities. Be careful about mixing them. Whenever a consumer writes a contract, it places constraints upon the provider's API. This isn't itself a bad thing - the purpose of a contract test is to encode the shared understanding about usage, so that an automated signal exists for breaking changes. The constraint is "don't break my usage". But this constraint, enforced too particularly, limits the ability for the API to evolve.

Contract testing veers into functional testing when the contracts end up expressing the business logic of a provider. Instead, contracts should be written from the perspective of the consumer: how is the API used? What must be constrained to prevent a breakage in the consumer application?

In the example above, the consumer app is calling the simple /ping endpoint and displaying the response text. Does it matter if that changes from 'pong' to something elese? As the consumer app currently uses it - not at all! The constraint can be loosened into a type - the contract expresses the response must be a string.

There will be times when an exact value will be necessary to assert, I'm sure of it. There may be times where a certain format is expected, such as a URL containing http:// or https:// (so from a consumer's perspective, their usage in an <a> tag works correctly). There will also be times where a particular structure to the response object needs to be specified in the contract.

Pact contract tests let contracts be expressed through:

  • Exact values
  • Types (string, boolean, integer, uuid, and some built-in date-time matching!)
  • Regex
  • Object/array 'shapes'

The exercise of consumer-driven contract tests is not for a consumer to dictate how an API behaves. It's to have a conversation and form the shared understanding. Constraints placed in a contract should be expressed as loosely as possible, whilst still giving convidence that the consumer use case is satisfied. Keep this principle in mind while developing contract tests to ensure the contracts don't become proxy-functional tests.

Provider States

Taking a step back, it's common to see a Given / When / Then pattern form in testing. Also expressed as Arrange / Act / Assert, the idea is that sometimes there's some pre-test setup that's needed to establish the context of the item under test. Contract tests are no different - think of applications that communicate with a database, or have user-driven workflows. When contract testing a particular API, it's useful to be able to request that the provider performs some setup code first.

This is where provider states are useful. The simple /ping example from earlier in the series did specify a provider state - but it was being ignored. So let's do something more interesting!

I'm creating a new API so that for a given user, it returns some favourite quotations from TV shows. The rules of the contract are that:

  • When there's no user, return 404
  • When there's a specified user, return a list of quotations
  • Quotations need to have the quote text, an author and a source in the form of a URL.

Why is there no need to contract test the empty list case, you may ask? Well, this is where it's worth recalling the line between contract and functional testing. For me, that leans closer to the functional testing side of things, because the contract around the 'shape' of the array satisfies the contract. So, I opted to omit this case.

Consumer

On the consumer side, I can express this as a contract test:

import { AxiosError, AxiosResponse } from "axios";
import { pactWith } from 'jest-pact';
import { Matchers } from '@pact-foundation/pact';
import QuoteService from './quoteService';

pactWith({ consumer: 'pact-consumer', provider: 'pact-provider', cors: true }, provider => {
  let quoteService : QuoteService;

  beforeEach(() => {
    quoteService = new QuoteService(provider.mockService.baseUrl)
  });

  describe('Quotes API', () => {
    it('not found when no quotes', async () => {
      await provider.addInteraction({
        state: "no user",
        uponReceiving: "a request for quotes",
        withRequest: {
          method: 'GET',
          path: '/quotes'
        },
        willRespondWith: {
          status: 404
        }
      });

      try {
        await quoteService.getAQuote();
      } catch (err) {
        expect(err.response.status).toBe(404);
      }
    });

    it("receives quotes", async () => {
      await provider.addInteraction({
        state: "a comedy-loving user",
        uponReceiving: 'a request for quotes',      
        withRequest: {
          method: 'GET',
          path: '/quotes',
        },
        willRespondWith: {
          status: 200,
          body: Matchers.eachLike({
            quote: "I'm Pickle Rick!",
            author: "Rick Sanchez",
            source: Matchers.regex({
              matcher: "http(s)?:\/\/([^?#]*)",
              generate: "https://www.imdb.com/title/tt5218268/"
            })
          })
        },
      });

      const response: AxiosResponse = await quoteService.getAQuote();
      expect(response.status).toBe(200);
      expect(response.data.length).toEqual(1);
    });
  });
});

Here, I'm using a mix of Pact features. The source URL is specified using a (rather naive) regex - as it's important to the consumer app this link is fully-resolvable. Using eachLike is a handy convenience method for specifying an array of JSON objects, based on a sample object. This sample object uses strings for the quote and author, but unlike the earlier contract for /ping, these aren't absolute values. Rather, they're matching the type of string, but using the sample values as an example.

Recall I'm using the jest-pact library, which is a wrapper on pact-js. This means after each test I don't need to call verify myself. But, I do need to await the call to provider.addInteraction. As there's multiple of these now, I opted to move individual interactions into each test, rather than in the catch-all beforeEach.

Note that each test has a different state value. These are provider states which must be implemented on the provider side, during provider verification.

Provider

There's now a Plain Ol' Java Object to represent a quote:

public class Quote {
  private String quote;
  private String author;
  private String source;

  public Quote(String quote, String author, String source) {
    this.quote = quote;
    this.author = author;
    this.source = source;
  }

  public String getQuote() { return quote; }
  public String getAuthor() { return author; }
  public String getSource() { return source; }
}

Alongside a service class which has a pre-canned Quote response. As a @Component, it's injectable in the Spring Boot context and effectively made a singleton. This means that I can define a public method to modify the pre-canned response - which is what I'll be doing later during provider verification to satisfy the provider states.

@Component
public class QuoteService {
  private List<Quote> quotes = Arrays.asList(
    new Quote(
      "You Come At The King, You Best Not Miss.",
      "Omar Little",
      "https://www.imdb.com/title/tt0749433/"
    ));

  public List<Quote> getQuotes() {
    return quotes;
  }

  public void setQuotes(List<Quote> quotes) {
    this.quotes = quotes;
  }
}

Finally, the new GET endpoint for quotes. This uses Spring Boot dependency injection to @Autowire the singleton instance of QuoteService. Upon receiving a call to get quotes, it gets the list from the service. If it's empty a 404 error is returned. Otherwise, the list is returned as a JSON array. Sure, this isn't production-quality either, but it demonstrates the contract well enough.

@SpringBootApplication
@RestController
@CrossOrigin
public class DemoApplication {

  @Autowired
  private QuoteService quoteService;

  @GetMapping("/quotes")
  List<Quote> quote() {
    List<Quote> quotes = quoteService.getQuotes();
    if (quotes.isEmpty()) {
      throw new ResponseStatusException(HttpStatus.NOT_FOUND);
    }
    return quotes;
  }

}

Calling localhost:8081/quotes produces:

[
  {
    "quote": "You Come At The King, You Best Not Miss.",
    "author": "Omar Little",
    "source": "https://www.imdb.com/title/tt0749433/"
  }
]

Now, onto provider verification. Note that I'd previously mentioned the provider application performs two variants of provider verification: one using pact-cli (where the app is treat as a black box), another using junit5-spring (which uses JUnit and can interact with the Spring setup at test-time). This is complete overkill - choose one of them only. I only did this for completeness in an earlier post. But, I'll continue, and implement provider states in both ways.

Provider Verification via pact-cli

To the DemoApplication class, I added a POST endpoint to handle provider states. Provider verification, when performed using the pact-cli tool, posts a JSON object for each provider state it encounters:

{ consumer: “consumer-name”, state: “state-name” }

For convenience sake, I mapped a POJO object for this. For the provider states known about, the QuoteService’s quote list is modified:

@PostMapping("/provider-states")
public void providerStates(@RequestBody ProviderState providerState) {
  if ("pact-consumer".equals(providerState.getConsumer())) {
    if ("no user".equals(providerState.getState())) {
      quoteService.setQuotes(Collections.emptyList());
    } else if ("a comedy-loving user".equals(providerState.getState())) {
      quoteService.setQuotes(Arrays.asList(
        new Quote(
        "I don’t know how many years I got left on this planet, I’m going to get real weird with it.",
        "Frank Reynolds",
        "https://www.imdb.com/title/tt1504565/")
      ));
    }
  }
}

When running pact-cli verify, there's an option to pass in

Provider Verification via junit5-spring

Curiously, I couldn’t find a way to get the junit5-spring library to work with a provider state url, so I ended up duplicating the provider state setup in the tests. As this is a Spring Boot test, the test class itself can utilise the DI context, so it’s possible to inject the same instance of QuoteService that the application receives, and configure it appropriately. Each provider state is given its own method to perform its setup, and annotated with @State("<the state name>") so that it is called at the appropriate time.

@Provider("pact-provider")
@PactBroker
@SpringBootTest(webEnvironment = WebEnvironment.RANDOM_PORT)
public class ProviderVerificationTests {
  @LocalServerPort
  private int port;

  @Autowired
  private QuoteService quoteService;

  @BeforeEach
  void before(PactVerificationContext context) {
      context.setTarget(new HttpTestTarget("localhost", port));
  }

  @TestTemplate
  @ExtendWith(PactVerificationInvocationContextProvider.class)
  void pactVerificationTestTemplate(PactVerificationContext context) {
    context.verifyInteraction();
  }

  @State("no user")
  public void setupEmptyState() {
    quoteService.setQuotes(Collections.emptyList());
  }

  @State("a comedy-loving user")
  public void setupUserState() {
    quoteService.setQuotes(Arrays.asList(
        new Quote(
          "I don’t know how many years I got left on this planet, I’m going to get real weird with it.",
          "Frank Reynolds",
          "https://www.imdb.com/title/tt1504565/")
    ));
  }
}

Test Output

When running Provider verification, the output looks quite like the Given / When / Then format. This shows that it's useful to think of the provider state in that way. Don't overspecify the provider state - exactly how the provider wants to implement that is their business - but it forms part of the conversation and shared understanding between the consumer and provider. My provider state represents a use-case: "a comedy-loving user", rather than an overspecified "a user who has a Frank Reynolds quotation saved".

Verifying a pact between pact-consumer and pact-provider
  Given no user
    a request for quotes
      with GET /quotes
        returns a response which
          has status code 404
  Given a comedy-loving user
    a request for quotes
      with GET /quotes
        returns a response which
          has status code 200
          has a matching body

Wrapping Up

Well, this has been fun! Over four posts, I've built a full contract testing workflow using Pact Broker and Gitlab. It's been fun! I've learnt a lot. I hope you have too. Please get in touch (details in footer) if you'd like to discuss more in private, or have any suggestions for where to go next.

In my opinion, contract testing is an essential step in creating, maintaining and operating a distributed service-oriented architecture. Done well, it avoids the need for "big bang" deployments and costly phases of manual end-to-end testing. With integrations such as Pact Broker, it can also act as a vehicle for Continuous Delivery by letting individual services be deployed to an environment with surety they're compatible with the existing setup in that environment. Obviously, other contract testing techniques than Pact exist, but what I like especially about the Pactflow is how it's specifically designed to be an enabler of good practices.