It’s such a common scenario that most developers run into it within a few months of writing their first Go programs: your program makes HTTP requests to an external service to perform an everyday task, such as fetching a list of repositories from GitHub. As a responsible developer, you want to write unit tests without actually hitting the service with network requests.

You are also most likely coming to Go from other programming languages, and as a responsible developer, you’ve used a mocking framework like WireMock, nock, webmock, or something else in your language that allowed you to mimic external server responses. These helped to make you confident that your app was making the correct outbound requests and correctly handling different responses.

You figure this is a very common use case (and many other people have been in this situation) so surely the obvious way to do this in Go will be in the top few search results on Google. This is a perfectly reasonable assumption and often works. Unfortunately, what I’ve seen is that some good developers have come up with suboptimal solutions to the problem, shared their solutions with the community, and due to positive community reinforcement have dominated Google search results.

In fact, the Go creators already had devised a solution but didn’t optimize for the search engine (a bit ironic when Google is the shepherd of Go). Before we look at the native solution, let’s take a look at some common alternatives and their shortcomings.

A simple scenario

To illustrate, suppose we started working on a codebase and found the following untested function, and now we want to add a test.

https://gist.github.com/sonya/271b8a1f82d4c6a3a36731ec2d3cfbdc#file-contrived_scenario-go

There are a number of conditional branches in this function, but for now we just want to test the successful case. We want to make sure

  • We’re hitting the correct URL path
  • We’re passing the correct Accept header
  • We’re returning the string that is the value of the “value” key in the JSON response

Fashionable, but suboptimal: the interface approach

This approach will be familiar to those coming from duck-typed languages. This blog post by Sophie DeBenedetto does an excellent job explaining this approach (the post is actually quite good even though I disagree with the solution). In short, it works by mocking the built-in http.Client by defining an interface with a Do method, which is implemented by both http.Client and a mock version.

In the scenario we laid out, we need to change the source file so that Client is a package-level variable that implements the common interface:

https://gist.github.com/sonya/788779802ae320cd31ff761345fe6774#file-interface_approach-go

In the test file we have:

https://gist.github.com/sonya/adbe341c3df10370888e4115a8a6f3c7#file-interface_approach_test-go

How does this test do on the three assertions we wanted to make? It does everything we needed:

  • The handler checks the URL path and returns an error if it doesn’t match
  • The handler checks the Accept header and returns an error if if it doesn’t match
  • The return value of the function is checked against the string fixed, which we seeded in the mock handler

However, take note of how many changes we made:

  • Not only did we have to add new definitions to the source file, we actually had to alter the source code of the function we were trying to test
  • We are now exposing a HTTPClient as a package-level variable, which has a number of problems:
  • Anyone can read and write to it, which makes it hard to track down where the variable is being changed.
  • This variable exists for the sole purpose of enabling tests. Without looking at the test code, a developer might be confused about why it’s a variable
  • Also, because anyone can read and write to it, someone can accidentally or maliciously set it to something that breaks other parts of the program.

The package-level variable is an example of using the singleton pattern to manage state across tests, which can result in subtle bugs when not managed carefully. Suppose the external service you’re calling is actually a microservice under your control, and someone comes along trying to write an integration test for it. They run the service in a local Docker container on port 8080, make it return {“value":"actualvalue"}, and write this very simple test:

https://gist.github.com/sonya/b61ede65abf83b33e7716a55170b9288#file-interface_approach_integration_test-go

To their surprise, the tests fails with the message: Expected 'actualvalue', got fixed. They think there is a bug in their own service, but no matter what changes they make, the result remains unchanged. It might take them some extra hours of hair-pulling before they realize the global Client variable was stuck in the state your last unit test set it to.

Convenient, but suboptimal: the external library approach

There are third party Go libraries out there that provide APIs for mocking outbound HTTP requests, such as gock and httpmock. Both of these offer some of their own special sauce, but either way they still come with the overhead of a third party dependency.

httpmock

The httpmock library allows you to set up responders for a given request method and URL. It takes advantage of the fact that there is an existing global variable called http.DefaultTransport that configures how all newly created instances of http.Client make connections.

To test our sample function, we could write a test like this:

https://gist.github.com/sonya/99e23752f6cd967c6fe5596c646cee15#file-httpmock_test-go

In this example, the header and final value are checked the same way as the test where we mocked http.Client, whereas the URL path is checked implicitly by matching.

The matching approach makes test failures a bit harder to interpret because the error message will be the result of not running the responder, which might be something like invalid memory address or nil pointer dereference instead of the friendlier message Expected to request '/fixedvalue', got: %s in the previous version. We can address this using a regex in the responder and capturing what was called with httpmock.GetTotalCallCount().

The use of http.DefaultTransport is concerning here. Since http.DefaultTransport is overridden during the duration of the test, this means that when using httpmock, developers are unable to test any changes that they intentionally made to http.DefaultTransport in the source. Moreover, if someone forgets to call Activate() and Deactivate() at the beginning of the test, they could leave the http package in an undesired state for later tests.

gock

gock offers a feature-rich and concise API that includes matching of headers and path patterns. We can write a very concise test for our sample function:

https://gist.github.com/sonya/3ea4f2bcebfcaed2c8c3cf7be7f62f34#file-gock_test-go

This is appealing because of how short and readable it is. All three of our criteria are tested here: the path and header are implicitly tested through matching, and the return value is explicitly asserted at the end.

What I see as the main difficulty with this API is the issue with matching that we also saw in httpmock: if the source does not use the correct path and headers, the mock response is not triggered and the error message from the test failure will not give us information about which criteria we failed.

Note that if you have operations that need to make many separate requests, potentially to separate servers, a matching approach like gock is likely to be very useful. The use case we’re focusing on for this post is for operations that can be completed in at most a few requests to the same service.

There is an easier, idiomatic way

You do not need to define your own interface and mock classes. You do not need to depend on a third party library. The creators of Go foresaw the need to mock outbound HTTP requests a long time ago, and included an API in the standard library.

The API is provided in the package httptest, and there are many examples of how to use it, including not only in the httptest package’s own Godoc examples, but other golang contributed packages as well, including substantial unit tests in the token and clientcredentials tests in the oauth2 library.

Here is how we could write a test for our sample function:

https://gist.github.com/sonya/e1fc0d40ffaf6be2227302949478ce65#file-httptest_test-go

Like the version where we mocked http.Client, we have full access to the original http.Request and thus can make direct assertions on what was requested. We don’t have to make any changes to the source file. We don’t have to go get any extra libraries. We don’t need to worry about shared state — even if we forget to call defer server.Close() we are unlikely to run into unexpected state issues elsewhere in the test suite. (In fact, intentionally calling server.Close() prematurely to simulate a remote connection hiccup is a simple way to test error handling.)

If you’re doing too much work, you’re probably doing it wrong

The Go language was developed at a time when communicating over HTTP was probably as commonplace as writing to the filesystem. As such, native support for testing HTTP was included as a first-class citizen in the standard library.

Unfortunately, information about httptest ended up too far below the fold — appearing in none of the intro or topical articles in the Go documentation page, nor in the blue book — thus evading many a developer’s browsing patterns. Developers coming from other languages where robust mocking libraries like WireMock and Mockito for JVM languages, nock for JavaScript, webmock for Ruby, etc. have been the norm, were likely primed to search elsewhere for a mocking library.

All of this means that if you didn’t know about httptest and have been using one of the non-native mocking approaches instead, it is perfectly understandable. However, the next time you find yourself having to test outbound HTTP calls, reach for the native approach instead and see how much time it could save you.

Interested in these types of discussions? Zus is hiring! Check out our current job openings.

Thanks to Anoush Mouradian, Brendan Keeler, and Bryan Knight for their invaluable feedback on this article.

Notes:

[1] Before I started writing production Go code, I happened to subscribe to the justforfunc channel on YouTube, which is run by one of the Go contributors. One of the videos was on httptest. The channel is a few years old, but pretty laid back and entertaining, and still relevant in my opinion.

[2] Code for the examples in this post are available at https://github.com/sonya/go-http-mocking

Mockingbird image courtesy of Sheila Brown (CC0)