Testing HTTP–based applications

I have been thinking a bit about testing HTTP-related applications recently. I’d like to share those thoughts publicly because I strongly believe that they bring perspective into how you write and design your tests. I’ve seen blog posts which encourage to write tests in exactly the opposite way I think they should be written, so I decided to give them some counterbalance and also try to structurize the concepts in my head. This text might be a bit specific (referring to frameworks/libraries) but in general I feel that you might apply this to any language or framework. The specific examples are based on Ruby, Ruby on Rails & RSpec – technologies I use in my daily work.

This has been triggered by quite a few informal talks at my company, a tweet which I liked and a short blogpost by Kent Dodds which I enjoyed.

Why even test?

I’ve been an advocate of writing tests since forever I think – the idea of making sure your software actually does what you wanted it to do is very appealing. Lots of very smart people already wrote about it, but to give you a brief recap:

  1. first write a test confirming that what you want to test is actually broken,
  2. then write the code that makes the test pass, and
  3. make sure the code quality is good enough for you

This is relatively easy to follow, but the real problem is: what do you want to test? Everything and aim for 100% coverage? Only some crucial parts which you think might break? How to write the tests? Do you need to follow some structure, how much time to invest in building it? These questions don’t have easy answers but I hope to give you at least some guidance.

What to test?

I’m gonna write about API–centric apps since it’s my primary use-case and also a often-chosen architecture recently, although any Rack–centric application, and by that I mean working with HTTP protocol, would benefit.

The first question you want to ask yourself is: what is the most important function of my app? In case of APIs it is… the API! By providing it for end user you promise that if the request is this, the response will be that and most probably some database will have something stored/updated for future use. This is the entry point to your application, this is how end user interacts with it. You want to focus your effort on making sure that this is 100% good. There are probably other entry points to your application, maybe you have a micro-service architecture and services inside communicate via other means like message queue? You also want to test it, it’s also kind-of an API.

How would I name these tests? A few years ago I might’ve used a name integration tests or maybe functional tests. These days I’d probably name them just unit tests. Why? Well, they are most probably relative easy to setup, you can run them within the same process as your app, you don’t need to set up a separate process serving HTTP requests, because your framework likely supports writing such tests. They are also extremely fast, I’m not saying they will take 10 microseconds to run, but it’s safe to assume that you would be able to test every endpoint of your API in less than 30 seconds. If you think about the value of such tests, this is more than great!

I’m not against testing other parts of your application, if you have some parts of it doing Very Complicated Things™, you want to write a few test cases for a specific module responsible. But remember: start from the top, start from the core value you give, in our case an API. Your tests will be very resilient to code change (which is an extremely desirable property) and if they fail, they fail because you broke the API promise which is exactly what you want. Should your application become more complex feel free to introduce more specific tests, but remember that comes at a cost: when you change implementation you will have to change the tests – and deal with something called test fragility.

Do you have some examples?

You might wonder: ok, so what should I test exactly?

What is the value? Your users will send some requests, which are basically requset headers and request body, and expect to have a response in result: a status code, response headers and most probably response body. They also expect some state to change, be it a database or whatever else.

I will use Rails 5 + Rspec for the examples just because I want to show you the actual code related, but the thinking process can be applied to any framework.

Let’s assume a simple API controller for contact resource. The excerpts below define rough template of how such setup would look:

# app/controllers/api/contacts_controller.rb

class Api::ContactsController < ApplicationController
  def index
    # ...
  end

  def show
    # ...
  end
end
# config/routes.rb

Rails.application.routes.draw do
  namespace :api do
    resources :contacts, only: [:index, :show]
  end
end

Let me show you a sample test I would write for the index action.

# spec/api/contacts_spec.rb

RSpec.describe "/api/contacts", type: :request do
  it "responds correctly" do
    contact = Contact.create!(name: "Lona Wrinters")

    get "/api/contacts.json"

    expect(response.status).to eq(200)
    Oj::Doc.open(response.body) do |doc|
      expect(doc.fetch("data/0/id")).to eq(contact.id)
      expect(doc.fetch("data/0/name")).to eq("Lona Wrinters")
      expect(doc.fetch("data/0/created_at")).
        to eq(api_timestamp(contact.created_at))
      expect(doc.fetch("data/0/updated_at")).
        to eq(api_timestamp(contact.updated_at))
    end
  end

  def api_timestamp(time)
    time.strftime("%Y-%m-%dT%H:%M:%SZ")
  end
end

Let’s look at this file.

Structure

The structure of this test is fairly understandable.

It is vital that all your tests follow it. Why? It brings clarity when reading and understading the test. You immediately know the data dependencies, you know what you actually test and you know what is expected.

Anti-pattern to this would be:

RSpec.describe "/api/contacts", type: :request do
  let!(:contact) { Contact.create!(name: "Lona Wrinters") }

  before(:each) do
    get "/api/contacts.json"
  end

  it "responds correctly" do
    expect(response.status).to eq(200)
    expect(json["data"][0]["id"]).to eq(contact.id)
  end

  def json
    JSON.parse(response.body)
  end
end

You might think: hey, I’ll create a before block which calls the get for every test I write, that way I don’t have to repeat myself, I’m DRY! And! I will always create a contact with let! so that I don’t have to create this contact in every test, DRY for the win!

On the surface it might be tempting to do that, but what you are effectively doing is getting rid of this structure and introducing:

The dependencies (let!) in this example are initially ok to deal with, but the more complex your file gets, the more dependencies you introduce, the harder it gets to understand if these are really things you need, or maybe you just have them because other tests need it? Minimal required input is what you should aim for, trying to reduce test complexity to minimum.

Running get in the before hook is also anti-pattern for me since it hides what the test really tests. As before: in the beginning it may be easy to grasp, but the more complex your suite becomes, the harder it gets and more time you have to spend figuring out how the test really works.

The tweet I mentioned earlier is a great example of how you should not write your tests:

hellish tests

As a rule of thumb: a test should be understandable just by looking at it and the methods it uses, you don’t want to jump around the code like crazy trying to find all those magically invoked methods which happen to be called.

Helper methods

The def json helper. In general it’s very good to write methods that help you test your application – you should treat your test code the same as you treat your app code, so writing a method here or there is good. But don’t be tempted to introduce weird dependencies on other methods (in this case, response). This will bite other people eventually, because it’s gonna be hard to understand where things come from. Better: write some pure methods, which take some input, process it, and then spit some output. Don’t be afraid to use such methods generously, they will always work the way you intended them to. This is obviously a good advise not only in tests, but remember to stick to normal programming also when testing.

def parse_json(json)
  JSON.parse(json)
end

it "..." do
  parsed = parse_json(json)
  # expect(parsed)...
end

As a rule of thumb: treat your tests with equal love as your code, write methods. But stay focused on clarity and understandability, stick to pure methods.

Meaningful errors

Your tests are likely to fail from time to time. Focus on making sure that if they fail, the person which has to deal with the failure (often not you!) will have full visibility into what’s wrong. Compare this:

it "..." do
  Oj::Doc.open(body) do |doc|
    expect(doc.fetch("/data/contacts/0/attributes/name")).
      to eq("Lena")
  end
end

To this:

it "..." do
  parsed = parse_json(body)
  expect(parsed["data"]["contacts"][0]["attributes"]["name"]).
    to eq("Lena")
end

As usual: on the surface it seems the same. But if the test fails (you returned a JSON that is formatted differently for example), you will get very different messages:

       expected: "Lona Wrinters"
            got: nil

vs.

NoMethodError: undefined method `[]' for nil:NilClass

The difference is striking. You actually have no idea what failed in the second example and where it failed.

You can even take the first example to a higher level, and introduce JSONSchema validation which checks your response for validity against previously defined schema. The failure messages will be more apparent and meaningful.

Focus on assertions which provide meaning into what failed.

Stay simple

Don’t introduce patterns you are unsure of.

FactoryBot/Girl? Do I actually need it? What is my gain? Maybe I can provide simple values myself in the tests?

it_behaves_like? Do I wanna sacrifice the readability of my tests for the sheer fact of having my code formatted in a certain way? Why do I want to extract my assertions into some outer file?

There are dozens examples of such abstractions I found throughout my programming life. Hell, I’ve authored them myself! The lesson I took is not to do it prematurely.

Keep your tests simple, Ruby-esque (or Java-esque, Elixir-esque or whatever languge you write in), wait for patterns to emerge and only then try to incorporate them.

Work top-down

Don’t be tempted to write tons of tests of your models, helpers etc. They will pass initially, they will make your suite bigger, but is that the point? The more tests, the more time you have to spend maintaining them, that’s the sad truth – your tests are fragile, and the more low-level aspects they touch, the more fragile they get. So keep your tests suite small but meaningful: understand what your users expect from your application and test that, leaving what may be easy to test but won’t give you any value.

Go for a small but stable and meaningful set of tests.

Test one thing at a time

When writing tests there is this temptation to assert too many things in one test. You might be thinking: hey, I already set up all this data, I’ll also check this one small thing that I care about… Remember: when writing expectations in your test you want to make sure that you check for a cohesive set. By that I mean: if you write a test checking that a HTTP response looks in a specific way, don’t try to verify that you also ran this specific background task which is only semi-related to that request-response. Think about how to divide your tests, don’t do it blindly and don’t try to cram everything into one test. The purpose of test is to give you a warm, reassuring feeling that things are working, but also to pinpoint to a specific place if things are not working. Don’t let this place be too large, otherwise your maintenance cost goes up.

As an anti-pattern I’d say this would be a good example (notice how test description is only partially relevant):

it "responds correctly" do
  post "/api/contacts.json", {data: {name: "Lola Copacabana"}}

  expect(response.body).
    to match_schema("contact_create_response.json")
  expect(MyMessageQueue).
    to receive(:publish_messages).with(...)
  expect(BackgroundJob).
    to receive(:push_object_to_elasticsearch).with(...)
end

Test one, cohesive functionality in each test.

Closing thoughts

Treat your tests with respect because their value cannot be overestimated. Of course in Ruby and other dynamically typed languages they are even more important than in languages where types are checked at a compilation level, but even there you want to invest in tests, because it’s one of primary ways to stay relaxed knowing that the things you did are working.

As always you should apply common sense, but top-down approach, focusing on places where you can get the biggest value, keeping things simple, understandable and doing regular programming, the same as in your application code, will keep your test suite both useful and hassle-free. I wish you a stable, ever-green test suite which tests which you actually care about.