Thoughts On Testing

People who write software understand tests are a good thing.

Code without tests is broken by design.

—Jacob Kaplan-Moss

Recently I was thinking about test coverage and structure for any given project. Many tests are written in the moment, with possible conflicts when run outside of a transaction or against a clean database. Writing tests has never been easier, but there are things you have to keep in mind. Those things vary between projects, frameworks, and languages, but there are a few that I’ve found work universally.

Lose the “Tests Pass” mentality

Failing tests aren’t fun, especially when you’re preparing for a release towards the end of a day. It’s better to postpone something important than to start adding skips when you’re unsure why something has failed. The correct thing would be to regression test (e.g. git bisect) and see what changed around the time the tests started failing.

On the other hand, sometimes tests are viewed as “flaky” and get skipped because they don’t pass consistently. Tests fail because something changed. Figuring out what changed might be an exercise in frustration, but more often that’s because of bad test design.

Put things where you expect them

Sometimes a test is written against a known set of data, like an existing database. Depending on existing data breaks the test for any other environment where that data doesn’t exist. One of the projects I worked on in 2015 had a test suite with multiple different flags to differentiate between who was running it. On the continuous integration server, all these flags were set to False. The irony that a green light because of passing tests was meaningless did not get lost on me.

When things include a database save, the objects need to be entirely controlled by the test. Without that, the test relies on whatever the database might have at the time the test runs. The test then breaks if run in isolation or against a different database. Tests need to be backed by data, not only by intent. I’ll put a quick example here.

# Create a post that is a DRAFT
Post.objects.create(title="On Testing", status=Post.DRAFT)
# Check that when we filter by ONLY draft posts it is returned
response = self.client.get(post_url, data={"status": Post.DRAFT})
self.assertEqual(len(response.content["results"]), 1)

There isn’t anything inherently wrong with this. In fact, if this is run as a unit¹ it’s perfectly valid. However, what happens if this test is as part of a suite where something left a draft Post in the database? Instead of measuring a static expectation (new Post saved, one is returned), measure the dynamic change (new Post saved, there is now another draft post returned). A quick improvement to the test makes it more reliable and less flakey:

# Save how many DRAFT posts exist
drafts = Post.objects.filter(status=Post.DRAFT).count()
# Create a post that is a DRAFT
Post.objects.create(title="On Testing", status=Post.DRAFT)
# Check that when we filter by ONLY draft posts it is returned
response = self.client.get(post_url, data={"status": Post.DRAFT})
self.assertEqual(len(response.content["results"]), drafts + 1)

Testing components are easier than mocking

Mocking is a great way to creating expectations that aren’t controlled by the test. A typical example is mocking out API responses. It’s easier than inserting a shim layer to make the code testable and produces more consistent results. The ability to directly control the effect of an internal library or function is helpful and in some cases necessary for complete test coverage.

That does not mean mocks are a good thing all around. Breaking large sets of code into separate components makes it more reusable and easier to test. The worst offenders are transactional types which have grown organically to accommodate each and every branching condition and edge case. How does this get tested? Either the test is relatively fragile, or it gets ignored for being “too complicated.” Complicated is what needs tests the most.