Stop Thinking About Code Coverage
Every now and then I get asked what level of testing code coverage I aim for in my projects. I usually answer with, “it depends” and follow up with my explanation of how it’s case by case and depends on what the area is, how critical the code is, what it’s for, etc. But what I really want to say is, “who cares” because it simply doesn’t matter.
When I say code coverage I’m referring to the level of testing that has gone into a piece of code, that defines the lines it covers, if it covers every logic branch, and so on.
Code coverage has it’s time and place, it’s a great way to validate if you’ve missed anything, but my gripe isn’t with the usefulness, it’s that it shouldn’t be a metric to aim for. We shouldn’t be relying on code coverage as a means to tell us whether or not our code is well tested. It depends on many factors and sometimes no tests are better than some tests.
Let’s take a simple example of a function that I use in production to validate if a variable can be transformed from a string to a number.
// @flow
export default (value: string): boolean => {
if (Number.isNaN(Number(value))) return false;
if (value.indexOf('.') > -1) return false;
if (value.indexOf(' ') > -1) return false;
return true;
};How many tests should we write here? Is it 4 because that covers each logic branch? Should I write a test that if it’s passed a non-string variable it shouldn’t break, or possibly even a test for each variable type just in case? Since this function returns false if the string contains a space should I test space being used in various positions to ensure I’ve correctly checked for greater than -1 instead of 1?
These are all good questions to ask, ones that can’t be answered by looking at code coverage, and ones that only the author of this code can answer. It simply all depends.
We should first take a step back and question why we’re writing tests in the first place? Recall above I mentioned that sometimes no tests are better than some tests.
Consider a complicated UI element full of animations and colourful effects, is a unit test the best thing to do here? If you were able to write a well covered test for this, how did you do it? Did you grab the CSS attributes and validate that they’re all what you expect? Great job if you managed to validate that your code looks the right way to a computer.
Now consider you need to improve this UI, maybe add another effect, maybe your styling needs to change, or the colour palette has updated. Would the tests be of any value to you and does every change you make warrant a change to your test? That’s called a brittle test and often times the effort spent writing that test in the first place plus maintaining it into the future just isn’t worth it. Instead we should explore other avenues, in this case visual feedback. Instead of using unit testing as the de-facto and only tool available to validate code, can we instead build a suite that allows us to visually test our code against a range of use cases to battle test usage prior to adding it into production features? Storybook is a great tool for this.
Visual problems require visual tests.
We can’t just look at code written with tunnel vision and decide what level of testing is necessary, this is exactly how code coverage works. We need to have a wider scope and sometimes the way it’s consumed will drive this. Consider again the code snippet above is written with Flow, given the usage in a completely flowtyped project, would there be any need to validate usage of other types apart from string being passed into the function? Probably not. But if this were instead a reusable function that was open sourced, there would be no telling what tooling your consumers have, you’d want to battle test this so there are no unexpected behaviours.
We as developers are strategically lazy. We want to write the best, most robust code with a minimum level of testing to fit our needs. With too many unnecessary tests, will the next developer be as enthusiastic to keep up the standard? And too little tests will give the next developer the impression that the area isn’t important enough to upkeep.
Tests do more than just validate your code or serve as a checklist. They,
- Document the functionality, use-cases, and expectations
- Enable us to enhance with confidence
- Find edge cases during development
If we keep a nice balance and articulate in our code what’s important, we can maintain a healthy codebase that is resilient and easy to maintain.
Everything in software development is case by case, there is no silver bullet.
