[Authors note: So, I wrote this in December, and promptly forgot about it for five months. It happens. I've annotated slightly.]
I was excited to be a guest on Ruby Rogues this week [note: actually last December] to discuss my book Rails 4 Test Prescriptions, available now as an ebook, and coming very soon [note: available now] as a physical object that you can buy or, say, give as a gift to all your friends. [note: still a great idea, please buy for all your friends].
I want to follow up on something that I think we talked about. (As I write this, I haven’t heard the final recording, [note: But I've heard it now, you can listen to it here] and I find my memory of what we talked about to be vague. I remember making fun of David Brady.) [note: I definitely made fun of David. Hi David! Okay, I'm done annotating now.]
We were talking about Test-Driven Development, and I was mentioning that tests written via TDD need to be valuable in two time frames.
- In the TDD process, as you write the feature
- As an ongoing part of the test suite, meaning forever.
Each time frame values different things.
As a part of the ongoing suite, the test runs as a tiny part of a larger whole, executed for developers who, by and large, are removed from the immediate context in which the test and code were written. The most valuable features of the test in that context are speed, consistency, and clarity. Speed, because the test is a tiny part of a larger whole, and even small grains of sand will make a mountain if you have enough of them. Consistency because an intermittently failing test is really challenging, and clarity because when something bad happens, you need to be able to recover as much context as you can from the test.
When you are initially writing a test as part of a TDD process, those features are still important, but they aren't as necessary. Often, when writing a new feature, you may only choose to run the tests associated with that feature, making speed less important. A test that takes two seconds is less of a burden when you are only running one of them. Consistency and clarity are similarly less important in the moment of creating the new code.
If you are looking for a structural reason why test suites get out of hand, this is a good one: as you are incrementally writing one test, you are thinking of it in the context of just running the one test, which causes you to underestimate the effect the test will have as part of the suite when you have dozens, hundreds, or thousands of similar tests.
What is valuable when writing the tests during TDD? Incrementalism and accuracy: that each test adds a little bit of information to the application, or put another way, that each test exposes a limitation of the application, or that each test fails by the smallest amount you can imagine. Accuracy, in this case, means that executing the test gives you a true picture of the state of the code. One common cause of reduced accuracy is a test with multiple assertions -- in many frameworks, the first failure ends the test, meaning you don't know whether subsequent assertions are passing or failing.
In the moment testing also values reducing the amount of typing needed to create the next test. The less typing you need to write a test, the tighter your feedback loop is during TDD. This has a good component—a better design leads to less setup—and a bad one—the temptation to do clever hacks to minimize typing with metaprogramming or extra loops, or short variable and test names.
These two values are somewhat in tension. Writing in TDD argues for one-assertion per-test for better accuracy, but that’s a problem over the long haul since the one-assertion per test-style is usually slower.
There are ways to reconcile some of the tension, and they revolve around doing some test cleanup when you are finished writing the feature using the TDD process. For instance, you can TDD using a one-assertion per test style, then combine the assertions into a single test once you are satisfied that the feature is complete. You’re trading off in-the-moment value for long-term value.
Pro tip, if you ever find yourself on Ruby Rogues: Script your picks in advance. I was rushed and nervous and fried by the time it was my turn, and I feel I did not truly sell two of my favorite authors, Max Gladstone and Martha Wells, and one of my favorite comic books, Astro City.