Software developers hold a lot of information in their head while working. Any serious interruption means they lose their train of thought and have to start over. This was already explained in The Mythical Man-Month, which came out in 1975 (over fifty years ago by now!). The same concept is expressed regularly in articles such as Maker’s Schedule, Manager’s Schedule by Paul Graham in 2009.
By now software professionals know not to interrupt their colleague without good reason, but what about the interruptions we inflict on ourselves? When we build automated systems which are slow to respond, we can either switch context while we wait, or stare at a screen waiting for a machine to be done. I believe machines should wait for humans, never the other way around.
Slow feedback can come from many sources:
- An individual integration test which takes more than a minute to run locally because it has to rebuild an entire database from scratch.
- Testing against a third party system which takes five minutes before it sends back a response.
- Performing multiple manual steps in a user interface before waiting for an invoice to be generated and downloaded. Adding several minutes to each design iteration.
- A continuous integration (CI) pipeline that takes an hour to run because it contains thousands of slow tests which it executes one by one.
When I have to wait for 30 seconds I can stay focused; I will stare at my screen until the automated process is complete. If it takes more than a minute I start checking my email, notifications, or chat. At five minutes or more I look for a minor task to complete or get a coffee. However, if I have to wait longer I have no choice but to pick up a different problem, which means switching context completely until the system gives feedback.
Slow CI is often the worst offender yet the easiest to improve, making the cost/benefit payoff the greatest. When you reduce a CI process from over an hour to under ten minutes it has an immediate positive impact on your quality of life (and productivity). Less context switching, faster collaboration, less frustration and fatigue. Therefore, I will demonstrate this with a simple code-base, which behaves like some real-life CI pipelines I had to live with.
This repository on GitLab contains a test suite with 3600 tests, each of which run the sleep(1) command. If you want to see for yourself you could clone the project and run:
docker build -t sleepytest .
docker run --rm sleepytest vendor/bin/phpunit
I do not recommend doing this. You would be waiting a long time just to confirm that 3600 seconds do in fact add up to one hour. I did run it once, the output looked like this:
There is always some overhead, which accounts for the 3.855 extra seconds.
Imagine a real codebase with thousands of tests which does all kinds of database operations and file generation processes taking a full hour. It would take significant time and effort to meaningfully reduce the test runtime by improving the code. Furthermore, test suites always grow as products accumulate features, optimizing them is generally a losing battle. Instead, everyone just has to deal with it by picking up a different task, switching back and forth between contexts while they wait for test results.
It is often easier to split the test pipeline into smaller parallel jobs rather than optimizing the tests. I’ve taken my demo repository and created two pipelines on gitlab.com:
- pipeline 1 runs all tests serially, which took 61 minutes
- pipeline 2 splits the tests into 10 parallel jobs, which took 7 minutes to run
Instead of having to switch context between multiple features while waiting for feedback you could do something minor; review your own code a last time before submitting a merge request, stretch your legs, or answer an email.
These delays don’t exist in isolation. I have worked with multiple slow feedback loops stacked on top of each other. A poorly written task requires clarification, individual tests take more than a minute to run locally, the CI pipeline takes over an hour, code reviews take time, QA adds another delay, infrequent releases mean you can wait a long time before you find out how real users interact with a feature.
Each step introduces its own feedback loop. Stacked together they make it impossible to finish a single task without switching context multiple times. As a result work spreads across many tasks but little actually gets completed. Some of these problems are hard to fix, others are easier, but it is important to realize you are making a trade-off whenever you tolerate slow feedback.
p.s. My apologies to the operators of gitlab.com for occupying your runners for 120 minutes just to prove my point.