The Fallacy of Unit Testing

Unit Testing is a standard practice within the industry. Unit testing is often associated with test driven development. In fact, I would venture to guess that most people cannot imagine test driven development without unit testing. However, I have slowly begun to come to the conclusion that unit testing, while certainly valuable in some contexts, is one of those techniques that can obscure the clarity of your code and make it less agile and whose purpose serves more to compensate for failures of language and architecture design rather than one of those techniques that has inherent value. At this point, I feel confident in saying that I believe for a significant number of projects you are more likely to see a net benefit by eliminating unit testing from your workflow than by adding it.

I understand that this perspective flies in the face of the currently trendy agile development methods. Let me see if I can add some perspective and then let's see whether we can understand why unit testing seems to be valuable within the agile methods while it might not in fact be essential to them. I'm not entirely alone in this perspective.

Let's first consider the whole point of agile methods. The agile methods are meant to be customer focused and problem centered. This simply means that, instead of following a process and focusing on that process above everything else, we try to focus on delivering value immediately and rapidly to the customer by solving the customer's problems. In order to do this, we try to eliminate aspects of our development methodology that stray from the customer centered approach and instead replace it with one that abstracts the development process away from a customer. Of course, there are many ways to accomplish this and so we find many different varieties of agile methods. My current process is best classified under the "direct development" model. You could say that this is an extreme case of customer focused development methodologies. It is extreme because it attempts to eliminate as much as possible the feedback loop distance between customer and developer. One way that it accomplishes this is with user directed pair programming.

In this method, the most valuable and important tests are those that are directly relevant to the customer. Since the customer does not care how the system is architected, as long as it solves the problem, customer tests are by definition black box tests. In this case, we want the user as much as possible to be doing programming with us. The customer comes in with a problem, we sit down at the computer, the customer describes the problem through demonstration, a test case is created that automates this demonstration, and the developer and the customer sit down together working on the system until the test passes.

The introduction of unit testing into this workflow reduces what is called semantic density of our code. Semantic density is the degree to which the key parts of our code that the customer sees is written in a language or a vocabulary that the customer understands and uses to think about the problem themselves. Unit testing, by definition, is code that is unreadable or not useful to the customer. It is a form of documentation of how the inner pieces is fit together, but it has no bearing on the actual domain of the problem. Unit testing introduces a node in our feedback loop which does not provide any real value to the customer. This sort of testing is just a convenience for the developer. In a system where the relationship between the customer and the developer is more distant, this convenience may seem like a win, but in reality it only serves as a check or a design roadmap for how the pieces of the code fit together. Because of its positioning in the overall workflow, unit testing may in fact be a dangerous crutch.

The problem with unit testing is that it provides a degree of automated checks that allow the developer to hide complexity behind those checks. Without unit testing it becomes readily apparent when the architecture of your system starts to become unwieldy. The true test of correctness of your code is not in the unit testing but in the functional or black box testing that you conduct. By removing unit testing from the code base, you are not removing any tests related to the correctness of your code, but you are removing the scaffolding that allows you to hide complexity behind automated machinery. But unit testing is a leaky abstraction, and any complexity that the unit testing helps to check or support is not actually hidden. Instead, you still have to use the architecture that the unit testing is checking.

Now, with unit testing, you have distanced herself from the tests that actually tell you whether you are solving customer problems, and you are introducing developer contracts to enforce an architecture that has no real direct relevance on the customer problem. Instead, you are making it more difficult to change the architecture and more difficult to adapt your code base to be simpler. Unit testing allows you to "fail fast." However, this comes at the expense that adapting, simplifying, and changing your architecture and the way that your code base connects together is much more difficult. In short, you have introduced a secret form of boilerplate.

Unit testing is not the same thing as automatic type inference or automatic error checking. In the latter cases, the developer need not write anything. The testing is done automatically and adapts to the code base. With the unit testing, however, the developer now must take on the additional maintenance burden to ensure that the tests and the code correspond correctly together. All this without actually contributing anything to testing the correctness of the code base itself.

I'm not entirely alone in this thinking. Developers of the Chez scheme compiler, for instance, have told me that they value black box testing significantly more than any form of unit testing. Indeed, this is where this idea first began to appear on my horizon. As I began to study other development methodologies, I found Cleanroom Software Engineering. In this method, there was no unit testing. Instead, statistical black box testing was combined with careful formal methods at the development stage to produce high quality code. One of the consequences of requiring formal methodologies for the development of your code is that it pushes you to simplify your code as much as possible to reduce the proof burden. Unit testing does not similarly create this pressure to simplify your code.

If you remove the unit testing from your code, the result is that you must be able to reason about your architecture without automatic mechanisms providing support for that activity. This provides a pressure to simplify your architecture so that it is easier to keep in your head. Every time that you need to make a change to your code, if that change is large enough, you will be tempted to refactor the code in order to make your life easier. If you have unit testing, refactoring requires editing potentially a large number of tests, and you are thus disincentivized to make that refactoring change to simplify the code base.

By focusing on a continual refactoring of the code base to simplify the architecture and thereby making it easier for humans to understand the entire code base, you are attacking the same problem that unit testing tries to fix but doing so in a way that results in cleaner, better code. Furthermore, you are increasing the ability of your developers to make changes quickly to the code without reducing quality. This will have a tendency to reduce unnecessary abstraction and constantly push the developers to have a simpler code base.

So why do so many people fall in love with unit testing? I think this has in part something to do with the languages that they're using and the systems that they design. If you rely on too much abstraction, too much indirection, and too much unnecessary modularity, then unit testing provides an important support for that kind of code. So, if you are using a language that requires a great deal of artificial abstraction in order to make anything happen, you're likely to enjoy unit testing. However, if you're like me, and you are strongly interested in reducing the overall abstraction complexity of your system, and you attempt to maintain a strong degree of connection between the customer involvement in the development cycle, then unit testing presents an artificial barrier that is counterproductive to increasing the immediacy of the development feedback loop and only serves to add additional complexity that fights against your efforts to simplify the code continuously.

Instead, take the time you would have spent writing unit tests and write more functional black box tests and do more refactoring of your code to simplify the architecture. By not writing unit tests, you have just saved yourself time, you should spend that time better documenting the customer's needs in the form of black box tests and improving your ability to respond to customer needs by having a more agile and simpler to hack architecture.