Aside from everything else, writing tests is just as much about writing testable code. Not everything that makes code easier to test makes code better, but often times many things do; after all, nothing is easier to test than a simple, small, pure function with inputs and outputs.
I prefer trying to focus on table driven testing where I can write a single test that exercises the code various ways. This is not applicable to all types of software, but it is wonderful for things like parsers, emitters, algorithms, data structures... things that test well.
Unit tests like this are cheap but make it easy to assert that the code works. If you can assert that your individual functions do what you expect, it makes debugging and understanding software easier.
I like to think of testing as executable debugging. It’s like a debugging session that is executed over and over again. If your tests are difficult to maintain it may say something about what they are asserting or what they are testing.
When people say "testable code" I always wince. Unit tests fucking suck at testing most code. Really suck. They're ok (ok, still not great) for things like parsers and algorithms... which isn't most code.
But, rather than improving test tools to handle more types of code, developers say that they want to completely restructure the code to make it more amenable to the fact that unit tests suck. Unit tests can't handle database? You need "Testable code"! No...
Developers code to a spec, but unit tests suck at encoding a spec except in the few cases you alluded to simply because most specs are not easily, clearly encoded in the form of your turing complete programming language with its mediocre tooling.
The practical upshot of this theoretical brain damage is that when people write unit tests on code where it isn't a suitable tool it tends to be an expensive waste of time. The tests cost time to build, time to maintain and when they fail it means... "oh, you changed some code". Thanks, test.
And then religious unit testing people always argue that it wasn't the tool that was at fault it was you.
Working on projects with excellent integration test tooling really opened my eyes to the possibilities of another world - one without unit tests.
I don’t have much to say beyond what I already did re “testable” code; forcing code to be “testable” can be bad, but in reality making code more testable probably mostly amounts to breaking it down into simpler, functionally pure bits.
Not everything unit tests terribly well. I haven’t seen it work very well for things like React or Angular components. It does work well for small bits of apps. Like for example, lets say you have a component where a good deal of what it does is extracts information out of a URL. You might have tests that exercise the entire UI, entering URLs and checking the HTML output.
The “more testable” version of this, imo, would be separating the URL parsing and extraction bit to a single free-standing routine that outputs some data given an input URL. You can then table test that bit. Then testing that this ties into the UI correctly could be done in the integration or end to end testing.
In case of table based unit testing, I think it often works great, as it can act as a running log of regressions and newly discovered edge cases that can even serve as a sort of document of expectations for other developers, and while it cannot be used to test all sorts of code, it has wide applicability and you can see it in webapps, Go servers, the Wine and libinput sources, etc.
It’s easy to get stuck on a single strategy to rule them all, but I think that often is a bit presumptuous and maybe dogmatic. Tests are a toolbox. Not every problem is a nail.
>in reality making code more testable probably mostly amounts to breaking it down into simpler, functionally pure bits.
I know. The problem is:
* This process often introduces bugs. How are you going to catch those bugs? Not with your tests, you're changing this code precisely so you can write tests. It's a catch 22.
* Sometimes people do this only to discover that the simpler "functionally" pure code is pointless to test because it's so trivially simple. Somebody literally did that today on the code base I work on. The code as a whole still has bugs but those tests won't ever catch one. They'll just break when the code changes. Plus that "refactoring" probably introduced bugs. This I think is what the concept of "unit test induced design damage" was getting at.
This isn't a problem with tests as a whole. Or TDD. It is partly a problem with people who use the terms "unit test" and "test" interchangeably (this engenders entirely the wrong kind of thinking). It's mostly a problem with unit testing as a concept (i.e. not the specific frameworks themselves).
Having nice clean code interfaces is also often conflated with unit testing - this is a mistake. One does not necessarily lead to the other.
Michael Feather's "Working Effectively with Legacy Code" goes into some detail on this. When you have bad, untested code, you don't start refactoring. You start writing high level "characterisation tests", then you refactor. After that you can still write the unit tests for the better components.
I've followed this process a few times and then noticed that the last step didn't really add much value.
It felt good at the time doing it because that's what I was "supposed" to do. I'd achieved the supposed "testable code Nirvana" and... meh.
The first step was life (or at least, career) changing though. Bringing a piece of shit code base under control with integration tests was a process that blew my mind.
That's what led me to start questioning the efficacy of jamming architectural changes into code in order to sacrifice at the altar of the unit testing gods and that maybe, just maybe, unit tests' steep demands and limited value means that they suck.
What self-contained unities are there on your code? If you are doing low level system programming, I bet there are a lot of them. If you are doing high-level CRUD, I bet there is none, all of them you import from third parties.
Writing unit tests for non-self-contained code is crazy, and leads to all those problems people identify. In my experience, the problem is that the most vocal evangelist believers of unit test are all on places that work on the high level stuff.
You're talking about things like dependency injection. I'm not a huge fan of testing but "testable code" when written in a certain way (without dependency injection) actually has many external benefits that don't have to do with testing per se.
Really the way you need to structure your program is to divide functions in your code between things that can be unit tested and can't (IO calls).
This can easily be done without dependency injection which is likely what you're complaining about.
For example Don't do this:
#unit testable with mock (Bad!)
function add_one(key: str, database: Database):
return database.get(key) + 1
Or.. even worse:
#Unit testable with mock (Bad!)
class Adder
Adder(database) -> None {
this.database = database
}
add_one(key: str) -> int {
return this.database.get(key) + 1
}
Do this instead:
#IO function, Not unit testable
def get_database_value(key, database) -> None:
return database.get(key)
#Unit testable without mock! Good!
def add_one(value: int) -> int:
return value + 1
#IO function, not unit testable.
def composition(key: str) -> int:
return add_one(get_database_value("name"))
You need a combination of unit tests and integration tests. Even trivial tests can help prevent someone ‘fixing’ your code in a refactor and breaking it, by defining the properties of what the code should do through tests. But it depends on the project - sometimes it just doesn’t matter, sometimes you need more formal V&V.
I work in a formal methods environment with proof checkers, property based testing, Isabelle/Coq/TLA+, and we still have unit tests and integration tests as concepts. You need a common language and these work well.
The distinction is easy. A unit test test whether a single unit works as expected. Generally, a failed unit test comes with a good pointer to what is wrong, and failed unit tests are rare.
Meanwhile integration tests test whether multiple units are integrated properly to produce expected results. When an integration test fails, you don't really know what is wrong. Instead you look at or write unit tests for the smaller units to see whether they fail, or whether the way you tied them together has a flaw.
I'm essentially saying that we should write specifications and a translation/implementation layer that "tests" the specification against reality. Maintain a separation of concerns.
For instance.
This is done in theory by cucumber/gherkin but it's done pretty badly. Good concept, bad implementation.
My broader point, though, is that unit tests are a bad concept and bad abstraction for this among many reasons but it's so embedded in programming culture (to the point that when people say "test" they automatically mean "unit test") that other approaches almost become "unthinkable".
I think you mean something like system testing, but backed by BDD-style specs?
TBH, I personally think you're being a bit too hard on unit tests - the number of times they reveal bugs in our our code bases (and in my own code - the horror!) is ridiculous. I do find them really valuable.
That said, testable OO code (I'm primarily a C# guy) has certain constraints, and sometimes making it testable results in a high level of abstraction - so much so that individual tests almost don't seem to actually test anything meaningful, and it becomes difficult to see where the logic and behaviour is, without digging through 42 layers of abstract classes, interfaces and factories.
Recently I've come to favour a kind of inverted test pyramid, where instead of unit tests forming the most substantial foundation, system tests do instead, followed by integration tests, and finally by unit tests at the tip. I find this leads to a "sensible" level of abstraction, where unit tests are used where they are most valuable, and system tests keep tests meaningful. Depending on your code, it might be quite tricky to setup systems tests, and it might need quite a large time investment, but IMO it's worth it. If you're able to dockerise your entire solution, then it's much easier to do.
I’m bearish on unit tests as well. Integration and e2e tests get you closer to the spec and are easier to change when the spec changes or your product inevitably evolves. I reach for unit tests when something is foundational or hairy, but IMO they should be a last resort. They fill your application like sand in gears otherwise.
I find I often have an integration test that is failing, and then I start adding unit tests until I find the error. Especially when the failure is not amenable to investigation by a debugger.
Have you had a chance to check out the Screenplay Pattern at all? I have had great success using Cucumber to stitch together common abilities/actions/question types. My step definitions usually consistent of a single line of words grouped by a few parentheses and commas. Groovy/Kotlin make this particularly nice.
I like to think of tests and the team's emotional relationship to them as a barometer of the extent to which the repo is and has consistently followed best practices in the language. IOW, enforcing unit and functional testing forces developers to follow best practices.
If I find myself dreading writing and maintaining tests, it's a great indicator that the code diverged from best practices at some point in the past, and that the technical debt has just been accumulating ever since. Difficult to test code is code smell telling me that the technical debt is getting out of control or that the code is getting unmaintainable, and this can be difficult to walk back.
Another great indicator is the extent to which the team "detests" mocking. Mocking is great -- when it's trivial to do. When it's not trivial, and elaborate mocking behavior and reflection and testing of private methods, etc. is needed, it's another indicator that best practices have been sacrificed for expediency.
If I really needed that I’d use property-based testing. On my phone or I’d code an example. But basically it generates test inputs and gives you confidence by running some number of random tests that verify the property holds or finds a counter example.
More realistic but basic example is to show that a function is its own inverse (reverse reverse list == list) or that one function is the other’s inverse (decode encode plain == plain).
Testning actively malicious code seems to me like a different problem than testing regular business logic. Yes, the code you're testing could also look for the running test code in memory and rewrite it to always return success, but why would you write code like that?
Their point is that the test case provided would lead the developer to believe that the implementation is a correct identity function, when in fact it isn't.
Just clarifying, I do not take this as a good reason to not write tests.
Ah, right. I didn't realize some people believe tests should "prove" the correctness of a function (or that if they can't, they're worthless).
Testing has its flaws, but that's not one of them. There's often very good reason to ask the question "does this function operate the way I expect it to when I provide this input?"
I prefer trying to focus on table driven testing where I can write a single test that exercises the code various ways. This is not applicable to all types of software, but it is wonderful for things like parsers, emitters, algorithms, data structures... things that test well.
Unit tests like this are cheap but make it easy to assert that the code works. If you can assert that your individual functions do what you expect, it makes debugging and understanding software easier.
I like to think of testing as executable debugging. It’s like a debugging session that is executed over and over again. If your tests are difficult to maintain it may say something about what they are asserting or what they are testing.