Software engineering is programming over time. It's not so hard to write code that is correct today, if that's all that tests did then they wouldn't be worth the effort.
We write tests so that we know whether future changes have broken the system or not.
This article sounds like a reaction to the practice of writing many small, hermetic unit tests, which do little more than recapitulate the code under test. The weakest parts of a system are often in the joints, so the most important tests to write and to run are the integration tests, the ones that tell you with the most confidence whether the code works in the real world or not.
See, for example, this puzzling line from the article:
> Other tests grow obsolete because the target platform is retired, but not the test. There's even less pressure to remove stale tests than useless code. Watch as the workaround for Windows XP is removed from the code, but not the test checking it still works.
How do you remove the code for a feature, but not the test for that feature unless either you aren't running your tests or your tests aren't testing the things that you care about?
Think the question "What's the return on investment on writing tests?" doesn't get asked enough. There's a blind assumption that writing lots of tests is always good and code coverage tools tend to push the idea that you're not done until you reach 100% coverage.
Imagine some incubating startup where you have an initial 4-week runway for development, with the goal of getting a prototype working, so you can get it in the hands of some customers and begin validating a business case. How many tests do you need in this case? And how much time do you allow for it? Most of the code you're going to produce will be thrown away and the more compelling the prototype is in terms of functionality, features and design the stronger the signal you'll get back from potential customers.
To me there's still too much "testing is my religion" amongst developers and not enough "here's the quantified value of me spending 3 days writing tests and here's how that fits in the context of where our business is right now"
The benefit of testing is confidence that things are working. There are side benefits, like acting as documentation and examples, but they should not be the focus.
Writing tests (or deleting them, refactoring them, etc.) should always involve a cost/benefit calculation, even if it's a rough mental estimate that's not written down. In particular, that requires answering "How much effort will this test take me to write/debug?", "How much extra confidence will having this test give me?", "What level of confidence do I feel comfortable with?" and "Could I achieve higher confidence spending this effort on something else?".
This doesn't need much effort. For example, we might think "This array indexing took me a few attempts to get right; it could result in misleading reports, so I'd better throw a few edge cases at it to make sure I understand it correctly.". On the other hand we might think "This class doesn't have any unit tests, but there's already a bunch of validation on the results; it only affects the page layout, and it'll be obvious if something's wrong, so I'll leave it for now and put some extra checks on the gnarly payment system."
This approach makes sense regardless of the situation. In your startup example, it might make a lot of sense to write some tests that, for example, spider our site for broken links; or look for certain strings on certain pages. Things which are quick to write (they could even be shell scripts), but give us prior warning that our demo will go wrong. It would make less sense to write a pile of unit tests for the internals of some boilerplate piece of the system.
Code coverage, dogma, etc. should not be the deciding factors for what to work on.
>The benefit of testing is confidence that things are working.
No it gives you confidence that the test are passing.
I have worked on code bases that are well tested and code bases with non existing testing. The tested ones contain just as many bugs as the non-tested ones as far as the end user is concerned. In a year and a half working at the "well tested" company, regression tests caught an error one time. That was good but considering the testing team was bigger than the dev team, I question if it was actually efficient or not.
You misunderstand me. I never said that testing is an efficient way to gain confidence, or that tests give much confidence, or that only testing can provide confidence. Rather, I said:
> The benefit of testing is confidence that things are working.
Some people treat testing as a gamified metric (e.g. code coverage). Some people treat tests as executable documentation. Some people treat tests as some essential requirement, without which the software cannot be shipped. I'm saying that those are not good reasons to write tests. Tests should be written to give us confidence that things are working.
How much confidence will we gain? How much do we want? How much will it cost? Where should our efforts be directed? Those vary from project to project, and is exactly why I say thought should be given to the cost/benefit tradeoff.
(Personally I think automated testing can be an efficient way to quickly gain a lot of confidence in a system; especially if we do property checking of high-level functionality, instead of unit testing of low-level details. However, that's a separate point; we can disagree on that, whilst agreeing on the need for weighing costs versus benefits.)
Regression tests don't usually identify bugs in new features, unless they interact with the older code or required changing the older code. In my experience, they're useful to have, but not useful for identifying novel issues. I recall reading someone's analysis of applying fuzzing and that using a particular fuzzer would eventually stop discovering issues, but that didn't mean the system under test was perfect. It meant that they had worked out most (if not all) the errors that fuzzer could identify for them and needed a new fuzzer.
The same thing with regression testing. I wouldn't stop running regression tests just because they aren't identifying issues anymore, but I wouldn't rely on them to discover novel issues. That's what the new, updated test suite should be doing. A regression test suite will not have a test to cover a new feature. It will only make sure that the new feature doesn't break the old one.
> To me there's still too much "testing is my religion" amongst developers
Maybe some developers. On the French web market: almost no one is.
> What's the return on investment on writing tests?
I only write end to end tests. Because I come mainly from a maintenance background: unit test for test coverage are a hindrance. But testing functionalities of an app? That's what a client want. They don't care about how you implementing things, only that it works. And that old bugs don't come back.
Yes it means new functionalities have higher estimates. But they tend to not come back once done. And doing it with end to end tests means you should easily be able to trash all the code and replace it with something else doing the same work.
The problem is the tooling. Mocking external APIs tend to be a pain even with Wiremock. Testing a GUI is not easy: a Sikuli server to drive a GUI with screenshots would be a boon. And it tends to be slow. But that's because most of the work has been going to JUnit-like tests for years.
> "What's the return on investment on writing tests?" doesn't get asked enough
I consider tests to be highest return on investment for velocity.
The smaller the change you are making, the more likely your tests will assist you in delivering faster software.
Sample sizing is probably biasing me here, but all of our more legacy code bases have higher costs for feature delivery to the point we have begun implementing logic at the API level (instead of the monolith) just to able to minimize the overall cost. The code bases that get features faster are more modern React code bases that have tests while the legacy code bases are Java or Scala with close to 3% code coverage.
> all of our more legacy code bases have higher costs for feature delivery
Compared to one another, the nominal cost of feature delivery in legacy is higher. This is a correct assertion from the perspective of the person maintaining those systems.
But from a business perspective, as long as the operational costs for that component don't exceed it's revenue / value generated, keeping legacy code running is perfectly valid.
There's an opportunity cost calculation at it's base: When does the compounded sum of all the time "lost" due to the nature of the code (lacking tests, architectural limits,...) outgrow the cost of decommissioning / replacing the legacy code? As long as the latter costs more then the former, it makes sense to keep the legacy code around.
Writing tests comes at a cost as well. The same calculation applies here as well. Is it strategically sound to sink time and money in writing tests if the costs of doing so outstrip the marginal gains on efficient feature delivery over the projected lifespan of a legacy component?
Maybe. Maybe a feature-rich but rough and unpolished prototype is what will delight customers the most.
Or maybe focusing on an absolutely minimal core, but executing it really well with high performance and having it run rock-solidly will attract the potential customers more.
I think this depends a lot on demographics, but in broad strokes, I think the former is overvalued compared to the latter. I.e. contrary to your experience!
I would be inclined to agree except people generally have low standards of performance (ofc depending on the industry!).
People also rarely care about the hand of the puppeteer, what keeps the lights on parts especially if we're talking about a prototype.
I think people care a great deal more than they have words to express. We're not very good at educating our customers in ways to discuss and analyse technical performance. We're getting better at it when it comes to uptime, but latency discussions still lag (hah!) behind.
(This is from purely personal experience. I have only had one customer ever, when asked about performance requirements, who could list some. Everyone else has been "I guess I want it to be... good?")
I wrote a one-off test script to do a REPL session as tests. You just make a file with the commands you want, run it, and the system replaces the input-only file with the inputs+outputs. Subsequent runs replace the output.
Then you can check that file into version control, and whenever you run it you get a diff of the broken tests. If they're not actually broken, just check in the new file and you've updated your tests. It works really well for this project, a search engine.
Yep, this is something I always try to instill in my peers when writing tests.
How much time does it take to get 100% code coverage, how much do you get paid, and if it was your startup would you burn through payroll cash to write them?
When you start adding dollar amounts to things in terms of an engineer's salary, it's easier to see what is really important and what is waste. You are paid to write correct business logic. Tests are important, but only in that they verify your business logic is correct and stays correct.
I think in the situation you're describing, with a prototype being the difference between the business existing at all or not, you could justify all manner of sins. If you had a prototype already 90% done in COBOL it would potentially be the right call to finish that off - that doesn't mean using COBOL is the right thing to do in general. To use a common HN term, that is a situation where it is perfectly sensible to acquire technical debt, just as their are non-software contexts where it is financially sensible to acquire a bit of regular debt for a while. It doesn't shed any light on whether a lack of tests is technical debt.
(What you've described is a classic problem of junior devs trying so hard to do the right thing that they don't step back and ask why that is the "right thing" in the first place and whether that actually applies in their situation. I like to call this phenomenon the "overenthusiastic amateur". At least it tends to go away with experience - unlike the underenthusiastic dev who doesn't care what the aftermath is so long as their code works long enough for them to move on to the next thing!)
Imagine some incubating startup where you have an initial 4-week runway for development, with the goal of getting a prototype working
Are you talking about an actual prototype or an MVP? A prototype meant to showcase some idea doesn't need any unit tests. An MVP? probably yes.
Now back to the practicality of software development. Unit tests are fundamental when working in a team setting. What are people doing in PRs without unit tests?
> Imagine some incubating startup where you have an initial 4-week runway for development
Who have been asked to build something that does X, Y, and Z. Without tests how do they know when they're done? How do they convince other people that they've done what they were asked?
Absolutely agree. My biggest lightbulb moments with testing have come when needing to do significant refactoring. Without tests, I feel aimless because it's hard to get feedback about whether the refactoring has 'worked'. With tests there's a nice, tight feedback loop. Not saying it gives any guarantees, in my experience it has dramatically increased how quickly I could improve the code - to the extent where in some cases I wouldn't have even tried in the absence of tests because I wouldn't have been confident I could improve it.
Exactly. I picked up some legacy invoicing code that's 13 years old, and people are terrified of breaking it, so it's just quietly rotted away and half of it isn't even used anymore, but no-one can quite tell which half.
First thing I did was add tests so that I can refactor with confidence.
Agree with this and the parent comment. For the most part automated tests are for testing for regression. The highest ROI tests for complex projects are end-to-end in my experience - where the gateway interface is the API or UI - because they test an entire flow and can let you know if there is an issue somewhere, ideally with an error that can narrow it down to a specific function or file/module. It does seem like the author is talking about unit tests based on what he/she is saying which can imo have less ROI but they're still not useless. If you have limited resources (don't we all) then start with a developer working on end-to-end tests, it's further away from the code being tested but it's great 'bang for buck', and run them on an automatic schedule - each commit may be overkill because end-to-end tests can take a long time.
The second part of the article mentions that stale tests are often not removed. However if running tests using either CI or on a regular schedule and viewing the results everyday, this can't happen, they would be removed or modified quickly. This does raise a valid point that within our industry test results data is often not handled well and is an afterthought. My hypothesis is that this is actually the reason some developers get frustrated with tests or do not see the value or why you may get stale tests not being removed. The problem of poor management of test results data itself (the output from testing) rather than testing itself is the problem. A small plug here but I started tesults.com to address this very problem., integration takes a few minutes if you use a popular test framework. There's a forever use free tier so try it out, and if you need more but don't have or can't get budget but love what it does then send me an email and I'll expand your free tier. The important thing is to try to see if it makes thing better for you and you use it as part of your release management.
Are you defining unit tests are per class tests? This is the biggest mistake.
You should be testing a unit of behaviour, like a business rule or an effect to be expected. If this involves multiple objects collaborating then so be it. But they should not cross architectural boundaries like http calls, or database calls. If you can only make public api classes accessible, and make all helper classes inaccessible to users. That also forces you test things from only a public api standpoint.
Tests that just check an object calls another object, but exhibits no desired behaviour in its self are pointless, and just couple everything to the implementation. I.e Certain methods, are called in a certain order with specific parameters. When you do that, you make it really hard to change things(just design changes, not desired behaviour changes) without breaking every test.
This is part of the difference between the classicist and the mocking styles of testing. The argument for the latter goes that if (big if!) your collaborators are well designed and have good interfaces, then asserting that something is called is a meaningful assertion in the language of your domain (e.g. "check that an email is sent with exactly this content" is a meaningful assertion, even when you mock the mailer).
I agree that in many cases, APIs are badly designed and expose implementation details that you shouldn't couple your tests to, but many still do that. Mockist proponents would claim that this is not the right way to do it (for example, they argue you should wrap bad APIs in better ones and write your assertions against the latter, see also "don't mock what you don't own").
I can understand the point of view that even then you wouldn't like these kinds of tests, but for me they do provide some assurance that "the right stuff happens" (of course, you need integration tests to test the wiring etc.) at a still lower cost than "you have to extremely strictly separate pure from impure code" (which has other benefits, yes, but is also really hard to enforce especially in a team).
I mean I am a functional programmer so i naturally make code pure until the edges anyway. In a functional language I will just replace the edge function call with a fake version.
But the problem with mocking, is you couple your objects to one another. You expect certain methods to be called. With certain parameters.
What if you want to add in extra layer, or remove a layer, or add a helper. All your tests have now been broken yet the overall behaviour has not changed.
You also have issues around mocks not acutally setup to behave like the real object.
If your object under test sends a null it may pass but the real thing may fail.
Instead create an architectural boundry with a clear public interface. Exercise the interfaces in tests and then check for results on the otherside of the boundary. You might do that with a mock or a fake.
You are now free to refactor from the start to finish of that architectural boundry without breaking tests.
This is just fancy talk for only testing your public API. But defining those boundries and public interfaces is where the skill is.
If your building a web app using hexagonal architecture, I might say drive your primary ports from your tests, mock and fake your secondary ports.
If you expect some group of objects to used in multiple places like a library I will test those as if they're a boundry to. For example If I've built money exchange rate module.
> Instead create an architectural boundry with a clear public interface. Exercise the interfaces in tests and then check for results on the otherside of the boundary. You might do that with a mock or a fake.
I think mockist (test-doublist really) TDDers would suggest that the architectural boundaries derived from the design pressure are the "correct" public interfaces you're describing here, they just often happen to coincide with class (or your favourite languages equivalent) boundaries.
The pattern often ends up with many one-function role interfaces, orchestrated by collaborators down the dependency tree until you hit value structures, pure functions or external integrations at the edge of the system. In many ways, mockist TDD is a gateway to functional programming.
Hm, yes and no. I do feel that mockist TDD is still heavily emphasising the OOP idea (more in the original sense than in the Java sense) of having separate "collaborators" with their own internal state and side effects that exchange messages (in a way similar to the Actor pattern).
A functional approach (at least in a pure functional language) wouldn't necessarily emphasise this sort of interaction pattern between independent components and try to isolate state and side effects much more.
If you do functional programming, obviously many OOP techniques just don't apply. This is more of a difference in philosophy. I'm sympathetic to functional programming, and do try to use immutability, value objects, referential transparency, explicit state handling etc., whenever possible, but I'm still constrained by the languages, frameworks and teams that I work with, and I think many others are as well.
Of course, the question whether we should all just program in Haskell (or lisp, or erlang, ...) can be debated, but for a variety of reasons that is not currently the case, so I think mocks are still a valid answer for OOP, if (!) you use them correctly (and I agree that many may be too cavalier about mocking).
But to answer your question about your "extra layer": If your code is written in a domain driven style, then potentially adding in a new layer should be considered a change in behaviour, so changing the tests makes sense. If it's purely a technical thing, then there are IMHO often ways of not exposing them to surrounding code. As an example, if you're introducing some e.g. logging layer to your BillingService, instead of pasting in that layer in the original code that calls the billing_service, you could decorate your BillingService with a logging wrapper class and just change the injected dependency. Nothing about the tests using the billing_service would have to change. This is a stupid example, but I hope it gets the point across.
> What if you want to add in extra layer, or remove a layer, or add a helper.
You make the extra layer implement the same interface and delegate to the original object. Classes don't depend directly on other implementation classes in this style.
Absolutely. I've worked in places which enforced this decouple-everything-from-everything approach, and it just grinds work to a halt, makes test failures commonplace and uninformative.
How do you avoid external dependencies in this situation? If you're only testing the public interface, then you have to expose some kind of public dependency injection pattern. Now you've introduced a level of indirection into the production code specifically to make testing easier. This feels like an antipattern.
It's how I do it, but I'm not satisfied with it. It makes the code harder to read, I sometimes miss bugs because my injected mocks don't handle an edge case correctly, and the effort of maintaining the mocks themselves is non-trivial.
I agree, fake objects are a good alternative to mocks especially for complicated interfaces (databases, external services, etc.). But they require more setup, which is why I still think mocks can be useful in some situations.
They can require more setup in that you have to write an object to represent the thing.
They require less setup in that mocks require tons visual noise in setting them inside the test. Where as a fake will just be created with a standard constructor.
I hate a lot of mocking I see in real code. They go over board with the mocking and the test is 80% mock setup. Making it hard to see the real purpose of the test.
Tests should be short, simple, to the point, and easy to read. When most of the test is setting up a mock, you've lost that.
But there not too bad if it's a very simple one line setup. Which is how they should be used.
But that would not tell you that if you passed a null object into the real thing it would crash. Where as fake, with the tests will.
I think it really depends. If you keep the number of collaborators in a class small and your interfaces are well designed mocking is ok (and really just a limited form of a fake).
I don't agree that dependency injection is necessarily bad, but if you really want to avoid it/mocking, there are also other ways, i.e. structuring your code into pure and impure parts, only unit testing the pure stuff and relying on integration tests for the "wiring".
You’re talking about integration tests, not unit tests. Unit tests should focus on a single method. Integration tests can combine methods/objects/APIs/etc for business rules.
Your best bet is a combination of unit tests, integration tests, and e2e tests (there’s some old advice about a 70/20/10 split but this is pretty arbitrary).
Yes this what has spread around and what people think of as unit tests. But it an awful way of splitting tests. Code bases following this end up with brittle tests, that break at slightest design change with no functional change.
I define unit tests as fast, and don't cross architectural boundaries, and operate on a well defined public interface. And test some kind actual property you care about. I don't follow "strict" rules like unit test per class, per function etc.
A lot classes just extract to some to some helper class, and they operate well together and the helper is not likely to be used anywhere else. Just make the helper class private and test as a unit against some actual desired property you want to test.
Integration tests is when you bring external things into the mix like databases, or http calls.
What I would encourage you to do is define the terms you're using, and don't assume that others are using certain words in the way that you mean them. I like that you've been explicit about your definitions here (although it may be better to describe rather than assert, e.g. "I treat unit tests as focusing on a single method", rather than "Unit tests should focus on a single method").
Indeed, especially having an axis extending from an individual function/method up to third-party services would presumably stop the "everything is a class" creep that's happened to so much testing terminology already.
Small unit tests are valuable iff your small units are well designed and valuable. If they are brittle, temporary components they don't have much value, I agree, but writing mostly integration tests suffers from the combinatorial explosion problem and from making it hard to express succinctly what the important bit about a piece of code is (and besides, almost no test is 100% integrated, you always isolate something, even in full end-to-end tests).
If you have many functional, stateless components, the unit tests can be particularly valuable.
But I think many tests end up being badly written because people don't necessarily ask the fundamental question: "does this increase my confidence in the code?" I think that's the most important consideration when writing tests.
I agree that tests are an insurance against changes. They find regressions and are also lights in the darkness that points to where we have to work on.
I'm moving 3 db fields (let's call them a, b, c) from 3 tables (A, B, C) to a fourth one (D) right now. Adding a, b, c to D is easy, the data migration is also easy (a single update from a CTE coalescing a, b, c from the 3 tables -- the value in a wins over b and over c). And now let's see if anything breaks (it has to.) I run the test suite. Only a dozen errors. Some are where I expected them to happen, some are totally unexpected. There are parts of the code base I forgot about in the last year or never heard about (it's a team of about 5 developers.)
Without tests, no matter the language, typing system, etc any change like that would be a nightmare.
Not to downplay tests, but this would be caught by a typing system if all access to DB fields was through an ORM such that all access could be type checked.
Not to say that everyone should use an ORM, but this would be a very simple property for a type checker to catch if you let it.
Possibly, but (simplifying a lot) the actual refactoring involves JSONB fields that store serializations of objects which include those fields plus a yet unknown amount of code that might be using them, etc. And we're ending up redesigning part of the UI because 1) it makes sense and 2) it makes the migration to the new backend simpler. Luckily we're pretty well covered by tests.
You can always start with black box testing so you check that bit of the program matches some expected input/output pairs, and then add more white box integration tests if the code is important/fiddly. It’s an easy way to get some validation on a particular subset of the system for a small amount of effort.
We write tests so that we know whether future changes have broken the system or not.
This article sounds like a reaction to the practice of writing many small, hermetic unit tests, which do little more than recapitulate the code under test. The weakest parts of a system are often in the joints, so the most important tests to write and to run are the integration tests, the ones that tell you with the most confidence whether the code works in the real world or not.
See, for example, this puzzling line from the article:
> Other tests grow obsolete because the target platform is retired, but not the test. There's even less pressure to remove stale tests than useless code. Watch as the workaround for Windows XP is removed from the code, but not the test checking it still works.
How do you remove the code for a feature, but not the test for that feature unless either you aren't running your tests or your tests aren't testing the things that you care about?