The Race Against Tests: A Guide to Optimizing Automated Acceptance Testing
“You are always losing the race against tests.” This quote by Alan Cyment made me reflect on the time we invest in maintaining acceptance tests—a practice that, due to its repetitive nature, the fragile tools we use, and the heavy time commitment it demands, is often abandoned without fully realizing the value it delivers.
At Grupo Esfera, we have been walking the test automation path for years. While we know there is still plenty of room for improvement, we have a few insights to share.
First, we should address motivation: Why do we do automated testing?
To increase code quality. Not just to comply with a quality standard, but to make it easier to ship new features. Software is an ever-evolving beast; if it doesn't grow, it dies. From a development standpoint, we must manage and protect that growth so it doesn't split into chaotic "branches" that we later fear touching because the codebase has become a thorn bush.
To touch code without fear, we rely on a solid test suite to back us up. Ultimately, we write tests to make changing software easier.
Furthermore, if we write those tests before writing the implementation code (as TDD and ATDD practices advocate), we unlock several additional benefits:
It forces us to define expectations: I first think about what I expect from this code snippet or feature—what the preconditions are and what outcome I expect. I write a small test, build a minimal implementation, and advance toward a better, more complete implementation only as I am able to define tests for each expected outcome.
It keeps us focused on high-priority items so we start there.
It brings structure to our thinking, covering all test cases rather than just the happy path.
It guides us to implement in small increments, which are naturally less error-prone.
It reduces the perception that testing is a waste of time—a common feeling when tests are written a posteriori, after the implementation has already been manually tested. Writing them upfront ensures that by the time we finish, we already have a robust set of tests built out.
Additionally, test automation gives us two primary advantages. First, it shortens the feedback loop—reducing the time it takes to realize whether a change broke existing functionality.
I make a change, run the whole suite, and if every test passes, I know my change didn't break other features. Could this regression testing be done manually? Yes, but it would take significantly longer. Beyond offering almost instant feedback, automation reduces human error, making the test execution repeatable every single time.
How to implement automated acceptance testing without losing efficiency
In our project team, we decided to implement automated acceptance tests from Day 0. This practice forces everyone involved (analysts, developers, testers, users, stakeholders) to sit at the table, discuss every user story, and translate those discussions into acceptance criteria, which effectively become an executable specification. This document serves a dual purpose: it describes what the system does and simultaneously verifies that the description holds true.
Moreover, as mentioned earlier, any automated test suite doubles as a safety net for regression testing.
The Problem with UI (User Interface) Tests
However, because acceptance tests are built from the end-user's perspective, we often fall into the (bad?) habit of describing them via the graphical interface—implementing them as end-to-end (E2E) integration tests that start from the user's interaction with the screen.
These types of UI tests tend to be highly inefficient:
They are slow (they require spinning up a browser, rendering the full HTML—including images and stylesheets that may take a long time to load—locating DOM objects, etc.).
They are unreliable because current tools are not very stable when selecting UI elements or reproducing behaviors. In other words, they produce false positives and tend to be flaky.
They don't help pinpoint issues when something breaks. When a failure occurs, all we see is the generic error message shown to the user, lacking granular details about what actually failed, why, or on which line of code.
Because of this, they require high maintenance: we spend a tremendous amount of time fixing tests that fail not due to faulty implementation, but due to the flaky nature of the test itself.
Should You Have Automated UI Tests?
The short answer is yes—but to what extent?
This is where Mike Cohn's Test Pyramid (1) comes into play. The pyramid provides a blueprint for structuring the volume of tests we should write per layer. It demonstrates that we should favor unit tests above all others.
Why? For several reasons: they are fast (executing in milliseconds); isolated (they don't depend on one another); repeatable; and self-validating, requiring no manual verification. They make optimal use of our most scarce resource: time.
While extremely valuable, unit tests alone cannot test end-to-end business logic or guarantee the system as a whole works correctly.
Therefore, other test types are necessary to ensure system correctness. We need varying degrees of integration tests—ranging from integrating a few components together up to full E2E integration, including the graphical interface.
How do we strike a balance between the need for integration tests and the desire to minimize the time spent maintaining them?
There is no silver-bullet answer to this question. Rather than a golden rule, we offer a few practical takeaways from our own journey, which will undoubtedly continue to evolve over time.
Practical Advice
Ask: What are we actually trying to test here? If you genuinely want to test the graphical interface, consider decoupling the UI interactions from the underlying system logic. This is easily achieved by mocking the API, effectively narrowing the test scope to how the UI responds to specific user actions and payload responses. Bonus: this makes UI tests run dramatically faster. A tool we found particularly useful for mocking APIs quickly is JSON Server, though many alternatives exist.
If you don't need to verify screen rendering, skip the UI. Implement integration tests directly against the HTTP layer, using an HTTP client to issue requests directly to the API and asserting the response payload. Martin Fowler calls these subcutaneous tests. Our teams have successfully implemented them in backend projects (built in C# or Java); they run significantly faster and are infinitely more stable than browser-driven tests. Plus, they allow integration tests to contribute directly to code coverage metrics.
Use unit test runners (such as JUnit, xUnit, NUnit, Jasmine, Karma, etc.) to execute narrow-scoped integration tests. This allows you to complement unit tests with lightweight tests that verify interactions between a small cluster of components while keeping execution times low.
The core motivation behind exploring these alternatives is recognizing that not every execution path needs to be tested in the exact same way, as doing so severely degrades test suite performance and maintainability. The Test Pyramid advises keeping E2E integration tests to a minimum.
So, which ones should you automate? Align them with critical business capabilities—the core features without which the application loses its fundamental value. This aligns team incentives, making the maintenance of these critical tests a shared, high-value responsibility.
Perhaps the most important takeaway is to remember that acceptance tests and the test pyramid are orthogonal concepts. Avoid the pitfall of implementing all acceptance tests as E2E UI tests. Acceptance tests can—and should—be implemented at any level of the pyramid, including the unit test layer. You can leverage various tools for this: the common mistake is expecting a single tool to solve every testing need. Generally, BDD frameworks like Cucumber (across its language variants) serve as essential allies to ensure test definitions use a domain language everyone can read, discuss, and refine—ultimately transforming tests into executable documentation.
Test Types Glossary
Integration Tests: Verify that individual modules or components assemble and interact correctly after passing unit testing. Integration exists on a spectrum; when it spans the entire stack—from UI interaction down to low-level data persistence—it is considered an End-to-End (E2E) test.
Acceptance Tests: Validate whether the software meets business requirements and satisfies user goals. Their primary purpose is to automatically verify that user expectations and acceptance criteria are met.
Unit Tests: Isolate a specific unit of code to verify its execution logic and state transitions in complete isolation. They are fast, focused tests that validate object behaviors.
Regression Tests: Executed to ensure that recent code modifications or refactoring have not introduced side effects or broken existing functionality in untouched parts of the system.
Further Reading:
(1) The Test Pyramid (Mike Cohn): Unit tests sit at the base of the pyramid, offering fast, highly localized feedback. At the top sit user-level tests, which test the application end-to-end via the UI. The middle layer consists of what Cohn calls "service tests" (testing application capabilities/API endpoints below the UI surface).
Comments