Quality without bottlenecks

Have you ever had to delete end-to-end acceptance tests because your Continuous Integration became unacceptably slow? It has happened to all of us.

Have you ever had to delete end-to-end acceptance tests because your Continuous Integration became unacceptably slow? It has happened to all of us.

End-to-end acceptance tests are highly meaningful because they verify how the system is expected to behave from the end-user's perspective and in an environment very close to production. However, overusing this type of test to cover everything in a system can turn your pipeline into a bottleneck, make test maintenance an excessive burden for developers, and delay feedback by hours (or even days!).

So, how do we maintain high quality without sacrificing efficiency? Here are a few strategies:

• Identify which errors we cannot afford and pinpoint the system's critical paths—those aligned with business goals—to focus the most meaningful tests there (meaning end-to-end acceptance tests). The remaining paths can be tested using less costly test types (API, unit, or fast acceptance tests).

  
  

• Find the right balance between different types of tests by prioritizing reliability and execution speed (which translates into early feedback), without losing sight of test relevance. For example, increasing coverage by adding getter and setter tests—no matter how fast they run—adds very little value.

  
  

• Replace external dependencies with in-memory fakes (mocks or stubs) and add contract tests to validate that interfaces with those dependencies behave as expected.

  
  

• Reduce the scope of the system under test by running certain acceptance tests in smaller, more focused test assemblies. Learn more about how we work here.

  
  

Has this ever happened to you? How did you handle test bloat in your end-to-end suite? Share your experience with us!

Natalia Lehmann