The main goal of Continuous Integration and Deployment (CI/CD) is to let software development teams frequently release working products to users, thereby benefiting the developers by giving them feedback on how the product is being used.
In order to keep up with the competition, many companies have started using DevOps practices.
However, business pressure to speed up delivery should not adversely affect the quality of the product being produced. Users may be excited about new features, but they expect the software to continue to function and remain stable. That’s why robust and thorough automated testing that gives you confidence in new builds is a necessary part of your continuous integration and delivery practice.
Using automated tests in CI/CD
Tests give us confidence in the quality of our software and have long been a part of the development process. In waterfall development, manual testing (or the QA stage) was performed after the code had been written and incorporated, to inspect whether the application behaved according to the specifications or not. This linear strategy slows down the release process. It takes a long time for the developer to find out if the new code works when a lot of other things have been already added to the code.
In contrast, the CI/CD process takes a flexible approach with short iterative cycles that provide quick feedback and allow for frequent small updates. A key component of these short iterative cycles are tests – automatic verification of whether the new code works and does not break anything around.
Continuous integration includes regular commits of changes to the main master branch, which triggers the assembly and testing of the application. To get the most out of CI, team members should commit at least once a day. Let’s say that you don’t even have that many developers. In order to manually manage the launch of tests, you will need an impressive number of testers who will be forced to constantly do the same work. That’s why automated testing comes into play.
Automation is perfect for repetitive tasks. It provides more consistency than manual testing because people always run the risk of missing details or introducing variability within commits when repeating the same steps.
Not only are automated tests faster than manual ones, but they can also be run in parallel. With the right infrastructure in place, you can scale and save time later. Writing automated tests takes time, but being able to commit regularly and release a product much more often is really worth it.
Automated testing will free you from a number of boring repetitive tasks, but this does not mean that you will no longer need testers. Testers are responsible for identifying and prioritizing test cases and writing automated tests, and they often do all of it in cooperation with developers. Moreover, testers are left with the kind of tests that cannot be automated.
Source: mitrais.com
Role of testing in CI/CD process
Testing is performed at several stages of the workflow. If you’ve never done continuous integration, you might think that this is too much. But the essence of CI/CD is short cycles of communication, allowing your team to learn about problems as early as possible.
It’s much easier to fix a bug right after it occurs, so you don’t build new code on top of it. In order to accomplish this, in practice developers create a structure of tests that can often be called a ‘testing pyramid’.
Creating a testing pyramid
The testing pyramid helps you get an idea of how to prioritize the tests in the CI/CD pipeline, and determine the required number of tests and the order in which they are executed. Originally introduced by Mike Cohn, the testing pyramid has unit tests at the bottom, integration tests in the middle, and interface tests at the top.
![Tech Blog] Test-Pyramid](https://cdn-contents.anymindgroup.com/corporate/wp-uploads/2021/09/28103833/pyramid.png)
Source: anymindgroup.com
The name “testing pyramid” is not too precise, but the initial idea is clear: it is worth starting with the creation of a solid base of automated unit tests that are fast and effortless to run, then move on to more difficult-to-write and long-running tests, and finish with a small number of the even more difficult ones. Let’s look at the types of tests to do.
Unit tests
Unit tests rightfully occupy a place at the base of the pyramid. These tests are invented to verify that your code works as expected by addressing the smallest parts of the code’s behavior as possible. Unit tests, as a rule, are written by developers – in parallel with writing the code itself. This naturally follows from the Test Driven Development (TDD) methodology. However, you don’t have to follow TDD to write unit tests.
If you’re working on existing development but don’t have unit tests for it, it can be almost impossible to write unit tests for the entire codebase. While broad coverage is recommended, you can start with as many unit tests as you can and add them over time. A practical strategy is to add unit tests for every piece of code you interact with, thereby providing coverage for the new code and the old code you have used in development.
Integration tests
Integration tests allow you to make sure that the components of your software interact correctly with each other (for example, the interaction of application code with a database). It is useful to divide integration tests into broad and narrow ones. Narrow integration tests test interaction with another module using test twins rather than the real module, while broad integration tests interact with a real component or service.
Depending on the intricacy of your software and the number of internal and external services used, you may want to write a layer of narrow integration tests. They will run faster than broad ones as the narrow ones do not require the availability of other components of the system. After that, you can add a set of integration tests aimed at higher priority subsystems.
End-to-End Tests
Also known as full-stack, end-to-end tests cover the entire application. These tests can be performed not only through the user interface but also through the API, which also works with several components of the system (APIs are also tested using integration tests). The testing pyramid dictates that fewer of these tests be done, not only because they take longer to complete, but also because they are usually more fragile.
Performance Testing
Although the testing pyramid does not mention performance tests, it is worth considering including them in your test suite, especially for products where stability and speed are highly important requirements.
Benchmarking refers to a range of testing strategies to verify how your software will behave in a real environment. Load testing checks the behavior of the system when demand increases.
The purpose of these types of testing is not only to check that the software can cope with the specified load thresholds but also to test the behavior when these thresholds are exceeded – it is desirable that the fall looks a little more graceful than a crash with a crack and roar.
Test environments
Both performance and end-to-end tests need test environments that mimic the production environment as closely as possible and may need test data. For automated tests to really instill confidence in the software being developed, it is important that the tests are run the same way every time, in particular, that the test environments maintain the same state from run to run (while they must be updated according to changes in production).
Handling environments manually is rather time-consuming. Therefore, it is worth thinking about automating the steps to create and reset pre-production environments for each new build.
Does CI/CD mean the end of manual testing?
There is a shared misconception among those new to CI/CD that automated testing eliminates the need for manual testing and professional testers. Automated testing may free up QA team members some time, but it will not replace them. Instead of wasting time on routine tasks, testers can focus on identifying test cases and writing automated tests, as well as doing exploratory testing, unleashing creativity and ingenuity.
Unlike automated tests, which are scripted to run, exploratory testing requires a more relaxed approach. Exploratory testing is valuable in its ability to find what balanced and structured tests miss. In essence, you are looking for errors that you have not thought about before and for which you have not written a test case. When choosing which areas to explore, think about both new functionality and the parts of your system that would suffer the most if something went wrong in production.
Exploratory testing should not become routine manual testing. You don’t have to run the same set of tests every time. If you find errors during exploratory testing, in addition to fixing them, take the time to write the appropriate automated test (selecting the appropriate level of the testing pyramid) so that if the error occurs again, it will be detected much earlier. To make better use of testers’ time, manual testing should only be performed after all auto-tests have been completed.
Continuous improvement of auto-tests
Automated testing is central to any CI/CD pipeline. While writing these tests takes time and effort, by starting to get quick feedback and seeing your code rollout, you’ll quickly feel the value of the work you’ve done. However, building a test suite is not something you can do once and forget about after.
Automated tests should be a significant part of your application code, and to keep them relevant and informative, you will need to maintain them. Continuous improvement of the code requires continuous improvement of the tests.
By expanding test coverage as new functionality becomes available and adding what comes to light through testing, you’ll keep your test suite up to date and efficient. It’s also a good idea to take the time to analyze how fast your tests are running, and whether you need to reorder or break up process steps to get feedback sooner.
Basically, your task is to regularly deliver functioning software to users. Test automation helps with this by providing fast and reliable feedback so you can be confident that your software will be successfully deployed to production.
