Pre- and post-deployment testing methodologies for CI/CD

Implementing strong pre- and post-deployment testing methodologies will help ensure your CI/CD workflows stay fast without sacrificing software quality.

Your team has worked hard on a software product for months, and it’s finally ready to release to your users! But then the worst-case scenario happens: a wide release soon indicates that the software is plagued with bugs and performance issues, resulting in poor reviews and widespread user dissatisfaction.

Regardless of how meticulous we are, things don’t always go according to plan when developing and deploying software. Software testing can contribute significantly to the overall success of a product, but developers often stop testing their software after deployment. Although we may be unable to eliminate all potential bugs and issues that may arise, the proper pre- and post-deployment testing strategy will drastically lower their chances of occurrence. 

Pre-deployment testing methods

Fixing bugs in the early stages of software development is inexpensive compared to doing so later on. Spotting bugs early can also save us countless hours of debugging time, resources, and the potential cost of maintenance that would otherwise be incurred in production. This is because such issues can easily be addressed in a controlled environment without causing users grief.

Pre-deployment testing aims to find underlying bugs and issues so they can’t make their way into production. Performing end-to-end testing, verifying testing, and manually verifying features and functionalities is part of the pre-deployment testing process. Typically, a testing environment, be it local, staged, or in the cloud, is used to perform these tests.

Often, most of the deployment problems that teams face don’t come from a lack of software tests but rather from having plenty of manual tests that require a manual acceptance step. Only then is the application deployed into production. This manual testing workflow is time-consuming and can quickly burden us. 

Therefore, continuous integration/continuous deployment (CI/CD) pipelines should be used to automate testing as much as possible. Why? Because automated testing, which enables continuous testing, ensures that bugs are found early and fixed before they get to end users. 

Instead of depending on traditional manual tests, Agile teams use modern DevOps features by implementing CI/CD pipelines to run tests automatically after every commit. CI/CD can remove some testing burdens by automating certain tests as part of the pipeline. 

Automation is key to achieving speed and flexibility in continuous software testing, which is essential for delivering high-quality software quickly. The agility and flexibility of a CI/CD  pipeline help teams mitigate the risks associated with new code, especially when testing by verifying that tests work. 

These are several examples of pre-deployment testing that can be automated as part of a continuous testing pipeline. 

Functional testing 

Functional testing aims to verify that each software function works per specifications and the end user’s expectations. It’s performed by providing various data inputs that are used to test each function to verify that the actual outputs match the expected results. 

Tests of this type are purely black-box tests that do not concern themselves with the software’s source code.

Unit testing

A unit test verifies that each independently functioning component operates as expected. When unit testing is performed, developers search for errors within these units. Unit testing entails developers testing the code of each of these units to identify if there are any errors.

Before further testing can occur, each code unit is tested thoroughly. As a result, possible bugs are identified right at the beginning of a test cycle, saving time and resources. 

Stress testing

Imagine bending a stick to find the point at which it breaks. That’s what stress testing does to software. During a stress test, a software application is tested to see if it can handle conditions beyond its normal operations. It’s performed by testing how the application works under high load or is placed under abnormal conditions.

The results are used to optimize the software to cope and be more stable under these extreme conditions.

Integration testing

Integration testing is needed when multiple functional units or modules are required for a software to function effectively. This ensures that each module works as expected when operating in conjunction with other modules. This test aims to reveal the faults in the integration and interaction between the units.

However, performing these various tests can be tedious, repetitive, and time-consuming. By automating the testing process through a CI/CD pipeline, we can reduce the testing time and cost by executing test cases automatically. Automating these tests enables us to maintain the quality and security of our software without sacrificing delivery speed. 

Post-deployment testing methods

In pre-deployment testing, despite all the testing carried out before the final deployment, we probably won’t be able to uncover every issue in the software. Once pre-deployment testing is complete, depending on our deployment strategy, we can release the software to a small group of users or everyone at once. Then comes the post-deployment testing stage. 

The next priority of any team is to rapidly identify and fix bugs and defects that only emerge in a live environment—some of which are introduced by the deployment process itself. 

Post-deployment testing is the final line of defense. It reveals previously undetected problems and unintended outcomes. So, we can detect and rectify problems before they negatively affect our user experience and possible conversion rates.

Post-deployment testing often takes on more of a QA aspect. It entails retesting the software features, gathering user feedback, and monitoring the software to ensure it functions as intended in production.

Some post-deployment testing activities include:

• Performing release verification to ensure the application functions as expected and is in line with test results from the pre-deployment development environment;
• Performing canary testing after deployment by only releasing the new software to a certain percentage of real users in production to collect user feedback and limit potential blast radius before rolling it out to the general audience;
• Ensure software is deployed with robust observability and dive into the details to identify problem areas and formulate tests;
• Collecting feedback from observability and monitoring tools to understand the success or failure rate of the software and to ensure the software’s performance doesn’t derail from expected performance;
• Look for problems introduced by the deployment process itself. Our deployment may be prone to common issues such as missing configuration, unaccounted production dependencies, or incorrectly installed scripts;
• Feature flagging to facilitate the testing of components and features in a controlled manner in production with the ability to roll them back if necessary;
• Collaborate closely with QA teams to determine potential issues and create tests to isolate and illuminate the problem.

Conclusion

Teams are pushed more than ever to ship software faster and more reliably, but this can introduce risks and problems for our users without the proper testing workflow. When development teams fail to conduct the correct tests before and after deployment, it can lead to more work, higher maintenance costs, and unhappy users. 

A critical part of the deployment process is testing our software. We can ensure a better product by performing tests pre-and post-deployment. When done right, software testing can significantly improve a software’s quality, performance, and reliability. 

However, a pipeline that depends on manual tests is slow and prone to errors. As a result, the demand for faster software delivery requires teams to adopt flexible automated testing practices. A CI/CD pipeline ensures the process of testing, integrating, and delivery is Agile and flexible. Automating these tests allows us to achieve continuous testing, which is essential for delivering high-quality software quickly.

To learn more about designing secure CI/CD pipelines, check this out.

This blog post was created as part of the Mattermost Community Writing Program and is published under the CC BY-NC-SA 4.0 license. To learn more about the Mattermost Community Writing Program, check this out.