Most decision makers agree that test automation is good for their business, as it increases test coverage while reducing testing costs and business risk.
That being said, decision makers often hesitate to commit the appropriate resources and tools to test automation. This is largely due to the uncertainty around the upfront cost of developing a test automation framework, and the ongoing cost of maintaining an automated testing program.
This blog is about how to measure test automation ROI--both directly and indirectly. Let’s jump in and learn how to communicate the value of test automation to fiscally-minded decision makers.
And for more content on this topic, check out our eBook on how to save money with test automation, and our blog post on how to measure the business value of test automation.
Calculating Test Automation ROI
Use Case: Calculating direct cost
To measure the direct ROI benefit of test automation, let’s begin with some basic arithmetic:
Scenario 1 - 100% manual testing
Assumption 1 - A team consists of 5 entry level testers costing $50/hour, and 5 senior testers costing $75/hour.
Assumption 2 - Each tester runs 8 hours of tests/day.
Assumption 3 - Each tester works 20 days/month.
Cost per month:
Scenario 2 - Replacing entry level manual testers with test automation
Assumption 1- The cost of a test automation solution equals the cost of an entry-level tester, working 8 hours/day.
Assumption 2- The test automation solution includes 2 virtual machines (VMs) running tests 24 hours/day.
Assumption 3- The test automation solution can replace an entry-level testing team.
Based on the table below, we see that test automation saves the typical enterprise nearly $24/hour for each hour of testing ($62.50-$38.64).
Cost per month:
Conclusion
Each hour of manual testing costs more than an hour of automated testing. You will easily save over 30% in your testing budget by replacing the entry level testers with automated testing. The numbers speak for themselves: the monthly cost for each month of manual testing adds up quickly. Simply put, manual testing processes are costly and they do not ensure optimal risk coverage like test automation. Your exposure to risk is not the only ROI-related concern. There are other indirect factors impacting your ROI when weighing the benefits of automation.
- Test scripts are created through educated guesses. As a result, testing teams either end up testing too much–wasting time and money–or testing too little, exposing themselves to business risk. Automated testing prevents your team’s time from being wasted and covers gaps to minimize risk.
- A failure to innovate will impact your ROI, leaving you trailing behind the competition. With major ERP vendors like Oracle, SAP, and Microsoft rolling out updates more frequently, it’s imperative to deploy these updates on time to keep up with your competitors. You may miss out on new customers, markets, and revenue streams.
- Your ROI is always subject to the unforeseen costs of post-release bug fixes. According to the Systems Sciences Institute at IBM, the cost to fix bugs found during the testing phase of software is 15x more than fixing bugs during the design phase. It is impossible to assign a monetary value to the cost of bugs that will be avoided due to optimal test coverage. Embracing shift-left testing saves time, and we all know, time is money. Read more: Why does DevOps require shift-left testing principles
.png)
