In March, we exhibited at the 33rd SQA Annual Meeting and Quality College and spoke with many people from regulated industries who tested web applications using paper based manual test. Almost uniformly people expressed frustration with paper based testing, especially the amount of time it takes. One woman in particular had just finished testing a web application with paper tests that took her five months to execute. In our experience, it is common even for teams of five to ten people to take weeks or months to finish a suite of paper tests.
Paper based web application testing takes too long to implement, and our desire to speed up this process is one of the factors that lead us to create Toffee. In our prior blog post, Manual Testing: Toffee is about Orchestration not only Automation, we explained how Toffee improves the speed and accuracy of manual testing by eliminating paper and promoting heads up testing. With Toffee, testers no longer need to switch contexts between the application under test, paper test scripts, and a text document for tracking screenshots. Now, we want to share an experiment we performed to test the time efficiency gains of using Toffee tests versus traditional paper based tests.
As part of our experiment, we created two sets of tests for the exact same functionality of a food ordering web application. One set of tests was a pure paper based test consisting of 47 manual steps. The second set of tests was Toffee based and although 7 of the 47 steps were automated, the rest were still manual.
Twenty different people executed both sets of tests with half doing the paper based test first and half starting with the Toffee based test. We timed how long it took each person to complete each test and found that the Toffee test took 31% less time to complete. So, even though 85% of the Toffee test required the exact same interactions with the application under test as the paper test, people were able to complete the Toffee test in about two thirds the time it took to perform the paper test.
This is a huge gain in time efficiency especially if tests take weeks or months to complete and just one of the many reasons to use Toffee to orchestrate your testing.
For more information about Toffee, visit http://toffeetesting.io, or feel free to contact us with questions at firstname.lastname@example.org.