Archive for May, 2017

Time Consuming Paper Tests

In March, we exhibited at the 33rd SQA Annual Meeting and Quality College and spoke with many people from regulated industries who tested web applications using paper based manual test. Almost uniformly people expressed frustration with paper based testing, especially the amount of time it takes.  One woman in particular had just finished testing a web application with paper tests that took her five months to execute.  In our experience, it is common even for teams of five to ten people to take weeks or months to finish a suite of paper tests.

Paper based web application testing takes too long to implement, and our desire to speed up this process is one of the factors that lead us to create Toffee.  In our prior blog post, Manual Testing: Toffee is about Orchestration not only Automation, we explained how Toffee improves the speed and accuracy of manual testing by eliminating paper and promoting heads up testing.  With Toffee, testers no longer need to switch contexts between the application under test, paper test scripts, and a text document for tracking screenshots.  Now, we want to share an experiment we performed to test the time efficiency gains of using Toffee tests versus traditional paper based tests.

As part of our experiment, we created two sets of tests for the exact same functionality of a food ordering web application.  One set of tests was a pure paper based test consisting of 47 manual steps.  The second set of tests was Toffee based and although 7 of the 47 steps were automated, the rest were still manual.

Twenty different people executed both sets of tests with half doing the paper based test first and half starting with the Toffee based test.  We timed how long it took each person to complete each test and found that the Toffee test took 31% less time to complete.  So, even though 85% of the Toffee test required the exact same interactions with the application under test as the paper test, people were able to complete the Toffee test in about two thirds the time it took to perform the paper test.

This is a huge gain in time efficiency especially if tests take weeks or months to complete and just one of the many reasons to use Toffee to orchestrate your testing.

For more information about Toffee, visit, or feel free to contact us with questions at

Testing Multi-User Interaction

How do you properly test the interaction between multiple users  within a web application?  What if you need to test that user B receives a notification immediately after user A performs some action? How do you ensure that user D’s access is cut off immediately when user C revokes it? Testing this type of functionality can be difficult, because it is not as simple as opening the application under test and typing in fields and clicking buttons.  In order to ensure that the interaction functions properly and timely, you often need to concurrently open, interact with, and monitor at least two unique instances of the application.

Consider a chat app that allows two users to instantly send messages to each other through your web interface.  To test this application, you must log two different users into their own instance of the app at the same time.  To test that the messages are instantaneous, you need to send a message from the first user and immediately check if the message appears in the second user’s inbox.  You cannot simply log in to the app with one user, send a message, and then log in as the receiving user because the immediacy of the message exchange is not tested.

Collecting screenshot evidence of user interaction adds another layer of complexity because, you have to position each unique user’s instance of the application on the screen so that they do not overlap with each other.

Manually testing user interaction quickly becomes time consuming and complicated as you need to test and gather evidence for more unique users.  Many web testing tools do not even offer the ability to simultaneously run multiple instances of the application under test, however, Toffee has features to make it easy including multiple test sessions and comparative screenshots.

A Toffee test session is a single active instance of a browser that is then used to access the application under test.  Toffee allows you to open multiple test sessions at the same time.  This can be used to simulate multiple users because each test session can access the application as a different user.  As in the chat app example, Toffee can log in to two instances of the app, send messages from one user, and confirm that the messages are immediately received by the second user.

Toffee makes it simple to document the user interaction with comparative screenshots.  Using Toffee dock or move commands you can easily set up a comparative screenshot so that each unique test session does not overlap.  Dock commands place a session at either the top, bottom, right, left, or any corners of the screen.  Move commands allow the user to set a session to any user specified screen coordinates.

Testing multi-user interaction can be a complicated nuisance, but Toffee’s test sessions and comparative screenshots make it simple.

For more information about Toffee, visit, or feel free to contact us with questions at