Archive for April, 2017

Record/Replay vs Programmable Tests: Toffee the Best of Both

Record/replay and programmable web testing tools each have their own benefits and shortfalls. These trade offs are analysed in a paper by Maurizio Leotta titled “Capture-Replay vs. Programmable Web Testing: An Empirical Assessment during Test Case Evolution” (citation below).  Leotta concludes that while initial test creation is faster for record/replay tests, programmable web tests save time during regression testing as the application is updated.  

As part of Leotta’s study, testers with some programming experience made two sets of tests for six different web applications.  Each web application had one set of tests made using a record/replay tool called Selenium IDE and another set of programmed tests written in Java using Selenium WebDriver.  The testers then updated each test suite for a newer version of each web application.  Leotta measured the time the testers spent creating and updating the test suites, comparing the record/replay time with the programmable test time.

Leotta found that the programmable tests took up to 112% more time to make from scratch than the record/replay tests.  This was mainly because creating the record/replay tests did not require any programming knowledge and was as simple as the tester recording themselves using the web application.

Next, Leotta’s research showed that when test suites were updated for new versions the programming tests took up to 51% less time to repair than the record/repay tests.  This time saving was mostly due to the developers using the page object design pattern.  In summary, a page object is a programming construct where code that locates elements and initiates actions on the web application is stored in one place.  The page objects can be reused in multiple tests.  Testers only update page objects when repairing programmable tests, which can fix multiple tests at once. Record/replay tests do not reuse anything, so every record/replay test must be fixed individually or even recorded again every time the application under test changes.

As I discussed in a previous post, Toffee avoids the brittleness common to record/replay testing, replacing it with plain-language test commands and an interactive script development environment. Toffee also saves time during test repairs with aliases and test modularization, which provide the same benefits as  the page object pattern, but without requiring programming.  Toffee commands that are repeated throughout tests can be saved as an alias, so only the alias needs to be updated.  Also, a series of frequently used commands (think login) can be saved and maintained in a single “include” file and used on multiple tests.

If the application will be updated more than once, Leotta recommends using programmable tests, otherwise, he recommends record/replay tests.  Toffee has the efficiencies of both record/replay tests and programmable tests, so it is the best solution for whatever your web testing needs may be.

For more information about Toffee, visit, or feel free to contact us with questions at

Leotta, D. Clerissi, F. Ricca and P. Tonella, “Capture-replay vs. programmable web testing: An empirical assessment during test case evolution,” 2013 20th Working Conference on Reverse Engineering (WCRE), Koblenz, 2013, pp. 272-281.
© 2013 IEEE
doi: 10.1109/WCRE.2013.6671302

Familiar Reports for Regulated Industries


Web application testing for regulated industries often involves auditors, who are not developers, to confirm the testing results.  We developed Toffee so that non-developers can read the automated test results and quickly understand them.

We have personal experience of the challenges of incorporating automated testing into the computer system validation process for regulated industries.  Before we built Toffee, we wrote our automated tests in Java using Selenium.  After writing a large suite of automated tests that validated one of our client’s applications, we turned over the results to their auditor.  The auditor came back to us and explained that our results did not help her because, understandably, she did not know what any of the Java meant.  We ended up sitting down with her and explaining each line of code.

It quickly became apparent that describing lines of code was not a tenable exercise for regulated industry web testing.  We think auditors should not have to become developers, so instead we made Toffee to allow auditors or any non-developer to read automated tests.

After Toffee runs a test, it generates a report that conforms to the traditional web testing style used in manual/paper tests.  The report has five columns for each test step which state the action taken, the expected result, the actual result, the verdict, and if applicable the screenshot evidence.

If the test uses screenshot evidence, a picture is taken for each step.  We discussed this in a prior post called  Automatic Screenshots: Evidence Gathering Made Easy.  Each screenshot is included in the report appendix with a reference number, so all the information is in one place.  This makes it easy for an auditor to quickly evaluate the test.

We realized that non-developers are regularly a part of web testing, so we made Toffee to be user friendly for everyone regardless of their programming knowledge.

 For more information about Toffee, visit, or feel free to contact us with questions at

The Fragility of Record/Replay Tests

The convenience of using record/replay web testing tools is often defeated by the brittleness of the tests they produce.  Mouna Hammoudi’s paper “Why Do Record/Replay Tests of Web Applications Break?” explores why these tests break so easily as the application under test changes.  Toffee overcomes many of these issues with record/replay testing.

Hammoudi used a record/replay tool called Selenium IDE, which allows a user to record themselves using the application under test and replay their actions at a later time to test functionality.  After creating a test for each of 300 releases across 5 web applications, Hammoudi found that roughly 73% of all breaks in her Record/Replay tests were due to locators.

Hammoudi explains that locator breaks occur when either an HTML element’s attributes or the HTML structure has changed.  In our experience, tests that use structure based locators such as XPath are especially vulnerable: as web applications are updated the structure changes regularly and even small changes can invalidate structural locators.

Toffee offers two solutions that make Toffee tests more robust and less expensive to maintain compared to record/replay tests.  First, Toffee makes it easy to write test steps using attribute based locators like id, name, label, and others.  These locators tend to be more stable because they are not affected by common structural changes. Instead, developers have to intentionally change the attribute value.

Second, Toffee provides users with the ability to create aliases.  Aliases are custom commands which are defined by the the tester and refer to other Toffee commands.  For example, the command “click button with id button-logout-user” can be aliased as “logout”.  Then the “logout” alias can be used throughout your tests, instead of the original command.  This minimizes the necessary repair after the button’s id changes because only the “logout” alias must be updated.

The drawback of record/replay tests is their brittleness.  Toffee tests are more robust and reusable without sacrificing the accessibility of record/replay tools.


For more information about Toffee, visit, or feel free to contact us with questions at

Manual Testing: Toffee is about Orchestration not only Automation

We created Toffee to not simply automate QA testing but to orchestrate both manual and automated tests.  Manual tests are an important part of QA testing because it is either impractical or impossible to automate everything.  See our prior post on the virtues of manual testing: Two Cheers for Manual Testing

Creating manual tests for Toffee is as straightforward as writing traditional manual tests.  Simply mark the test step as manual and then write the instructions for the tester to perform.  Toffee will know it is a manual step and will take care of the rest.

Manual test are typically carried out using stacks of paper or lengthy text documents with instructions and a space to fill in the result.  Testers are stymied by going back and forth between the application under test and the test documentation, squandering time and becoming susceptible to mistakes every time they lose their place in the instructions.

Toffee eliminates these problems. Testers no longer have to deal with shuffling paper or scrolling through online documents.  Toffee provides test step instructions, and a place to record the results for only the current step, in a “heads up” display alongside the application under test.  This reduces human errors and is less taxing on testers.

As part of testing orchestration, Toffee allows manual steps to be integrated directly into otherwise automated tests.  Toffee executes all of the automated steps until it comes to a manual step, then it will pause and request the manual action from the tester.  After the tester completes the manual step and records the result, the computer continues on with automated steps.  This way if the majority of a test can be automated, a whole separate test does not need to be made just for the manual steps.  In addition, the results from the manual step and automated steps are all in the same report generated by Toffee.

Manual tests are still a vital part of QA testing, and Toffee does not just accommodate manual testing, but it makes it less of a headache for testers and more accurate.

For more information about Toffee, visit, or feel free to contact us with questions at

Automatic Screenshots: Evidence Gathering Made Easy

When working with clients and performing tests ourselves, we found that documenting tests with screenshot evidence is one of the most time consuming and challenging aspects of testing especially for manual tests.  With this in mind, we designed Toffee to make evidence gathering easy with automatic screenshot capture for both manual and automated tests.

By using the simple command “enable automatic screenshot capture”, Toffee will begin to automatically take screenshots at every test step regardless if it is manual or automated.  Testers no longer have to remember to do it themselves or repeat steps if they forget.  The default timing for the screenshot is within milliseconds of the test step being completed, but the timing can be delayed as needed to accommodate slower applications.  The screenshot delay can be modified at any point within the test.  This allows flexibility to ensure the appropriate evidence is captured with every screenshot.

One of the main frustrations Toffee’s screenshot capture solves is keeping screenshots connected to the correct test step results.  For example, without Toffee a tester often has to paste screenshots for manual tests into some additional word processing tool and document it appropriately so others know which screenshot corresponds to which test step.  This method is tedious and prone to error.  Toffee stores the results and screenshots together and can easily generate a single report containing all of the results and evidence in a single document.  

Toffee makes gathering screenshot evidence easy by automatically capturing them after test steps and keeping them all in one place with the test results whether they are manual or automated.

For more information about Toffee, visit, or feel free to contact us with questions at

Announcing Toffee

Toffee: Test Orchestration for the Enterprise

Since my last post on functional testing, KSM has been hard at work transforming the ideas from that post into a new product and service offering. Toffee (“Test Orchestration for the Enterprise”) allows QA professionals to build and execute automated tests in an interactive, online test environment, without requiring programming expertise. For those test cases that automation cannot easily reach, Toffee lets you include manual test steps in your scripts alongside automated ones. Automated screenshots capture your entire desktop, providing evidence for steps whether executed within or outside of the browser. Test results for both automated and manual tests are presented in a familiar step/expected result/actual result format, along with screenshot evidence.

Toffee started as a command-line based solution, which allowed us to focus on the syntax and scope of the Toffee command set. We used this first incarnation to test solutions we developed in house. The largest test suite achieved 100% automation with over 28,000 test steps, and completely replaced the Selenium tests we had written in Java.

On the trade show floor of the Society of Quality Assurance Annual Meeting last week, KSM previewed the next generation of Toffee, called Toffee Composer. Composer provides the same level of functionality as the initial version, but in a user-friendly web interface. Build your scripts, execute them, and store your results online, either in our cloud-based environment or in your own data center.

For more information about Toffee, visit We would be happy to schedule an online demonstration for you; just send us an email at