Record/Replay vs Programmable Tests: Toffee the Best of Both

Record/replay and programmable web testing tools each have their own benefits and shortfalls. These trade offs are analysed in a paper by Maurizio Leotta titled “Capture-Replay vs. Programmable Web Testing: An Empirical Assessment during Test Case Evolution” (citation below).  Leotta concludes that while initial test creation is faster for record/replay tests, programmable web tests save time during regression testing as the application is updated.  

As part of Leotta’s study, testers with some programming experience made two sets of tests for six different web applications.  Each web application had one set of tests made using a record/replay tool called Selenium IDE and another set of programmed tests written in Java using Selenium WebDriver.  The testers then updated each test suite for a newer version of each web application.  Leotta measured the time the testers spent creating and updating the test suites, comparing the record/replay time with the programmable test time.

Leotta found that the programmable tests took up to 112% more time to make from scratch than the record/replay tests.  This was mainly because creating the record/replay tests did not require any programming knowledge and was as simple as the tester recording themselves using the web application.

Next, Leotta’s research showed that when test suites were updated for new versions the programming tests took up to 51% less time to repair than the record/repay tests.  This time saving was mostly due to the developers using the page object design pattern.  In summary, a page object is a programming construct where code that locates elements and initiates actions on the web application is stored in one place.  The page objects can be reused in multiple tests.  Testers only update page objects when repairing programmable tests, which can fix multiple tests at once. Record/replay tests do not reuse anything, so every record/replay test must be fixed individually or even recorded again every time the application under test changes.

As I discussed in a previous post, Toffee avoids the brittleness common to record/replay testing, replacing it with plain-language test commands and an interactive script development environment. Toffee also saves time during test repairs with aliases and test modularization, which provide the same benefits as  the page object pattern, but without requiring programming.  Toffee commands that are repeated throughout tests can be saved as an alias, so only the alias needs to be updated.  Also, a series of frequently used commands (think login) can be saved and maintained in a single “include” file and used on multiple tests.

If the application will be updated more than once, Leotta recommends using programmable tests, otherwise, he recommends record/replay tests.  Toffee has the efficiencies of both record/replay tests and programmable tests, so it is the best solution for whatever your web testing needs may be.

For more information about Toffee, visit, or feel free to contact us with questions at

Leotta, D. Clerissi, F. Ricca and P. Tonella, “Capture-replay vs. programmable web testing: An empirical assessment during test case evolution,” 2013 20th Working Conference on Reverse Engineering (WCRE), Koblenz, 2013, pp. 272-281.
© 2013 IEEE
doi: 10.1109/WCRE.2013.6671302

Categories: Automated Functional Testing, Quality Assurance, Toffee
Trackback URL for this post:

Leave a Reply

Your email address will not be published. Required fields are marked *