• info@helpingtesters.com
  • helpingtesters

To Automate or not to Automate

August 23, 2016
Automate, automation, not to automate

Most companies are now faced with this dilemma with their code base – Should it be Automate or not? If so, how much to automate? Where does one start? There are no straight answers to these questions and no one-size-fits-all solution.

Wikipedia defines test automation as the use of special software to control the execution of tests and the comparison of actual outcomes with predicted outcomes.

Put in simple words, automation is simply repeating what a manual tester does, using a tool. The definition is simple enough, but it opens up a new Pandora’s Box of questions for the uninitiated.

What to Automate?

Selecting what to automate is crucial for guaranteeing a return on investment. The following points can be kept in mind while choosing the suite which needs to be automated.

  • Repeatable Tests: Tests which are repeatable are the ideal candidate for automation. If a manual tester is forced to execute the same test more than once, there is a risk of human error creeping into the tester’s reporting. Avoid this by using a tool to do the same sequence of steps. In controlled conditions of pre-defined assumptions, this is the best way to ensure quality.
  • Regression Tests: Regression cases are mostly run every release to ensure that the new code check-in has not broken the existing functionality. They are mostly repeatable in nature, and also keep growing as per the new functionality that got added in the previous release cycle. This is an ideal candidate for doing Automate.
  • Smoke Tests: An application is smoke tested after every release, i.e. a set of tests are run to ensure that the most critical functionalities are working fine. This is the basis on which a decision can be made if the module is stable enough for the QA to test exhaustively.A more detailed definition of smoke test and the difference between smoke and sanity can be found here. Being a subset of regression testing, smoke tests are another candidate for automation because of the sheer volume of times one might need to run these tests.
  • Cross browser Tests: Tests which need to run across multiple browsers or environments take a lot of time to test manually as the manual tester’s time multiples by as many browsers or operating systems which require testing. Automate these once on one browser and replaying them in multiple ones free up the manual tester’s time for more complex and exploratory testing.
  • Data-driven Tests: There are a lot of applications where the entire business functionality is a single flow, and they just need to be driven by multiple sets of data. Automate these kinds of tests is relatively simpler provided the flow is static.
  • Tests which create test data: In a lot of test environments, it has become necessary to spend the first couple of days just to create test data in the database to start testing. Moreover, this activity is repeated over releases every time a fresh database is brought in for testing. Automating these kinds of actions which pump in data into a DB can reduce the manual test effort required for the same and also speed up the process.
  • Tests which are most risky and more prone to bugs: Pareto principle says that 80 % of the bugs are caused by 20 % of the code. If this 20 % can be automated and run every release, it would give stakeholders the confidence that the riskiest part of the application is working as expected.Thus this 20% becomes a starting point of automation for a lot of companies.

For a successful chef, talent lies not just in what he puts on his plate, but also in what he chooses to leave out. Similarly, understanding tests which are not good candidates for automation are vital.

What Not to Automate

  • Complex tests: With applications getting more and more complex, it is no surprise that some tests get complex as well. Automate these complex cases with elaborate calculations and dependencies might end up in creating an automation script which is as convoluted as the product itself! Keeping automation simple is essential for guaranteeing ROI (return on investment). 
  • Tests with ambiguous expected results: There are some tests which are people-dependent, i.e. only the test planners know exactly what to expect when the application comes up. Such cases are not good candidates for automation.
  • Usability Tests: Applications strive to ensure that they reach out to their end user in the easiest way possible, and one way of checking this is through usability testing. It involves checking how easy it is to use the application. The only way to do this is by testing from the perspective of the user, which certainly cannot be mimicked by a machine. 
  • One time tests: We often come across a situation where it is known that certain functionality would change in a couple of releases. In such cases, it does not make sense to automate something that would stop given value in a very short period of time.

 When to automate?

Test Automate can be done at any stage in the life cycle. It can be done in the development cycle by the developers themselves (unit tests) or to regressively test a piece of software (regression test). When to automate is a question that goes hand in hand with what to automate.

Where to start?

Once a set of cases are identified for automation based on the above two sections, all that remains now is to start work. But where to start? A good point to start would be to pick those cases which are to be run every release to ensure module stability first (smoke cases), move on to the cases which are most likely to break, and then the rest of the regression cases.

Each company has its own vision, mission, and statement when it comes to automation, so whatever the start point is, it needs to fall in line with the broader picture. Automation is a powerful tool, but it can only harvest benefits if used with the right vision.

Before starting out automating anything, one needs to have a clear road map as to where this effort is expected to lead the organization. This vision needs to translate into a good automation strategy and then into short term plans. Sticking to this plan would ensure that the vision is met and test automation realizes its potential in drastically improving the quality of the delivery.


About the author

HelpingTestersTeam administrator

Blog is credited to HelpingTesters.com team

Leave a Reply

Your email address will not be published.