• info@helpingtesters.com
  • helpingtesters

Choosing an automation tool – The Nuances

August 13, 2016
Nuances, automation tool, automation Tool selection

“The choices we make today have important implications” – A quote by Adam White. This quote is most relevant to the field of Information Technology where sometimes the only difference between two product offerings is the choices the creators made in terms of the technology, framework, process, etc. Test automation is one field where the power of choice of automation tool is held in the hands of a select few who determine the fate of how automation is viewed in the organization as a whole.

Understanding the answers to the questions below is important in selecting that one magic automation tool which is the pill for all ills.  

Questions which determine How to choose an Automation Tool

  • How much does it cost?
  • Can the automation tool handle the size of the product(s)/software(s)?
  • Can the tool handle the complexity of technology that organizations have as their product offerings?
  • Would the organizations have to create a framework on the automation tool?
  • If so, how much would it cost to set up the framework? How long?
  • Can the tool support distribute execution?
  • Can the tool integrate seamlessly with some other tools that the organization might already have?
  • When can one start realizing ROI (return on investment) considering the above factors?

Even though there are multiple other questions which come to mind once a tool comes into the spotlight, above are the primary 8 questions which determine which tool would be chosen finally. Now, let us delve a little deeper into some of these points.

Cost Factor

TypesOfTools

  • Traditional COTS (Commercial off-the-shelf) tools always have an associated cost with them. Licensing typically comes in various flavors- Floating, node-locked, cloud-based, etc. depending on how the company pitches its cost-effective usage. However, it is not a one-size-fits-all situation.
  • Each company has to ask themselves the question -what is the best choice for their use? What is the scale of work they are looking for? All these determine which license or a combination of licenses companies purchase.
  • In addition to licensing cost, some automation tools have an additional cost associated with them. This is again time bound. Logic dictates that for the first couple of years, support becomes more or less a given task.
  • The cost to the company comes in multiple other avatars. If we are going for a tool which already has an inbuilt framework, the cost of setting up a framework is a moot point. However, companies still need to factor in the fact that they need to train people to learn the scripting language to build business-level functions or custom reporting (if required) and so forth (which still counts towards framework cost).
  • Last, but definitely not the least come to the open source tools in which everything needs to be set up from scratch. The obvious advantages are that we do not need to invest in this automation tool, and it can be customized according to what is needed in the company.
  • The cost of resources to set everything up from scratch is amazingly high, and moreover, it is not a one-time setup. Framework maintenance cost also needs to be factored in.

Technical Complexity

  • Companies typically need to automate their entire suite of products, which might include a mainframe component, a DB component, UI component (web-based and non-web based), a native app on mobile (Android, Apple) and much more.
  • One tool needs to suit all these needs seamlessly. Being sure of what one needs to automate is essential to understand which tool or a combination of tools to choose. Most tools can be knocked out this way and narrower the choice, easier the decision. This is the starting point (after cost) to start narrowing down from the gamut of automation tools to a select few.

Product Size

  • ‘My product is so huge that no automation tool can support it!’ cries out the decision-makers in big companies. Naturally, panic sets in when a huge product needs to get automated from scratch. But, a clean slate is always better than a messed up one.
  • The trick is to understand how clear the requirements are, to take on this mammoth task. Automation requires a very clear expected result, with no ambiguity in the starting or end points. Even though this might seem a given, the sad truth is requirements of huge products lie more with the resources working on it, rather than documented somewhere in a way easily understandable by those wanting to automate the product.
  • If the requirements are already in a Given-When-Then format, it might be easier to adopt tools like Cucumber which is more of a BDD tool. Even then, Cucumber and other Specification by Example tools are not meant for this purpose. When all you need to do is regression testing, a lot of thought needs to go into selecting the best automation tool.

Framework Creation

  • Most free tools do not have an inbuilt automation framework. The onus is on the organization to finance this framework building. A clear understanding of the type of products to be automated is paramount in defining the type of framework.
  • Each type of framework has its own pros and cons. Understanding what is best for the gamut of applications we need to automate is what will determine which framework to create.
  • A good framework created on an open source tool can even be packaged and sold as a proprietary product! Here comes an opportunity for revenue and extra mileage in reaching and far exceeding that ROI.

Distributed Execution

  • “Why should I automate? I take lesser time to execute my cases manually!” This shows a clear misunderstanding of what test automation means, and when it starts providing value.
  • Automation is not meant to replace manual testing anytime, and can only complement it. One needs to realize that when scripts are replayed, ROI is realized not the one time it is replayed but over a period of time where the script is replayed on a changeable code base and unearths defects.
  • To help the companies get maximum value from the replay, and to ensure continuous integration and quick release of testing area, distributed execution plays a very great part.
  • Distributed execution is a technique in which multiple threads of execution run parallel on multiple nodes. This ensures that the time was taken to finish executing the suite basically goes down by as many machines as you want to deploy the scripts. This is a vital component for choosing an automation tool.

Tool Integration

  • Organizations usually have a lot of already in-use tools for defect logging, test management, requirements management, etc. It might be beneficial if the automation tool was chosen can integrate with the existing tool(s), ensure complete visibility of how much business has been automated, how much is pass/fail, and in more mature cases, auto log defects on failure.

ROI

                                                 ReturnOnInvestment

  • The bottom line of all decisions is money. How long would it take for one to realize the money put into this venture? What is the breakeven point? Below is a simple grid which helps calculate this ROI.

Conclusion

No automation tool is bad and no tool is flawless. More than anything else, it is important to understand what the organizational need is to determine the best automation tool for the job. If this aspect is not understood clearly, even the best automation tools would fail to achieve its purpose. As quoted by a wise man once, ‘A tool is only as good as the person using is’, and my own twist to it ‘and the reasons which have driven its choice’.

About the author

HelpingTestersTeam administrator

Blog is credited to HelpingTesters.com team

Leave a Reply

Your email address will not be published.