008-12_SDT013.qxp_Layout 1 6/22/18 2:17 PM Page 10
10
SD Times
July 2018
www.sdtimes.com
5 Intelligent Test Automation Tools AI and machine-assisted automated testing tools are relatively new. The only way to understand exactly what they do and how their capabilities can benefit your organization is to try them. Following are five of the early contenders: n AppliTools Eyes is an automated visual AI testing platform targeted at test automation engineers, DevOps and front-end developers who want to ensure their mobile, web and native apps look right, feel right and deliver the intended user experience. n AutonomIQ is an autonomous platform that automates the entire testing life cycle from test case creation to impact analysis. It accelerates the generation of test cases, data and scripts. It also self-corrects test assets automatically to avoid false positives and script issues. n Functionize is an autonomous cloud testing platform that accelerates test creation and executes thousands of tests in minutes. It also enables autonomous test maintenance.
< continued from page 9
between what’s real and what isn’t, particularly for the uninitiated. “Really do due diligence and a POC to make sure it’s going to reduce or eliminate that human interaction you have to have,” said Voke’s Lanowitz. “Realize this is new technology and new innovation, because we haven’t had a lot of innovation in the testing space. Tools have gotten less expensive, they produce fewer false positives, but we haven’t had a lot of innovation. This is innovation.” While it’s fun to be technologically curious, it’s also wise to consider how the organization could benefit from such a product or service and whether the organization is actually ready for it. Machine learning requires data which may not be readily available. Alternatively, if the data is available, it may not have been curated because no one knows how to make sense out of it. “Increasingly, people talk about shift right and the idea is essentially I have all this data about how end users are using my application, where errors are occurring and the load in the system. I can use AI to make it much more meaningful,” said Gartner’s Herschmann. “The whole notion of testing and QA broadened in scope from the ideation phase to the requirements phase all the way back to when things are live in production. I can use the data in a machine learning context to
n Mabl (above) is machine learning-driven test automation for web apps that simplifies the creation of automated tests. It also identifies regressions and automatically maintains tests. n Parasoft SOAtest API testing is not a new product. However, the latest release introduces AI to convert manual UI tests into automated, scriptless API tests. —Lisa Morgan
identify patterns, and then based on the patterns, I can make certain changes. Then rinse and repeat all the time.” It’s a mistake to underestimate the dynamic nature of machine learning, because it’s a continuous process as opposed to an event. Common goals are to teach the system something new and improve the accuracy of outcomes, both of which are based on data. For example, to understand what a test failure looks like, the system has to understand what a test pass looks like. Every time a test is run, new data is generated. Every time new code is generated, new data is generated. The reason some vendors are able to provide users with fast results is because the system is not just using the user’s data, it’s comparing what the user provided with massive amounts of relevant, aggregated data. “Three or four years ago, Google said that their code base then was like 100 million lines and it’s well past that now. Every day, that code base is growing linearly and so is their test code base, so that means that test execution is growing exponentially and at some point it’s no longer affordable,” said Gartner’s Murphy. “They built tools to determine which tests need to be fixed or thrown out, which tests are of no value anymore, what tests should be run based on what changes have been checked into [a] build. These things are what organizations have to look at and
now you’re seeing companies other than Google do this.”
What to expect along the way While autonomous testing and AI technologies aren’t new, the combination of them is in the early stages. More and different types of products will hit the market in the coming months and years. Meanwhile, there will be a lot of trial and error involved by end users and vendors. “If you look at the Gartner Hype Cycle, all of the technologies that are in some shape or form related to machine learning are all just climbing the slope. Basically that means they are still ahead of getting into the trough of disillusionment,” said Gartner’s Herschmann. “I think we will see people fail at using these kinds of technologies [because] there’s a lot of over-promise. We tell people you’ve got to have the right expectations about what you can do with this because yes, we’ve seen some very cool things like Google and Facebook and some of the other big guys, but keep in mind this is very, very narrowly focused. We’re decades away from anything that’s general purpose AI.” Voke recommends taking a long-term view of the technology and consider how it’s going to impact the mix of skills in the organizations and workflows. “Understand where skills can go and how you can use the skills to benefit the continued on page 12 >