Continuous Testing Buyer's Guide - SD Times July 2019

Continuous Testing A GUIDE FOR BUYERS

Buyers Guide

Testing all the time Continuous testing enables organizations to keep pace with code changes and new features

C

ontinuous testing marks the evolution of software testing to where it can keep up with the rapid pace of development, while adapting to new software architectures and development methodologies such as DevOps and Agile.

Relying on automated test execution as a cornerstone, continuous testing is the embodiment of the “shift left” craze sweeping the industry. Move everything up earlier in the lifecycle, so nothing is left behind. That means even at ideation, security people, business people and testers are considering their role in bringing that idea through to a final product.

But for those just tipping their toes

BY DAVID RUBINSTEIN into the DevOps waters, it’s often hard to figure out how to put the pieces together and where to begin.

“Where I normally start is talking about what does that mean to them? It has some different definitions,” explained Mark Lambert, vice president of products at testing software provider Parasoft. “Continuous testing is, in essence, you’re building testing into your pipeline so it’s running continuously as you’re doing your development, and it happens automatically. It goes beyond automated testing but this is where somebody starts. Somebody may start by building some automated tests and run-

ning them as part of the pipeline.”

Mobile development creates its own challenges for organizations looking to begin continuous testing of their applications. Steve Orlando, senior director of product marketing at Mobile Labs, said you first should identify issues or bottlenecks. “Are you slow to release? Maybe you’re not getting fast enough answers from QA,” he said. “Once you identify these challenges, it’s important to set some goals. When you know what you want to do, such as improve turnaround time from testing or find bugs faster, it’s then important to find the right tools to aid your process and to get the job done.

Look for tools in a build system, where you can continuously build and integrate

Full Page Ads_SDT025.qxp_Layout 1 6/21/19 5:09 PM Page 34

THIS IS

GigaFox. She’s begging for more tasty bugs to devour.

Why don’t you feed her?

She’s so hungry.

Mobile Labs GigaFox™ is the fastest mobile device testing platform available designed to accelerate mobile app development and continuous testing. Available on-premises on in a hosted environment, GigaFox comes with built-in Appium for increased performance and easier scripting.

Superfast speed, easy management of real devices and quality support makes Mobile Labs your partner for success.

Find us online at www.mobilelabsinc.com/products/gigafox

TE

TEST

T

TEST

Steve Orlando of Mobile Labs suggests four practices for effective continuous testing: n Set up a proper build system that stops on any error and provides a notification to all interested parties. This allows issues to be addressed immediately. n Integrate unit tests for the application into the build steps. n Upload applications to a mobile device testing platform so the application can be used by the testers. n Run functional automated tests against real devices by utilizing a mobile device testing platform. z

T Back in 2012, I wrote an Industry Watch column in SD Times ST titled, “Need for Speed? Take a look at DTO.” DevOps had become a thing by then, and I asked Tom Lounibos, then the CEO of SOASTA, what he thought of the term DevTestOps. “DevTestOps. DTO. I kind of like the speed element to it,” he said. “It sounds like a Camaro in the ‘70s.”

It was catchy, and some in the industry tried to make it happen, but it was flawed, much as the term DevSecOps is flawed. In that 2012 column, I had even written, “Move over, DevOps. Well, more specifically, Dev, you slide over to the left, and Ops, you move a bit to your right. We’ve got to make room for Test.”

The problem, of course, is that the term still implied steps in a process, and too closely resembled waterfall practices of the day. DevTestOps was a handy way to let development teams know that test should no longer wait until the end of the development process, when fixing problems is costlier and harder to deal with. It was the beginning of the thinking that development should not be a linear process but rather a collaborative effort to ensure code worked correctly and was secure at all times.

In a continuous testing world, you’re testing during code creation, you’re testing during integration, you’re testing in your deployment pipeline, and you’re testing changes after software is delivered.

That is why DevTestOps, as a term, has not stood the test of time. z —David Rubinstein

TE S

CT best practices for mobile apps

We have DevOps, and DevSecOps...

Why not DevTestOps? TES

T TES

your code and tests. When code is checked in, it can then be built and compiled for continuous testing.”

Brian Reed, chief mobility officer at NowSecure, describes three keys: First, catalog your applications and assign risk profiles, with the idea to test the highrisk apps at every build, every day.

Next, plug security testing into your build tools, ticketing systems and vulnerability management tools. Finally, he said, “security and DevOps teams should review existing security processes and testing/repair criteria to create a tiered model that defines P1/P2/P3 service levels of what to fix when in a continuous testing program.”

According to Eran Kinsbruner, mobile technical evangelist at mobile and web application test provider Perfecto, test automation is absolutely critical to continuous testing. Building tests to execute inside your continuous integration pipeline automatically ensures that each time code is checked in, it’s tested, at a place and time when it’s less expensive and easier to detect than later on in the software’s life.

“Automation today, without even exaggerating, is the most important pillar of continuous testing,” Kinsbruner said.

“Continuous testing, by its meaning, means you need to validate code changes as soon as possible to uncover risks and release faster. You cannot do that if you are tied into a lot of manual testing. If you want to deliver fast, you want to

release fast, you want to integrate new code changes, and the only thing you have is manual testing, it won’t be continuous and will actually slow you down.”

Kinsbruner stressed that automation is not a one-time thing for each software iteration. Instead, he said, you need to continually build on the automation, lest you fall behind because much of your process remains manual.

Automating as much as you can in testing also is important because it can give you greater code coverage than manual testing can offer. But Mobile Labs’ Orlando said how much you cover with tests is less important than what you’re actually testing.

“Honestly, 100% code coverage is a lofty goal that I’m sure all developers and testers wish was easily achievable,” he said. “But, in reality I’ve found that it is most important to focus on the most used functions within your app. Sometimes certain functions are not able to be tested via automation — that’s where manual testing comes into play.”

NowSecure’s Reed said that when considering code coverage for mobile app security testing, you must examine the mobile app attack surface itself.

That, he noted, “spans data at rest on the mobile device (DAR), data in motion between device and back ends (DIM) and code itself (including custom code and third-party included libraries). Traditional Static App Testing (SAST) looks at custom code before build and misses DIM, DAR and thirdparty libraries. Achieving 100 percent code coverage of the full mobile app requires dynamic testing (DAST) to actually install and run the app on a real mobile device in the real world.”

Reed added that NowSecure benchmarks have found that more than 60 percent of mobile apps in the Apple AppStore and Google Play Store leak data via DIM, DAR and/or third-party code.

The mere mention of test automation as the first step toward continuous testing begs the question: Are we talking about automating test creation or automating test execution? The answer, in fact, can be both.

“If you would have asked me this question five or six years ago, it was very clear: You need to write scripts, and you need to create them in a development language such as Java, VBScript, C#, Python and so forth,” Kinsbruner said.

Full Page Ads_SDT025.qxp_Layout 1 6/21/19 5:09 PM Page 37

“In today’s IT, there are two ways of creating test automation. The first one is the traditional one, where you use the IDE, like Eclipse, IntelliJ and so forth, to write test automation scripts based on user stories or even functional specs. You go by the definition of the thing that you want to automate and you write code in JavaScript or other languages. That’s the creation.”

The second way, he continued, involves artificial intelligence and machine learning. “With the rise of AI and machine learning tools, you can add to this creation activity a new method, which is record and playback,” Kinsbruner said. “You can point your tool into a website or mobile device, and based on supervised algorithms for machine learning, you can actually create test automation without really writing one line of code.”

Continuing on the path After test automations have been created and put into the pipeline, where do you go from there?

According to Parasoft’s Lambert, the next step is to take tests you’d normally created in the later stages of development and execute them continuously moving forward, during active and ongoing development. And that, he added, is often a conversation with the move from waterfall to Agile, “or staged development into an agile-ISH. “I would normally do load and performance testing at the end just before release,” he said, “and what I’m saying is, you can now do that because your cycles are compressed; you can now do that automatically as part of your CI process; and yes, you’re going to do a more complete end-to-end full system verification as things start coming together.”

On the personnel side, getting people with different backgrounds and different skill sets to work together is a big challenge to overcome. As Perfecto’s Kinsbruner said, “While you want to make everyone work together, not everyone is wearing the same T-shirt.

Some come with a background in Java, some in Python, or C#, and if your technology does not match the differ-

Model-based testing… Good or bad?

Model-based testing, in which you define what an application’s functionality should be and test against that, can be used to augment more traditional test automation.

“I define model-based testing where I’m more focused about defining a model or a business process that represents how my functionality should work,” said Mark Lambert, VP of products at testing software provider Parasoft. “Typically, modelbased testing works at the higher levels of the application architecture, so when I’m defining through a business use case, it runs into challenges when you start trying to apply it to code that is not developed with a model-centric view in mind.”

So, should you be doing model-based testing? Here’s Lambert again: “Embedded safety-critical systems many times have a lot of models already … so therefore creating the test based upon the models is actually very logical. If you’re doing model-based testing for enterprise IT-type of applications, you’re trying to reverse-engineer the models, which sometimes can cause inconsistencies. But it also gives you a view of how the business flow should work. So there are pros and cons to it. I don’t view it as a good [method] for validating or building a regression against a business requirement, but I view it as a good way of fleshing out many different scenarios within the business use cases.” z —David Rubinstein

ent squads involved in the different activities, development and testing, then you’re starting to have some hiccups in the overall process. you need to assess and maybe try to even match the right people into the right dev team and feature and product to be developed, so they find it easier to implement test automation. Otherwise, you’re just causing delays and so forth.”

Meanwhile, organizations have to be careful not to rush into plans or make promises that aren’t close to their reality, meaning: Are they using technologies such as artificial intelligence, or frameworks such as Cypress.io? “People are talking about those things, but whether they really match the activities the team needs to do is something to be determined. Just because it’s a popular thing, you don’t need to rush and choose these frameworks,” said Kinsbruner.

Parasoft’s Lambert agrees that frameworks are critical, but suggests sticking with open source.

“I’m an advocate of leveraging opensource frameworks where they exist.

For web testing, I’m an advocate of leveraging Selenium. For testing in the Java world, it’s mostly JUnit. Leveraging a framework and an open-source framework that has a significant amount of adoption is valuable because you can then take advantage of what other teams have done. In the JUnit

world, for example, there are stubbing and mocking frameworks ... so you don’t have to reinvent the wheel.

Frameworks are a fundamental instrument, as far as I’m concerned.”

Perfecto’s Kinsbruner suggests aligning people with the right skill set, the right product and the right testing framework, because the framework the organization chooses needs to serve multiple personas. “Continuous testing is an activity that comes to validate your code quality upon each code change. It can be done by developers, or testers, performance engineers or by many other people who are involved in the process,” he said.

What should you test continuously?

For testing mobile applications, the critical areas to test are those that provide its main functionality. Those, said Mobile Labs’ Orlando, are the ones you must continuously test. “What does your app do? What’s its purpose? If the app is intended to help travelers book a hotel room, then you need to continually test and make sure that the app is able to book a hotel room 100 percent of the time,” he said. “It’s important to nail the functionality of your app before you explore other areas such as network virtualization and other features that are present but not tied to the main funccontinued on last page >

Full Page Ads_SDT025.qxp_Layout 1 6/21/19 5:10 PM Page 38

How does your solution help organizations find success with continuous testing?

Eran Kinsbruner, mobile technical evangelist, Perfecto Perfecto has been playing in this space of software test automation and continuous testing for more than 12 years now, and we have seen so many different organizations of varied maturity levels struggling with continuous testing and just straight test automation. So what we’ve decided in the last year or so is to focus on specific tool suites to help practitioners with different skill sets starting from developers or software test engineers as well as manual business testers, giving them the right tools.

If they are very skilled and can write code in Java, JavaScript and the like, we will give them full support for Selenium and Appium to create and execute their test automation in the cloud. Perfecto has a cloud for mobile and desktop web virtual machines. It is a software-as-a-service platform, so basically, if you have the skill set to create in Selenium and Appium, you can execute them on the Perfecto private cloud for enterprises with full security and governance. You don’t need to worry about the test environment yourself. You don’t need to build a Selenium grid, or you don’t need to buy smartphones and tablets and continuously maintain them. We provide the access from the cloud to all of these digital platforms to run your test automation grid.

If you do not have the skill sets to create test automation in development languages, with Selenium and Appium, we just recently launched Perfecto Codeless. This is a record-and-playback solution for web and mobile testing in the cloud, based on machine learning algorithms that will reduce the headache for maintaining scripts if they change, or if an object has changed. That’s the other level of creation we added, so we can support any level of skill set that you have in your organization.

We also provide a wide range of execution capabilities through CI. We integrate with all the CI servers. We allow you to execute in the cloud on multiple platforms — desktop browsers and mobile native devices — and at the end of the execution,

we provide a machine-learning based reporting and analytics solution so you can slice and dice the data. You can get root cause analysis out of each test report at cloud scale.

deep reliability and security issues in the code base before you start testing it in the traditional sense.

Mark Lambert, vice president of product, Parasoft

Mobile Labs’ mobile app testing platform, GigaFox helps organizations get started and maintain their continuous testing initiatives in a couple of ways. First, if testers are using Appium for automation, then GigaFox actually comes with built-in Appium, making it even easier for testers to run Appium scripts with faster speed and better performance. In addition, GigaFox enables teams to run more concurrent Appium tests and even makes iOS provisioning fast and easy.

But, even if mobile app dev and QA teams are not using Appium, GigaFox works well with the majority of other commercial and open-source frameworks and tools available for automation. Through easy integration, speed, and management of real devices, GigaFox helps both dev and QA teams streamline and set up an effective continuous testing strategy that works.

Available on-premises or in a hosted environment, GigaFox is the most open and flexible cloud in the mobile space, supporting continuous testing through DevOps integrations and the largest number of third party and open-source tools in the industry.

We align Parasoft’s offerings to the testing pyramid. The testing pyramid is advocated by two agile thought leaders, Martin Fowler and Michael Cohn. They talk about organizing your portfolio of tests in a pyramid. Do a lot of unit tests, that’s your base of the pyramid, big and wide.

Then you’re going to try to cover as much as possible with those unit tests.

Then, you’re going to do API or servicelevel tests, testing business logic. That’s your middle layer, big, but not as big as unit tests. Then you’re going to minimize your end-to-end UI tests as kind of the small piece at the top, because they’re brittle and hard to maintain, and you have all the external dependency infrastructure. What Parasoft does is we provide technology at each of these layers, so Jtest for unit testing, SOAtest for API testing, and we have technology within SOAtest for web UI testing. Service virtualization helps me take those top two layers of the pyramid and start isolating the code from its external dependencies in a similar way that you would get unit testing at the bottom of the pyramid.

This pyramid gives you a really scalable way of maintaining your testing strategy, but it doesn’t actually address quality. It’s very good for CT, but CT is only part of the solution. CT doesn’t actually help you build quality into the application.

What it does is help you detect a problem, and detect it sooner, when it costs less to fix. It’s all about shifting left defect detection. How you build quality into the process is through use of preventative techniques, such as deep code analysis, and that’s where our language products like Jtest. dotTEST, C++test come into play. They give you the ability to uncover

Steve Orlando, senior director, product marketing, Mobile Labs

Brian Reed, chief mobility officer, NowSecure The NowSecure automated security testing platform helps organizations overcome the limitations and time constraints of traditional mobile app security testing with the speed, accuracy and integrations required for continuous security testing.

NowSecure helps organizations on the journey from Agile to DevOps as they scale to meet the volume and velocity of continued on last page >

Your Continuous Testing Checklist.

CLOUD-BASED LAB REAL DEVICES & DESKTOP BROWSERS CODELESS & CODE-BASED SCRIPTING SCALABLE EXECUTIONS FOR WEB & MOBILE SMART REPORTING & ANALYTICS Perfecto checks all your boxes.

Learn More

A guide to continuous testing tools n CA Technologies offers next-generation, integrated continuous testing solutions that automate the most difficult testing activities — from requirements engineering through test design automation, service virtualization and intelligent orchestration. Built on end-to-end integrations and open source, CA’s comprehensive solutions help organizations eliminate testing bottlenecks impacting their DevOps and continuous delivery practices to test at the speed of agile, and build better apps, faster. n HPE Software’s automated testing solutions simplify software testing within fastmoving agile teams and for Continuous Integration scenarios. Integrated with DevOps tools and ALM solutions, HPE automated testing solutions keep quality at the center of today’s modern applications and hybrid infrastructures. n IBM: Quality is essential and the combination of automated testing and service virtualization from IBM Rational Test Workbench allows teams to assess their software throughout their delivery lifecycle. IBM has a market leading solution for the continuous testing of end-to-end scenarios covering mobile, cloud, cognitive, mainframe and more. n Micro Focus is a leading global enterprise software company with a world-class testing portfolio that helps customers accelerate their application delivery and ensure quality and security at every stage of the application lifecycle — from the first backlog item to the user experience in production. Simplifying functional, mobile, performance and application security within fast-moving Agile teams and for DevOps, Micro Focus testing solutions keep quality at the center of today’s modern applications and hybrid infrastructures with an integrated end-to-end application lifecycle management solution that is built for any methodology, technology and delivery model. n Microsoft provides a specialized tool set for testers that delivers an integrated experience starting from agile planning to test and release management, on premises or in the cloud. n Orasi is a leading provider of software

n

FEATURED PROVIDERS n

n Mobile Labs: Mobile Labs remains the leading supplier of in-house mobile device clouds that connect remote, shared devices to Global 2000 mobile web, gaming, and app engineering teams. Its patented GigaFox is offered on-premises or hosted, and solves mobile device sharing and management challenges during development, debugging, manual testing, and automated testing. A pre-installed and pre-configured Appium server provides "instant on" Appium test automation. n NowSecure: NowSecure is the mobile app security software company trusted by the world’s most demanding organizations. Only the NowSecure Platform delivers fully automated mobile app security and privacy testing with the speed, accuracy, and efficiency necessary for Agile and DevSecOps environments. Through the industry’s most advanced static, dynamic, behavioral and interactive mobile app security testing on real Android and iOS devices, NowSecure identifies the broadest array of security threats, compliance gaps and privacy issues in custom-developed, commercial, and business-critical mobile apps. NowSecure customers can choose automated software on-premises or in the cloud, expert professional penetration testing and managed services, or a combination of all as needed. NowSecure offers the fastest path to deeper mobile app security and privacy testing and certification. n Parasoft: Parasoft’s software testing tool suite automates time-consuming testing tasks for developers and testers, and helps managers and team leaders pinpoint priorities. With solutions that are easy to use, adopt, and scale, Parasoft’s software testing tools fit right into your existing toolchain and shrink testing time with nextlevel efficiency, augmented with AI. Parasoft users are able to succeed in today’s most strategic development initiatives, to capture new growth opportunities and meet the growing expectations of consumer demands n Perfecto: Perfecto offers a cloud-based continuous testing platform that takes mobile and web testing to the next level. It features a: continuous quality lab with smart self-healing capabilities; test authoring, management, validations and debugging of even advanced and hard-to-test businesses scenarios; text execution simulations; and smart analysis. For mobile testing, users can test against more than 3,000 real devices, and web developers can boost their test portfolio with crossbrowser testing in the cloud. testing services, utilizing test management, test automation, enterprise testing, Continuous Delivery, monitoring, and mobile testing technology. n Progress: Telerik Test Studio is a testautomation solution that helps teams be more efficient in functional, performance and load testing, improving test coverage and reducing the number of bugs that slip into production. n QASymphony’s qTest is a Test Case Management solution that integrates with popular development tools. QASymphony offers qTest eXplorer for teams doing exploratory testing. n Rogue Wave is the largest independent

provider of cross-platform software development tools and embedded components in the world. Rogue Wave Software’s Klocwork boosts software security and creates more reliable software. With Klocwork, analyze static code on-the-fly, simplify peer code reviews, and extend the life of complex software. Thousands of customers, including the biggest brands in the automotive, mobile device, consumer electronics, medical technologies, telecom, military and aerospace sectors, make Klocwork part of their software development process. n Sauce Labs provides the world’s largest cloud-based platform for automated testing of web and mobile applications. Optimized for use in CI and CD environments, and built with an emphasis on security, reliability and

scalability, users can run tests written in any language or framework using Selenium or Appium, both widely adopted open-source standards for automating browser and mobile application functionality. n SmartBear provides a range of frictionless tools to help testers and developers deliver robust test automation strategies.

With powerful test planning, test creation, test data management, test execution, and test environment solutions, SmartBear is paving the way for teams to deliver automated quality at both the UI and API layer.

SmartBear automation tools ensure functional, performance, and security correctness within your deployment process, integrating with tools like Jenkins, TeamCity, and more. n SOASTA’s Digital Performance Management (DPM) Platform enables measurement, testing and improvement of digital performance. It includes five technologies: mPulse real user monitoring (RUM); the CloudTest platform for continuous load testing; TouchTest mobile functional test automation; Digital Operation Center (DOC) for a unified view of contextual intelligence accessible from any device; and Data Science Workbench, simplifying analysis of current and historical web and mobile user performance data. n Synopsys: Through its Software Integrity platform, Synopsys provides a comprehensive suite of testing solutions for rapidly finding and fixing critical security vulnerabilities, quality defects, and compliance issues throughout the SDLC. n TechExcel: DevTest is a sophisticated quality-management solution used by development and QA teams of all sizes to manage every aspect of their testing processes. n Testplant: Eggplant’s Digital Automation Intelligence Suite empowers teams to continuously create amazing, user-centric digital experiences by testing the true UX, not the code. n Tricentis is recognized by both Forrester and Gartner as a leader in software test automation, functional testing, and continuous testing. Our integrated software testing solution, Tricentis Tosca, provides a unique Model-based Test Automation and Test Case Design approach to functional test automation—encompassing risk-based testing, test data management and provisioning, service virtualization, API testing and more. z

< from page 4

tionality. Until you’ve pinned down your main functionality, these other areas do not require continuous testing.”

According to NowSecure’s Reed, “Continuous Testing starts with the risk profile of the mobile app portfolio itself.” Most organizations have high, medium-, low-risk mobile apps matrixed by frequency of releases such as: • Tier 1 high risk apps typically are the primary customer-facing business apps on iOS and Android that have significant customer personal information, perform critical transactions and contain significant intellectual IP where continuous testing with automated security testing of every build (possibly every day) should be the default operating mode. In addition, an organization might add regular pen testing certification for these high-risk apps.

• Tier 2 medium-risk apps may have substantial company information, HR and IP, perhaps used by employees only, that should be continuously tested based on what is typically a less frequent release cycle where builds might be weekly or biweekly.

• Tier 3 low-risk apps such as conference room booking, cafeteria menus or parking, may not require continuous testing. In these cases, periodic but non-continuous testing strategy is most cost and risk effective.

< from page 5 their business.

NowSecure tests mobile app binaries on real iOS and Android devices using a comprehensive approach of SAST, DAST and behavioral testing and prioritize findings by industry-standard CVSS scores. Automated dynamic and behavioral analysis eliminates the human error and false positives that are typical of manual assessments and source code analysis. The NowSecure solution enables organizations to speed development and delivery by returning accurate testing results in minutes rather than weeks and providing remediation instructions to developers directly into the tools they use.

The NowSecure solution plugs directly into the SDLC, meaning there are no tools for developers to learn. They can config-

Parasoft’s Lambert suggests following the testing pyramid, created by Martin Fowler.

“You should have a significant number of unit tests, typically developercentric, that typically don’t require service virtualization because they’re isolated by definition,” he explained.

“Then as you start moving up the testing pyramid and up the technology stack, API testing ... and then you want to minimize your level of end-to-end UI tests. It’s not to say to eliminate them; you want to make them as efficient as possible, so you’ve got to worry about maintainability of those tests. And the higher up the pyramid you go the more complex the use cases are as well.”

NowSecure’s Reed said to start small on one mobile application and in one DevOps pipeline, and then incrementally grow the continuous testing capabilities. Next, he said, “Partner across security, dev and DevOps to define SLA and process agreements to optimize for flow vs. perfect security, with incremental improvements over time.” Follow that with a focus on automation and integration into your existing toolchain, and finally, look for automated testing tools that “provide complete SAST+DAST test runs in 15 minutes or less to ensure full security coverage while meeting the DevOps speed requirements,” he added. z

ure the NowSecure platform once and run security tests on daily builds in parallel with functional tests. Plug-ins and APIs make it easy to integrate with a myriad of SLDC tools including Archer, Brinqa, CloudBees, CircleCI, Code Dx, Jenkins, Jira, Microsoft Azure DevOps and more.

NowSecure provides interactive dashboards to analyze all app security testing results and trend lines across vulnerabilities, privacy and compliance trends. In addition, NowSecure maps findings to numerous compliance regimes including OWASP, NIAP, FFIEC, PCI DSS, HIPAA, GDPR, CWE, and more. Organizations can also feed testing results into vulnerability and compliance management systems to enable security and risk teams to analyze their overall portfolio. z