SD Times Deeper Look at Continuous Testing

Full Page Ads_SDT022.qxp_Layout 1 3/25/19 11:46 AM Page 22

Why Continuous Testing Is So Important

BY LISA MORGAN

rganizations continue to modernize their software development and delivery practices to minimize the impact of business disruption and stay competitive.

Many of them have adopted continuous integration (CI) and continuous delivery (CD), but continuous testing (CT) tends to be missing. When CT is absent, software delivery speed and code quality suffer. In short, CT is the missing link required to achieve an end-to-end continuous process.

O

“Most companies say they are Agile or want to be Agile, so they’re doing development at the speed of whatever Agile practice they’re using, but QA [gets] in the way,” Manish Mathuria, CTO and founder of digital strategy and services company Infostretch.

“If you want to test within the sprint’s boundary, certifying it and managing the business around it, continuous testing is the only way to go.”

Part of the problem has to do with the processrelated designations the software industry has chosen, according to Nancy Kastl, executive director of testing at digital transformation agency SPR. “DevOps should have been called DevTestOps

THE FIRST OF THREE PARTS [and] CI/CD should have been called CI/CT/CD,” said Kastl. “In order to achieve that accelerated theme, you need to do continuous testing in parallel with coding. Otherwise, you’re going to be deploying poor quality software faster.”

Although many types of testing have shifted left, CT has not yet become an integral part of an end-toend, continuous process. When CT is added to CI and CD, companies see improvements in speed and quality.

Automation is key Test automation is necessary for CT; however, the two are not synonymous. In fact, organizations should update their automated testing strategy to accommodate the end-to-end nature of continuous processes.

“Automated testing is a part of the CI/CD pipeline,” said Vishnu Nallani Chekravarthula, VP and head of innovation at software development and quality assurance consultancy Qentelli. “All the quality check gates should be automated to ensure there is no manual intervention for code promotion between [the] different stages.”

That’s not to say that manual testing is dead. The critical question is whether manual testing or automated testing is more efficient based on the situation.

“You have to do a cost/benefit analysis [because] the automation has to pay for itself versus ongoing manual execution of the script,” said Neil PriceJones, president of software testing and QA consultancy NVP Testing. “The best thing I heard was

someone who said he could automate a test in twice the time it would take you to write it.”

CT is not about automating everything all the time because that’s impractical.

“Automated testing is often misunderstood as CT, but the key difference is that automated testing is not in-sprint most times [and] CT is always done in-sprint,” Qentelli’s Chekravarthula said.

Deciding what to automate and what not to automate depends on what’s critical and what’s not. Mush Honda, vice president of testing at KMS Technology, considers the business priority issues first.

“From a strategy perspective, you have to consider the ROI,” said Honda. “[During a] project I worked on, we realized that one area of the system was used only two percent of the total time so a business decision was made that only three cases in that two percent can be the descriptors, so we automated those.”

In addition to doing a cost/benefit analysis, it’s also important to consider the risks, such as the business impact if a given functionality or capability were to fail.

“The whole philosophy between CI and CD is you want to have the code coming from multiple developers that will get integrated. You want to make sure that code works itself as well as its interdependencies,” said SPR’s Kastl.

Tests that should be included CT involves shifting many types of tests left, including integration, API, performance and security, in addition to unit tests. All of those tests also apply to microservices.

“For modern technology architectures such as microservices, automated API testing and contracts testing are important to ensure that services are able to communicate with each other in a much quicker way than compared to integration tests,” said Chekravarthula. “At various stages in the CI/CD pipeline, different types of tests are executed to ensure the code passes the quality check gates.”

Importantly, CT occurs throughout the entire SDLC as orchestrated by a test strategy and tools that facilitate the progression of code.

“The tests executed should ensure that the features that are part of a particular commit/release are covered,” said Chekravarthula. “Apart from the insprint tests, regression tests and non-functional tests should be executed to ensure that the application that goes into production does not cause failures. At each stage of the CI/CD pipeline, the quality check gates should also reach a specified pass percentage threshold for the build to qualify for promotion to the next stage.”

Since every nuance of a microservices applica-

Full Page Ads_SDT022.qxp_Layout 1 3/25/19 1:10 PM Page 27

A brief history of web and mobile app testing.

BEFORE SAUCE LABS Devices. Delays. Despair.

AFTER SAUCE LABS Automated. Accelerated. Awesome.

Find out how Sauce Labs can accelerate continuous testing to the speed of awesome.

For a demo, please visit saucelabs.com/demo Email sales@saucelabs.com or call (855) 677-0011 to learn more.

The Cloud’s #1 Continuous Testing Platform Accelerate digital transformation across the enterprise with a comprehensive suite of software testing tools – from agile test management to automated continuous testing for enterprise architectures.

tion is decomposed into a microservice, each can be tested at the unit level and certified there and then in integration tests created above them.

“Functional, integration, end-to-end, security and performance tests [are] incorporated in your continuous testing plan,” said Infostretch’s Mathuria. “In addition, if your microservices-based architecture is to be deployed in a public or private cloud, that brings different or unique nuances of testing for identity management, testing for security. Your static testing could be different, and your deployment testing is definitely different so those are the new aspects that microservices-based architecture has to reflect.”

KMS Technology’s Honda breaks testing into two categories — pre-production and production.

In pre-production, he emphasizes contracts in testing, which includes looking at all the API documentation for the various APIs, veri-

His test automation emphasizes business-critical workflows including what would be chaotic or catastrophic if those functions didn’t work well in the application (which ties in with performance and load testing).

“A lot of times what happens is performance and load are kept almost in their own silo, even with microservices and things of that nature. I would encourage the inclusion of load and performance testing to be done just like you do functional validation,” said Honda.

On the production side, he stressed the need for testers to have access to data such as for monitoring and profiling.

“With the shift in the role that testing plays, testers and the team in general should get access to that information [because it] allows us to align with the true business cases that are being applied to the application

‘I call out pre-prod and prod buckets specifically instead of calling out dev, QA, staging, UAT, all of those because of the nature of continuous delivery and deployment.’

— Mush Honda, KMS Technology

fying that the APIs perform as advertised, and checking to see if there is any linguistic contract behavior within the documentation or otherwise that the microservices leverage. He also considers the different active end points of all the APIs, including their security aspects and how they perform.

Then, he considers how to get the testing processes and the automated test scripts in the build and deployment processes facilitated by the tool he’s using. Other pre-production tests include how an API behaves when the data has mutated and how APIs behave from a performance perspective alone and when interacting with other APIs. He also does considerable component integration testing.

“I call out pre-prod and prod buckets specifically instead of calling out dev, QA, staging, UAT, all of those because of the nature of continuous delivery and deployment,” said Honda. “It’s so much simpler to keep them at these two high levels.”

when its live,” said Honda. “How or what areas of the system are being hit based on a particular event in the particular domain or socially? [That] also gives us the ability to get a good understanding of the user profile and what a typically useful style of architecture is. Using all of that data is really good for test scenarios and [for setting] our test strategy.”

He also recommends exploratory testing in production such as A/B testing and canary deployments to ensure the code is stable.

“The key thing that makes all of this testing possible is definitely test data,” said Honda. “When you test with APIs, the three key sorts of test data that I keep in mind are how do you successfully leverage stubs when there’s almost canned responses coming back with more logic necessarily built into those? How do you leverage fakes whereby you are simulating APIs and leveraging anything that’s exposed by the owner of any dependent services? Also, creating mocks and virtualization where we need to make sure that any of the mocks are invoked in a certain manner [which allows] you to focus on component interaction between services or microservices.”

In addition to API testing, some testing can be done via APIs, which SPR’s Kastl leverages.

“If I test a service that is there to add a new customer into a database, I can test that the new customer can be added to the service without having a UI screen to collect all the data and add that customer,” said Kastl. “The API services-level testing winds up being much more efficient and with automation, it’s a lot more stable [because] the API services are not going to change as quickly as your UI is changing.”

Jim Scheibmeir, associate principal analyst at Gartner, underscored the need for authentication and entitlements. “[Those are] really important when we talk about services: who can get to the data,

SLA and some agreed-upon independent factor that says our test cases are not necessarily depending on one another. It increases the concept of good scalability as your system matures,” said Honda. “One of the routine mistakes I’ve seen a lot of teams make is saying I’m going to execute six test cases, all of which are interdependent. [For example,] I can’t execute script number three unless tests one and two have been executed. [Test independence] becomes critical as your system matures and you have to execute 500 or 50,000 test cases in the same amount of time.”

As his teams writes tests, they include metadata such as the importance of the test, its priority, the associated user story, the associated release, and what specific modules, component, or code file the test is executed against. That way, using the metadata, it is possible to choose tests based on What goes into the right mix of CT tests are goals. those that are going to test that functionality you’re “The type of test and the test’s purpose both have to be incorporated in your deciabout ready to deploy, as well as critical business sion-making about what tests to run,” said functionality that could be impacted by your change Honda. “You don’t write your tests thinkplus some performance and security tests.’ ing, ‘I want to test this,’ you write your test and incorporate a lot of metadata in — Nancy Kastl, SPR it, then you can use the tests for a specific how much they can get to, is goal like putting quality into your CD process or there a threshold to usage or doing a sanity test so you can promote it to the availability,” said Scheibmeir. “Perfornext stage. Then you can mix and match the metamance is also key, and because this is a complex data to arrive at the right suite of tests at runtime environment, I want to test integrations, even without hard-coding that suite.” degrading them, to understand how my composite Performance and security testing should be an application works when dependencies are down, or integral part of CT given user experience expectahow fast my self-healing infrastructure really is.” tions and the growing need for code-related risk management.

Ensuring the right combination of tests “What goes into the right mix of CT tests are When deciding which tests to run, KMS Technol- going to be those tests that are going to test that ogy’s Honda considers three things: business prior- functionality you’re about ready to deploy, as well ity, the data, and test execution time. as critical business functionality that could be Business priority triages test cases based on the impacted by your change plus some performance business objective. Honda recommends getting and security tests,” said SPR’s Kastl. “Taking a riskbusiness buy-in and developer buy-in when based approach is what we do normally in testing explaining which tests will be included as part of and it’s even more important when it comes to the automation effort, since the business may have continuous testing.” thought of something the testers didn’t consider.

Past application issues are also an indicator of Second, the data collected by monitoring tools which tests should be prioritized. While test coverand other capabilities of the operations team pro- age is always important, it’s not just about test covvide insight into application usage, the target, pro- erage percentages per se. Acceptable test coverage duction defects, how the production defects are depends on risk, value, cost effectiveness and the being being classified, how critical they are from a impact on a problem-prone area. business severity perspective, and the typical Gartner’s Scheibmeir said another way to workflows that are being used. He also pays atten- ensure the right mix of tests are being used is tion to execution speed since dependencies can benchmark against oneself over time, measuring such things as lead time, Net Promoter Score, or result in unwanted delays.

“There should be some agreed-upon execution the business value of software and releases. z

CLOUD-BASE TESTING LAB With real devices and browsers to test, we’re always on. Always updated. Entirely secure. Get the stability and enterprise-grade scalability you need.

SIMPLIFIED TEST CREATION Whether you need help with test authoring, maintenance, management, validations, or debugging, Perfecto has a solution for you.

STREAMLINED TEST EXECUTION Orchestrate large suites across platforms for high velocity and parallel text execution with unlimited elastic scaling.

SMART ANALYSIS Get an instant overview in a powerful management dashboard with Perfecto’s visual analytics. With fast feedback, you’ll get all the insights you need.

webinar ad.qxp_WirelessDC Ad.qxd 3/25/19 12:41 PM Page 1

Be a more effective manager

Visit the sdtimes.com Learning Center Watch a free webinar and get the information you need to make decisions about software development tools.

Learn about your industry at www.sdtimes.com/sections/webinars

Century-Old Company Prioritizes CT ompanies in just about every industry are being disrupted by digital natives. To stay competitive, old companies must adapt, which can be a painful transformation. Moving to Agile is tough enough. Now, more businesses have adopted DevOps, continuous integration (CI) and continuous deployment (CD), albeit at different speeds.

Software testing at Lincoln Financial Group is progressive by design because software delivery speed and quality differentiate the company, so much so that Michelle DeCarlo, SVP of IT Enterprise Delivery, leads the company’s technical transformational initiatives.

“DevOps, cloud, continuous testing all feed into our larger journey which is the digital transformation effort,” said DeCarlo. “At the core, it’s changing the way we work. So, when we talk about continuous testing, it’s how do we get faster, leaner, and eliminate friction.”

Like many mature companies, Lincoln Financial had waterfall-type practices that involved a series of hand-offs. In addition, a lot of the testing was manual. The company is now using continuous processes, including CT to get to market faster. In fact, DeCarlo said that continuous testing has improved Lincoln Financial’s delivery speed by 30%.

“Continuous testing is one of those levers that we view as being critically important and fundamental to our strategy,” she said. “Testing used to be viewed as a cost because we were always waiting. Now it’s what differentiates us and helps us get to market faster.”

Of course, getting there wasn’t all about the CT process. In addition to changing the way people work, talent had to be upskilled and new tools were required.

“There are a few things we do differently with CT compared to how it used to work. One of those things is recognizing that the advanced engineering practices that are important to learn, like test-driven development and service virtualization, are table stakes now,” said DeCarlo. “These skills allow us to accelerate delivery cycles and the test engineers are embedded in the delivery cycle.”

Lincoln Financial automates many types of tests, including performance, security, regression, feature and smoke testing.

“When you drop code, you need to determine in seconds whether that code is valuable. If it fails, it stops your production cycle,” said DeCarlo. “You have to think about it as one continuous flow

C

because if you have a weak link, it halts your production and you lose hours. You cannot advance your practice unless you have automation.”

The company is also taking advantage of microservices and containers, which enable a flexible and scalable application architecture.

“The ability to decompose your code base and test at the most granular level is how firms get an advantage,” said DeCarlo. “When you test monolithic systems in totality it takes a long time, so we now decompose code and features, and test at the smallest level possible. Minimum viable product is part of our DNA now.”

API testing is critical “The focus on service testing is huge,” said DeCarlo. “In the past, people were worried about testing the front-end, which is still helpful and necessary, but the majority of your effort should be focused on the service level because that’s where you need to make sure everything works.”

Data drives insights Lincoln Financial also makes a point of Michelle DiCarlo using encrypted test data that includes the scenarios it wants to test so it can very quickly test multiple parameters and scenarios against whatever use cases apply.

“It’s critical to ensure that testers have all the data needed as opposed to waiting on other groups,” said DeCarlo. “We have figured out how to get the data that might be upstream, downstream, in the middle, or in our service architecture. Our strategy is to make data [available] on demand, which will allow us to respond at the same pace as other firms.”

Continuous testing means continuous learning Lincoln Financial has a mantra when it comes to software: build, measure, learn. The philosophy maps to developing the software, getting the data ready, testing it, going through the continuous delivery process, and measuring it.

“You need to have performance dashboards that are available to all your stakeholders, whether it’s a developer, tester, or product owner, and then you have to make modifications and improvements because what worked yesterday might work today but likely will not work tomorrow,” said DeCarlo.

“You have to embed this continuous improvement mindset and not look at it as failures, but learning.” z