SDTimes Continuous Testing Deeper Look

Part 2: Testing at Every Step

Full Page Ads_SDT023.qxp_Layout 1 4/18/19 11:31 AM Page 22

Continuous Testing at every step C

ontinuous integration (CI), continuous testing (CT) and continuous delivery (CD) should go hand-in-hand, but CT is still missing from the CI/CD workflow in most organizations. As a result, software teams eventually reach an impasse when they attempt to accelerate release cycles further with CI/CD. What they need to get to from both a mindset and process standpoint is a continuous, end-to-end workflow that includes CT. While CT requires test automation in order to meet time-to-market mandates, the two are not synonymous. A common misconception is that CT means automating every test, which isn’t necessarily practical or prudent.

Instead, the decision to automate tests should be viewed from a number of

BY LISA MORGAN perspectives including time and cost savings.

How to set up continuous testing Like CI, CD, DevOps and Agile, the purpose of CT is to accelerate the release of quality software. To enable a continuous end-to-end workflow, one should understand how CT fits into the CI/CD pipeline and how it can be used to drive higher levels of efficiency and effectiveness.

“The key thing is prioritizing,” said Mush Honda, VP of testing at software development, testing services and consulting company KMS Technology. “If you are in a state where you don’t have a live system, it’s easier to go into a mindset of automation first. I still

believe not everything can be automated in most cases. For those things that you are trying to migrate off of manual testing and add a component of automated testing with a system that’s already live or near going live, I would attack it with business priorities in mind.”

Automated testing should occur often enough to avoid system disruption and ensure that business-critical functionality is not adversely impacted. To prioritize test automation, consider the business severity of defects, manual tests that take a lot of time to set up, and whether the tests that have already been automated still make sense. Also, make a point of understanding what the definition of CT is in your organization so you can set goals accordingly.

“You need to understand what you’re going to achieve [by] doing CT in measurable terms and how that translates to your application or software project,” said Manish Mathuria, CTO and co-founder of digital strategy and services company Infostretch.

“Beyond that, then it depends on having the right strategy for automation.

Automation is key, so [you need] good buy-in on what layers you’re going to automate, the quality gates you’re going to put on each of these types of tests for static analysis, what you are going to stop at for unit tests, what kind of pass rates you’re going to achieve. It goes upstream from there.”

Each type of automated test should be well-planned. The rest is engineering, and the hard part may be getting everyone to buy into the continuous

code meets performance criteria • Security testing to identify vulnerabilities • Logging and monitoring to pinpoint errors occurring in production Implementing CT may require adjusting internal testing processes to

THE SECOND OF THREE PARTS

achieve the stated goals. For example, Lincoln Financial developers used to follow a waterfall methodology in which developers met with a business user or analyst to understand requirements.

Then, the developer would write code

“T est engineers need to know equally what is being changed by

software engineers and then make the changes at the same time your software engineers are changing the application.”

—Nancy Kastl

testing process.

“Continuous testing is designed to mature within your CI/CD process,” said Nancy Kastl, executive director of testing at digital transformation agency SPR. “Without having testing as part of the build, integrate, deploy [process], all you’re doing is deploying potentially bad code quicker.”

The CT process spans from development to deployment including: • Unit tests that ensure a piece of functionality works the way it is intended to work • Integration tests that verify the pieces of code that collectively enable a piece of functionality are working as intended together • Regression testing to ensure the new code doesn’t break what exists • API testing to ensure that APIs meet expectations • End-to-end tests that verify workflow • Performance tests that ensure the

and send it off for testing. The company now does Test-Driven Development (TDD), which juxtaposes testing and development. Test scripts are written and automated based on a user story before code is written. In addition, acceptance testers have been placed in development.

“When the code passes the test, you know you’ve achieved the outcome of your user story,” said Michelle DeCarlo, senior VP of technology engineering and enterprise delivery practices at Lincoln Financial.

Managing change When code changes, the associated tests may fail. According to SPR’s Kastl, that outcome should not happen in a CT process since developers and testers should be working together from day one.

“Communication and collaboration are really key when it comes to manag-

ing changes,” said Kastl. “As part of Agile methods, your team includes software engineers and test engineers, and the test engineers need to know equally what is being changed by the software engineers and then make the changes at the same time your software engineers are changing the application.”

To improve testing efficiency, Lincoln Financial uses tools to isolate software changes and has quality checks built into its process. The quality checks are performed with different types of resources to lessen the likelihood that a change may go unnoticed.

“We try to isolate when an asset changes [so we can] make sure that we’re testing for those changes. Quite frankly, nothing is foolproof,” said Lincoln Financial’s DeCarlo. “After we’ve released to production, we also do sampling and examine the code as it works in production.”

While it’s probably safe to say no organization has achieved a zero-defect rate in production, Lincoln Financial tries to minimize issues by performing different types of scans, including listening to customer feedback via various channels, including social media, so that feedback can be integrated into the delivery stream.

Generally speaking, it’s important to understand what these software changes impact so the relevant tests can be adjusted accordingly. If a traditional automation script fails, the defect may be traceable back to the build process.

If that’s the case, one can determine what has changed and what specific code caused the failure. Nevertheless, it’s also important to have confidence in the test scripts themselves.

“If you don’t have high confidence in the scripts that are traditionally run, that sort of spirals into the question of what you should do next,” said KMS Technology’s Honda. “You don’t know whether it was a problem with the way the script was written or the data it was using, or if it was genuinely a point of failure. Being able to have high confidence in the script I created is what becomes a key component of how I know something did go wrong with the system.”

Full Page Ads_SDT023.qxp_Layout 1 4/18/19 11:32 AM Page 25

A brief history of web and mobile app testing.

BEFORE SAUCE LABS Devices. Delays. Despair.

AFTER SAUCE LABS Automated. Accelerated. Awesome.

Find out how Sauce Labs can accelerate continuous testing to the speed of awesome.

For a demo, please visit saucelabs.com/demo Email sales@saucelabs.com or call (855) 677-0011 to learn more.

Issue tracking tools like Jira help because they provide traceability from the user story on. Without that, it’s harder to pinpoint exactly what went wrong.

Some tools now use AI to enable model-driven testing. Specifically, the AI instance analyzes application code and then automatically generates automated tests. In addition, such tools use other data, such as the data that resides in other tools to understand such things as what happens in the software development process, where defects have arisen, and why tests have failed. Based on that information, the AI instance can predict and determine the risks and impacts of defects module by module.

“Model-based testing is essentially about not writing tests by a human being. What you do is create the model and let the model create tests for you, so when a change happens you are changing something a lot more upstream versus changing the underlying test cases,” said KMS Technology’s Honda. “Likewise, when a test is written and automated [by the AI instance], if certain GUI widgets change or my user interaction changes, since I did not automate the test in the first place, my AI-driven program would automatically try to define automated tests based on the manual test case. Predictive QA is more resilient to change, [which is important because] brittleness is the biggest challenge for continuous testing and automation.

How to tell if your CT effort is succeeding The general mandate is to get to market faster with higher quality software. CT is a means of doing that. In terms of speed, tests should be running within the timeframe necessary to keep pace with the CI/CD process, which tends to mean minutes or hours versus days. In terms of quality, CT identifies defects earlier in the life cycle, which minimizes the number of issues that make it into production.

Another measure of CT success is a cultural one in which developers change their definition of “done” from

Automating tests when change is the norm Continuous testing requires automated testing to help speed the CI/CD process. The trick is to constantly expedite the delivery of code in an era when software change is not only a constant, but a constant that continues to occur at an ever-accelerating rate.

In today’s competitive business environment, customers are won and lost based on code quality and the value the software provides. Rather than packing applications with a dizzying number of features (a mere fraction of which users actually utilize) the model has shifted to continuous improvement, which requires a much better understanding of customer expectations in real time, an unprecedented level of agility and a means of ensuring software quality practices are both time-efficient and cost-effective despite software changes.

“You used to automate everything thinking you’re going to get an overall lift,” said Michelle DeCarlo, senior VP, Technology Engineering, Enterprise Delivery Practice at Lincoln Financial.

“While that was true, there’s also a maintenance cost that can break you because there’s the cost of keeping things current. Now [we have] a lot more precision upfront in the cycle to identify where we should automate and where we’re going to get that return.”

Rather than simply automating more tests because that’s what seems to facilitate a shift to CT, it’s wise to have a testing strategy that prioritizes tests and distinguishes between tests that should and should not be automated based on time savings and cost efficiency.

“Before people were implementing DevOps, [they] used to say if you needed a stable application you should have one round of manual testing before you could venture into automation. Once people started implementing DevOps,

testing had to happen with development,” said Vishnu Nallani Chekravarthula, VP and head of innovation at software development and quality assurance consultancy Qentelli. “One of the approaches that we have found to be successful is writing the tests before you write the code, and then write the code to ensure that the tests pass.”

While test-driven development (TDD) isn’t a new concept, it’s a common practice among organizations that have adopted CT. Whether TDD enables CT or the other way around depends on the unique starting point of an organization. With TDD and CT, automation isn’t an afterthought, it’s something that’s top of mind from the earliest stages of a project.

Adapting to constant change While applications have always been subject to change, change is no longer an event, it’s a constant. That’s why more organizations are going to market with a minimum viable project (MVP) and improving it or enhancing it over time based on customer behavior and feedback. Since development and delivery practices have had to evolve with the times, so must testing. Specifically, testing cycles must keep pace with CI/CD without increasing the risks of software failures. In addition, testers have to be involved in and have visibility into everything from the development of a user story to production.

“You’re always able to analyze and do a sort of an impact analysis from user stories [so if] we change these areas or these features are changing, [you can come up with a] list of tests that we typically no longer need, that would have to be abated to reflect the new feature set,” said Mush Honda, VP of testing at software development, testing services and consulting KMS Technology. “So, it follows that the involvement and the engagement of the tester as part of the bigger team definitely needs to be a

JUST A LAB? THINK AGAIN.

Perfecto is the only end-to-end solution for continuous testing.

Create, execute, AND analyze tests in our all-in-one platform. And do it all from our always-on, always-stable cloud-based lab. Explore all that Perfecto has to offer.

Learn More

core component.”

While it’s always been a tester’s responsibility to understand the features and functionality of the software they’re testing, they now have to understand what’s being built earlier in the life cycle, so tests can be written before code is written or tests can be written in parallel with code. The earlier-stage involvement saves time because testers have insight into what’s being built.

Also, the code can better align with the user story. It’s also more apparent what should be automated to drive better returns on investment.

Be careful what you automate

mindsets away from overreliance on automated UI tests.

“If you’re in a situation where you have the type of application that is driven by a lot of business needs and a lot is changing, from a technical perspective, you don’t want to automate at the UI level,” said Nancy Kastl, executive director of testing at digital transformation agency SPR. “Instead you want to automate at the unit level or the API services level.”

“Think of applying for a bank loan. In the loan origination process you go through screen after screen. [As a tester,] you don’t want to involve the whole workflow because if something changes in one part, your tests are going to have to change throughout,” said Kastl.

A common mistake is to focus automation efforts on the UI. The problem with that is the UI tends to change n order to write tests that are more often than the back end, which less brittle, you have to test it makes those autofrom the bottom up and then as mated tests brittle. things change, you have to change The frequency of tests at the individual layers.”

UI change tends to —Manish Manthuria be driven by the business, because when they see what’s built, they realize The concept is akin to building they’d prefer a UI element change, microservices applications that use such as moving the location of a button. small, self-contained pieces of code “If you have a car and if the car versus a long string of code. Like breaks down, you don’t just look at the microservices, the small, automated steering wheel and dashboard, so tests can be assembled into a string of unless you have tests and sensors at tests, yet a change to one small test individual parts of the car, you can’t does not necessarily require changes really tell why it has broken to all other tests in the string. down. The same idea applies “We need to think like proto software testing,” said grammers because if someManish Manthuria, CTO thing changes, I’ve got one and co-founder of digital script to change and everything strategy and services else fits together,” said Neil Pricecompany Infostretch. “In Jones, president of software testing order to write tests that and quality assurance consultancy are less brittle, you have to NVP Testing. test it from the bottom up and However, test automation can only then as things change, you do so much. If change is the norm have to change tests at the indibecause of ad hoc development pracvidual layers.” tices that aren’t aligned with the busiUsing a layered approach to testing, ness’ expectations in the first place, errors can be identified and addressed then test automation will never work, where they actually reside, which may according to SPR’s Kastl. Fix the way not be at the UI level. A layered you develop software first, then you’ll approach to testing also helps shift be able to get test automation to work. z

“I

the delivery of code to the delivery of tested code.

“You need the cultural belief that developers can’t say something is done until it’s been tested. Another key success indicator is when all your testing is completed in the same Agile sprint,” said SPR’s Kastl. “It’s not saying ‘I’m going to do some testing in the sprint based on the amount of time I have so I’m going to automate regression in the next sprint.’ You should not be a sprint behind. The way to make sure in-sprint testing is being done as part of a CT process is developers are merging their code and it’s ready to test on an hourly or daily basis, so testers can do their work.”

For Infostretch’s Mathuria, the highlevel indicator of CT success is data that proves a build or release is certified in an automated way. A lowerlevel indicator is that decisions are not being made about software promotion at any pre-defined level, such as this much functional testing is enough or that much security testing is enough. Instead, what qualifies as “enough” is determined by the CT process an organization has established.

“Only exceptions are managed by people and not the base level workflow,” said Mathuria. “Once you achieve that then you see the right value from continuous testing.”

And don’t forget metrics, because success needs to be measured. If speed is the goal, what kind of speed improvement are you trying to achieve? Define that and work backward to figure out what’s necessary to not only meet the delivery target but also be confident that the release is of an acceptable quality level.

“You also need to think about skill sets. Are they able to adopt the tools necessary or not? Do they understand automation or not? Do they understand the continuous testing strategy or not?” said Honda. If you want to get to continuous anything, there has to be a timeline and a goal you have to measure up against, which ultimately defines whether you’re successful, not successful or facing roadblocks.” z

Full Page Ads_SDT023.qxp_Layout 1 4/18/19 11:33 AM Page 29

Full Page Ads_SDT022.qxp_Layout 1 3/25/19 11:46 AM Page 22

Why Continuous Testing Is So Important

BY LISA MORGAN

rganizations continue to modernize their software development and delivery practices to minimize the impact of business disruption and stay competitive.

Many of them have adopted continuous integration (CI) and continuous delivery (CD), but continuous testing (CT) tends to be missing. When CT is absent, software delivery speed and code quality suffer. In short, CT is the missing link required to achieve an end-to-end continuous process.

O

“Most companies say they are Agile or want to be Agile, so they’re doing development at the speed of whatever Agile practice they’re using, but QA [gets] in the way,” Manish Mathuria, CTO and founder of digital strategy and services company Infostretch.

“If you want to test within the sprint’s boundary, certifying it and managing the business around it, continuous testing is the only way to go.”

Part of the problem has to do with the processrelated designations the software industry has chosen, according to Nancy Kastl, executive director of testing at digital transformation agency SPR. “DevOps should have been called DevTestOps

THE FIRST OF THREE PARTS [and] CI/CD should have been called CI/CT/CD,” said Kastl. “In order to achieve that accelerated theme, you need to do continuous testing in parallel with coding. Otherwise, you’re going to be deploying poor quality software faster.”

Although many types of testing have shifted left, CT has not yet become an integral part of an end-toend, continuous process. When CT is added to CI and CD, companies see improvements in speed and quality.

Automation is key Test automation is necessary for CT; however, the two are not synonymous. In fact, organizations should update their automated testing strategy to accommodate the end-to-end nature of continuous processes.

“Automated testing is a part of the CI/CD pipeline,” said Vishnu Nallani Chekravarthula, VP and head of innovation at software development and quality assurance consultancy Qentelli. “All the quality check gates should be automated to ensure there is no manual intervention for code promotion between [the] different stages.”

That’s not to say that manual testing is dead. The critical question is whether manual testing or automated testing is more efficient based on the situation.

“You have to do a cost/benefit analysis [because] the automation has to pay for itself versus ongoing manual execution of the script,” said Neil PriceJones, president of software testing and QA consultancy NVP Testing. “The best thing I heard was

someone who said he could automate a test in twice the time it would take you to write it.”

CT is not about automating everything all the time because that’s impractical.

“Automated testing is often misunderstood as CT, but the key difference is that automated testing is not in-sprint most times [and] CT is always done in-sprint,” Qentelli’s Chekravarthula said.

Deciding what to automate and what not to automate depends on what’s critical and what’s not. Mush Honda, vice president of testing at KMS Technology, considers the business priority issues first.

“From a strategy perspective, you have to consider the ROI,” said Honda. “[During a] project I worked on, we realized that one area of the system was used only two percent of the total time so a business decision was made that only three cases in that two percent can be the descriptors, so we automated those.”

In addition to doing a cost/benefit analysis, it’s also important to consider the risks, such as the business impact if a given functionality or capability were to fail.

“The whole philosophy between CI and CD is you want to have the code coming from multiple developers that will get integrated. You want to make sure that code works itself as well as its interdependencies,” said SPR’s Kastl.

Tests that should be included CT involves shifting many types of tests left, including integration, API, performance and security, in addition to unit tests. All of those tests also apply to microservices.

“For modern technology architectures such as microservices, automated API testing and contracts testing are important to ensure that services are able to communicate with each other in a much quicker way than compared to integration tests,” said Chekravarthula. “At various stages in the CI/CD pipeline, different types of tests are executed to ensure the code passes the quality check gates.”

Importantly, CT occurs throughout the entire SDLC as orchestrated by a test strategy and tools that facilitate the progression of code.

“The tests executed should ensure that the features that are part of a particular commit/release are covered,” said Chekravarthula. “Apart from the insprint tests, regression tests and non-functional tests should be executed to ensure that the application that goes into production does not cause failures. At each stage of the CI/CD pipeline, the quality check gates should also reach a specified pass percentage threshold for the build to qualify for promotion to the next stage.”

Since every nuance of a microservices applica-

Full Page Ads_SDT022.qxp_Layout 1 3/25/19 1:10 PM Page 27

A brief history of web and mobile app testing.

BEFORE SAUCE LABS Devices. Delays. Despair.

AFTER SAUCE LABS Automated. Accelerated. Awesome.

Find out how Sauce Labs can accelerate continuous testing to the speed of awesome.

For a demo, please visit saucelabs.com/demo Email sales@saucelabs.com or call (855) 677-0011 to learn more.

The Cloud’s #1 Continuous Testing Platform Accelerate digital transformation across the enterprise with a comprehensive suite of software testing tools – from agile test management to automated continuous testing for enterprise architectures.

tion is decomposed into a microservice, each can be tested at the unit level and certified there and then in integration tests created above them.

“Functional, integration, end-to-end, security and performance tests [are] incorporated in your continuous testing plan,” said Infostretch’s Mathuria. “In addition, if your microservices-based architecture is to be deployed in a public or private cloud, that brings different or unique nuances of testing for identity management, testing for security. Your static testing could be different, and your deployment testing is definitely different so those are the new aspects that microservices-based architecture has to reflect.”

KMS Technology’s Honda breaks testing into two categories — pre-production and production.

In pre-production, he emphasizes contracts in testing, which includes looking at all the API documentation for the various APIs, veri-

His test automation emphasizes business-critical workflows including what would be chaotic or catastrophic if those functions didn’t work well in the application (which ties in with performance and load testing).

“A lot of times what happens is performance and load are kept almost in their own silo, even with microservices and things of that nature. I would encourage the inclusion of load and performance testing to be done just like you do functional validation,” said Honda.

On the production side, he stressed the need for testers to have access to data such as for monitoring and profiling.

“With the shift in the role that testing plays, testers and the team in general should get access to that information [because it] allows us to align with the true business cases that are being applied to the application

‘I call out pre-prod and prod buckets specifically instead of calling out dev, QA, staging, UAT, all of those because of the nature of continuous delivery and deployment.’

— Mush Honda, KMS Technology

fying that the APIs perform as advertised, and checking to see if there is any linguistic contract behavior within the documentation or otherwise that the microservices leverage. He also considers the different active end points of all the APIs, including their security aspects and how they perform.

Then, he considers how to get the testing processes and the automated test scripts in the build and deployment processes facilitated by the tool he’s using. Other pre-production tests include how an API behaves when the data has mutated and how APIs behave from a performance perspective alone and when interacting with other APIs. He also does considerable component integration testing.

“I call out pre-prod and prod buckets specifically instead of calling out dev, QA, staging, UAT, all of those because of the nature of continuous delivery and deployment,” said Honda. “It’s so much simpler to keep them at these two high levels.”

when its live,” said Honda. “How or what areas of the system are being hit based on a particular event in the particular domain or socially? [That] also gives us the ability to get a good understanding of the user profile and what a typically useful style of architecture is. Using all of that data is really good for test scenarios and [for setting] our test strategy.”

He also recommends exploratory testing in production such as A/B testing and canary deployments to ensure the code is stable.

“The key thing that makes all of this testing possible is definitely test data,” said Honda. “When you test with APIs, the three key sorts of test data that I keep in mind are how do you successfully leverage stubs when there’s almost canned responses coming back with more logic necessarily built into those? How do you leverage fakes whereby you are simulating APIs and leveraging anything that’s exposed by the owner of any dependent services? Also, creating mocks and virtualization where we need to make sure that any of the mocks are invoked in a certain manner [which allows] you to focus on component interaction between services or microservices.”

In addition to API testing, some testing can be done via APIs, which SPR’s Kastl leverages.

“If I test a service that is there to add a new customer into a database, I can test that the new customer can be added to the service without having a UI screen to collect all the data and add that customer,” said Kastl. “The API services-level testing winds up being much more efficient and with automation, it’s a lot more stable [because] the API services are not going to change as quickly as your UI is changing.”

Jim Scheibmeir, associate principal analyst at Gartner, underscored the need for authentication and entitlements. “[Those are] really important when we talk about services: who can get to the data,

SLA and some agreed-upon independent factor that says our test cases are not necessarily depending on one another. It increases the concept of good scalability as your system matures,” said Honda. “One of the routine mistakes I’ve seen a lot of teams make is saying I’m going to execute six test cases, all of which are interdependent. [For example,] I can’t execute script number three unless tests one and two have been executed. [Test independence] becomes critical as your system matures and you have to execute 500 or 50,000 test cases in the same amount of time.”

As his teams writes tests, they include metadata such as the importance of the test, its priority, the associated user story, the associated release, and what specific modules, component, or code file the test is executed against. That way, using the metadata, it is possible to choose tests based on What goes into the right mix of CT tests are goals. those that are going to test that functionality you’re “The type of test and the test’s purpose both have to be incorporated in your deciabout ready to deploy, as well as critical business sion-making about what tests to run,” said functionality that could be impacted by your change Honda. “You don’t write your tests thinkplus some performance and security tests.’ ing, ‘I want to test this,’ you write your test and incorporate a lot of metadata in — Nancy Kastl, SPR it, then you can use the tests for a specific how much they can get to, is goal like putting quality into your CD process or there a threshold to usage or doing a sanity test so you can promote it to the availability,” said Scheibmeir. “Perfornext stage. Then you can mix and match the metamance is also key, and because this is a complex data to arrive at the right suite of tests at runtime environment, I want to test integrations, even without hard-coding that suite.” degrading them, to understand how my composite Performance and security testing should be an application works when dependencies are down, or integral part of CT given user experience expectahow fast my self-healing infrastructure really is.” tions and the growing need for code-related risk management.

Ensuring the right combination of tests “What goes into the right mix of CT tests are When deciding which tests to run, KMS Technol- going to be those tests that are going to test that ogy’s Honda considers three things: business prior- functionality you’re about ready to deploy, as well ity, the data, and test execution time. as critical business functionality that could be Business priority triages test cases based on the impacted by your change plus some performance business objective. Honda recommends getting and security tests,” said SPR’s Kastl. “Taking a riskbusiness buy-in and developer buy-in when based approach is what we do normally in testing explaining which tests will be included as part of and it’s even more important when it comes to the automation effort, since the business may have continuous testing.” thought of something the testers didn’t consider.

Past application issues are also an indicator of Second, the data collected by monitoring tools which tests should be prioritized. While test coverand other capabilities of the operations team pro- age is always important, it’s not just about test covvide insight into application usage, the target, pro- erage percentages per se. Acceptable test coverage duction defects, how the production defects are depends on risk, value, cost effectiveness and the being being classified, how critical they are from a impact on a problem-prone area. business severity perspective, and the typical Gartner’s Scheibmeir said another way to workflows that are being used. He also pays atten- ensure the right mix of tests are being used is tion to execution speed since dependencies can benchmark against oneself over time, measuring such things as lead time, Net Promoter Score, or result in unwanted delays.

“There should be some agreed-upon execution the business value of software and releases. z

CLOUD-BASE TESTING LAB With real devices and browsers to test, we’re always on. Always updated. Entirely secure. Get the stability and enterprise-grade scalability you need.

SIMPLIFIED TEST CREATION Whether you need help with test authoring, maintenance, management, validations, or debugging, Perfecto has a solution for you.

STREAMLINED TEST EXECUTION Orchestrate large suites across platforms for high velocity and parallel text execution with unlimited elastic scaling.

SMART ANALYSIS Get an instant overview in a powerful management dashboard with Perfecto’s visual analytics. With fast feedback, you’ll get all the insights you need.

webinar ad.qxp_WirelessDC Ad.qxd 3/25/19 12:41 PM Page 1

Be a more effective manager

Visit the sdtimes.com Learning Center Watch a free webinar and get the information you need to make decisions about software development tools.

Learn about your industry at www.sdtimes.com/sections/webinars

Century-Old Company Prioritizes CT ompanies in just about every industry are being disrupted by digital natives. To stay competitive, old companies must adapt, which can be a painful transformation. Moving to Agile is tough enough. Now, more businesses have adopted DevOps, continuous integration (CI) and continuous deployment (CD), albeit at different speeds.

Software testing at Lincoln Financial Group is progressive by design because software delivery speed and quality differentiate the company, so much so that Michelle DeCarlo, SVP of IT Enterprise Delivery, leads the company’s technical transformational initiatives.

“DevOps, cloud, continuous testing all feed into our larger journey which is the digital transformation effort,” said DeCarlo. “At the core, it’s changing the way we work. So, when we talk about continuous testing, it’s how do we get faster, leaner, and eliminate friction.”

Like many mature companies, Lincoln Financial had waterfall-type practices that involved a series of hand-offs. In addition, a lot of the testing was manual. The company is now using continuous processes, including CT to get to market faster. In fact, DeCarlo said that continuous testing has improved Lincoln Financial’s delivery speed by 30%.

“Continuous testing is one of those levers that we view as being critically important and fundamental to our strategy,” she said. “Testing used to be viewed as a cost because we were always waiting. Now it’s what differentiates us and helps us get to market faster.”

Of course, getting there wasn’t all about the CT process. In addition to changing the way people work, talent had to be upskilled and new tools were required.

“There are a few things we do differently with CT compared to how it used to work. One of those things is recognizing that the advanced engineering practices that are important to learn, like test-driven development and service virtualization, are table stakes now,” said DeCarlo. “These skills allow us to accelerate delivery cycles and the test engineers are embedded in the delivery cycle.”

Lincoln Financial automates many types of tests, including performance, security, regression, feature and smoke testing.

“When you drop code, you need to determine in seconds whether that code is valuable. If it fails, it stops your production cycle,” said DeCarlo. “You have to think about it as one continuous flow

C

because if you have a weak link, it halts your production and you lose hours. You cannot advance your practice unless you have automation.”

The company is also taking advantage of microservices and containers, which enable a flexible and scalable application architecture.

“The ability to decompose your code base and test at the most granular level is how firms get an advantage,” said DeCarlo. “When you test monolithic systems in totality it takes a long time, so we now decompose code and features, and test at the smallest level possible. Minimum viable product is part of our DNA now.”

API testing is critical “The focus on service testing is huge,” said DeCarlo. “In the past, people were worried about testing the front-end, which is still helpful and necessary, but the majority of your effort should be focused on the service level because that’s where you need to make sure everything works.”

Data drives insights Lincoln Financial also makes a point of Michelle DiCarlo using encrypted test data that includes the scenarios it wants to test so it can very quickly test multiple parameters and scenarios against whatever use cases apply.

“It’s critical to ensure that testers have all the data needed as opposed to waiting on other groups,” said DeCarlo. “We have figured out how to get the data that might be upstream, downstream, in the middle, or in our service architecture. Our strategy is to make data [available] on demand, which will allow us to respond at the same pace as other firms.”

Continuous testing means continuous learning Lincoln Financial has a mantra when it comes to software: build, measure, learn. The philosophy maps to developing the software, getting the data ready, testing it, going through the continuous delivery process, and measuring it.

“You need to have performance dashboards that are available to all your stakeholders, whether it’s a developer, tester, or product owner, and then you have to make modifications and improvements because what worked yesterday might work today but likely will not work tomorrow,” said DeCarlo.

“You have to embed this continuous improvement mindset and not look at it as failures, but learning.” z