Deeper Look at DevSecOps - SDTimes

DeeperLook cover March2020.qxp_Layout 1 2/24/20 1:30 PM Page 1

THE LAST OF THREE PARTS

How to get DevSecOps right SPONSORED BY

020-27_SDT033.qxp_Layout 1 2/20/20 2:18 PM Page 20

20

SD Times

March 2020

www.sdtimes.com

How to Get DevSecOps Right

BY LISA MORGAN

D

evOps and security teams are learning how to work together, albeit somewhat awkwardly in these early days of DevSecOps. One reason why it can be difficult to get the partnership “right” is that people define DevSecOps in different ways.

“If you asked a room of 10 people to define DevSecOps, you’d get 15 definitions. I think we’re starting to coalesce, but I still think there are a lot of misconceptions and there are still a lot of people who are going about it the wrong way,” said Caleb Queern, principal at KPMG Cyber Security.

One challenge is approaching DevSecOps from a tools-only perspective, without considering the impacts of change on processes and people.

“A lot of buyers will procure something, put it in place in their organization and declare victory,” said Queern. “It doesn’t lead to the business outcomes that most folks are looking for.”

In a December 2019 report, Gartner estimated that by 2021, DevSecOps practices will be embedded in 60% of rapid development teams, as opposed to 20% in 2019. To succeed, DevOps teams must prioritize security and security teams must adapt to the accelerating pace of development.

Is shifting left enough?

DevSecOps reduces the number of application vulnerabilities, but shifting left is not enough.

“Shift left has truly moved the needle on capturing a lot of these early-stage, often ignored, vulnerabilities [but] it doesn’t necessarily solve all of the security challenges for an organization,” said Jake King, co-founder and CEO of Linux security platform provider Cmd. “We have to be vigilant, we have to be aware, we have to be con-

tinuously testing.”

Application exploits are the most common way hackers infiltrate organizations. So, in addition to running a Common Vulnerabilities and Exposures (CVE) scan late in the SDLC, shifting security left adds another security point. However, security should be integrated into every aspect of the application life cycle because as a recent Trend Micro report states, “Attackers will find ways to take advantage of any weak link to compromise the DevOps pipeline.” “[Application] security has kind of been looking at the trees and missing [the] forest,” said Zane Lackey, former chief information security officer (CISO) at ecommerce marketplace Etsy. “Trying to sprinkle a little bit of security dust at the end or at the beginning [means] you’re missing the forest, which is that you need to be empowering teams to think about and have security capabilities from when they’re designing new services to when they’re implementing them to when they’re actually deploying and running them in production.”

Sandy Carielli, principal analyst at Forrester Research, agrees.

“We need to make sure we’re not just addressing security in the design and development phase, but also in the testing and deployment phases,” said Carielli. “It absolutely needs to be addressed throughout the life cycle. The vulnerability is going to survive for a much shorter lifetime if there is regular scanning.”

End-to-end application security results in more secure code. And, full life cycle visibility into exploits helps teams understand how and where their applications are being compromised.

“The way security was done historically fruscontinued on page 22 >

020-27_SDT033.qxp_Layout 1 2/20/20 2:18 PM Page 21

www.sdtimes.com

March 2020

THE LAST OF

SD Times

THREE PARTS

21

020-27_SDT033.qxp_Layout 1 2/20/20 2:19 PM Page 22

22

SD Times

March 2020

www.sdtimes.com

What the relationship with the security team should look like There’s a diversion of opinion about who should initiate the DevOps-security relationship. Should DevOps reach out to security or should security reach out to DevOps? There has been some friction in the relationship because security has served as a barrier to faster application releases.

“There’s no time for old school security testing every release. Fundamentally, security should be something that’s just like quality. It’s one of the fundamental requirements of your code,” said 451 Research’s Fernando Montenegro. “The role of security is collaborating with that development team to let them know what it is that they need to do. Here are our corporate objectives around security and the kinds of things that we want you to be aware of. Here’s how you threat-model your application to define what needs to happen where and then let the development team run with it.”

Traditionally, security’s first instinct is to prevent developers from introducing risks in the first place. For example, Etsy’s former

< continued from page 21

trates developers. You wait until the very end and it takes a week for them to get back results. And a lot of the tools that we’ve used historically, things like static analysis, create an unacceptable [number] of false positives,” said Neil MacDonald, vice president, distinguished analyst and Gartner fellow emeritus in Gartner Research. “It’s the antithesis of what we want for DevOps. We’re supposed to be enabling speed and agility and so security has to shift left, integrate natively into the developer’s environment and catch these problems as soon as possible so the developer isn’t slowed down at the end.”

However, baking security into the entire SDLC takes time.

“This isn’t something that you finish, ever,” said KPMG’s Queern. For one thing, black hat tactics are constantly evolving.

It’s also important to pay attention to

CISO Zane Lackey recently met with the CSO and CIO of a Fortune 100 company separately, about 30 minutes apart. When he asked each of them how they think about cloud and DevOps, the CSO said he wasn’t allowing them to happen because they’re not secure. The CIO subsequently said they had 50 cloud apps now. By the end of the year, they’ll have 200 and in 2021, a couple thousand.

“I looked at him kind of funny, and he said, ‘Oh, you just met with the CSO,’ “ said Lackey. “We don’t tell them [anything] anymore because they try to say no to everything. If they want to enable us, then we’ll incorporate that, but if they’re just going to be gatekeepers and say no, we’re not going to speak to them anymore.”

Good progress is getting security and DevOps working together. Better progress is when security and DevOps cooperatively integrate security practices throughout the SDLC in a manner which doesn’t create friction in the DevOps pipeline. z — Lisa Morgan

access rights.

“It’s not so much about technology components, but how you’re divvying up who does what,” said Fernando Montenegro, principal analyst, information security at 451 Research. “If I have an artifact here, am I allowed to interact with it? Am I allowed to do something with it? Do I have the right credentials to do dev versus QA testing versus production etc.? And then at runtime, is this thing behaving the way it’s supposed to behave and if something goes wrong? Do I have the infrastructure to understand what happened? You absolutely need that across the stack.”

What the DevSecOps team should look like Moving from DevOps to DevSecOps isn’t a matter of putting a security team member on a DevOps team, because developers tend to outnumber security professionals by 100 to one. Since most

developers aren’t security experts and most security experts aren’t developers, the two must learn to work together effectively.

“Research shows that the places where we talk more about security, we’re better at security,” said Queern.

Queern and others interviewed for this story underscored the importance of “security champions” who serve as the security expert on a DevSecOps team. Security champions are usually a developer who volunteers for the role.

In Queern’s experience, the security champion participates in labs, demos or presentations that focus on security and application development best practices.

While the application security team may host the trainings, the presentation duties are very quickly assigned to the security champions who will serve as security experts for their teams.

Health management platform continued on page 24 >

Full Page Ads_SDT033.qxp_Layout 1 2/20/20 11:24 AM Page 23

020-27_SDT033.qxp_Layout 1 2/20/20 2:20 PM Page 24

24

SD Times

March 2020

www.sdtimes.com

< continued from page 22

4

DevSecOps Mistakes to Avoid

DevSecOps isn’t just a practice, it’s a continuous learning experience. If you want to be successful faster, avoid these common misperceptions.

#1

Business as Usual Is Good Enough. Cybercriminals are constantly chang-

ing their tactics. If your organization’s application security practices are static, they aren’t as robust as they should be.

“I think a lot of times, people think that things are going to continue as normal.

That the same processes, the same organizational structure and the same way you’ve been doing things up until now are going to help you in the future,” said Fernando Montenegro, principal analyst, information security at 451 Research.

#2

Failing to Monitor Developers’ System Access. Jake King, co-founder and CEO of Linux security platform provider Cmd, said during the early stages of rolling out DevSecOps, organizations overlook the access developers have in the environment.

“They’ll grant developers a lot of trust and empower them to do their job as well, but at the same time, they’re not keeping a close eye on what they are doing and how they are doing it – simply ignoring that people are doing very sensitive things,” said King. “It’s like having your CFO being able to process a wire transfer to a country you’ve never made a payment to, independently.”

#3

Failing to Monitor Code Changes. Code is constantly changing, including

new or changed configurations, patches and system maintenance, many of which are outside a DevSecOps’ team’s control. The result is that no one is sure what exists in the environment.

“What libraries and packages are out there, not only from a vulnerability perspective but also from an exposure perspective? When you deploy a library, how many supply chain components are you bringing into the fold? How many kinds of upstream vendors?” said Cmd’s King.

#4

provider Advantasure has a security liaison who works across more than 500 developers, according to Chief Information Security Officer (CISO) Wallace Darlymple. The security liaison is responsible for upskilling the developers who will become security champions. Specifically, the liaison oversees their e-learning training programs and is the first escalation point when a security-related question or issue arises.

The ROI on training dollars tends to be poor if the person attending training doesn’t apply the newly acquired knowledge on a regular basis, however.

“The best type of training is training in the moment. Having that training built into the development process a little more closely will allow the teams to start absorbing it and living it a lot more,” said Forrester’s Carielli.

Charles Betz, global DevOps lead serving infrastructure and operations professionals at Forrester, thinks it’s important to provide DevSecOps teams with secure baseline infrastructure environments.

“In many cases, people just don’t know what needs to be secured,” said Betz. “So, give them an image that’s secured, give them an environment that’s secure, give me acceptable patterns for using the database, using Apache Kafka, using a load balancer in the firewall. Then, have them focused on what they need to be focused on.”

Trying to Force Traditional Security Methodologies on DevOps Teams.

The dual-speed nature of DevOps and security can be problematic. If security imposes too much overhead on DevOps teams or suggests solutions that aren’t practical, DevOps may well ignore security.

“Everything — applications, new services, things being updated and shipped — is now moving orders of magnitude faster than they were 10 years ago. Yet a lot of the security mentality hasn’t adapted,” said Zane Lackey, former chief information security officer (CISO) at ecommerce marketplace Etsy. “That gatekeeper mentality ends up getting routed around. We need to shift them to enabling [DevOps] teams so they can really self-serve.” z — Lisa Morgan

Supply chain security matters Developers are using more opensource and third-party elements in their code than ever before to meet time-tomarket mandates and focus on what matters most, such as competitively differentiating their applications. However, Gartner estimates that fewer than 30% of enterprise DevSecOps initiatives had incorporated automated security vulnerability and configuration scanning for open source and commercial packages in 2019. That number is expected to rise to more than 70% in 2023.

“Security needs to go all the way back to the point where they’re downcontinued on page 27 >

025_SDT033_r1.qxp_Layout 1 2/24/20 1:31 PM Page 25

www.sdtimes.com

March 2020

SD Times

25

INDUSTRY SPOTLIGHT

Security – Just Another Aspect of Quality These flaws don’t get as much attention as performance bugs rogrammers err as much as any of us — between 15 and 50 errors per 1,000 lines of code to be more exact. QA tests for these bugs, attempting to ensure that releases are as bug-free as possible. Customers who trust their operations to software won’t tolerate poorly written code, and teams go out of their way to ensure that programs work the way they are supposed to.

But there’s another type of bug that often doesn’t get the same attention — the security bug. Those bugs generally don’t affect performance, at least right away, so teams tend to deprioritize them in favor of fixing functional bugs.

But in reality, security bugs are no different, and will eventually cause an app to do something unintended.

In fact, they have the potential to be far worse. A button that doesn’t respond properly (functional bug) is inconvenient and annoying, and can drive employees using an application batty. But a hackable module in the software (security bug) could give hackers the keys to the corporate kingdom, providing them with access to employee accounts, data, and even money.

Part of the challenge is the fact that security is never called out explicitly as an outcome. It’s very often that companies do not have a rules book, best practices or policies developers could follow. Security is expected, but very rarely asked for. The main difference between security bugs and functional bugs, is that for the latter very often there are policies in place and is an explicit requirement for development.

Security is never explicitly asked for, but expected to be there when the app is ready for production deployment.

If averting security bugs hasn’t been

P

Content provided by SD Times and

a priority for programmers, it certainly should be, and customers should demand testing for bugs as part of quality control. In today’s programming environment, that is likely to mean utilizing automated tools that will test for security issues as program modules are developed — just as they are used for functional bug testing.

With much development done in the cloud, for example, teams can use cloudbased tools, to determine if they have been following best security practices.

OWASP (the Open Web Application Security Project) provides a long list of automated security testing tools that can help developers detect vulnerabilities.

For maximum security testing, teams can use newly-developed tools that check code against security scenarios as

it is written and uploaded to repositories. Such tools can catch problems in specific modules before a security bug gets “buried” in the full application made up of dozens of modules —only to surface when a hacker gets wise to the bug and starts taking advantage of it. If developers “shift left” enough, seeking to resolve security issues as early as possible in the development process, we can squash security bugs as successfully as we do functional bugs.

In conclusion, when shifting left the work, make sure to call out “better security” as something developers are expected to deliver, rather than just hoping for it.

For more information, go to www.hcltechsw.com/wps/portal/ products/appscan/home. z

020-27_SDT033.qxp_Layout 1 2/20/20 2:20 PM Page 26

26

SD Times

March 2020

www.sdtimes.com

Focused on application vulnerabilities?

You’re missing the bigger picture.

BY LISA MORGAN

In today’s era of digital transformation, every organization must focus on application security. However, focusing on security vulnerabilities alone is unwise because it’s nearly impossible to prioritize what needs to be done.

“DevOps teams are sitting in front of a table with the keys to the kingdom on their computers,” said Jake King, cofounder and CEO of Linux security platform provider Cmd. “We need to think not only about the risks pertinent to the software that we build and the way in which we build that software, but also the way in which we access the systems that deploy that software. It becomes the weakest link in that chain.”

At the time of this writing, there were 130,969 Common Vulnerabilities and Exposures (CVEs) listed in the MITRE Corporation database.

That database is synchronized and used by the National Institute of Standards and Technology’s (NIST) National Vulnerability Database (NVD). NIST’s NVD ranks vulnerabilities using NIST’s Common Vulnerability Scoring System (CVSS) which is an open framework for communicating the characteristics and severity of

software vulnerabilities. CVSS v3.0 reflects five levels of severity (none, low, medium, high and critical) versus CVSS v2.0 which used three (low, medium, and high).

Security professionals use the ratings to prioritize patching, and hackers know it. So, some hackers will exploit vulnerabilities that have lower ratings while security professionals are focusing on the high and critical vulnerabilities.

“I have a very strong allergy to any kind of risk management that relies on an ordinal scale, rate the risk from one to five,” said Charles Betz, global DevOps lead serving infrastructure and operations professionals at Forrester Research. “I think that kind of stuff is worse than useless. It leads people to believe they’re secure when they’re not.”

No organization has the money or human resources to manage the 131,000 published vulnerabilities (which does not represent the entire universe of vulnerabilities). So, there are two other things enterprise security professionals consider for prioritization purposes. One is the organiza-

tion’s attack surface (software and hardware assets ranging from traditional to IoT). From a software development perspective, application composition, including open-source, third-party software and other dependencies, must be considered.

The reason it’s important to understand the attack surface is because not all vulnerabilities apply to any one organization. If there were no Microsoft software anywhere (unlikely in an enterprise setting), then Microsoftrelated vulnerabilities wouldn’t apply.

Similarly, if only certain Microsoft products or versions of products are used by the organization, then the products or versions not in use would not apply. Essentially, understanding the entire attack surface allows security professionals to focus on the relevant vulnerabilities, which is still an overwhelming number.

Therefore, it’s important to also consider threats. Of all the vulnerabilities that apply to an organization’s ecosystem or DevOps supply chain, which are hackers actively exploiting?

What security mistakes are people in the organization making that cause

020-27_SDT033.qxp_Layout 1 2/20/20 2:21 PM Page 27

www.sdtimes.com

March 2020

SD Times

‘A common metric is the number of bugs and if they’re going down over time. This has probably done more disservice to application security programs than anything else.’

—Zane Lackey, former chief information security

officer (CISO) at Etsy < continued from page 24

systems, software and data to become vulnerable?

“We’re never going to get bugs to zero whether they’re in open source or our own code. We need to get visibility into how people are actually attacking and abusing our services,” said Zane Lackey, former chief information security officer (CISO) at Etsy. “The critical ones are easy [because] they jump out at you.

Every security team on the planet is [grappling with] a mountain of medium severity bugs that never actually get fixed because they’re the same severity, the same risk.”

Worse, security teams tend not to have visibility into what’s happening in production, so they don’t have a way to quantify or prioritize those risks.

“There have been a lot of major breaches due to client-side protection issues, API issues and third-party open-source issues,” said Sandy Carielli, principal analyst at Forrester. “Application security is to some extent a third-party risk issue because you’re bringing in all these third-party components and container images.” z

loading code from GitHub. Let’s tell them this is a known vulnerable component. We’re not going to download that because it contains a known set of critical vulnerabilities,” said Gartner’s MacDonald. “Let’s tell them that before they download it, before they stitch it together in their application as they’re writing code in their IDE.” Advantasure’s decision to implement application security platform Veracode was made jointly between security and DevOps. “We can’t dictate to the business, you must do it this way,” said Advantasure’s Darlymple. “It was really us reaching out to them and saying you’re going down this path. How can we be leveraged in your environment without having to hire an army of application security architects just to support you?”

Measuring success It’s hard to tell how effective DevSecOps is if it isn’t being measured. There are some of the classic methods like bug tracking and code coverage, although Forrester’s Betz said it’s important to start with the target operating condition.

“What is your risk tolerance? What are your metrics there? What are some of the desired metrics with things like code coverage and what level of assurance do you require in your systems?” said Betz. “Really look at your highest governance metrics that drive your security posture and how you understand your attack surface. Those drive your operational processes and procedures which will include things like

software, bill of materials, supply chain, vulnerability scanning, pen testing and red team testing.”

Cmd’s King said time to detection and resolution of an infrastructure issue is important, as is code coverage and code hardening. Measuring customer perception is also wise because security is a trust issue.

“If you keep on top of it, if you incrementally add security, you bake it in from the start and add it early in your processes, you will see dividends paid up majorly in the long term because you will not be dealing with legacy,” said King.

Another metric has to do with the collaboration between DevOps and security.

“A common metric is the number of bugs and if they’re going down over time. This has probably done more disservice to application security programs than anything else because there’s a million questions within that that don’t get answered,” said Lackey. “The healthiest metric you can track is how often DevOps teams proactively interact with the security team because if those teams are proactively coming to you and focusing on that metric, whenever they think of new technologies or architectures you’re going to be able to reduce risk.”

Teams also should be counting security incidents such as breaches and data leakages, insider and external exploits and advanced persistent threats. “[Whichever way] you’re counting them you want to be baselining and continuously improving,” said Betz. “I strongly believe in benchmarking against yourself.” z

27