How Splunk Built A Practical Approach to DevSecOps At Scale

What Splunk Does

Joel Fulton is the Chief Information Security Officer for Splunk. At Splunk, they've put effort into transforming their organization from a waterfall approach to agile, to now a DevSecOps approach.

In case you're not familiar, Splunk is a software development company focused on machine data aggregation. They collect your data on to your on-prem and they count on you to manage and protect that. Splunk relies on your perimeter, your moat, your infrastructure, your detection, your response. Splunk's licensing model guarantees nobody puts innocuous data into Splunk. They charge on ingestion, so you only put the most important pieces of your data into Splunk. When that situation is then delivered through the cloud, you're delivered you not only your Splunk as a service but all of the vulnerabilities that are part of that Splunk code as well as the infrastructure in which it's hosted. So now it's a different risk problem for Splunk because the customer assumes, correctly, and demands that the security they've got in place meets or exceeds customer requirements. That times ten thousand customers.

Splunk's Development Process

How Splunk got there is they backed into this process. Splunk started with waterfall. Splunk was built under a waterfall methodology. They quickly moved to agile which for them meant waterfall without the rules, and that creates lots of chaos. Splunk's  process of getting into cloud was to take their on prem software and host it in an AMI in Amazon. So that's really on prem hosted. So all of the problems that originated in their waterfall style: product checks, secure development lifecycle, code reviews, before release to production, the assessment of the amalgamation of vulnerabilities, that was all done at a checkpoint at a moment before making it live for the customer. Because it wasn't built for the cloud, they lacked things like orchestration so they could provision systems with rapidity and consistency, but then the customers would each demand snowflakes. They wanted to tweak this way they wanted this bits added to it. And that was a necessary request because Splunk was on prem hosted in the cloud at the time.

So then Splunk gets drift in their systems and all of these hosts serving, transacting, storing, and processing customer data are now starting to drift in their configuration. Joel likens the process to an El Camino. You know what an El Camino is? It's like a truck and a sedan put together; you've got all the carrying capacity of a sedan and all the passenger capacity of a truck - the worst or both worlds.

So it's this agile machine within a waterfall framework because security was the problem. And the problem with this is the difference between agile and DevSecOps, is when do you put security into the process? When do you understand the cumulative impacts of vulnerabilities that you have. And that was their real problem because they can assess source code as it goes through but what Splunk couldn't assess in the agile process is what is the concomitant result of the amalgamated vulnerabilities when it goes into GA?

If the customer would come to Splunk and say "hey why were we exploited? Why was there a problem?" The customer doesn't care that it wasn't the 'security' team, it was the 'product' team. It's the amalgamation of vulnerabilities that Joel and his team and then Splunk at large cares about. The difficulty that Splunk had was in the planning phase.  There wasn't a security member in the daily standup, seeing the user stories. And so they don't know what's coming. Therefore Joel couldn't predict what the result of the amalgamation of these changes in the environment in the milieu in which they offered this to customers was going to be. That bit of blindness, that bit of lack of visibility, is what stopped Joel and his team from being part of their process.

How Splunk Went from a Waterfall Approach to Agile, to Now a DevSecOps Approach

So, to integrate into what was maturing agile and maturing DevOps process, here's what Joel did:

First, Splunk has members of the security team that are part of engineering and development organization involved in the daily standups and involved in taking bug reports, triaging, doing the selection of features - the user stories that include security as an emphasis. Joel has a security team that are directly under him and they're separate. And the reason they've separated this is they are conforming to the org that the product team have. People think Splunk is a security company, but they're not, they're a software development company. The distinction and the difference between those two is significant in terms of persuasion priorities and what features are released. So the product team have segregated the application from the infrastructure and the hosting environment. And so security is meeting them on their grounds and they're in their daily standups. But security are the ones that take a step back and put them together to understand the consequences of the differing vulnerabilities when they come together for the customer.

The next step, and where Splunk is today is moving the organization further into a security mindset. How do we hold them accountable and reward them. The features that matter to engineers are not the features that matter to security necessarily. It is less cool to go back and fix something and get really no credit except nothing bad happens so there's a negative credit as opposed to new feature new functionality that people will create and trumpet about. When security has a good day it's a boring day. That's not what most people seek out of their career. And so being able to motivate, being able to train, doing threat modeling and sharing the information: This has been a massive advantage to Splunk to persuade a change in the engineering environment.

The focus what on what matters and how to align those things. For engineers, they're all in the result of that code bears on them personally. It bears on their relationship with their peers; they take quality as an emblem of pride. The production matters to them. They viewed Joel and his team as kind of having an interest but not skin in the game. Getting that skin in the game, having Joel's team embedded in the engineer's team - that enabled security to meet engineering where they were. And that's what moved the line for Splunk in getting a better outcome and getting Sec in the DevSecOps.

‍