Scaling Secrets Management with Policy-as-Code

Niamh Cahill, WW Sales Engineering Leader, Chef

Solution architect and sales leader with strong experience in design & implementing management solutions for mobile wireless providers, network providers and large enterprise environments.

Hey, everyone. Well, the thing being like being this session between everybody and their break in the cup of coffee, but no pressure. What a great conference, huh? And we have so many great presentations. I’ve really enjoyed all of them. And I think what I’ve enjoyed most is the fact that we all seem to have very common problems. Right? We’re all trying to solve the same issues, but we have different environments and different challenges depending on your vertical in your business.

So, what we’re going to talk about today is essentially how do you really smooth that transition to automate secrets management practically. How do you embed that into your infrastructure’s code deployments and your app deployments? You’re trying to do all these different things. We heard a lot about deploying different applications into multi-cloud, on-prem, data centers, etc. And we’re trying to do all of these different things at once across all of your application portfolios in your relative businesses.

So, let’s talk a little bit about Chef and who we are. So, Chef was founded in 2008. We focused originally on infrastructure as code. What a lot of folks don’t know about Chef is we also have a core functionality around compliance-as-code, and really bringing that functional layer to security and to secrets management in general, and your deployments. We were acquired by Progress last year.

What that has meant is we can really invest in add a lot more functionality to our products and invest in our engineering teams. Also, we see a lot of different customers all around the world. In my role especially, I see lots of different kinds of deployments, different geographies, different requirements and needs, different security standards. But still, there is commonality in those challenges across the board.

So, when we say we’re trying to do all the things and do all of these deployments, what does that mean exactly? Everybody’s application portfolios tend to follow a certain pattern. And you might have legacy applications. You might have a combination of legacy, some SaaS, but then also deploying on to just regular virtual machines, either in the cloud or on-prem in your data center. And you’re really trying to ensure security across all of these stacks. Right?

And so, as you’re deploying change into your environments, really the only way to achieve secrets management and 100% coverage and really securely deliver your deployments is to automate all of this. So, how do you approach automating all of this across all these different tools, application stacks, deployment patterns, etc?

It’s really a challenge. And if you wait until after your deployment, which I think most folks do do some checking checks and balances as part of their pipelines and their deployment pipelines. But if you wait ‘til after the fact, after your deployment to do the balance and the checks, or to validate that you have the correct secrets management policy deployed, you’re going to lack visibility.

You really have to shift that whole process left in order to ensure that it’s consistently deployed. And also, we talked a lot about the human role and the human component, as part of this whole process. And automating and automating secrets management and making it easier, it’s really about making your operations and developers lives easier too. And if you don’t add that big, easy button into this whole process, you’re really not going to get there as consistently or become 100% compliant 100% of the time, which is obviously the Nirvana goal.

So, what is infrastructure as code? I know, a lot of you’ve heard, obviously, of Kubernetes, and application deployments, leveraging containerized workloads, that kind of approach. But overall, infrastructure as code is really about defining the change you want to deliver, whether it’s an application deployments, or an operating system configuration updates, patches, whatever that might be, and codifying it.

And in doing that, what you can do is you can bring automated checks and balances to that deployment code. So, stating what you want to have changed. So, think something as simple as a CloudFormation template. But it could also be a Chef cookbook that you’re delivering to a virtual machine within VMware, for example.

And in doing so, what you can do is you can functionally validate your deployment, and then you can also perform security checks on those. So, is this a change that I want to deliver? Is this update in this change going to actually work as you’re going through that deployment? And is it secure?

So, for example, my secrets management perspective, we’re going to cover an example in a couple of slides, but what should the policy be around the secrets management? No basic authorization, but for example, let’s make sure that we’re including secrets rotation and a policy as part of that deployment that states that I need to integrate with a solution like Akeyless, and you can have that as a check and balance as part of your application deployment and your infrastructure-as-code.

Typically, the changes that we see, you’re either updating your application config, or you’re updating some layer of your operating system or underlying infrastructure. So, if we think about the change in those kinds of categories, so we’ve heard this example, or this kind of example, mentioned a lot today.

So, one of the key takeaways from infrastructure as code as it relates to secrets management is, you don’t want to end up in this scenario. Right? So, this seems so obvious when we’re standing here looking at it and it’s the middle of the day. But when it’s 12 o’clock at night, and you’re finishing up a project, and you commit something inadvertently, it could be to a public GitHub repo, you can easily end up in this situation where you have secrets in GitHub, or something that’s publicly accessible. And this is one example. There are other examples.

But you’ll see that what we’re doing here is we’re actually defining, as code, user account creation in Windows, which is a whole other layer that we could talk about there in terms of authorization and password rotation, etc. But you will be surprised how many examples of this kind of deployment pattern we see at Chef. And it’s definitely a problem that folks across the board have to deal with an ongoing basis.

So, compliance-as-code. What exactly does this mean. So, you want to deploy change into your environment, but you also want to make sure that the change is secure, and that it adheres to your security policy. And that’s where Chef InSpec and the whole concept of compliance-as-code comes into play.

So, what Inspect does is it validates your YAML. It’ll validate the deployments that you’re delivering on an ongoing basis. So, it could be a container build, it could be a Kubernetes update, it could be a new virtual machine, and just really making sure that those configurations are secure. And it’s an extremely flexible language, we’re going to take a look at an example here. And it’s meant to be very legible.

So, basically, it’s bringing that functional and pragmatic level layer of security to all of the other things that you’re doing. And so, these validation checks, what they’ll do is make sure that you’re, for example, not deploying an S3 bucket into AWS and setting it to public. It abstracts away a lot of the complexity of these checks.

In this example, what we’re validating is that my Kubernetes deployment… or update isn’t being deployed with basic authentication turned on. Now, the way that this works, practically speaking, is you would run this check as part of your deployment. If the check fails, the deployment will stop at that point and back around it goes and the update gets fed to your developer or whoever is responsible for helping update those standards. And the goal of compliance-as-code. And this approach is really to make it legible.

So, you may not be a developer. You might be on a security team and know technical stuff, but you may not know Ruby or lower-level languages. The goal here is to make it as human readable as possible, so that somebody who might be in security or in a different team can come in and get up to speed and understand this kind of approach right away.

So, putting it all together, I also heard a talk about you bringing the teams together and making sure that they have a common language and are having conversations and it’s not just one team handing off to the other, but really a collaboration around all of these different all of these different deployments and change that you’re delivering. And really defining what you’re changing as a policy and defining exactly what’s kind of stack and what kind of update you want deployed into your environment, and providing that common language that you all have visibility into, and that you can agree upon.

And it’s not just one team handing off to another in a very binary step. And then you start to break down the silos. Because at the end of the day, when we talk about DevOps and DevSecOps, that’s really the promise of that whole approach is providing that ability to have the teams talk to one another and have that basically holistic approach to all of this.

So, what I’m going to do, I’m going to run through a Kubernetes example. This is an example that one of our very large customers uses and they run this on an ongoing basis. So, the example is really leveraging Chef InSpec to validate the changes that they’re delivering into their Kubernetes environments. And they do this at a number of different levels. So, what we’re going to talk about is essentially updating a container image. And then from left to right, how that deployment occurs and how we automate secrets management through this update workflow.

So, you have a developer. The person is delivering change into a newly built container image. And that is being validated by your Chef InSpec and compliance code down here in the bottom. Some of the things we might check in that scenario are make sure that the application user is not root. For example, just very pragmatic stuff. Let’s make sure that monitoring is turned on for the app. Make sure that the image isn’t built from another image that somebody just happened to download from online, and who knows what’s in it?

So, yeah, there’s a number of different things you can do there. And if any of those checks fail, making sure that that image does not get promoted into your image repository for containers, so Docker, or whatever kind of container runtime you’re using. Also, if you have a policy that you want to enforce around that container, that could potentially be part of the container image itself, you can also perform that kind of check.

So, the second step that we’re going to do in terms of enforcing secret management policy is making sure that, as we’re deploying, we heard a lot about it Zero Trust, and let’s talk about what Zero Trust means in cloud deployments or just programmatic deployments in general. Really, what we’re doing is making sure you have the minimal amount of access and authorization required, and also that it’s access that is short-lived.

So, as part of your deployment pipeline, typically, what we do is we’ll integrate with a solution like Akeyless. And then what we’ll also do from an InSpec perspective is validate that you have the correct secrets management configuration as part of your Kubernetes manifest. There’s a lot of different checks as well, that we can do there.

And so, we’re doing our deployment into Kubernetes, and formed an update. And then finally, on the right-hand side, this is where it comes together with our operations team. Because there’s two sides to this. So, there’s the team creating the change, but then there’s the team who’s on in an ongoing based way, responsible for making sure that your Kubernetes environments are up and running, and also that they’re secure.

That team will also integrate with authorization around secrets management to make sure that the right people are able to log in to view status, etc. And in conjunction with that, we also have Chef InSpec, where essentially, InSpec will sit and watch the Kubernetes configurations, the container images, making sure the right versions of different images are running, and also that the Kubernetes configuration that you’re deploying is still secure. And obviously, the checks that we run depend on how you’re running Kubernetes in this example.

So, you could be running it yourself on your own hosts. But if you’re using a cloud platform, typically what we’ll do is we integrate with that cloud platform API. So, the beauty of Kubernetes is obviously that a Kubernetes deployment is standard, no matter where you’re deploying to. So, that’s very flexible, but then how we’re accessing that information is really, you know, basically comes down to access into the different cloud layers.

So, when you bring that all together, and you’re codifying your deployments, you’re doing, obviously, your vulnerability scanning, for example, over here on the left-hand side, and you add in this really your policy as code and compliance-as-code approach, you really have a very powerful solution to ensure that your developers and the teams that are running your environments really are following the correct policies that they’re leveraging, for example, cert configuration, etc., that are required for your security policy.

So, I don’t know if any of you have heard of Kelsey Hightower. I live in Portland, Oregon. Kelsey is one of my neighbors in town, and he does a lot of talks for Google Cloud on Kubernetes. And he’s a Developer Advocate. But this really struck me as very relevant and poignant and funny because, yes, you can learn a container in a day with Kubernetes. Every time I think I understand all the different layers, there’s something new that comes along.

So, I think the point here is all of our environments are constantly evolving. We have to keep up with all of this in a codified manner. And the more automated and the more that we can smooth the way for all of our teams to be able to adhere to secrets management policies and security policies we’re putting in place, the easier it will be and the faster we can go.

So, really need to build security into your automated processes, preferably into your pipeline. Secrets management is most certainly a core component of DevOps. You really can’t have infrastructure as code without having a solid understanding and approach around secrets management. It’s really fundamental. And then finally, especially when it comes to multi-cloud, obviously when you combine that with infrastructure-as-code, it’s even more important. So, thank you very much, and enjoy the coffee.