Secrets Management at Enterprise Scale

Dr. Conor Mancone, Lead Application Security Engineer, Cimpress

Data scientist turned software engineer turned security specialist. Apparently I just like all-the-things.

Hello, everyone. I’m really happy to be here today and talking with all of you. My name is Conor Mancone. I’m with the Application Security team at Cimpress. And I’m here to talk to you about Secrets Management at Enterprise Scale. So, I do have to do a real quick shout out because my whole team couldn’t come today. I’ve got some great co-workers, Dan Fabbo, Miguel Fernandez, Chris Tom, and Pablo de la Concepción, who helped made our secret management service great. So, I just wanted to give a shout out to them since they’re not all here.

So, I’m going to talk a little bit about us and our core philosophies, because that’s going to help you understand the problems that we were facing, and so why we gravitated to the solution we did. And then kind of walk you through a little bit of the results of all of that.

So, jumping right in, Cimpress is not necessarily well-known name. But for you in the US, you’re probably more familiar with the Vistaprint brand name. We’ve got 13 main subsidiaries. Vistaprint is one of the largest. They’ve been handing out free business cards to get small business owners hooked forever. So, there’s a good chance that some of you have some of those business cards somewhere one of these days. But we’ve got about 13 subsidiaries total in most of the continents. I did have a proposal for a polar expedition to get some employees of ours on Antarctica, but it didn’t pass. So, we don’t have any employees there, but we have people on every other continent on the globe. And so, that’s Cimpress.

I’m part of simpler security. We’re the primary security team in Cimpress. And in comparison to the 12,000 employees in Cimpress, there’s about 30 of us plus or minus 2. So that alone tells you a lot about how we operate, because we’re obviously not a centralized security team. People don’t come to me to look at their merge request. They don’t need our approval to do deployments, because there’s not enough of us to do that. And that’s kind of our goal. We want to keep that security knowledge and our engineering teams. So, we’re an MSSP, a Managed Security Service Provider. So, we find services that our teams need. We purchase them. We manage that relationship with the vendor, and we provide those services to our teams.

So, really, our core philosophy, our main one is to be a lean, mean fighting machine. That’s who we want to be. That’s what we want to do. So, we go for those really high-value services. And we curate. We path find we publish. We let people know, “Hey, this is how you’re trying to work. These are the problems you’re having. And here’s a solution that’ll help for you. Here’s some documentation to make it easy.” Our goal is basically enablers, to make life nice and easy for engineering teams, so they don’t have to think too hard to get security done right.

Another very important core philosophy we have as part of that is secrets. We got a lot of rules around secrets management. Secret management has always been one of the services that we’ve provided. It’s really critical. Obviously, you guys know that. You’re here. Because compromised credentials are a leading cause of malicious data breaches. And in case you’re unfamiliar, another term for malicious data breach is expensive data breach. These are the things that we want to avoid. And to avoid it, we have to manage our secrets.

And so, we’ve got rules about secret rotation. Everything has to be rotated within 90 days. But even that, although we prefer to avoid as much as possible, I like pushing people towards those temporary credentials. You don’t have to worry about secret rotation. You don’t even have to worry as much about your credentials if they’re going to expire and be useless in an hour anyway. So, as much as possible, we like to push people towards solutions that make secrets nice and temporary.

So, along those lines, like I said, we’ve been doing a secret management service for a while. We’ve, for a long while. If you can think of sort of an enterprise name secrets management brand out there, we’ve tried it. We’ve POC-ed it. We’ve probably had it in production. We’ve used just about everything. And we’ve basically always had the same set of challenges, which is what has brought us to here today.

So, to go into some specifics, usability, usability is always a key concern. To quote somebody else, “Security at the expense of usability comes at the expense of security.” If it’s not easy to use, people are just not going to use it. It turns out that’s how it works more often than not. So, you got to make sure your stuff is easy to use. And that hasn’t historically been the case. With our last provider we were using, I would get a little bit dread, if you want me to be honest, every time we had somebody requesting onboarding. Because I knew that week, I was going to be locked up for 3 or 4 hours in meetings, helping them understand how it works. It did not matter how much documentation we wrote, how many examples we gave, people were going to get confused because a lot of, especially the role-based access control was just not intuitive. And it was painful, if you want me to be honest.

Missing functionality was… and just kind of suboptimal feature set was a problem for us. And they’d often forced us like down directions we didn’t want to go. It’s kind of a strong statement. So, to be more specific network isolation, we wanted to use dynamic database credentials, which our secret provider, our old ones supported. It will create a temporary database user for you to use, which is great. Because, again, you got that whole, those 3 Js that have been talked about all day, your credentials, they’re just given to you when you need them and they expire automatically, so you only use them once. That’s what we wanted for our database access. But our secret manager required network access to the database that it was trying to generate credentials for.

And the problem with that was some of our companies are in that Zero Trust world where we’ve got hundreds of databases, they’re each in their own individual network. So, for me to use dynamic database credentials with my old secret manager meant that I had to set up VPC peering between every single one of these networks then the network of our secret management provider. And again, we’ve got hundreds and hundreds of these systems. Like, it’s not possible to set up that level of VPC peering. I don’t think our subnet IP space would support it. We just couldn’t use this feature that we wanted. It was really unfortunate. We

have a lot of on-prem infrastructure. We’re a print company. And so, we were always going to have some on-prem infrastructure. We didn’t have any good solutions for kind of a secret zero problem for those. So, when people wanted to get their print machines hooked up to our old secret management solution, they just created a permanent access credential, and they’d stick it in all of their machines in their print warehouse, and then they had to worry about are they going to actually remember to rotate that after 90 days? Are they going to remember to rotate those credentials if somebody leaves? It really created more problems than assault.

And again, we had issues with cross-account trust on the other end. For our cloud infrastructure, we want to use IAM Auth to let our cloud services login to our secret manager. And our secret manager had IAM Auth to do it. Basically, you make your cloud provider kind of a trusted third-party identity provider. It’s really nice and slick. But there was this assumption built into our secret manager that the secret manager and your infrastructure would all live in the same cloud account. And if they did it, you had to set up mutual cross-account permissions between your secret manager and all the other accounts you own. I mean, we’ve got like 500-ish cloud accounts. Like that’s not a small thing that have to configure in order to use IAM Auth to get into your secret manager. And so, our cloud teams just didn’t use our service. Again, it was all very unfortunate.

And of course, it was expensive. More than just licensing costs, wasn’t even the biggest cost for us. The cost of it was these were on-prem solutions. So, we needed to spin up lots of infrastructure. We had to have a team managing that. We had to set up good monitoring, so that we knew if something was broken. We had an on-call rotation. Again, if we were a team of like 100, maybe this wouldn’t be a big deal. But for us, we’ve got that kind of lean, mean fighting machine approach to like just not interested. Or at least it was expensive, and we didn’t like it.

So, we needed something better. We needed something that worked better. We needed to be easy to use. We needed to have the functionality. We needed to bring good security. And I didn’t want to have to be managing infrastructure.

So, plot twist, we ended up with Akeyless. I’m guessing nobody realized I was going to say that. And it’s a SaaS. So, that automatically fixed a lot of these problems. We don’t have a 24/7 on-call rotation anymore. That was actually the only service… secrets management was the only service that we had a 24/7 on-call rotation for. So, now we just don’t have one. I can’t tell you how many people that made happy. It made me happy. It made my wife happy. It made my boss happy. Everybody’s happy. Don’t have to worry about upgrades or maintenance, all that stuff. Immediate win right there.

And of course, it had the features we needed. For those databases where we want to use temporary credentials, now we can deploy an API gateway. And that gives us like a Bastion for the secret manager into local network. So, we don’t have to worry about VPC peering, which was never going to happen anyway. IAM Auth just works as normal cross-account permissions, so we can use for the first time ever, we can basically use it well for our cloud infrastructure. And for on-prem service, we got universal identity. Oded mentioned that in passing in his last talk. It’s just this continuously rotating token. You can have your identity token rotate once a minute if you want to. So, you don’t have to worry about it getting stolen anywhere near as much. And so, now we’ve got a whole print centers, where there’s just not a single static credential anywhere. They just all automatically rotate itself. It’s beautiful.

So, it was really what we needed. But like anything else, why stop there? We fixed the problems that we knew we wanted to solve, but it’s even better if you get new problems solved too that you didn’t even know about. And we did. This whole flexible segregation model was great. So, with our old service, since I was the admin of our secret management solution, I inherently had access to any secrets anybody put in there. And I mean, I have a very trustworthy looking face, I’m sure as many of you can contest, so I feel like that should be fine. But some of our teams were a little bit iffy about that. So, it’s nice that now we’ve got that the DFC technology. They can make an API gateway. They can put a key on it. And as long as I don’t have access to the API gateway, I can’t decrypt their secrets anymore. They get to hide their secrets even from me if they want to.

But at the same time, it’s even easier to share it amongst teams if they want to. The role-based access control, multi-tenancy models a lot more fluid. So, we can kind of break down those walls when we want to. Teams can share folders amongst each other when they want to, even across tenants. I can pre-configure access methods and other configuration to share with teams, so they get up and running quicker. And we’ve got this fun new concept of ad hoc collaboration where it’s really easy to be like, “Hey, I want to give you access to this secret right here,” and so, teams can do that when it’s convenient for them. And it’s kind of led to this new model of Akeyless really becoming kind of a centralized collaborative for all of us. Like, I, as a service owner, stick some of my secrets in Akeyless, and then I give that person access so they can put their own related things that I need in the Akeyless as well. And then my service can log into Akeyless, and it gets everything it needs in one place.

So, it’s nice and simple. And we’ve shared that. We cut down on a lot of the other methods of secret sharing and collaboration where like, not that anybody would do this, but teams emailing each other secrets when they’re trying to collaborate or Slacking secrets. I’m sure that never happened, but we don’t have to worry about it now. So, life is much better.

And you can see that in the results. Basically, we we’ve been with Akeyless actually now it’s about a year. And in that time, we’ve more than doubled basically any metric you choose to measure this by the usage of our secret service. And I want to emphasize here that we don’t have any mandates. You don’t have to use our service inside this company. As long as you are following the policies, you’re fine. We have this year for people who want it. So, the fact that we’re getting increased adoption, dramatically increased adoption means that people really liked the service. They find it easier. They find it better. They want to use it. And they’re not we’re not making the music.

And this lowered costs. Previously, again, most of our cost came from not just like… not licensing, but a lot of infrastructure, but just the engineering time of managing the solution, monitoring, support, etc. And so, now that we have increased usage, the licensing is very comparable in terms of cost. But now we have far less infrastructure to manage. We spent a lot less time managing it. Really, the only infrastructure is we pre-deploy API gateways for people to make their lives easier. We save so much money, we figured we would put that money somewhere. And so, we’ve saved a ton of money. Don’t tell Oded this. I don’t want him raising the rates. But it’s been great for us. And so, yeah, we got more functionality and usability out of this thing. We have a higher adoption rate. We spend less time maintaining it. And our systems are more secure. So really, what else can you ask for? Thank you.