KeyConf NYC Interviews – Secrets Management at Scale

David Spark: Conor, you presented at KeyConf today. Just 30 seconds just give us a quick summary of what you presented about.

Dr. C1onor Mancone: So, I was talking about our journey in secrets management. We’ve used a number of different providers for Secret Management Solutions in our company with a lot of different pains and issues that we’ve run into. And you know, how that got us to Akeyless, which we’ve now been using at Akeyless in a lot of our systems for about a year or so now, and just what that process has looked like.

DS: That is the discussion I want to have with you actually.

CM: Okay.

DS: So, you join Cimpress a year and a half ago and what was the situation when you walked in in terms of Secrets Management using other solutions and what was falling short?

CM: So, yeah, first of all, I think it helps to know that Cimpress overall is very hands off for their development teams. We let them use whatever they want to use as long as they meet certain basic requirements. And so, we as a security team provide services to them and an attempt to be helpful but nobody actually has to use them. So, we have as a result just about every technology possible somewhere inside Cimpress.

In terms of our services, we have provided secret service solutions to our teams. The one that we had in place when I first showed up was HashiCorp Vault, and we were having some issues with it. You know, just like the usual technical infrastructure management stuff. Our application was broken. Some teams were using it, not a lot of teams necessarily were. We were trying to get people on-boarded it to kind of manage their secrets and like anything else, there was different features that we needed that we didn’t really have available and just that all the issues with infrastructure management that come with managing a system like that.

DS: Okay and so, these were the issues that were concerning you at the time. What were you looking for then when you were trying to find a new solution? Like we can’t deal with this anymore because it’s just making the problem worse. I don’t know, I’m putting words into your mouth, tell me what the situation was and how you went out looking for a new solution.

CM: Yeah, although I can’t necessarily, I’m not going to take credit for that myself because it was more like our CSO Ian Ahmed just saying, hey, try out the solution and let me know if it works well for you. But I think the thing probably one of the big motivating reasons for him was especially a good software service that provided the features need. We are a small team and our previous Secret Management Solution was the only one we had that had a 24/ 7 on-call rotation. We had a lot of engineering time spent trying to manage our cluster, make sure everything was healthy, make sure everything was always working. And it was consuming a lot of time from our small team. So, I think SaaS Secret Management Solution was very attractive and that as we went through the PoC with them just realizing it had a lot of the features that we really needed in the network currently.

DS: So, let me pause you right there. What were the features you needed?

CM: So, one to pick some specific examples. I’m a big proponent of good solutions to the secret zero problem. You have time for me to go into little technical skills. Sure. So, the idea is you know if you put all of your stuff in your secret management solution but then you’ve just got a permanent access credential to get into your secret manager like, are you really making your things any better. You’re putting your eggs in one basket and then you accidentally leave the key, sticking in your door locked. I’m not going to help anybody. So, that’s the secret zero problem isn’t getting into your secret manager very securely.

And so, for our on-prem infrastructure. We’ve got prints, print warehouses and whatnot. It was a problem because there wasn’t any good secret zero solution for them with our previous provider. And so, basically, they were just creating a permanent access token, sticking it in their deployment machine in their warehouses, in their data centers, and generating more permanent access tokens. And they had to manually rotate those tokens once every 90 days assuming they remember to, or remember to rotate them if an employee left. It was a very manual process and again…

DS: Any manual process you’re going to have fail points.

CM: Exactly.

DS: And again, as good as everybody’s intentions are too, as well.

CM: And those teams really well intention, so I think they probably did a pretty good job. But if it’s manual and it was a bit of a pain like it’s not going to go well or it’s going to hinder adoption which means people won’t be using it in the first place. So, yeah, so that was one specific example and Akeyless solved that Universal Identity. It’s their idea of an access token that’s never meant to be permanent. It just automatically rotates. You can make it rotate as often as you wanted to and so your access tokens are never permanent. It basically was designed to solve Secret Zero for on prem infrastructure which one of our problems. So, that was great.

We also had some practical issues with some of the other systems that did have select for our cloud infrastructure. You would normally use IAM Auth to get into your secret manager. Basically, turn your cloud provider into like a trusted third-party identity provider. But again, hours in order to use IAM Auth it required kind of mutual cross account permissions between your different cloud accounts. And if you’ve only got a few that’s not a big deal, but we’ve got roughly 500 cloud accounts. Managing all of those permission between all of them wasn’t something anybody wanted to do.

So, we basically just didn’t have any adoption from our cloud teams. Because if they wanted to use it, they were going to be just storing permanent access credentials which again what’s the point. So, stuff along those lines and I guess just to pick one more this was a very specific issue is we wanted to use database dynamic credentials. Our past secret manager offered that, that way we wouldn’t need permanent access credentials to get into databases. But it didn’t work in practice because the secret manager needed access to your database network access. So, like if you’re all in one data center not a big deal, but we’ve got hundreds of databases and hundreds of networks. So, we would need to set a VPC pairing between all of them. It was going to happen. It’s not even possible. So, we just never use that feature.

So, basically, we had all of these good features available in our secret manager that we wanted to use but for various technical reasons, we couldn’t actually use them in our organization.  So, we weren’t really getting what we wanted out of our solution.

DS: So, from hearing correctly, there were products out there that were well intentioned to but I guess maybe they weren’t listen to customer environments to understand, it’s just not going to work in the way it’s architected.

CM: Yeah, I think that’s exactly.

DS: And that’s fortunate for you. So let me ask the forward-thinking question. I mean nothing, zero trust identity is always sort of a journey. What what’s the next step in the journey of secrets management or zero trust or identity you’d like to see that you currently don’t have now.

CM: So, I think for me, it’s not necessarily a huge new big step either. It’s just kind of making sure you’ve got those secret zero solutions everywhere. Because bringing our secret manager into play for our services really helps a lot. We can use temporary access credentials just about everywhere, but that’s the tricky part. Making sure that all of the services and all of the people and all the infrastructure trying to consume the Secret Manager have a good secure way to get in there with temporary credentials is important. And it’s impossible to get 100% coverage everywhere although Akeyless has done a pretty good job of it.

And then as much as possible bringing temporary credential access methods to other systems is kind of the next big step. There’s a lot of third-party services we use, where there just isn’t an option for dynamically generating credentials or dynamically managing credentials. So, in those cases, we have no choice but to use static credentials. So, I think just you know continued community support for like hey this is the way we want to operate. Let’s give our users better control over their own access credentials and just making sure services everywhere can support that is going to be really not a new step, but just the continued step in that.

DS: Do you think maybe the reason that’s taking just a little bit of, there’s a little bit of a hurdle right there is that it takes time for the users to realize this is the new method?

CM: I think that can definitely just general inertia for sure. It’s just that users and teams and services realizing, hey there’s really a better way to do this. But again, it makes us more secure and it really makes life easier. Because it’s a lot easier when you just got temperate access credentials everywhere and you got automated access systems in place. So yeah, just getting people, help people to understand different ways of doing it, better ways of doing it, and getting everybody on board. It’s definitely a journey.

DS: Thank you very much, Conor.