DevSec For Scale Podcast Ep 3: Improving Software Supply Chain Trust

Supply chains are found everywhere in our world from food to clothing to school supplies. For each item that is built or created there is a chain of suppliers for each piece of the final item. The famous example is the pencil. Something as simple as a pencil actually takes numerous suppliers of graphite, wood, and other raw materials are required just to get the process started.

The same idea applies to software. More often than not, software companies use tools, products, libraries, and packages built by other developers. For various reasons, it also just makes sense to buy those (or use them, in the case of open source) rather than build them yourself.

With that, we have a software supply chain. And, as with other products, where companies have very detailed records (i.e. bill of materials) with information about all of their suppliers and materials they provide, a software company should have the same thing. However, they usually don’t.

In this episode, Barak Brudo, Developer Relations Advocate at Scribe Security, gives us insight into the importance of a software bill of materials (SBOM). With this SBOM, companies can easily track all changes in their supply chain and quickly detect malicious activity.

Watch the episode below

TRANSCRIPT

Jeremy Hess: I’d like to welcome fellow developer advocate Barak Brudo. He works for a really cool startup called Scribe Security, and I’m really happy that Barak is joining us today because, as you know, we usually talk with CTOs, CISOs, things like that, but this time we get to speak to someone who has a little bit more experience from a developer level, and so we get to hear a little bit more detail and a little more about what it means to deal with security from the developer side.

So, Barak really happy to have you, and before we get into you, talking a little bit about yourself and about the company, can you give us just a little bit of information of… you know we’re here to talk about integrity and software supply chain security. So, can you give us a little bit of a breakdown you know, in your mind, just something quick about what does that mean to secure a software supply chain and what type of integrity are we referring to here?

Barak Brudo: Well, the main problem with the software supply chain, as a lot of people know, is that when you’re using a library, a tool, a piece of code that somebody else wrote, you’re basically getting a black box, in most cases. You’re getting the code. You have no idea what’s in it, and there’s a good chance that that code or tool that you’re using is also using other libraries, tools, and code that other people have developed, and so on and so forth.

A famous case like SolarWinds showed us that it’s enough to make a tiny change in the code somewhere down the line, and then unsuspecting people by the thousands can eventually get impacted. So, the integrity that we’re talking about here is just that to make sure that code, be it your original code or open-source packages, has not been tampered with, has not been changed from the original pristine condition. That’s essentially it.

Jeremy Hess: Fantastic! I like that. Concise, clear. And so, let’s talk a little bit about your background, because you’ve only started working for Scribe Security for just a few months now, as far as I understand. So, can you give us a little bit about where you came from, what brought you to Scribe, and a little bit about what Scribe is trying to do in terms of, you know, integrity for the security of software supply chains?

Barak Brudo: Sure, I have a very eclectic background. I didn’t always know what I wanted to do. After the army, I started as a programmer. I worked for a few years, but it was very boring. So, I made a complete direction change. I went, and I studied education and art. I’m actually an art teacher by degree. But, after I got my degree, I discovered, you know, to the surprise of no one, that being a teacher in Israel is not that easy and definitely not as rewarding.

So, I worked in a lot of other related fields to art. I was a photographer assistant, I designed websites, I worked as a graphic designer, and an ad company and then, obviously, because you know, I like to be able to eat and afford rent, I went back to development, and I worked for quite a number of years as a developer in various fields. I actually came to Scribe for an interview as a front-end developer, and they saw that I was, you know, relatively articulate and presentable, I guess. So, they offered me this job in a new field which I’d never heard of until they offered it, and here I am, yeah.

Jeremy Hess: That’s great, fantastic. Well, we know each other as part of the developer advocate crew in the Israel region, so that’s great. So, let’s talk a little bit about SBOMs. What is it that an SBOM is and explain that a little bit to us, what that means and sort of how it was done previously and how it’s changing?

Barak Brudo: An SBOM – a software bill of materials. It is just like any other bill of materials that you can think of. The easiest parallel that I usually use to draw to is, if you go to the supermarket, you look at a packet of some sort of food. You have no idea what’s in there because the packet is closed. But if you turn it around, you can see all the ingredients. Sometimes even down to the chemical composition.

So, you know exactly what’s in it. You know if there’s anything in it that you’re allergic to, you know if there’s anything in it that you might object to on a conscious level, or some other reason you might, you know, object to the country was manufactured or some other sort of problem. So, a software bill of materials is basically the same thing. Except it does the same breakdown to software, product, or artifact. It breaks down that product down to the file level, and it gives you a bunch of other information.

Other than just “these are all the files that are in the product,” it gives you added value like the relationship between those files, what is a dependency, what was created as a part of a test or as a different runtime process from a different file or library. It breaks down unique identifiers. The system on which that software was developed. And it includes unique identifiers to files or libraries like file hashes.

Now, all that information can be used for a lot of different things. The easiest one is, again, like I said in the beginning, to check if you’re allergic or if you have a problem to any of the ingredients in that code. SBOMs were actually started to develop in 2014. That’s a while now. But they were quite obscure.

They were, you know, in use in some parts of the industry, but they were not broadly known. A couple of years ago, I think in 2018 or 2019, CISA, which is the cyber security and infrastructure security agency in US, started to really push the SBOM idea as a way to create a standard that everybody in the software industry can get behind. To create something to increase the transparency without only the software supply chain but of, you know, stuff like open source that everybody is using on a regular basis to stop the hiding of what it is you’re actually using or how.

So that if there’s a problem, like recently log4j, nobody would just be hiding it… this is the product, and you know, I don’t even know if it’s in it. With an SBOM, you can just check.

There are still various problems with this. Especially if you’re trying to break down a product and you just don’t know what constitutes a certain piece, and some people are still selling products with a license agreement that prohibits you from checking under the hood. So, that’s something that needs to be dealt with, and again there are other problems. Like, if you’re thinking of software these days, a lot of it might be on the cloud and completely distributed. There’s not even a single product. It’s just a bunch of lambdas and processes that are running serverless everywhere.

So, calling all that collection of products and breaking it down to the fine level is slightly more complicated. But in a sense, trying to keep it simple. Having an SBOM is good for you, the developer, because it helps you discover problems as soon as they come up. You don’t have to spend hours digging through legacy code to see if you actually have the problem, and it’s doubly good for the consumer because you know exactly what you’re getting, and if there’s a problem, again, you can easily find it.

I spoke a couple of weeks ago to a security researcher in the cyber field about the log4j problem, and they were describing it, describing the solution, how they were able to find if a certain company was actually impacted by the problem, and he told me, not to me personally, but to the crowd listening, that one of the main things they were dealing with is trying to figure out if a certain company was actually using log4j and, if indeed so, what version. Finding that out was the biggest problem. If those companies had that SBOM, all they had to do was run a search on JSON or an XML file, that’s it. You know exactly where it is. You know exactly what version. Makes things a lot simpler.

Jeremy Hess: Well, I feel like those things do change quickly, especially in, you know, DevOps and things like that where, you know, you’re constantly updating packages – things are always changing. So, how does an SBOM keep up with those changes, those consistent changes?

Barak Brudo: Exactly like that, an SBOM is not a static thing. It’s not something that happens, and then you forget about it. An SBOM, a proper one, is attached to your build process just like any other test in a CI/CD automation, and every time there’s an update, every time you update your dependencies, you change a file, you change anything – the SBOM is recreated with the exact build. And obviously, if you’re conscientious about security and about documentation, you’ll keep all those other SBOMs.

So, you have an exact, let’s call it, evidence trail of exactly what changed, exactly what happened. So, if, heavens forbid, something breaks, you can trace it down. You can do a simple file comparison and see exactly what changed between your previous product and your current one. See where the problem was.

Now, I did mention, or you had mentioned, that this is getting more and more attention, and that’s true. Essentially, because last year in May 2021, the Biden administration released an executive order – number 14028. Its topic, its major topic, is improving the nation’s cybersecurity. There’s an entire section there, section four, that deals with the security of the software supply chain and cybersecurity in general, and it mentions SBOM by name. It told NIST to build regulations that are based on this executive order. They already did that. It’s going to become basically a part of all new contracts in the US in May this year, really close.

And again, SBOM is mentioned by name, so obviously, a lot of companies, especially if they’re working with US government, are starting to take notice this is getting more and more attention there are more and more webinars and talks about this. And like I said, CISA is definitely pushing this for the betterment of everybody, I believe. Because even if you don’t remember anything else from this short podcast, if you remember an SBOM and you want to incorporate it into your product, that’s just great. I did my job.

Jeremy Hess: Well, what would you say the level of, you know, let’s say percentage in your mind of, let’s take smaller companies as an example, right. We’re more of a startup type of podcast. So, like in terms of startups that you’ve seen in your time so far, do you think that SBOMs are relatively common, or are they still pretty rare?

Barak Brudo: Unfortunately, they’re still pretty rare. I’ve done a lot of reading and listening, obviously, but I’m looking for stuff about SBOMs because that’s part of my job. Last week we were both at the cyber tech conference – lots of security companies, lots of interested parties who are there to shop for security products of various kinds. And one of my first questions to anybody who came to our group is “Do you know what an SBOM is?” After three days of the conference, I found three people who initially knew what an SBOM is. I had to explain it to everyone else. I’m sure it’s going to become a lot more common, especially with the changing regulation.

But for now, it is unfortunately still a relatively unknown piece of, you know, security idea, which is a shame because it’s so easy to use. Even if you don’t do anything with that SBOM, even if you don’t externalize it. You don’t send it to your clients. You don’t send it to anybody. Just having a simple piece of code in your pipeline to create that SBOM for you, for your own documentation, so you know exactly what’s in your product – even if you’re just doing that – you already are head and shoulders above a lot of other people.

Jeremy Hess: Well, There’s a question there then. I mean, this is exactly what we’re trying to do. You know, with this podcast. We’re trying to help companies, especially startups, young companies who look at security as a bit of a burden, trying to help them figure out ways to take basic steps in security and securing their code to, you know, without having developers, you know, have too much friction with their work, right? Don’t take their lifecycle. You know, their developer cycles, you know, away from them, right? They’re focused on making a product, right?

Now, security is super important. Everyone knows that. But it’s usually thought of last. So, what would you say would be a quick way to have an SBOM, you know, created and managed simply by developers?

I mean, is this also, and is it more front and back-end developers that need to deal with this, or is this something that’s more DevOps type of, you know, action?

Barak Brudo: It’s more of a DevOps thing, definitely. For a single developer there wouldn’t be that much benefit unless he or she wants to also start looking at the individual packages and libraries that he or she is incorporating into their code. But when you’re looking at the entire product, or you know, usually it would have, if you were talking about a big one, multiple pipelines, multiple images, some of them might be then internalized and used in a different pipeline. For that sort of a scenario, or any sort of scenario where you actually do have a CI/CD pipeline – incorporating an SBOM build at the very end, right when you’re actually building that image before you’re sending it off to where whatever registry you’re sending it to, that’s like the perfect time. Like I said, even if you don’t want to do anything with that SBOM.

Just generating it and keeping it, so you have it, you know exactly what you’re using, you’re already better than a lot of other places. And as the industry improves and technology improves, there are going to be machine-readable SBOMs that you can automatically send between customers, between vendor and supplier. You can immediately check when you’re receiving a product the SBOM attached to it. If it includes anything that is problematic to you, and again, that’s not even covering the Scribe product.

This is just generally including an SBOM with your product. And because, like I said, there are even open-source code products that are used to create those SBOMs. You don’t even have to use anything that costs money or incorporate any sort of complex customization. Just go to any of the three major SBOM standards, which are CycloneDX, SPDX, or SWID. They have thriving forums and communities that are currently working to improve them. They also have a bunch of tools which are published on those places. Some of them are open-source, some of them are not. Ours is, you know, currently not open source, but it is free. So, you know, you get the best of both worlds.

So yeah, even if you’re not doing anything with that SBOM except, you know, having it for your own peace of mind, that’s great. Doing more with it… if you’re looking at the Scribe solution, we’re using that SBOM as a way to guarantee integrity. There are two places that we’re checking integrity. One, we’re creating an SBOM in the SCM (supply chain management) – your repo, and then we’re creating an SBOM in the build.

We’re basically putting it on both ends of the CI/CD pipeline. Any person who’s worked in code knows that there’s almost no reason for most files to change between your SCM and your build. If something changed, something might be problematic with your CI/CD… might be an indication of a tamper of some sort, from somewhere. So that’s one thing.

And the other really big thing is that currently, we’re aggregating open-source information and package management information, currently from npm, but eventually, we’ll go over all of them, so that we can guarantee that whatever library open-source package you’re using came from the correct place. What I mean by that is, that sometimes there are mirrors or men in the middle attacks, and again, we’re looking for tampering. We’re looking for a different code than the original.

If we find something like that, it usually means something bad. And one of the problems you didn’t mention for a lot of smaller companies and developers is fatigue from way too many alerts. If you’ve ever worked with npm, with Node, a lot of times when you like, let’s say you clone a repo from Github and you’re trying to do ‘npmi’, and then ‘npm start’ – you get a huge list of CVEs.

Most people, me included, if I know it’s something that I’m going to use anyway, I’m just ignoring it. Yeah, I’ll try to patch, you know, the basic things if I can, but going into the code looking for what might break if I change something… too much of a problem. I’m trying to trust the people who designed this to do that for me.

So, getting all those notifications and warnings… way too much. Most people just don’t even look at it anymore. So, what we’re trying to do is create something simple. We do the comparison, we check for the integrity, we give you a report, with a simple headline that says out of that many files in your repo, that many files are fine, haven’t changed, that you know basically like one-two files or no files were changed in the interim. Out of that many files or packages in the open-source that you’re using that many files and packages are obtained that many files are not. You can easily see that there’s a big number at the top of the report.

So, you don’t have to go digging into that SBOM to see if there are any specific problems. You could keep it, though. You should keep it. You should even sign it cryptographically so that it’s, you know, immutable, and then if there’s ever a problem that comes up like colors or log4j, which wasn’t a problem until it was, just go over that SBOM and you have it figured out in a minute.

Jeremy Hess: That’s great! Yeah. And so, this obviously is an attack vector that’s not, that hasn’t been previously thought of too much, right? it’s been something that people have sort of left to the wayside, I mean.

Barak Brudo: One of the reasons that Scribe decided to go after this route is because of the whole executive order thing that made it. The executive order was released, I think, in part, following the Solarwinds incident, which was huge for the US government. There were nine different major branches, including the Pentagon and the DoD, which were attacked or compromised, not really attacked, through SolarWinds.

Jeremy Hess: They used SolarWinds software.

Barak Brudo: Yep! And that means there was a back door into their system, and the attacker supposedly Russians, but who can tell definitively, got into their emails into various other information. I’m not going to go into it so much because it was covered to death in a lot of other places.

Jeremy Hess: Yeah! Absolutely. We did one as well.

Barak Brudo: So essentially, having any sort of the change in the code is considered a bad thing. But there are a lot of other companies that are checking for a lot of other very important things like secrets or CVEs or code problems of a different sort, like problems in loops or problems in different names for variables.

There are a lot of other products out there, and the fact that you’re checking for integrity doesn’t mean that you’re forgetting everything else. You should still have, you know, your build system or your build environment separate from everything else, if possible, ephemeral, you know, if you’re really tight on security. You should have two FA or three if they for everybody who’s going into the system. You should also check your code for secrets and for other CVS or problems.

But the problem of a code changing is such that it’s so easy to miss. Even, you know, even if you’re looking at SolarWinds. The code was fine when it got to the build server, like two or three lines were added. Even people who knew the code looked at it later and couldn’t definitively say that they didn’t put them there themselves. So having a way to definitively and easily compare files any files of whatever language whatever system to make sure that they weren’t tampered with, I think that’s a good thing.

Jeremy Hess: Absolutely. Yeah. Integrity is pretty much everything when it comes to your code. I mean…

Barak Brudo: It’s not everything, it’s just one of the big things.

Jeremy Hess: Well, definitely one of the most important things. I mean, if you’re not, you know, ensuring that your code is the same here and there you know in both places, then you’re definitely, you know, you could potentially be up a creek, right?

So, what I like to do for the last question that I asked in these interviews, and you kind of covered it a little bit in your previous answer was. If you could give one or two tips on, you know, in general for developer security, and it doesn’t necessarily have to be related to, you know, what you do or what the company does, right? But in general, if you had one or two tips for developers if they’re focusing of security and they work for start-ups. So, you know, what are in your eyes, you know, the first one or two things that developers should at least be aware of and try to implement? You know, and it wouldn’t make it difficult, you know, wouldn’t be a burden on their work.

Barak Brudo: Well, one of the tips that I found really useful, but that depends if you have somebody who can actually do the job, is to check all the files, tools, libraries that the company is going to use and create a list a separate list of the approved files, versions, libraries, open-source packages everything like that so you don’t leave it down to the developers to pick whatever it is that strikes their fancy at the time.

I was a developer. So, I know that sometimes I would get a task or a problem, and I would just start looking at npm, in my case because I was working with JavaScript. I was just looking at npm and looking at packages.

Yes, I did do my supposed due diligence. I checked that there were stars and there were not too many problems, and there were enough downloads of the package, and it looked okay. But then I would just download it and start messing with it. If it didn’t work, fine, I’ll remove it or something else. But there are instances that just downloading that package just by doing that, you’re already infected with something. There’s already some sort of code that ran on your machine on your you know Node.js or your anything like that, you’re already infected, you’ve already compromised anything you do from then on is a problem.

So if there’s actually somebody in the organization, a CISO or a DevOps, somebody who knows what they’re doing and can create that curated list, can save a lot of problems later on. Another good tip is to set version numbers – as in, don’t just use a certain library and let npm, and again I’m using that as an example because I was a JavaScript developer, don’t let npm automatically update it because the recent case of colors was that the actual developer decided that he didn’t want to contribute open source anymore. He created an endless loop, an infinite loop in the library, created a new version, it was automatically pushed to everybody who was using it who didn’t set their version, and it broke a lot of things. And it was completely… we wouldn’t, Scribe wouldn’t have caught it because it was a legitimate upload from a legitimate source that was the development of the product.

We can’t catch malicious developers. That’s still we still don’t know how to do that, but if you know that you’re going to be using a certain library. Set the version number and only change it if you are absolutely certain that this is something that you want to do. Those are the two main tips other than obviously using an SBOM and asking, you know, other people that you’re working with to do the same. So that the more transparency happens across the entire supply chain, the better all of us would be.

Jeremy Hess: Absolutely! Wonderful. Thanks so much for those tips. And Barak, it was really great to have you for this episode. And before we sign off, do you have any, I think you said it, or we talked earlier, and you have an event coming up pretty soon. Do you want to quickly let everybody know how they could access it?

Barak Brudo: Sure! All they have to do is go to the scribesecurity.com website and go to resources. There are currently webinars. There’s only one because this is our first one and my first one too. It’s going to be taking place on the 31st of this month, so about three weeks from now pm Israel time, 12pm EDT if I remember correctly, and we’re going to be covering the new regulations that I mentioned. We’re going to be covering an idea that is called continuous assurance connected to CI/CD but something slightly different, and we’re going to show our new product, and you know, demonstrate how it works, which hopefully will show that it’s not that scary that some people might fear, and it could be really fun to play with it.

Jeremy Hess: Fantastic! All right. Well, we’ll be sure to make it to that event and really looking forward to it. Thank you so much for being one of our first guests on the DevSec for Scale podcast. Really happy to have you, and we look forward to having you on later when we hear more updates about how you know Scribe Security is moving in the world. So, thank you very much for your time, and have a really great day!

Barak Brudo: You too!