Posted by Joyce Ling
November 22, 2024
It’s hard to imagine software development or operations without secrets. From the first commit through the first build pipeline, up to deployment and beyond, there are numerous secrets—credentials, keys, etc.—that must remain private. Unfortunately, it is impossibly difficult to keep something absolutely secret forever. That is just one of the reasons behind secrets rotation, one of the most important security measures available to DevSecOps practitioners.
This post will explore why rotating secrets is so critical, the need for automated rotation, and how to effectively perform secrets rotation.
Why is it so important to rotate secrets?
As outlined in all modern security guideline frameworks, secrets need to be regularly rotated. The reason for this is that the longer a credential exists, the longer it might be used against you when exposed.
Moreover, the longer the lifespan of a secret, the longer its chain of custody. Each link in that chain adds even more risk that one of the parties the secret was disclosed to will eventually spill it—willingly, or not.
Let’s take a look at a couple of real-world examples.
Uber
The Uber incident from December 2022, when a mobile device management (MDM) was breached, perfectly shows the above principle in action. The attacker managed to gain access to a backup server hosted on AWS. Using hard-coded, unrotated credentials stored in PowerShell scripts, they then leaked the work-related and Microsoft Active Directory information of more than 77,000 Uber employees on a popular hacking forum.
Dependabot
In July 2023, hackers used stolen private access tokens belonging to legitimate GitHub accounts to impersonate Dependabot, a popular dependency management tool. Proposed commits were meant to infect code and inject configurations designed for stealing passwords, GitHub repository secrets, and variables either directly or with a rogue automation pipeline.
It is worth noting that earlier that year, GitHub had substantially raised account security requirements (e.g., enforcing strong 2-factor authentication). However, the attacker didn’t have to overcome that additional layer of security. All it took to breach hundreds of projects was a few long-lasting tokens left somewhere, forgotten.
Both of these incidents are a stark reminder that you should never consider permanent, long-lasting secrets safe.
Learn more about the importance of secrets rotation from DevOps expert Sam Gabrail 👇
Why is automated secrets rotation better?
Rotation is not only an essential preventive measure, but also one of the most important mitigation steps when an incident finally occurs. However, to fulfill its objectives, it has to be swift, precise, and exhaustive—qualities better delivered in scale by automation than human operators.
The importance of automation only grows as a project develops. At some point, even the smallest oversight often can burst into a separate incident, putting enormous strain on a business—something Cloudflare experienced directly.
Cloudfare exposure: Minor error, massive fallout
In October 2023, identity service provider Okta suffered two breaches in the span of just a few months, leaking customer support data. It impacted Cloudflare was as an Okta customer, and they immediately responded with rotation of all credentials suspected to be at risk. Due to human error, they missed just four (one service token, and three accounts) secrets. Cloudflare believed they were not in active use.
Unfortunately, on November 14, 2023, a hacker used those exact credentials to infiltrate Cloudflare’s internal systems, resulting in an attack that lasted 10 days. After accessing Cloudflare’s Atlassian server, the hacker went through Cloudfare’s internal Wiki pages and Jira tickets in search of information that could be forged into other attack vectors: multifactor authentication bypass, vulnerability management, secrets rotation, and other kinds of security flaws to exploit. They also targeted code repositories, with 76 out of 120 ultimately compromised.
In the end, Cloudflare executed an enormous mitigation procedure, rotating more than 5,000 credentials, physically separating test and staging systems, and putting almost 4,900 systems through forensic triaging. Additionally, they re-imaged and rebooted every single machine in their entire global network. An impressive response, no doubt, but still extremely exhausting for the business and only necessitated by a mistake that could have been easily avoided.
The key takeaway here Is that organizations need an automatic credential rotation process, implemented with proper care, attention to detail, and supervision.
As for manual rotation, it should either be treated as a last resort or performed with extreme caution.
Secrets rotation policy matters
The attacks described above are far from isolated incidents. Similar breaches are happening every day around the globe. The proper approach to rotating secrets can make the difference between safeguarding your data and being the next victim. Codifying such an approach into a policy will make it easier to establish, promote, embrace, and enforce.
Businesses should consider the aspects of a secrets rotation policy:
- The specific types of secrets you are handling
- The intricacies of your business
- The tools and platforms involved in keeping them safe
- The people taking part in their lifecycle
Note: Make sure to revise your policy periodically, as cybersecurity is constantly evolving, along with cybercrime. Safer encryption methods, better tools, and more effective methodologies and approaches are continuously emerging from the collective knowledge, experience, and understanding gathered by DevSecOps/Infosec practitioners every day.
Best practices for effective secrets rotation
Although providing a one-size-fits-all secrets rotation policy is impossible, this short list of recommendations will prove useful for any use case:
- Rotate regularly. The longer a secret exists, the bigger the risk that it is no longer a secret (or will soon cease to be) and, by extension, the more harm a malicious actor will be able to impose.
- If you can, rotate automatically. If you cannot, do so manually with caution, as a small human error can result in dire consequences. For best results, implement policies for regular, automated rotation for the greatest reliability.
- Keep secrets manageable. Avoid secrets sprawl, as excessive complexity brings obscurity, and obscurity is the enemy of observability. It is hard to protect something you can’t efficiently manage, or don’t even know about.
- Keep your policy strong and up to date. Periodically revise your policy to make sure it is dependable and matches applicable best practices and standards; also make sure the tools in use are up to their required tasks.
- Always keep secret safety in mind. Make this a priority from the very beginning. It is much easier and safer to start properly than to fix a looming disaster or deal with its aftermath.
See how automatic secrets rotation works in Akeyless 👇
Akeyless: Helping your business stay ahead of risks
Akeyless is a vaultless secrets management platform. It focuses on making the complex and cumbersome task of keeping secrets safe simple, time-efficient, and cost-effective.
Businesses can implement our automated credential rotation features in a vast variety of environments, tools, and platforms. It allows you to seamlessly unify your infrastructure while maintaining, managing, and rotating privileged credentials. Our Universal Machine Identity feature enables automated rotation for bare-metal and legacy systems while solving the “secret zero” problem.
Akeyless’ Distributed Fragments Cryptography™ (DFC) is a Zero-Knowledge encryption technology with NIST FIPS 140-2 certificatio. It provides unmatched security and scalability while giving you full control and exclusive ownership over your secrets. Attackers would have to penetrate multiple cloud providers and your environment to combine rotating fragments of encryption keys. Rest assured, your secrets are safe. If you’re interested in exploring how to tackle the biggest issues of modern secrets management, take our self-guided tour or contact us for a personalized demo today.
