A Platform Engineering Guide to Managing Secrets with Akeyless

Introduction to Platform Engineering

Platform engineering is about giving development teams the tools and workflows they need to ship code fast, securely, and at scale. By building reliable internal platforms, we (the platform engineers) create an environment where developers can focus on writing code. At the same time, we handle the heavy lifting behind the scenes—security, compliance, and infrastructure.

This guide will cover the basics of platform engineering, why internal developer platforms (IDPs) are so valuable, and why secrets management is a key part of keeping everything secure. We’ll also dive into how Akeyless fits into this workflow with a demo to show how it works in practice.

---

Video

Code

https://github.com/samgabrail/akeyless-platform-engineering-port

What is Platform Engineering?

Definition and Role in Modern Organizations

Platform engineering involves designing and building toolchains and workflows that enable self-service capabilities for software developers in software engineering organizations in the cloud-native world. It requires collaboration between development and operations teams to streamline workflows and enhance software development efficiency. It focuses on creating an integrated product—a platform—that provides a frictionless developer experience, abstracting the complexities of infrastructure and operations.

Principles of Platform Engineering

• Standardization: Establishing consistent processes and tooling across teams.
• Automation: Reducing manual intervention to accelerate delivery and minimize errors.
• Self-Service: Empowering developers to access resources and deploy applications without bottlenecks.
• Scalability: Ensuring the platform can grow with the organization’s needs.

Importance in Facilitating Efficient DevOps Practices

By bridging the gap between development and operations, platform engineering enhances collaboration and streamlines workflows. It supports DevOps objectives by enabling continuous integration and continuous deployment (CI/CD), infrastructure as code (IaC), and observability.

---

The Value of Internal Developer Platforms (IDPs)

Benefits and Importance

An Internal Developer Platform provides a unified interface for development teams, simplifying interactions with underlying infrastructure and services. They offer:

• Enhanced Productivity: Developers can focus on coding rather than managing infrastructure.
• Reduced Cognitive Load: Abstracting complexities allows teams to work more efficiently.
• Consistency: Standardized environments reduce discrepancies and errors.

Improving Key DevOps Metrics

Implementing an IDP positively impacts DevOps metrics such as:

• Mean Time to Recovery (MTTR): Faster recovery from incidents due to standardized processes.
• Change Failure Rate: Reduced errors through consistent deployment practices.

---

Building a Platform Engineering Team

Identifying the Need and Defining the Role

Building a platform engineering team is not for everyone. It can be an overkill in some cases. Assess your organization’s development processes, team dynamics, and business goals to determine if a platform team is necessary. Key considerations include:

• Complexity of Infrastructure: High complexity may necessitate a dedicated team.
• Developer Bottlenecks: Frequent delays in deployment suggest a need for streamlined processes.
• Security Requirements: Enhanced security protocols may require specialized handling.

The Role of Platform Engineers in Platform Engineering Teams

Platform teams focus on:

• Building and Maintaining the IDP: Developing the tools and interfaces that make up the platform.
• Optimizing Software Delivery: Streamlining CI/CD pipelines and deployment processes.
• Ensuring Security and Compliance: Implementing best practices for secrets management and access control.

The platform engineering team binds various tools and services to create a cohesive and efficient development environment, facilitating a streamlined, self-service experience for developers.

Implementing Platform Engineering

Treating the Platform as a Product

Approach the platform with a product mindset:

• User-Centric Design: Developers are your customers; their feedback is crucial.
• Start with an MVP: Be careful not to boil the ocean. Start small and test. The last thing you want is to build something developers won’t use.
• Continuous Improvement: Regularly update the platform based on user needs and technological advancements.

Focusing on Common Problems and Providing Solutions

Identify recurring challenges faced by development teams and address them through the platform, such as:

• Environment Consistency: Provide standardized development and testing environments.
• Secrets Management: Secure sensitive data like API keys and database credentials.
• App Templates: Identify your organization’s common apps and create templates for them.

---

Introduction to Secrets Management

What is Secrets Management?

Secrets management involves the tools and practices used to securely store, access, and manage sensitive information (secrets) such as passwords, API keys, certificates, and encryption keys. It’s a critical component of secure platform operations.

Common Secrets Management Tools and Practices

• Vaults: Centralized repositories for storing secrets.
• Encryption: Protecting secrets at rest and in transit.
• Access Control: Defining who or what can access specific secrets.

Key Challenges in Secrets Management

• Managing Secrets at Scale: Handling many secrets across various environments.
• Preventing Exposure: Ensuring secrets are not hard-coded or exposed in logs and repositories.

---

Why Platform Teams Should Consider Akeyless

Platform teams face a lot of challenges, and Akeyless provides smart solutions to make their work easier and more secure. Here are some key reasons why Akeyless is a great choice:

1. Easy and Cost-Effective
As a SaaS (Software as a Service) tool, Akeyless doesn’t need heavy or complicated on-site infrastructure. It’s simple to set up and more affordable compared to traditional on-premises systems, saving both time and money.

2. Great for Multi-Cloud Setups
Akeyless works smoothly across different cloud environments, which is perfect for teams that handle multi-cloud or hybrid setups. It easily scales across different business units and regions, making it flexible for all kinds of operations.

3. Ready-to-Use Integrations
Akeyless comes with many out-of-the-box integrations, so it connects quickly with tools your team is already using, like CI/CD pipelines and container platforms. This means your team can get started right away without major disruptions to your workflow.

4. Simple Compliance and Auditing
Meeting regulatory requirements can be tough, but Akeyless simplifies these processes. It has built-in features that help with auditing and compliance, making sure your systems meet the necessary standards.

5. Strong Security with DFC
Akeyless uses a patented technology called Distributed Fragments Cryptography (DFC) to keep secrets safe breaking an encryption key into separate fragments and storing them across different regions and cloud providers.

Here’s the best part: one of these fragments is created and stored entirely in your own environment, where only you have access. This gives you exclusive control and ownership of that piece, ensuring that only you can manage or use it. Because of this setup, Akeyless has Zero Knowledge of your keys, meaning they can’t access or see your secrets, even if they wanted to.

How Akeyless Solves Common Platform Team Problems

Akeyless isn’t just a security tool; it’s designed to tackle specific issues platform teams deal with daily:

• One Place for Secrets Management: With Akeyless, secrets can be managed centrally. This means teams can control access easily and keep everything secure, even if they have many environments to manage.
• Automatic Secret Updates: Secrets like passwords or keys need to be rotated often to stay safe. Akeyless automates this process, keeping secrets up-to-date and reducing the risk of being compromised.
• Seamless Tool Integration: Akeyless fits right into existing workflows without causing interruptions. It’s built to work well with the tools your team already uses.
• Support for Users and Machines: Akeyless can manage secrets for both people and systems, making sure that all identities on the platform are secure.

---

Integrating Akeyless with Platform Engineers’ Workflows

Integrating Akeyless into Your Internal Developer Platform (IDP) Using Port

IDP Overview

Internal Developer Platforms are not the same as Internal Developer Portals. Unfortunately, they have the same acronym, and many platform engineering teams may confuse them.

An Internal Developer Platform, like Mia Platform, are often more opinionated. They come with “batteries included,” providing comprehensive environments for developers to build and manage applications. In contrast, Internal Developer Portals, such as Port, serve as the frontend for developers. They offer flexibility to platform teams to build anything they want. This enables developers to access and interact with these environments and services efficiently.

In our demo, we will use Port to provide developers with a self-service portal and we will build what we need in the backend with a GitHub actions pipeline that integrates with Akeyless and ArgoCD.

---

Build a Secure App Template for a Python Flask App with a MySQL Database running in K8s and managed by ArgoCD

Now, let’s move on to this blog post’s fun and practical part and see how to implement a concrete example.

As a platform engineer, your mission is to build a secure app template that’s production-ready, optimized for efficiency, and packed with best practices. You aim to secure secrets across GitHub, Kubernetes, and Argo CD without burdening your developers. Additionally, you need to eliminate the security risks associated with long-lived credentials by providing developers with dynamic secrets seamlessly integrated into their application workflow.

General Workflow

Let’s take a look at our general workflow for this demo:

Here’s a breakdown of our secure app template deployment process in seven steps, as illustrated in the diagram:

1. Self-Service Request on Port: The developer initiates the process by filling out a form on Port, a frontend portal, to define application details.
2. Authenticate and Retrieve Secrets: GitHub Actions, triggered by the form submission, authenticates with Akeyless to retrieve static and create a MySQL dynamic secret needed for the application.
3. Create a New Repository: GitHub Actions creates a new repository for the application, setting up the groundwork for the development environment.
4. Link to ArgoCD: The new repository is connected to ArgoCD, which will handle deployment and continuous updates to the application. In fact, the pipeline already creates an ArgoCD app with our Python Flask and MySQL database running.
5. Dynamic Secrets Authentication: The application uses dynamic secrets from Akeyless for secure access to the MySQL database, fetching these credentials just-in-time as needed.
6. Monitor Logs on Port: Developers can view logs and updates in Port to track the creation process and application status and see links to resources.
7. Start Development Work: With everything set up, developers can start working on feature branches and push updates, triggering CI/CD pipelines to deploy new versions securely.

Diving Deeper into the Workflow

This section walks you through a developer’s workflow in action and then switches to a platform engineer’s viewpoint to show you how to make it all come together. I walk you through all these steps in the demo video at the top of this blog post.

1. Setting Up the Frontend of the Workflow Using Port

Developers start with Port, a self-service front-end portal where they input essential details shown in the image below:

Upon submission, Port automatically triggers a GitHub Actions pipeline to create the necessary environment, including the initial repository, secrets storage, and Argo CD application.

2. Automating GitHub Actions for Application and Secrets Management

The GitHub Actions pipeline is really the brains behind the whole operation. It has multiple steps and took me the longest to build. As I mentioned in the video, it took me over 50 tries to finally get it right. As a platform engineer, this is normal, and if you’re just starting out, please don’t get discouraged. However, once you get the pipeline working. It will always be available for all your developers, so it’s worth spending the time upfront.

You can find the entire pipeline in the repo, but here are the steps:

1. Initial Setup and Secret Management

• Triggers manually via workflow_dispatch with configurable inputs
• Retrieves secrets from Akeyless vault using OIDC authentication:
• ArgoCD credentials
• Port credentials
• GitHub Personal Access Token
• Database credentials

2. Container Image Management

• Sets up Docker Buildx for container image building
• Authenticates with GitHub Container Registry (GHCR)
• Builds and pushes the Flask application Docker image

3. Repository Management

• Creates a new GitHub repository for the application
• Clones the repository and prepares it for GitOps
• Copies and configures:
  • Kubernetes manifests
  • Application source code
  • CI/CD workflows
  • Configuration files

4. Kubernetes Secret Management

• Creates necessary Kubernetes namespaces
• Sets up MySQL root password secret
• Configures Akeyless gateway certificate secret

5. GitOps Configuration

• Updates Kubernetes manifests with environment-specific values
• Configures image references and namespaces
• Sets up database connections and secret references
• Commits and pushes changes to the new repository

6. ArgoCD Integration

• Logs into ArgoCD with retry mechanism
• Registers the new repository in ArgoCD
• Creates an ArgoCD application with:
• Automated sync policy
• Namespace creation
• Source and destination configuration

7. Dynamic Secrets Setup

• Creates Akeyless database target
• Configures dynamic secret generation for MySQL
• Sets up automatic rotation policies

8. Final Deployment

• Triggers initial ArgoCD sync
• Validates deployment status

9. Monitoring and Notification

• Sends deployment status to Port, providing links to:
  • GitHub repository
  • Service entity in Port
  • ArgoCD dashboard

This pipeline implements modern Platform Engineering practices including:

• GitOps-based deployment
• Infrastructure as Code
• Secret management
• Container orchestration
• Continuous Deployment
• Developer portal integration

3. Integrating Dynamic Secrets into the App

As we saw in the pipeline above, an Argo CD application is created and connected to the new repository, deploying a Python Flask app and a MySQL database. The Python Flask app first authenticates into Akeyless using the Kubernetes authentication method and then fetches the database credentials dynamically, eliminating the need for long-lived credentials.

The app periodically retrieves secrets from Akeyless using just-in-time (JIT) credentials, refreshing them as they expire based on the time-to-live (TTL) settings configured in Akeyless for the database dynamic secret.

You can see the new credentials getting generated in the logs of the flask container below:

4. Monitoring Success and Testing New Features

Developers can:

• Follow Logs in Port: Confirm that the repository, Argo CD dashboard, and the Port entity are set up correctly.
• Create Feature Branches: Work on new features without worrying about the underlying infrastructure.
• Trigger Deployments: Push version tags to redeploy the application with updates.

Exploring the Application Repository and Making Changes

The Application Repo Structure

The new application repository created includes the following file structure:

.
├── Dockerfile
├── Pipfile
├── Pipfile.lock
├── README.md
├── app
│   ├── __init__.py
│   ├── __pycache__
│   │   ├── __init__.cpython-310.pyc
│   │   ├── __init__.cpython-39.pyc
│   │   ├── akeyless_integration.cpython-310.pyc
│   │   └── routes.cpython-310.pyc
│   ├── akeyless_integration.py
│   ├── routes.py
│   └── templates
│       └── index.html
├── flask-deployment.yaml
├── mysql-deployment.yaml
├── requirements.txt
├── run.py
├── services.yaml
└── tests
    ├── __init__.py
    └── test_app.py

Here is a brief explanation of these files:

• Dockerfile: Defines the instructions needed to build a Docker image for the application, specifying the base image, dependencies, and commands to run the application.
• Pipfile & Pipfile.lock: Used for managing Python package dependencies, with Pipfile specifying the packages needed and Pipfile.lock ensuring consistent environments across deployments.
• README.md: Contains documentation and instructions for setting up, running, and contributing to the application.
• app/: This directory contains the main application code, including:
  • init.py: Initializes the Python package.
  • akeyless_integration.py: Handles integration with Akeyless for secrets management.
  • routes.py: Defines the application’s routes and associated logic.
  • templates/index.html: The HTML template used for rendering the application’s web pages.
• flask-deployment.yaml & mysql-deployment.yaml: Kubernetes manifests for deploying the Flask application and MySQL database, respectively.
• services.yaml: Defines Kubernetes services to expose the application and database.
• requirements.txt: Lists Python packages required by the application, used for dependency management.
• run.py: The entry point script to run the Flask application.
• tests/: Contains test scripts to validate the application’s functionality, including:
  • test_app.py: Unit tests for the application logic.

These files collectively support the development, deployment, and operation of the application within our Kubernetes environment, integrating secrets management and CI/CD processes. The best part is that the developer gets all of this for free.

Making Changes

Developers can now start working on their application by:

• Creating Feature Branches
• Updating the Codebase
• Issuing Pull Requests
• Pushing Version Tags

These actions trigger pipelines that validate, build, and deploy the application securely.

Managing Kubernetes and Argo CD

As a platform engineer, you can:

• Shield Developers from Complexity: Allow them to focus on code rather than infrastructure.
• Manage Deployments: Use Kubernetes to handle deployments, services, and persistent volumes.
• Maintain Application Consistency: Argo CD monitors and updates deployments based on repository changes.

Observations on using Dynamic Secrets

The dynamic secrets functionality ensures that:

• Credentials Are Short-Lived: Minimizes security risks.
• Secrets Refresh Periodically: Applications request new credentials as needed.
• Developers Remain Unburdened: No need to manage or update secrets manually.

Configuring Akeyless for Secure Integration

Let’s turn our attention to the specifics of how Akeyless secures both the delivery of the application via the pipeline and as it runs.

1. Secrets Storage and Authentication – Secures the Delivery of Credentials

• Static Secrets: These are used by the GitHub actions pipeline for multiple tasks, including configurations like GitHub tokens, ArgoCD, port credentials, and the original MySQL root password.
• Dynamic Secrets: Short-lived credentials for MySQL. The application continuously requests new Database credentials from Akeyless as the TTL expires for the old credentials.

Below is a screenshot showing the demos folder where we store static and dynamic secrets to be used by both the application and the pipeline.

It’s also important to note that an Akeyless admin must prepare all the static secrets mentioned above before the pipeline runs. Furthermore, the pipeline first creates a target and then a dynamic secret in Akeyless. Below is a screenshot of the target.

2. Solving the Secret Zero Problem

• JWT Authentication: This method uses JSON Web Tokens to prevent the storage of any Akeyless credentials in code or pipelines to access Akeyless (secret zero). This is very important to note. If you pay close attention, you will see that we didn’t store any long-lived credentials as secrets in the GitHub Actions secrets store. We also relied on the K8s auth method in Akeyless for the app to authenticate into Akeyless. Again, there is no hard coding of long-lived credentials there. Below are two screenshots. The first shows the GitHub authentication method using JWT in Akeyless. The second shows the K8s authentication method in Akeyless.

Key Takeaways from this Demo for Platform Engineers

If you forgot everything I said in this blog post, you would do well to remember these four points:

• Use Dynamic Secrets Whenever Possible: Enhances security by ensuring credentials are short-lived.
• Address Secret Zero: using a platform with an identity to authenticate into such as JWT tokens for GitHub and Kubernetes Auth into Akeyless.
• Integrate Native Security Tools: Incorporate solutions like Akeyless into app templates.
• Implement Retry Logic: Ensure applications can handle expired credentials gracefully.

---

Conclusion

Secrets management is a critical component of platform engineering, directly impacting the security and efficiency of your development workflows. By integrating Akeyless into your platform engineering practices, you can:

• Enhance Security: Implement dynamic secrets and eliminate long-lived credentials.
• Improve Efficiency: Streamline workflows for both developers and platform engineers.
• Scale Seamlessly: Manage secrets across multiple environments and cloud providers.

A final note: development teams are building application features and business logic. They shouldn’t worry about infrastructure such as Kubernetes, Argo CD, and Akeyless. Our job as platform engineers is to enable developers by providing secure best practices for infrastructure. I hope you noticed this in this blog post.

If you liked this post, consider checking out this one also Why Platform Teams Should Embrace Akeyless Secrets Management | Akeyless

---

We encourage you to explore Akeyless further and consider how it can fit into your organization’s platform engineering strategy. By adopting robust secrets management practices, you pave the way for more secure, efficient, and scalable software delivery.

Ready to take the next step? Visit Akeyless to learn more about how you can enhance your platform engineering efforts with advanced secrets management solutions.