December 16, 2021
Posted by Ohad Baruch
TL;DR
Akeyless added support for Kubernetes data encryption at rest, and is now available for use in your own Kubernetes cluster! Check it out here!
First Things First: Some Context
Kubernetes, as an infrastructure management solution, allows the creation of various resources, including Pods (servers), Persistent Volumes (storage), Services (load balancers), and others. You can also create Secrets for resources that need them—usually pods. These are all saved in etcd or another shared database.
All the resource secrets are normally saved in plain text, which is not a great idea. 
So, why would anyone keep Secrets in Kubernetes if they are not safe? One reason could be that there are existing deployments using the Kubernetes secrets engine and people fear the change. Or sometimes it is because of 3rd party dependencies such as cert-manager…
Now, there is a way to use Kubernetes secrets while keeping them encrypted in the database.
K8s External KMS Plugin
Starting with Kubernetes 1.10.0, it is possible to use a KMS to encrypt and decrypt resources (usually Secrets) in the Kubernetes database. This can be a static key inside the configuration (which kind of misses the effect of what we’re trying to do here), or a dynamic key exchange with an External 3rd party KMS provider.
This is the process:
- The kube-apiserver communicates with the 3rd party KMS provider plugin through a UNIX socket.
- It sends the plugin plain-texts or cipher-text, for encrypt and decrypt requests, respectively.
- The plugin then either encrypts or decrypts the received texts and returns the value to the kube-apiserver.
- The data is then either saved in the database, or returned to the requesting client, depending the requested action.
The Akeyless K8s External KMS plugin works exactly as described above. It runs as a static pod, or as a standalone docker container on each of the master nodes in the cluster, listening for requests from the kube-apiserver. Once a request has been received, it communicates with the user’s local Akeyless Gateway, or directly with the global platform at https://api.akeyless.io, depending the user’s configuration.
Usage
In order for the plugin to work, there are a few requirements:
- Kubernetes version 1.10 or higher
- Direct access to the Kubernetes control plane
- kube-apiserver must be restarted after the External KMS plugin has been configured and started
- For kubernetes-external-secrets to be able to retrieve your secrets it will need access to your Akeyless vault via Akeyless RBAC associated with an Authentication Method.
- An AES Encryption Key in Akeyless vault platform.
Authentication to Akeyless is done using the AKEYLESS SDK, using any of the available authentication methods in AKEYLESS (AWS IAM, GCP GCE, Azure AD, Universal Identity, etc.).
Encryption/Decryption flows
The KMS plugin uses an Envelope Encryption scheme. This means the kube-apiserver generates a “Data Encryption Key” (DEK) to encrypt the data, and the KMS plugin encrypts that key with a “Key Encryption Key” (KEK).
Below are the communication flows in detail:
Encryption:
- kube-apiserver receives request to create a secret
- kube-apiserver generates a DEK and encrypts the secret with it
- akeyless-kms-plugin encrypts the DEK with a KEK
- kube-apiserver saves the encrypted DEK and secret in etcd
- kube-apiserver receives request to get a secret
- kube-apiserver reads the cipher text from etcd and extracts the encrypted DEK from it
- akeyless-kms-plugin decrypts the DEK with a KEK
- kube-apiserver decrypts the secret with the decrypted DEK
Authentication to Akeyless
Just as any client authenticates to Akeyless, the KMS plugin can do the same. It only requires access_id to be configured, and will then automatically detect the Auth Method being used. It is recommended to give the auth method a role with only read permissions to the encrypting key, or any previous encrypting keys.
Authentication can basically be split to 2 major environment types:
- Cloud – Running a self-managed K8s cluster on a cloud provider
- Self-hosted – Running a self-managed K8s cluster in a self-hosted data center
Cloud
When running in the cloud, it is recommend to use the cloud id token based auth methods, relevant to the cloud provider (aws_iam, gcp_gce, azure_ad).
The plugin automatically detects if it is running in a cloud environment, and tries to use the machine’s cloud identity.
Self-hosted
In a self-hosted environment you can use Akeyless Universal Identity™ (recommended), or Access Key
- Akeyless Universal Identity: For production-grade, automated environments, this is the recommended way.
To work with Universal Identity, we need 1 top-level token generator which generates child tokens for every master node being created. The parent token generator will run in an always-on machine and uses the akeyless_universal_identity.sh script to generate tokens in a cronjob. Additionally, it listens to scale-out and restart events of the master nodes and will send a token accordingly.
The operations of sending machine-generated tokens to the master nodes are implementation specific and rely on your environment.
The generated tokens will:- Rotate their own token
- Not generate child tokens
Checkout Akeyless Universal Identity™ for more detail.
- Access Key: This isn’t a recommended way as the access key itself will have to be saved somewhere when new machines are being deployed, as well as be saved in memory as an environment variable.
This method should be used for testing purposes only.
The KMS plugin requires the access key for the used access id when booting up, simply add the AKEYLESS_ACCESS_KEY environment variable to the plugin machine and you’re done.
Once you have decided your deployment strategies and have followed the plugin configuration guide, we can take it a bit further and check Akeyless Audit Logs to see our plugin running Encryption and Decryption requests:
- Login to your Akeyless account at https://console.akeyless.io
- Click on the Audit Logs tab
- Use the Client text box to filter based on the Access ID used by the KMS plugin
- If you like to rotate the key, this can be done in the background and no change to Kubernetes or the KMS plugin is required.
- Click on the Secrets & Keys tab
- In the Filter text box enter the name of the key you used for encryption, in our case it’s /k8s-etcd-key
- Click the Rotate Key Now button to rotate your key
- Go back to the Audit log and check it again, notice the item_version has changed from 1 to 2
Conclusion
If you want to secure your secrets in etcd, regardless if your K8s cluster is on-prem or in a cloud environment—Akeyless KMS Plugin is the way to go! It is a quick, easy-to-use and secure way to encrypt your secrets in etcd.
Hope you find this useful and enjoyed the read, be safe out there!
About the Author
Ohad is a Software Engineer @ Akeyless with a DevOps background and over 10 years experience, loves to code and create stuff.
