Cloud Native Secret Management with External Secrets Operator

We are always dealing with sensitive data and secrets while managing our infrastructure and application resources. That’s why there are so many secret management tools out there and all of them are great solutions in their area. In the past we might not need these approaches but right now we must show the extra effort for keeping our data safe. Because the amount of secrets we use in our system has gone from one or two to hundreds. When we shift gears to Cloud Native architectures and microservices it started to become more difficult to keep the data safe.

Almost every application needs secrets. Those secrets can be usernames and passwords, tokens, certificate data, etc. Secret Management tools are the best way to keep those secrets safe. They have great features like encryption at rest, secret lifecycle management, RBAC, policy based management. But the major problems with these tools are the integration part and of course having different approaches for consuming secret data.

Managing secrets in Kubernetes is relatively an easy task. You can create secrets from the command line, from reading a file or creating a YAML file with the secret definition. Of course, consuming the Kubernetes secrets in pods or applications running inside pods is an easy task as well.

As you already know, secrets in Kubernetes are stored in etcd with base64 encoding and if you don’t encrypt your etcd those secrets are easily accessible and consumable by third parties. This is one of the key tasks we all should complete for securing our Kubernetes clusters. But when it comes to consuming secrets from external Secret Management Providers, there are some other things we should consider. For example, there can be secret sprawl between providers and implementing the access and other configurations for each provider can become difficult. And of course, if you have Kubernetes clusters in different environments, for example, on top of different cloud providers you would probably need to consume secrets from multiple providers and have different configurations in each cluster for different providers.

These will lead to some problems. To be more specific, security problems because of the different access methods, roles and permissions and extra operational burdens such as all configuration differences between these providers.

Enter External Secrets Operator

There were different solutions for consuming secret data from third party providers. With the beauty of open source and the community, External Secrets Operator (ESO) was borned with an issue on GitHub.

External Secrets Operator is an Open Source Kubernetes Operator that integrates external Secret Management Systems like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, etc. and the project is initiated by Container Solutions and GoDaddy. It is designed to synchronize secrets from external APIs to Kubernetes.

The architecture is relatively simple. ESO uses the operator / controller pattern. It extends the Kubernetes API using Custom Resources. You define where your secrets live, access configurations and how to synchronize those secrets. ESO fetches the secrets from the provider and creates a Kubernetes secret object. And also if the secret changes in the provider, ESO detects this change and updates the secrets.

External Secrets Operator recently accepted into CNCF as a Sandbox project. This is a huge accomplishment for the project and I believe it’s going to affect the project’s future in a good way. Of course there is always room for more contributors. If you want to use or contribute to the project you can check out the project’s GitHub repository.

Let’s see ESO in action

For this blog post, I will fetch secrets from AWS Secrets Manager with External Secrets Operator on an EKS Cluster. I will use Amazon EKS Blueprints for Terraform to create the required resources. I have also created this GitHub Repository for this demonstration.

The Terraform configurations will create the following resources:

• A VPC
• An EKS Cluster
• A Secret Resource on AWS Secrets Manager

I also have two Bash scripts in this repository and the installation script does the following:

• Deploying ESO on EKS Cluster
• Configuring required IAM Policy and Role for IRSA
• Creating and configuring Kubernetes Service Account and configuring the permissions

After running the scripts, I have ESO installed on my EKS Cluster and now it’s time to create the Secret Store and External Secret resources. I will create them manually with kubectl.

cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider: 
    aws:
      service: SecretsManager
      region: eu-west-1
      auth:
        jwt:
          serviceAccountRef:
            name: eso-sa
            namespace: external-secrets
EOF

This command above will create a SecretStore resource on the cluster and it will connect to the AWS Secrets Manager service in the eu-west-1 region.

cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: eso-demo-secret
  namespace: external-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: eso-created-secret
    creationPolicy: Owner
  dataFrom:
  - extract:
      key: eso-secret
EOF

With the command above I am creating the External Secret resource and this will sync the secret and create a Kubernetes secret in my cluster.

As you can see from the screenshots above, ESO automatically synced the secret from AWS Secrets Manager into a Kubernetes secret object. That means I can use this newly created secret in my pod definitions.

Conclusion

Secret Management in Cloud Native environments can become a really difficult job in time because of secret sprawl, different secret management provider configurations and integration problems. Managing secrets inside Kubernetes clusters is relatively an easier task. With External Secrets Operator (ESO), you will have the chance to manage your secrets from your Kubernetes clusters centrally without having to implement different approaches for every different secret provider and you will access and synchronize your secrets securely. Also, ESO’s acceptance into the CNCF will grow the community around the tool and I believe this is really a solid proof of the ESO’s power.