External Secrets Operator Integration with HashiCorp Vault

In my last blog post, I’ve explained what External Secrets Operator (ESO) is and how it works. External Secrets Operator is an Open Source Kubernetes Operator that integrates external Secret Management Systems like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, etc. and it is designed to synchronize secrets from external APIs to Kubernetes. In the last blog post, I’ve shared how to integrate ESO with AWS Secrets Manager. This time, I will explain the integration between ESO and HashiCorp Vault.

Before getting started, I want to mention again ESO’s acceptance to the CNCF as a Sandbox project. This is a huge accomplishment for ESO and also the community around the project getting larger and larger everyday. If you also want to contribute to the ESO project you can check out this part in the official External Secrets GitHub repository. You can also reach out to the maintainers from social media accounts like Twitter and there is a Slack channel #external-secrets in the official Kubernetes Slack Workspace.

In Action!

In this demonstration, I will use a Vault installation on a Kubernetes cluster with the official Helm chart and show the authentication with Kubernetes auth method. ESO allows several authentication methods and if you already have a Vault installation in place you can configure the authentication accordingly. I will also use a local cluster with minikube for all the installation but you need to have these in your environment:

• Helm v3
• Kubectl
• A Kubernetes Cluster
• Vault CLI

I have already deployed ESO and Vault with the official Helm charts to my Kubernetes Cluster. If you haven’t done so already, you can follow the documentation for ESO and Vault for the installation.

You can use this Tutorial from HashiCorp Learn to install, initialize and authenticate with Vault.

It’s time to create the secure authentication and authorization between HashiCorp Vault and ESO. I will first create a read policy for all the paths in the Vault. This is of course not the best practices you should follow in your production environment. You can create your own policies accordingly.

vault policy write eso -<<EOF
path "*"
{  capabilities = ["read"]
}
EOF

I will now enable the Kubernetes Authentication method for Vault.

vault auth enable kubernetes

I will create some environment variables for completing the authentication configuration.

kubernetes_host="$(kubectl exec vault-0 -n vault — printenv | grep KUBERNETES_PORT_443_TCP_ADDR | cut -f 2- -d "=" | tr -d " ")"

kubernetes_port="443"

kubernetes_cert="$(kubectl config view — raw — minify — flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 — decode)"

There is a service account created in the Vault Kubernetes namespace with the name vault and there is also a secret attached to that service account. I will create another environment variable with the token data inside this secret. You should change the secret name according to your environment.

account_token="$(kubectl get secret -n vault vault-token-5lwdt -o jsonpath='{.data.token} {"\n"}' | base64 — decode)"

It’s time to configure the Kubernetes authentication method in Vault.

vault write auth/kubernetes/config token_reviewer_jwt="${account_token}" kubernetes_host="https://${kubernetes_host}:${kubernetes_port}" kubernetes_ca_cert="${kubernetes_cert}" disable_issuer_verification=true

Next I will create a Vault role to access the secrets. Just like the previous environment variable I fetched the secret name for the external secrets service account and again you should change the secret name according to your environment.

demo_token="$(kubectl get secret -n external-secrets external-secrets-token-wz2fp -o jsonpath='{.data.token} {"\n"}' | base64 — decode)"
vault write auth/kubernetes/role/eso-role bound_service_account_names=external-secrets bound_service_account_namespaces=external-secrets policies=eso ttl=24h
vault write auth/kubernetes/login role=eso-role jwt=$demo_token iss=https://kubernetes.default.svc.cluster.local

After creating the Kubernetes Authentication method and configuring the required permissions, policies and roles, I can now move on to secret creation and synchronization.

ESO currently only supports KV Secrets Engine in HashiCorp Vault. I will now configure the KV Secrets Engine and create a secret.

vault secrets enable -version=2 kv
vault kv put kv/secret password=extremely-secure-password

Now, let’s configure External Secrets Operator to fetch the secrets from Vault. I will create the Secret Store first.

cat << EOF | kubectl apply -f -
 apiVersion: external-secrets.io/v1alpha1
 kind: SecretStore
 metadata:
   name: hashicorp-vault
   spec:
     provider:
       vault:
         server: "http://vault.vault:8200"
         path: "kv"
         version: "v2"
         auth:
           kubernetes:
             mountPath: "kubernetes"
             role: "eso-role"
EOF

Now let’s create the External Secret object.

cat <<EOF | kubectl apply -f -
 apiVersion: external-secrets.io/v1alpha1
 kind: ExternalSecret
 metadata:
   name: vault-secret
 spec:
   secretStoreRef:
     name: hashicorp-vault
     kind: SecretStore
   target:
     name: vault-secret-example
   data:
   - secretKey: password
     remoteRef:
       key: secret
       property: password
EOF

After applying these resources, my secret is now created and synced with Vault.

I can now use my newly created Kubernetes secret in my application pods. ESO automatically synced the secret data into a Kubernetes object. You can also add refreshInterval parameter to the External Secret definition to be able to keep the secret up to date.

Conclusion

Managing secret objects in Kubernetes is relatively an easy task. With integration benefits and a rich feature set of ESO, you can improve operational excellence in your environments. External Secrets Operator supports integration with many more secret management providers with easy configuration options. ESO provides secure connection between secret management provider and Kubernetes cluster and automates the secret synchronization operations. ESO’s acceptance to the CNCF as a Sandbox Project is a strong proof of the project’s capabilities.