Reversing the Workflow with External Secrets Operator’s Push Secret Feature

External Secrets Operator (ESO) is a great tool to manage secrets centrally across different providers from Kubernetes clusters. ESO is extremely easy to use and it removes the complexity from secret management. As I explained in my previous blog posts you can install, configure and integrate ESO with different Secret Management Providers by using plain Kubernetes yaml files.

You can read my previous blog posts about ESO from these links to understand the nature of the project more clearly:

• Cloud Native Secret Management with External Secrets Operator
• External Secrets Operator Integration with HashiCorp Vault

Up until the 0.7.0 release, ESO only allowed fetching secret data from external secret management providers. But within that release Push Secret feature is published. You can now push Kubernetes secrets to third party secret management providers with this feature. This allows you to reverse the workflow. With Push Secret you can also migrate secrets between providers even more easily.

As you can see in the diagram above, with Push Secret you can also send secrets to multiple secret management providers at the same time. As a side note, Push Secret is a namespaced resource. Push Secret resource should be in the same namespace with the Secret Stores.

You can still use the same method to authenticate with an external secret management provider which is creating a SecretStore resource. You can then create a Push Secret resource and reference the Kubernetes secret and SecretStore to push that data.

Demo

In this demo I will pull a secret from the AWS SSM Parameter Store with ESO. ESO will create a Kubernetes secret object from the new External Secret resource. Finally, I will use the Push Secret feature to push that newly created Kubernetes secret to AWS Secrets Manager.

I will start with creating a Secret Store for Parameter Store from this example Yaml file.

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
 name: aws-parameter-store
spec:
 provider:
   aws:
     service: ParameterStore
     region: eu-west-1
     auth:
       secretRef:
         accessKeyIDSecretRef:
           name: awssm-secret
           key: access-key
         secretAccessKeySecretRef:
           name: awssm-secret
           key: secret-access-key

As you can see I’ve used a Kubernetes secret with AWS Access Key and Secret Access Key information to authenticate with my AWS Account for this demonstration. I don’t recommend this approach in production environments. You can use IAM Roles for Service Accounts (IRSA) to authenticate with AWS. Also, you can change any information according to your configurations when following along.

After creating the Secret Store, I’ve created a secret on AWS SSM Parameter Store.

As you can see there is a simple JSON object with one key value pair. It’s time for me to create an External Secret object to fetch that secret data into a Kubernetes secret. I’ve used this External Secret Yaml file to do that.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
 name: eso-push-secret
spec:
 secretStoreRef:
   name: aws-parameter-store
   kind: SecretStore
 target:
   name: eso-push-secret
 data:
 - secretKey: password
   remoteRef:
     key: eso-push-secret
     property: secret.password

The secret is synced. Let’s see if the data is correctly fetched.

Perfect! I’ve used a tiny tool called cowsay for fun. Data is correct and that means I can move on to creating the Secret Store for AWS Secrets Manager and the Push Secret resource.

I’ve used this example Yaml file to create the Push Secret resource.

apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
 name: push-secret
 namespace: default
spec:
 refreshInterval: 10s
 secretStoreRefs:
   - name: aws-secrets-manager
     kind: SecretStore
 selector:
   secret:
     name: eso-push-secret
 data:
   - match:
       secretKey: password
       remoteRef:
         remoteKey: eso-push-secret

After creating the Push Secret object as you can see it’s status is synced. Let’s check the AWS side.

I can see the newly created secret on my AWS Management Console.

Also, ESO adds a tag to the pushed secrets. You can see which secrets came from ESO by looking up for the tag “managed-by”. If the secret already exists at the provider, ESO will only update it if the secret has that tag. That means you can stop reconciliation by removing the tag.

Let me check if the data is correctly pushed to AWS Secrets Manager. I’ve used AWS CLI to get the secret value.

aws secretsmanager get-secret-value — secret-id eso-push-secret — region=eu-west-1

You can change the secret name and region if you are using different configurations. This command gave me an output like this.

The actual secret data is the SecretBinary and it’s base64 encoded. Let me decode that value and see if the secret is pushed correctly to AWS Secrets Manager.

As you can see, with ESO, I’ve fetched the secret from AWS SSM Parameter Store and pushed that secret data to AWS Secrets Manager.

With this new Push Secret feature ESO makes it easier to manage secrets centrally from your Kubernetes cluster and across Secret Management providers. As I’ve mentioned in my previous blog posts, you can always contribute to ESO with issues and PRs. You can also check out the GitHub Repository and the website for learn more about ESO.