Deploying Hashicorp Vault to EKS cluster with DynamoDB backend.

Problem statement

Kubernetes secret is not the best place to keep sensitive data. Even though EKS supports adding KMS envelope encryption to enhance security for secrets, we still can not apply fine-grained control access to kubernetes secrets via RBAC. A better idea is to use an external secrets store, for example AWS Secrets Manager, Hashicorp Vault, etc. In this post we will look at deployment of Hashicorp Vault in EKS cluster in a highly-available manner with DynamoDB as backend and KMS for encryption.

Solution overview

1. Hashicorp Vault will be deployed by Helm Chart into EKS cluster.
2. Vault pods will assume IAM roles via IAM OIDC identity provider and IRSA.
3. DynamoDB will be used as the backend for Vault.
4. AWS KMS will be used for Vault auto-unseal.
5. DynamoDB and KMS are accessible through VPC endpoints.
6. Vault UI will be exposed via Elastic Load Balancer.
7. Vault injector pod will set secret into application pod.
8. Vault writes audit logs (without any sensitive data) into persistent EBS volume.
9. Data encrypted at rest and in transit.

Hashicorp Vault supports a variety of backends including Amazon S3 and DynamoDB. Highly-available setup is possible only with DynamoDB, so we will use it.

Before installing the Vault Helm chart we need to create a kubernetes namespace:

$ kubectl create namespace vault
namespace/vault created

We can use the official helm repository:

$ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp"has been added to your repositories

$ helm search repo hashicorp/vault
NAME             CHART   VERSIONAPP VERSIONDESCRIPTION                   
hashicorp/vault  0.19.0  1.9.2      Official HashiCorp Vault Chart

We need to generate SSL certificates in order to configure encryption is transit:

$ NAMESPACE=vault

$ SECRET_NAME=vault-server-tls
$ SECRET_NAME=vault-server-tls
$ TMPDIR=/tmp

$ openssl genrsa -out ${TMPDIR}/vault.key 2048
Generating RSA private key, 2048 bit long modulus (2 primes)
............+++++
..............+++++
e is 65537 (0x010001)


$ cat <<EOF >${TMPDIR}/csr.conf
> [req]
> req_extensions = v3_req
> distinguished_name = req_distinguished_name
> [req_distinguished_name]
> [ v3_req ]
> basicConstraints = CA:FALSE
> keyUsage = nonRepudiation, digitalSignature, keyEncipherment
> extendedKeyUsage = serverAuth
> subjectAltName = @alt_names
> [alt_names]
> DNS.1 = ${SERVICE}
> DNS.2 = ${SERVICE}.${NAMESPACE}
> DNS.3 = ${SERVICE}.${NAMESPACE}.svc
> DNS.4 = ${SERVICE}.${NAMESPACE}.svc.cluster.local
> IP.1 = 127.0.0.1
> EOF


$ openssl req -new -key ${TMPDIR}/vault.key -subj "/CN=${SERVICE}.${NAMESPACE}.svc" -out ${TMPDIR}/server.csr -config ${TMPDIR}/csr.conf

Next we need to create a certificate signing request in kubernetes and approve it:

$ export CSR_NAME=vault-csr

$ cat <<EOF >${TMPDIR}/csr.yaml
> apiVersion: certificates.k8s.io/v1beta1
> kind: CertificateSigningRequest
> metadata:
>   name: ${CSR_NAME}
> spec:
>   groups:
>   - system:authenticated
>   request: $(cat ${TMPDIR}/server.csr | base64 | tr -d '\n')
>   usages:
>   - digital signature
>   - key encipherment
>   - server auth
> EOF


$ kubectl create -f ${TMPDIR}/csr.yaml

$ kubectl get csr
NAME        AGE   SIGNERNAME                     REQUESTOR          CONDITION
vault-csr   9s    kubernetes.io/legacy-unknown   kubernetes-admin   Pending


$ kubectl certificate approve ${CSR_NAME}
certificatesigningrequest.certificates.k8s.io/vault-csr approved


$ kubectl get csr
NAME        AGE   SIGNERNAME                     REQUESTOR          CONDITION
vault-csr   18s   kubernetes.io/legacy-unknown   kubernetes-admin   Approved,Issued

Next we need to create a kubernetes secret with certificate and key:

$ serverCert=$(kubectl get csr ${CSR_NAME} -o jsonpath='{.status.certificate}')


$ echo "${serverCert}" | openssl base64 -d -A -out ${TMPDIR}/vault.crt

$ kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 -d > ${TMPDIR}/vault.ca


$ kubectl create secret generic ${SECRET_NAME} \
>         --namespace ${NAMESPACE} \
>         --from-file=vault.key=${TMPDIR}/vault.key \
>         --from-file=vault.crt=${TMPDIR}/vault.crt \
>         --from-file=vault.ca=${TMPDIR}/vault.ca
secret/vault-server-tls created

We need to override some chart values:

override-values.yml

# Vault Helm Chart Value Overrides
global:
  enabled: true
  tlsDisable: false

injector:
  enabled: true
# Use the Vault K8s Image https://github.com/hashicorp/vault-k8s/
  image:
    repository:"hashicorp/vault-k8s"
    tag:"0.14.2"

server:
# Use the Enterprise Image
  image:
    repository:"hashicorp/vault"
    tag:"1.9.2"

  extraEnvironmentVars:
    VAULT_CACERT: /vault/userconfig/vault-server-tls/vault.ca

  extraVolumes:
    - type: secret
      name: vault-server-tls

# This configures the Vault Statefulset to create a PVC for audit logs.
# See https://www.vaultproject.io/docs/audit/index.html to know more
  auditStorage:
    enabled: true

  standalone:
    enabled: false

  serviceAccount:
    create: false
    name:"vault-sa"

# Run Vault in "HA" mode.
  ha:
    enabled: true
    replicas:5

    config: |
      ui = true
      listener "tcp" {
        address = "[::]:8200"
        cluster_address = "[::]:8201"
        tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
        tls_key_file = "/vault/userconfig/vault-server-tls/vault.key"
        tls_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
      }

# S3 example. Not used in the solution
# storage "s3" {
#   bucket     = "vault-ait-bucket"
#   region     = "us-east-1"
#   kms_key_id = "alias/aws/s3"
# }

      storage "dynamodb" {
        ha_enabled = "true"
        region = "us-east-1"
        table = "vault-table"
      }

      seal "awskms" {
        region     = "us-east-1"
        kms_key_id = "ceedc5f5-7963-4482-a9df-17f34fad8169"
endpoint   = "https://vpce-06a1058e44fb409e5-3e7zjquu.kms.us-east-1.vpce.amazonaws.com"
      }

      service_registration "kubernetes" {}

# Vault UI
ui:
  enabled: true
  serviceType:"LoadBalancer"
  serviceNodePort: null
  externalPort:8200

# For Added Security, edit the below
#loadBalancerSourceRanges:
#   - < Your IP RANGE Ex. 10.0.0.0/16 >
#   - < YOUR SINGLE IP Ex. 1.78.23.3/32 >

Vault pods will use serviceAccount “vault-sa” and assume appropriate IAM role according to the least privilege principle.

IAM role trust policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::3**********3:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/0F51B5F333F6947BC27C138A1BC3E135"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.us-east-1.amazonaws.com/id/0F51B5F333F6947BC27C138A1BC3E135:sub": "system:serviceaccount:vault:vault-sa"
        }
      }
    }
  ]
}

Policies have to allow KMS and DynamoDB access:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowKMS",
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt",
                "kms:Encrypt",
                "kms:DescribeKey"
            ],
            "Resource": "arn:aws:kms:us-east-1:3**********3:key/ceedc5f5-7963-4482-a9df-17f34fad8169"
        }
    ]
}

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "dynamodb:DescribeLimits",
                "dynamodb:DescribeTimeToLive",
                "dynamodb:ListTagsOfResource",
                "dynamodb:DescribeReservedCapacityOfferings",
                "dynamodb:DescribeReservedCapacity",
                "dynamodb:ListTables",
                "dynamodb:BatchGetItem",
                "dynamodb:BatchWriteItem",
                "dynamodb:CreateTable",
                "dynamodb:DeleteItem",
                "dynamodb:GetItem",
                "dynamodb:GetRecords",
                "dynamodb:PutItem",
                "dynamodb:Query",
                "dynamodb:UpdateItem",
                "dynamodb:Scan",
                "dynamodb:DescribeTable"
            ],
            "Effect": "Allow",
            "Resource": [
"arn:aws:dynamodb:us-east-1:3**********3:table/vault-table"
            ]
        }
    ]
}

Kubernetes service account has the following annotation:

apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::3**********3:role/vault-role
name: vault-sa
namespace: vault

Next we can install the helm chart

$ helm install vault hashicorp/vault --namespace vault -f override-values.yml

$ kubectl get pods -n vault
vault         vault-0   0/1     Running   0   14m
vault         vault-1   0/1     Running   0   14m
vault         vault-2   0/1     Running   0   14m
vault         vault-3   0/1     Running   0   14m
vault         vault-4   0/1     Running   0   14m
vault         vault-agent-injector-f4c4f579c-b2lv8     1/1     Running   0   14m

Here we need to initialize the vault. It’s required just once, because we have auto-unseal with AWS KMS. If a Vault pod is restarted, unseal will happen automatically next time.

This feature enables operators to delegate the unsealing process to trusted cloud providers to ease operations in the event of partial failure and to aid in the creation of new or ephemeral clusters.

$ kubectl exec -n vault -it vault-0 -- /bin/sh

/ $ vault status
Key                      Value
---                      -----
Recovery Seal Type       awskms
Initialized              false
Sealed                   true
Total Recovery Shares    0
Threshold                0
Unseal Progress          0/0
Unseal Nonce             n/a
Version                  1.9.2
Storage Type             dynamodb
HA Enabled               true


/ $ vault operator init
Recovery Key 1: WrM/CbSByO7L4wab0IK/TgFJEPMWQSz5VMHaVu6ADc/q
Recovery Key 2: XJjZw4K7UX60GMfgK3kj4zOrFH0TPomiLn1BfY3wj7/1
Recovery Key 3: WI8qkKNbQvn1mMOb8QrSYaazaek6ZDYUdKVXlZJAfh8L
Recovery Key 4: GvyL2vXvrHz9hPszhOaAfxSt67Va8loeVIqQpx7DtTsO
Recovery Key 5: /8QFZzZkhVc9DhmtDB8rwGfewgXJxLvF+/ys8fZIZZxy

Initial Root Token: s.BoYAMCWvueOavJC7zBbEhcs2

Success! Vault is initialized

Recovery key initialized with 5 key shares and a key threshold of 3. Please securely distribute the key shares printed above.


/ $ vault status
Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
Version                  1.9.2
Storage Type             dynamodb
Cluster Name             vault-cluster-b2196873
Cluster ID               245e73b0-718a-e41f-3a8e-d9d05b80f774
HA Enabled               true
HA Cluster               https://vault-0.vault-internal:8201
HA Mode                  active
Active Since             2022-01-25T19:21:32.839644408Z


$ kubectl get pods -n vault
NAME                                   READY   STATUS    RESTARTS   AGE
vault-0                                1/1     Running   0          5m
vault-1                                1/1     Running   0          5m
vault-2                                1/1     Running   0          5m
vault-3                                1/1     Running   0          5m
vault-4                                1/1     Running   0          5m
vault-agent-injector-f4c4f579c-b2lv8   1/1     Running   0          8m

We have received Initial Root Token: s.BoYAMCWvueOavJC7zBbEhcs2 below, that can be used for the first login to the Vault and creating other users, groups, configurations, etc.

As an example I used web IU for creation a new secret

We can use CLI, API and Web UI for interaction with Vault. Let’s get the secret value from CLI:

/ $ vault kv get cubbyhole/prod/database/password
====== Data ======
Key         Value
---         -----
password    Supersecret123

Everything in Vault is “path based” and we can see it in DynamoDB table:

And all sensitive data is encrypted before storing in the backend, so we can not see actual secrets value:

If the backend is S3 bucket, a file with a secret looks like:

Audit logs of a secret store are very important for the production environment. The full list of production recommendations can be found here. In the Helm chart values above we set auditStorage. In our case Persistent volume claims are created and appropriate EBS volumes are provisioned and attached to Vault pods.

$ kubectl get pvc -n vault
NAME            STATUS   VOLUME             CAPACITY   ACCESS MODES   STORAGECLASS   AGE
audit-vault-0   Bound    pvc-0a2cc7aa-***   10Gi       RWO            gp2          27h
audit-vault-1   Bound    pvc-355da9c0-***   10Gi       RWO            gp2          27h
audit-vault-2   Bound    pvc-50b06581-***   10Gi       RWO            gp2          27h
audit-vault-3   Bound    pvc-01764bd1-***   10Gi       RWO            gp2          27h
audit-vault-4   Bound    pvc-b138651e-***   10Gi       RWO            gp2          27h
data-vault-0    Bound    pvc-da0a285c-***   10Gi       RWO            gp2          27h

/ $ df -h
Filesystem                Size      Used Available Use% Mounted on
....................................................................
/dev/nvme1n1              9.7G     36.0M      9.7G   0% /vault/audit
....................................................................

Next we need to enable the audit in the Vault once.

/ $ vault audit enable file file_path=/vault/audit/vault_audit.log
Success! Enabled the file audit device at: file/

/ $ vault audit list
Path     Type    Description
----     ----    -----------
file/    file    n/a

Then we can monitor activities and requests to Vault. We can see a timestamp, type of request, who did what and where.

$ kubectl exec -n vault -it vault-1 -- cat /vault/audit/vault_audit.log
....................
{"time":"2022-01-26T19:21:30.398813219Z","type":"request","auth":{"token_type":"default"},"request":{"id":"967fe6f0-8879-58f4-8dec-fb3b65279cbc","operation":"update","namespace":{"id":"root"},"path":"sys/audit/test"}}
...................
{"time":"2022-01-26T19:22:24.842012415Z","type":"request","auth":{"client_token":"hmac-sha256:712af35fc6509fc1eeda100c11d35a8aff1974c4abcd776694c2673bb3689096","accessor":"hmac-sha256:84108d0ee8c840e62ff1e7e8f2302defd9640ed6198c636da3f5820c82d2fcdd","display_name":"root","policies":["root"],"token_policies":["root"],"token_type":"service","token_issue_time":"2022-01-25T19:21:31Z"},"request":{"id":"bccb1d10-20fe-878a-481d-2535a581c7dd","operation":"read","mount_type":"system","client_token":"hmac-sha256:712af35fc6509fc1eeda100c11d35a8aff1974c4abcd776694c2673bb3689096","client_token_accessor":"hmac-sha256:84108d0ee8c840e62ff1e7e8f2302defd9640ed6198c636da3f5820c82d2fcdd","namespace":{"id":"root"},"path":"sys/internal/ui/mounts/internal/database/config","remote_address":"127.0.0.1"}}
...................

The process of secret injection is well described in official Vault documentation and no need to write it here.

Conclusion

There are many different external secrets storages that can be used for Kubernetes applications, one of them is Hashicorp Vault. Vault has the enterprise and open-source versions, and can be deployed to different clouds, platforms and operating systems. In this post we looked at deployment Vault in EKS cluster by Helm chart, with DynamoDB as backend, KMS for auto-unseal and EBS persistent volumes for audit logs. Vault was deployed in a highly available manner, data encrypted in transit and at rest, but as we saw sometimes it requires a number of manual steps. In the posts we will check other solutions, for example AWS Secrets and Configuration Provider (ASCP).