TLS in Ingress with Hashicorp Vault¶

Note

This document describes setting up self-rotating TLS certificates in a Kubernetes cluster using Hashicorp Vault. Some steps may vary based on a target applicaition. The guide covers creating roles and issuing certificates for an application in vault as well as Kubernetes side of things.

Tip

Each app should have it's own TLS certificate issued. Avoid issuing single certificate for multiple services. Avoid using wildcard certificates. This is a security risk and may lead to certificate hijacking.

Prerequisites¶

Kubernetes cluster
Hashicorp Vault installed and configured
Vault Secrets Operator setup [./]
Kubernetes Authentication method enabled in Vault [./]

Installation¶

Step. 1 Authenticating with Vault¶

Create a new role in Vault Kubernetes authentication method. This role will be used to authenticate the application with Vault.
```
vault write auth/kubernetes/role/<role-name> \
    bound_service_account_names=<service-account-name> \
    bound_service_account_namespaces=<namespace> \
    policies=pki-ingress
```
In policies field we explicitly mention pki-ingress as it's a generic policy allowing all applications access to PKI secrets (with additional permissions for issuing certificates).

And then a VaultAuth resource that will be responsible for Kubernetes side of authentication:

apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: grafana
  name: <sa-name>
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: <app>-auth
  namespace: <namespace>
spec:
  method: kubernetes
  mount: kubernetes # Name of the auth method in Vault
  kubernetes:
    role: <role-name>
    serviceAccount: <sa-name>

Tip

You might want to use pre-existing service account instead of creating new one, as all pods already have one assigned at creation time.

Step. 2 [Optional] Access Policy¶

# Allow issuing certificates via a specific role
path "pki/issue/*" {
  capabilities = ["update"]
}

# Allow reading and listing issued certs (optional, for debugging/logging)
path "pki/cert/*" {
  capabilities = ["read", "list"]
}

# Allow fetching CA chain (useful if storing chain manually or verifying certs)
path "pki/ca_chain" {
  capabilities = ["read"]
}

# Allow reading the root CA if needed
path "pki/cert/ca" {
  capabilities = ["read"]
}

# (Optional) allow revocation if you want the controller to revoke issued certs
path "pki/revoke" {
  capabilities = ["update"]
}

This should be already existing in the Vault instance.

Now, this definition is a little broad, as it allows issuing certificates for any role. This is mitigated by the fact that VaultPKISecret only allows issuing certificates by specific role, but if you want more strict security this should be split into generic:

# Allow reading and listing issued certs (optional, for debugging/logging)
path "pki/cert/*" {
  capabilities = ["read", "list"]
}

# Allow fetching CA chain (useful if storing chain manually or verifying certs)
path "pki/ca_chain" {
  capabilities = ["read"]
}

# Allow reading the root CA if needed
path "pki/cert/ca" {
  capabilities = ["read"]
}

# (Optional) allow revocation if you want the controller to revoke issued certs
path "pki/revoke" {
  capabilities = ["update"]
}

And then each (PKI) role should have it's own policy for issuing certificates

# Allow issuing certificates via a specific role
path "pki/issue/*" {
  capabilities = ["update"]
}

Note

While this is a good practice, it is not strictly necessary if you have full trust in the environment (VSO) that will be issuing the certificates. The VaultPKISecret will only allow issuing certificates for the role specified in the role field, so even if you have a broad policy, it won't be able to issue certificates for other roles, the risk is if something else will be able to authenticate it's unauthorized VaultPKISecret with the same kubernetes role.

Step. 3 Create a PKI role¶

This is a role that will be used directly by the secret engine to issue certificates. This role should be created

Now, the allowed_domains field should be set to full domain your application will use, so i.e:

allow_subdomains and allow_glob_domains should be set to false, as this will prevent issuing

Where by default this common_name would be rejected. in zero-trust manner and it's SANs should be strictly limited to the application that will be using it. This is a good practice as it limits the scope of the certificate and reduces the risk of certificate hijacking. href="#__codelineno-5-1">vault write pki/roles/<role-name> \ allowed_domains=<domain> \ allow_subdomains=false \ allow_glob_domains=false \ allow_any_name=false \ enforce_hostnames=true \ max_ttl=720h \ ttl=24h \ allow_bare_domains=true href="#__codelineno-6-1">vault write pki/roles/grafana-tls \ allowed_domains=grafana.example.com \ allow_subdomains=false \ allow_glob_domains=false \ allow_any_name=false \ enforce_hostnames=true \ max_ttl=720h \ ttl=24h \ allow_bare_domains=true certificates for subdomains or wildcard domains. This limits scope of certificate to only the domains mentioned in allowed_domains, while allow_bare_domains set to true will allow you to issue a certificate for exact match to the list. So it would allow me to issue a certificate like this: href="#__codelineno-7-1">vault write pki/issue/grafana-tls \ common_name=grafana.example.com \ ttl=24h

Step. 4 Create a VaultPKISecret¶

This is a custom resource that will be used to issue the certificate. This resource will be used by the Vault Secrets Operator to issue the certificate and store it in Kubernetes secret, it will also automatically manage it's lifecycle and issue new one as needed.

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultPKISecret
metadata:
  name: <secret-name> 
  namespace: <namespace>
spec:
  vaultAuthRef: <app>-auth # Same as in VaultAuth in step 1
  mount: pki # Name of the PKI mount in Vault
  role: <pki-role-name> # Name of the PKI role in Vault
  commonName: grafana.wmsdev.pl # Common name for the certificate
  ttl: 720h # requested TTL for the certificate
  format: pem # Format of the certificate
  destination: # Kubernetes secret definition
    create: true
    name: tls-cert

Step 5. Apply¶

Now with everything in place we can apply the resources to the cluster:

kubectl apply -f /path/to/vaultAuth.yaml
kubectl apply -f /path/to/vaultPKISecret.yaml

After short period of time you should be able to see that both resources were reconciled by the operator, if not refer to Events to check for any mismatches (most commonly issue with the certificate CN or SANs). If everything is correct you should be able to retrieve the certificate from the secret:

kubectl get secret tls-cert -n <namespace> -o yaml

Step 6. Ingress¶

Now here is where it gets application specific, as many Operators and Helm Charts implement their own ways to configure TLS, that's why we're gonna show it on a genericIngress resource. Usage is very simple, because what VSO creates is a standard Kubernetes secret, so you can easily reference it as you'd do with manualy imported TLS certificate:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  namespace: <namespace>
spec:
  tls:
    - hosts:
        - example.com
      secretName: tls-cert
  rules:
    - host: example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: example-service
                port:
                  number: 80