Secrets¶
Secrets management and identity infrastructure for Kubernetes and cloud environments — covering external provider sync, file-level encryption, identity-based vaults, and open-source IAM.
Topics¶
| Tool | Description |
|---|---|
| External Secrets Operator | Kubernetes operator that syncs secrets from external providers (Vault, AWS SM, GCP SM, Azure KV) into native K8s Secrets via CRDs. |
| SOPS | CLI tool for encrypting structured files (YAML, JSON, ENV) in-place — the GitOps standard for secrets-in-Git using age, AWS KMS, or PGP. |
| HashiCorp Vault | Industry-standard identity-based secrets management — dynamic secrets, encryption-as-a-service, PKI, and multi-backend replication. |
| Zitadel | Open-source identity and access management — event-sourced IAM with OIDC, SAML 2.0, passkeys, multi-tenant RBAC, and SCIM provisioning. |
Comparisons¶
| Comparison | Scope |
|---|---|
| Secrets Management Comparison | ESO vs SOPS vs Vault — use cases, operational overhead, and integration patterns |
Landscape¶
Secrets management has evolved from static credential storage to dynamic, identity-based access models driven by zero-trust security principles. The central challenge — "secrets sprawl" — refers to credentials scattered across environment variables, CI/CD pipeline configs, Kubernetes Secrets (base64-encoded but not encrypted), Helm values files, and developer laptops, making rotation and audit nearly impossible.
The GitOps movement intensified this problem by storing application configuration in Git, requiring solutions that either encrypt secrets in-repo (SOPS, sealed-secrets) or reference external secret stores (External Secrets Operator). HashiCorp Vault remains the industry standard for centralized secrets management, but its operational complexity has opened space for cloud-provider-native solutions (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault) that trade portability for managed simplicity.
Identity-First Security
The frontier is identity-based access: workloads authenticate to secret stores via OIDC federation (IRSA for AWS, Workload Identity for GCP), SPIFFE identities, or Kubernetes service account tokens — eliminating long-lived credentials entirely in favor of short-lived, automatically rotated tokens. This pattern is central to zero-trust architecture where no workload is trusted by default, regardless of network location.
The Kubernetes Secrets CSI Driver has emerged as an alternative to environment variables and mounted volumes — it mounts secrets directly from external stores (Vault, AWS SM, Azure KV) as tmpfs volumes without creating Kubernetes Secret objects, reducing the attack surface of etcd-stored secrets. Secret rotation remains a key operational challenge, particularly for database credentials and TLS certificates that require coordinated application restarts or connection pool refreshes.
Key Concepts¶
Dynamic Secrets¶
Secrets generated on-demand with automatic expiration, eliminating the risk of long-lived credential leakage. Vault's database secrets engine, for example, creates a unique PostgreSQL username/password pair for each requesting application with a configurable TTL (e.g., 1 hour). When the lease expires, Vault automatically revokes the credential.
This approach ensures that even if a secret is compromised, the blast radius is time-limited and auditable — Vault logs every secret issuance and revocation. Dynamic secret engines exist for databases (PostgreSQL, MySQL, MongoDB), cloud providers (AWS STS, GCP service accounts), PKI certificates, SSH certificates, and more.
Envelope Encryption¶
How It Works
A two-layer encryption scheme where data is encrypted with a Data Encryption Key (DEK), and the DEK itself is encrypted with a Key Encryption Key (KEK) stored in a KMS. SOPS implements this pattern: it encrypts each value in a YAML file with a unique DEK, then wraps that DEK with AWS KMS, GCP KMS, Azure Key Vault, or age keys. This means the encrypted file can be safely stored in Git — decryption requires access to the KMS. Vault's transit secrets engine provides envelope encryption as a service.
External Secrets Sync¶
The pattern of synchronizing secrets from an external provider into Kubernetes-native Secret objects, bridging the gap between centralized secret stores and workloads that expect secrets as mounted files or environment variables. External Secrets Operator (ESO) polls or watches an external source (Vault, AWS Secrets Manager, GCP Secret Manager) and creates/updates Kubernetes Secrets to match.
ESO supports templating, allowing secrets from multiple sources to be combined into a single K8s Secret with custom key names and formats. The sync is configured via two CRDs:
- SecretStore / ClusterSecretStore: Defines the connection to the external provider (endpoint, authentication method, CA bundle)
- ExternalSecret / ClusterExternalSecret: Specifies which secrets to fetch, how to map keys, and the refresh interval
OIDC Federation¶
A mechanism where workloads authenticate to external services using short-lived OIDC tokens issued by their platform's identity provider, eliminating the need for static credentials. Kubernetes issues projected service account tokens that conform to the OIDC spec; cloud providers validate these tokens and grant temporary cloud credentials.
Implementations by cloud provider:
- AWS IRSA: Maps Kubernetes ServiceAccounts to IAM Roles via an OIDC identity provider registered in IAM
- GCP Workload Identity: Binds Kubernetes ServiceAccounts to GCP IAM service accounts via a workload identity pool
- Azure Workload Identity: Uses federated credentials on Azure AD applications to trust Kubernetes-issued tokens
Vault supports OIDC/JWT auth methods, allowing Kubernetes pods to authenticate without any pre-provisioned secret.
Identity-Based Access¶
A zero-trust model where access to secrets is determined by the authenticated identity of the requesting entity rather than by network location or static credentials. Vault pioneered this approach with its policy engine: policies bind to identities (Kubernetes service accounts, LDAP groups, OIDC claims), and each identity receives the minimum set of secrets required.
Combined with dynamic secrets and automatic revocation, identity-based access ensures that credentials are both narrowly scoped and ephemeral — a compromised workload cannot escalate beyond its granted identity. The principle extends beyond Vault: cloud IAM policies, Kubernetes RBAC, and database role grants all implement variations of identity-based access control.
Open Questions¶
- As cloud-provider-native secret managers (AWS Secrets Manager, GCP Secret Manager) add features like automatic rotation and cross-region replication, does self-hosted Vault remain justified for organizations already committed to a single cloud?
- What is the right boundary between SOPS (encrypt-in-Git) and ESO (sync-from-provider) in a GitOps workflow — should SOPS be limited to bootstrap secrets, with ESO handling all application secrets?
- How does SPIFFE/SPIRE compare to Kubernetes-native OIDC federation for workload identity in multi-cluster or hybrid environments, and when does the added complexity of SPIFFE pay off?