Secrets & Configuration Security
Managing secrets in Kubernetes is where many security implementations fall apart. I’ve seen organizations with excellent network policies and pod security configurations completely undermined by secrets stored in plain text ConfigMaps or hardcoded in container images. The challenge isn’t just storing secrets securely—it’s managing their entire lifecycle from creation to rotation to deletion.
Kubernetes provides several mechanisms for handling sensitive data, but the built-in Secret resource is just the starting point. Real production environments require encryption at rest, proper access controls, secret rotation, and integration with external secret management systems. The goal is to ensure that secrets are never exposed in logs, configuration files, or container images.
Understanding Kubernetes Secrets Architecture
Kubernetes Secrets are stored in etcd, the cluster’s data store, and by default they’re only base64 encoded—not encrypted. This means anyone with etcd access can read all your secrets. The first step in securing secrets is enabling encryption at rest, which encrypts secret data before it’s written to etcd.
Here’s how to configure encryption at rest using the EncryptionConfiguration:
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: <base64-encoded-32-byte-key>
- identity: {}
The API server uses this configuration to encrypt secrets before storing them in etcd. The identity provider at the end ensures that if decryption fails, the API server can still read unencrypted data during the migration process.
Creating secrets properly involves understanding the different types and their intended use cases. Generic secrets work for most applications, but TLS secrets and service account tokens have specific formats and handling requirements:
apiVersion: v1
kind: Secret
metadata:
name: database-credentials
namespace: production
type: Opaque
data:
username: <base64-encoded-username>
password: <base64-encoded-password>
connection-string: <base64-encoded-connection-string>
Secure Secret Consumption Patterns
How your applications consume secrets is just as important as how they’re stored. The most secure approach is mounting secrets as volumes rather than using environment variables. Environment variables can be exposed in process lists, logs, and crash dumps, while mounted secrets exist only in memory-backed filesystems.
apiVersion: apps/v1
kind: Deployment
metadata:
name: secure-app
spec:
template:
spec:
containers:
- name: app
image: myapp:latest
volumeMounts:
- name: db-credentials
mountPath: /etc/secrets/db
readOnly: true
- name: api-keys
mountPath: /etc/secrets/api
readOnly: true
volumes:
- name: db-credentials
secret:
secretName: database-credentials
defaultMode: 0400
- name: api-keys
secret:
secretName: api-keys
defaultMode: 0400
The defaultMode: 0400 setting ensures that secret files are readable only by the file owner, adding an additional layer of protection. Your application code should read these files at startup and keep the sensitive data in memory rather than repeatedly accessing the filesystem.
For applications that must use environment variables, you can still reference secrets securely:
spec:
containers:
- name: app
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: database-credentials
key: password
However, I strongly recommend the volume mount approach whenever possible, especially for highly sensitive credentials.
External Secret Management Integration
While Kubernetes Secrets work for basic use cases, production environments often require integration with external secret management systems like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault. These systems provide advanced features like automatic rotation, detailed audit logs, and centralized policy management.
The External Secrets Operator provides a Kubernetes-native way to sync secrets from external systems:
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: vault-backend
namespace: production
spec:
provider:
vault:
server: "https://vault.company.com"
path: "secret"
version: "v2"
auth:
kubernetes:
mountPath: "kubernetes"
role: "production-role"
This SecretStore configuration connects to a Vault instance and uses Kubernetes authentication. The External Secrets Operator can then create Kubernetes Secrets based on data stored in Vault:
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: database-secret
namespace: production
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-backend
kind: SecretStore
target:
name: database-credentials
creationPolicy: Owner
data:
- secretKey: password
remoteRef:
key: database/production
property: password
Secret Rotation and Lifecycle Management
Secret rotation is critical for maintaining security over time, but it’s also one of the most challenging aspects of secret management. The key is designing your applications to handle secret updates gracefully without requiring restarts or causing service disruptions.
Kubernetes supports automatic secret updates when secrets are mounted as volumes. When you update a secret, Kubernetes eventually updates the mounted files in running pods. However, your application needs to detect these changes and reload the credentials:
apiVersion: v1
kind: Secret
metadata:
name: rotating-credentials
annotations:
reloader.stakater.com/match: "true"
data:
api-key: <new-base64-encoded-key>
Tools like Reloader can automatically restart deployments when secrets change, but this approach causes service disruption. A better pattern is implementing graceful secret reloading in your application code, watching for file system changes and updating credentials without restarting.
Configuration Security Beyond Secrets
Not all sensitive configuration data qualifies as a secret, but it still needs protection. ConfigMaps containing database connection strings, API endpoints, or feature flags can reveal valuable information about your infrastructure. The principle of least privilege applies here too—only the pods that need specific configuration should have access to it.
apiVersion: v1
kind: ConfigMap
metadata:
name: app-config
namespace: production
data:
database-host: "prod-db.internal.company.com"
cache-endpoint: "redis.prod.svc.cluster.local"
feature-flags: |
{
"new-feature": true,
"beta-feature": false
}
Use RBAC to control access to ConfigMaps just as you would for Secrets. Different teams or applications should have access only to the configuration data they need.
Audit and Compliance Considerations
Secret access should be thoroughly audited, especially in regulated environments. Kubernetes audit logs can track secret access, but you need to configure audit policies to capture the right events without overwhelming your log storage:
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
resources:
- group: ""
resources: ["secrets"]
namespaces: ["production", "staging"]
This audit policy logs metadata for all secret operations in production and staging namespaces. The logs help you understand who’s accessing secrets and when, which is crucial for security investigations and compliance reporting.
Regular secret scanning and rotation policies should be part of your security program. Automated tools can detect secrets that haven’t been rotated within policy timeframes and alert security teams to potential issues.
The secure handling of secrets and configuration data creates the foundation for trustworthy applications. In the next part, we’ll explore monitoring and compliance, focusing on how to detect security issues, enforce policies, and maintain visibility into your cluster’s security posture.