Container Security Best Practices: Protecting Your Containerized Applications

As organizations increasingly adopt containerization for application deployment, securing these environments has become a critical concern. Containers introduce unique security challenges that differ from traditional infrastructure, requiring specialized approaches and tools. From vulnerable base images to insecure runtime configurations, the attack surface for containerized applications is substantial and often overlooked.

This comprehensive guide explores container security best practices across the entire container lifecycle, providing practical strategies and tools to help DevOps teams build and maintain secure containerized environments.

Understanding Container Security Challenges

Containers present several unique security challenges:

Shared Kernel: Containers share the host OS kernel, creating potential isolation issues
Ephemeral Nature: Containers are short-lived, making traditional security monitoring difficult
Image Vulnerabilities: Container images may include vulnerable dependencies
Configuration Risks: Misconfigured containers can expose sensitive data or provide attack vectors
Supply Chain Concerns: Images from public registries may contain malicious code
Runtime Threats: Running containers face various attack vectors during execution

The Container Security Lifecycle

Effective container security requires a comprehensive approach across the entire lifecycle:

Build → Store → Deploy → Run → Dispose

Each phase requires specific security controls and practices, which we’ll explore in detail.

Secure Container Image Building

The foundation of container security begins with how you build your images.

1. Minimal Base Images

Use minimal, purpose-built base images:

# AVOID: Using full OS distributions
FROM ubuntu:22.04

# BETTER: Using minimal base images
FROM alpine:3.18

# BEST: Using distroless images
FROM gcr.io/distroless/static-debian11

2. Multi-Stage Builds

Implement multi-stage builds to reduce attack surface:

# Build stage
FROM node:18-alpine AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

# Production stage
FROM node:18-alpine
WORKDIR /app
ENV NODE_ENV=production
COPY --from=build /app/dist ./dist
COPY --from=build /app/package*.json ./
RUN npm ci --only=production
USER node
CMD ["node", "dist/server.js"]

3. Vulnerability Scanning

Integrate vulnerability scanning into your build process:

# GitHub Actions workflow with Trivy scanning
name: Container Security Scan

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  scan:
    name: Security Scan
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Build image
        run: docker build -t myapp:${{ github.sha }} .

      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: 'myapp:${{ github.sha }}'
          format: 'sarif'
          output: 'trivy-results.sarif'
          severity: 'CRITICAL,HIGH'
          exit-code: '1'
          ignore-unfixed: true

      - name: Upload Trivy scan results to GitHub Security tab
        uses: github/codeql-action/upload-sarif@v2
        if: always()
        with:
          sarif_file: 'trivy-results.sarif'

4. Secure Dependencies

Manage dependencies securely:

# Example .npmrc for secure Node.js dependencies
audit=true
fund=false
ignore-scripts=true
save-exact=true

# Example pip.conf for secure Python dependencies
[global]
no-cache-dir = false
require-hashes = true

5. Image Signing

Implement image signing to verify authenticity:

# Sign container image with cosign
cosign sign --key cosign.key myregistry.io/myapp:1.0.0

# Verify signed image
cosign verify --key cosign.pub myregistry.io/myapp:1.0.0

Secure Container Registry Management

Once built, container images need to be stored securely.

1. Private Registries

Use private registries with access controls:

# Example Harbor registry configuration
auth:
  # Harbor admin password
  adminPassword: "Harbor12345"
  # Use ldap, so we don't need to deal with user management in two places
  ldap:
    enabled: true
    url: "ldaps://ldap.example.com"
    searchDN: "cn=admin,dc=example,dc=com"
    searchPassword: "admin"
    baseDN: "dc=example,dc=com"
    filter: "(objectClass=person)"
    uid: "uid"
    groupBaseDN: "ou=Groups,dc=example,dc=com"
    groupSearchFilter: "objectClass=groupOfNames"
    groupAttributeName: "cn"

# Trivy vulnerability scanner integration
trivy:
  enabled: true
  resources:
    requests:
      cpu: 200m
      memory: 512Mi
    limits:
      cpu: 1
      memory: 1Gi

2. Image Promotion Workflows

Implement secure image promotion workflows:

# GitLab CI/CD pipeline for image promotion
stages:
  - build
  - scan
  - sign
  - promote

variables:
  DEV_REGISTRY: "registry.dev.example.com"
  PROD_REGISTRY: "registry.prod.example.com"
  IMAGE_NAME: "myapp"

build:
  stage: build
  script:
    - docker build -t $DEV_REGISTRY/$IMAGE_NAME:$CI_COMMIT_SHA .
    - docker push $DEV_REGISTRY/$IMAGE_NAME:$CI_COMMIT_SHA

scan:
  stage: scan
  script:
    - trivy image --exit-code 1 --severity HIGH,CRITICAL $DEV_REGISTRY/$IMAGE_NAME:$CI_COMMIT_SHA

sign:
  stage: sign
  script:
    - cosign sign --key $COSIGN_KEY $DEV_REGISTRY/$IMAGE_NAME:$CI_COMMIT_SHA

promote_to_production:
  stage: promote
  script:
    - docker pull $DEV_REGISTRY/$IMAGE_NAME:$CI_COMMIT_SHA
    - docker tag $DEV_REGISTRY/$IMAGE_NAME:$CI_COMMIT_SHA $PROD_REGISTRY/$IMAGE_NAME:$CI_COMMIT_SHA
    - docker tag $DEV_REGISTRY/$IMAGE_NAME:$CI_COMMIT_SHA $PROD_REGISTRY/$IMAGE_NAME:latest
    - docker push $PROD_REGISTRY/$IMAGE_NAME:$CI_COMMIT_SHA
    - docker push $PROD_REGISTRY/$IMAGE_NAME:latest
    - cosign sign --key $COSIGN_KEY $PROD_REGISTRY/$IMAGE_NAME:$CI_COMMIT_SHA
    - cosign sign --key $COSIGN_KEY $PROD_REGISTRY/$IMAGE_NAME:latest
  only:
    - main
  when: manual

3. Image Scanning Policies

Enforce scanning policies for registries:

# OPA Rego policy for image scanning
package imagecheck

default allow = false

# Allow images only if they have been scanned and have no HIGH or CRITICAL vulnerabilities
allow {
    input.scan_results.status == "COMPLETED"
    count(high_or_critical_vulnerabilities) == 0
}

# Find HIGH or CRITICAL vulnerabilities
high_or_critical_vulnerabilities[vuln] {
    vuln := input.scan_results.vulnerabilities[_]
    vuln.severity == "HIGH"
}

high_or_critical_vulnerabilities[vuln] {
    vuln := input.scan_results.vulnerabilities[_]
    vuln.severity == "CRITICAL"
}

Secure Container Deployment

Deploying containers securely requires careful configuration and policy enforcement.

1. Security Context Configuration

Configure security contexts for pods and containers:

# Kubernetes Pod with security context
apiVersion: v1
kind: Pod
metadata:
  name: secure-pod
spec:
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: app
    image: myregistry.io/myapp:1.0.0
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      readOnlyRootFilesystem: true
      runAsUser: 10001
      runAsGroup: 10001
    resources:
      limits:
        cpu: "1"
        memory: "512Mi"
      requests:
        cpu: "500m"
        memory: "256Mi"

2. Network Policies

Implement network policies to restrict communication:

# Kubernetes Network Policy
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-network-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api-service
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: database
    ports:
    - protocol: TCP
      port: 5432
  - to:
    - namespaceSelector:
        matchLabels:
          name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

3. Admission Controllers

Use admission controllers to enforce security policies:

# OPA Gatekeeper constraint template for requiring security contexts
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequiredsecuritycontext
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredSecurityContext
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredsecuritycontext

        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          not container.securityContext
          msg := sprintf("Container %v must have a security context", [container.name])
        }

        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          not container.securityContext.runAsNonRoot
          msg := sprintf("Container %v must set runAsNonRoot to true", [container.name])
        }

        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          not container.securityContext.allowPrivilegeEscalation == false
          msg := sprintf("Container %v must set allowPrivilegeEscalation to false", [container.name])
        }

# OPA Gatekeeper constraint for applying the template
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredSecurityContext
metadata:
  name: require-security-context
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    excludedNamespaces: ["kube-system"]

4. Secret Management

Implement secure secret management:

# Kubernetes Secret with external-secrets operator
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
spec:
  refreshInterval: "15m"
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: database-credentials
    creationPolicy: Owner
  data:
  - secretKey: username
    remoteRef:
      key: database/creds/api
      property: username
  - secretKey: password
    remoteRef:
      key: database/creds/api
      property: password

Runtime Container Security

Securing containers during runtime is critical for protecting against active threats.

1. Runtime Security Monitoring

Implement runtime security monitoring:

# Falco security rules
- rule: Terminal shell in container
  desc: A shell was spawned by a container with an attached terminal
  condition: >
    container and
    shell_procs and container_entrypoint
    and interactive
    and not container_name in (allowed_containers)
  output: >
    Shell spawned in a container with an attached terminal (user=%user.name
    container_id=%container.id container_name=%container.name shell=%proc.name
    parent=%proc.pname cmdline=%proc.cmdline terminal=%proc.tty container_entrypoint=%container.entrypoint)
  priority: NOTICE
  tags: [container, shell, mitre_execution]

- rule: File Access in Sensitive Directories
  desc: Detect file access in sensitive directories
  condition: >
    open_read and
    container and
    sensitive_files and
    not container_name in (allowed_containers)
  output: >
    Sensitive file accessed in container (user=%user.name
    command=%proc.cmdline file=%fd.name container_id=%container.id container_name=%container.name)
  priority: WARNING
  tags: [container, filesystem, mitre_credential_access]

2. Container Behavioral Analysis

Implement behavioral analysis for containers:

# Tracee runtime security configuration
apiVersion: tracee.aquasec.com/v1alpha1
kind: Policy
metadata:
  name: runtime-security-policy
spec:
  scope:
    - name: container
      value: "*"
  rules:
    - event: security_file_open
      filter:
        - field: flags
          operator: contains
          value: O_CREAT
        - field: pathname
          operator: prefix
          value: /etc/
      actions:
        - alert
    
    - event: security_socket_connect
      filter:
        - field: remote_address
          operator: not-in
          values:
            - "10.0.0.0/8"
            - "172.16.0.0/12"
            - "192.168.0.0/16"
      actions:
        - alert
        - log
    
    - event: security_bpf
      actions:
        - alert
        - log

3. Immutable Containers

Implement immutable containers:

# Kubernetes Pod with immutable filesystem
apiVersion: v1
kind: Pod
metadata:
  name: immutable-pod
spec:
  containers:
  - name: app
    image: myregistry.io/myapp:1.0.0
    securityContext:
      readOnlyRootFilesystem: true
    volumeMounts:
    - name: tmp
      mountPath: /tmp
    - name: cache
      mountPath: /var/cache
  volumes:
  - name: tmp
    emptyDir: {}
  - name: cache
    emptyDir: {}

4. Runtime Vulnerability Management

Implement runtime vulnerability management:

# Starboard Operator configuration
apiVersion: aquasecurity.github.io/v1alpha1
kind: Starboard
metadata:
  name: starboard
spec:
  operator:
    vulnerabilityReports:
      scanner: Trivy
    configAuditReports:
      scanner: Polaris
    scanJobTimeout: 10m
    scanJobTTL: 24h
  trivy:
    severity: CRITICAL,HIGH
    ignoreUnfixed: true
  polaris:
    config:
      checks:
        - hostIPCSet
        - hostPIDSet
        - hostNetworkSet
        - notReadOnlyRootFilesystem
        - privilegeEscalationAllowed
        - runAsRootAllowed
        - hostPortSet

Container Orchestration Security

Securing the container orchestration platform is essential for overall container security.

1. Kubernetes Security Posture

Implement Kubernetes security posture management:

# kube-bench configuration
apiVersion: batch/v1
kind: CronJob
metadata:
  name: kube-bench
  namespace: security
spec:
  schedule: "0 1 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          hostPID: true
          containers:
          - name: kube-bench
            image: aquasec/kube-bench:latest
            command:
            - kube-bench
            - --benchmark
            - cis-1.6
            volumeMounts:
            - name: var-lib-kubelet
              mountPath: /var/lib/kubelet
              readOnly: true
            - name: etc-systemd
              mountPath: /etc/systemd
              readOnly: true
            - name: etc-kubernetes
              mountPath: /etc/kubernetes
              readOnly: true
          restartPolicy: Never
          volumes:
          - name: var-lib-kubelet
            hostPath:
              path: /var/lib/kubelet
          - name: etc-systemd
            hostPath:
              path: /etc/systemd
          - name: etc-kubernetes
            hostPath:
              path: /etc/kubernetes

2. RBAC Configuration

Implement least-privilege RBAC:

# Kubernetes RBAC configuration
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: production
  name: app-reader
rules:
- apiGroups: [""]
  resources: ["pods", "services"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: app-reader-binding
  namespace: production
subjects:
- kind: ServiceAccount
  name: app-service-account
  namespace: production
roleRef:
  kind: Role
  name: app-reader
  apiGroup: rbac.authorization.k8s.io

3. Pod Security Standards

Implement Pod Security Standards:

# Kubernetes Pod Security Standards
apiVersion: pod-security.kubernetes.io/v1
kind: PodSecurityStandard
metadata:
  name: restricted
spec:
  enforce: restricted
  enforce-version: latest
  audit: restricted
  audit-version: latest
  warn: restricted
  warn-version: latest

# Namespace with Pod Security Standards
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: latest
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: latest

Supply Chain Security

Securing the container supply chain is crucial for comprehensive security.

1. Software Bill of Materials (SBOM)

Generate and verify SBOMs for containers:

# Generate SBOM with Syft
syft myregistry.io/myapp:1.0.0 -o spdx-json > sbom.json

# Verify SBOM against policy
grype sbom:./sbom.json --fail-on high

2. Artifact Provenance

Implement artifact provenance with Sigstore:

# Generate provenance attestation
cosign attest --key cosign.key --type slsaprovenance --predicate provenance.json myregistry.io/myapp:1.0.0

# Verify attestation
cosign verify-attestation --key cosign.pub myregistry.io/myapp:1.0.0

3. Policy as Code

Implement policy as code for supply chain security:

# Open Policy Agent policy for image verification
package imagecheck

default allow = false

# Allow images only if they are from trusted registries and signed
allow {
    # Check if image is from trusted registry
    startswith(input.image, "myregistry.io/")
    
    # Check if image has valid signature
    input.signatures[_].verified == true
    
    # Check if image has valid attestation
    input.attestations[_].verified == true
}

Container Security Automation

Automating container security is essential for scaling security practices.

1. Security in CI/CD

Integrate security into CI/CD pipelines:

# GitHub Actions workflow with comprehensive security checks
name: Container Security Pipeline

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  security-checks:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3
      
      - name: Lint Dockerfile
        uses: hadolint/[email protected]
        with:
          dockerfile: Dockerfile
      
      - name: Build image
        run: docker build -t myapp:${{ github.sha }} .
      
      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: 'myapp:${{ github.sha }}'
          format: 'sarif'
          output: 'trivy-results.sarif'
          severity: 'CRITICAL,HIGH'
      
      - name: Generate SBOM
        uses: anchore/sbom-action@v0
        with:
          image: myapp:${{ github.sha }}
          artifact-name: sbom.spdx.json
          output-file: ./sbom.spdx.json
      
      - name: Check SBOM for vulnerabilities
        uses: anchore/scan-action@v3
        with:
          sbom: ./sbom.spdx.json
          severity-cutoff: high
      
      - name: Check for secrets in code
        uses: gitleaks/gitleaks-action@v2
        with:
          config-path: .gitleaks.toml
      
      - name: Push to registry
        if: github.event_name != 'pull_request'
        run: |
          echo ${{ secrets.REGISTRY_TOKEN }} | docker login -u ${{ secrets.REGISTRY_USER }} --password-stdin myregistry.io
          docker tag myapp:${{ github.sha }} myregistry.io/myapp:${{ github.sha }}
          docker push myregistry.io/myapp:${{ github.sha }}
      
      - name: Sign container image
        if: github.event_name != 'pull_request'
        uses: sigstore/cosign-installer@main
        with:
          cosign-release: 'v1.13.1'
      - run: |
          echo "${{ secrets.COSIGN_PRIVATE_KEY }}" > cosign.key
          cosign sign --key cosign.key myregistry.io/myapp:${{ github.sha }}

2. Continuous Vulnerability Management

Implement continuous vulnerability management:

# Trivy Operator configuration
apiVersion: aquasecurity.github.io/v1alpha1
kind: TrivyOperator
metadata:
  name: trivy-operator
spec:
  vulnerabilityReports:
    schedule: "0 */6 * * *"  # Every 6 hours
    scanJobTimeout: 5m
  configAuditReports:
    enabled: true
  rbacAssessmentReports:
    enabled: true
  infraAssessmentReports:
    enabled: true
  resources:
    requests:
      cpu: 100m
      memory: 100Mi
    limits:
      cpu: 500m
      memory: 500Mi

3. Security Monitoring and Alerting

Implement security monitoring and alerting:

# Prometheus AlertManager rules for container security
groups:
- name: ContainerSecurityAlerts
  rules:
  - alert: VulnerableContainerDetected
    expr: trivy_image_vulnerabilities{severity="CRITICAL"} > 0
    for: 10m
    labels:
      severity: critical
    annotations:
      summary: "Vulnerable container detected"
      description: "Container {{ $labels.container_name }} in {{ $labels.namespace }} has {{ $value }} critical vulnerabilities"

  - alert: PrivilegedContainerDetected
    expr: kube_pod_container_info{container!="", container_id!=""} * on(container, pod, namespace) group_left() kube_pod_container_status_running * on(pod, namespace) group_left(securityContext_privileged) kube_pod_container_security_context{securityContext_privileged="true"} > 0
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Privileged container detected"
      description: "Container {{ $labels.container }} in pod {{ $labels.pod }} in namespace {{ $labels.namespace }} is running with privileged permissions"

  - alert: ContainerWithoutResourceLimits
    expr: kube_pod_container_info{container!="", container_id!=""} * on(container, pod, namespace) group_left() kube_pod_container_status_running * on(container, pod, namespace) group_left() (kube_pod_container_resource_limits_cpu_cores == 0 or absent(kube_pod_container_resource_limits_cpu_cores))
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "Container without resource limits"
      description: "Container {{ $labels.container }} in pod {{ $labels.pod }} in namespace {{ $labels.namespace }} is running without CPU or memory limits"

Conclusion: Building a Container Security Program

Securing containerized applications requires a comprehensive approach that addresses security at every phase of the container lifecycle. By implementing the best practices outlined in this guide, organizations can significantly reduce the risk of container-related security incidents.

Key takeaways for building an effective container security program:

Shift Left: Integrate security into the earliest stages of container development
Defense in Depth: Implement multiple layers of security controls
Least Privilege: Apply the principle of least privilege throughout your container ecosystem
Automation: Automate security checks and controls to scale security efforts
Continuous Monitoring: Implement continuous monitoring for runtime threats
Regular Updates: Keep container images and dependencies updated
Security Culture: Foster a culture of security awareness among development and operations teams

Remember that container security is not a one-time effort but an ongoing process that requires continuous attention and improvement. By following these best practices and staying informed about emerging threats and security techniques, you can build and maintain secure containerized environments that support your organization’s innovation goals while protecting critical assets.