Learn the core concepts and practical skills needed to deploy, manage, and scale containerized applications using Kubernetes.

Introduction and Setup

Introduction to Kubernetes

Container orchestration sounds complicated, but the problem it solves is simple: how do you run dozens or hundreds of containers across multiple servers without losing your sanity? Docker works great for single containers, but when you need to manage entire applications with databases, web servers, and background workers, you quickly realize you need something more sophisticated.

Kubernetes (or K8s if you’re feeling fancy) is basically your operations team in software form. Google built it after running containers at massive scale for years, and they open-sourced it because, frankly, everyone was going to need this eventually.

What Problem Does Kubernetes Actually Solve?

Here’s the thing—containers are great until you have more than a few of them. Then you’re stuck with questions like: Which server should this container run on? What happens when a server dies? How do I update my app without downtime? Kubernetes answers all of these automatically.

Think about it this way: without Kubernetes, you’re manually placing containers on servers like you’re playing Tetris. With Kubernetes, you just tell it what you want running, and it figures out the rest. Need five copies of your web app? Done. One of them crashes? Kubernetes starts a new one before you even notice.

# You write this simple config
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-web-app
spec:
  replicas: 5
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: web
        image: nginx

And Kubernetes handles all the complexity behind the scenes—where to run these containers, what to do when they fail, how to route traffic to them. It’s like having a really smart operations person who never sleeps.

How Kubernetes Actually Works

Let’s break down what’s happening under the hood, but without the enterprise architecture diagrams that make your eyes glaze over.

The Control Plane (The Brain)

The control plane is where all the decision-making happens. It’s usually running on separate servers (called master nodes) and consists of a few key components:

API Server - This is your main interface to Kubernetes. Every kubectl command, every dashboard click, every automated deployment goes through here. Think of it as the receptionist who knows everything about your cluster.

etcd - The cluster’s memory. It’s a database that stores the current state of everything—which pods are running where, what your configurations look like, etc. If etcd goes down, Kubernetes gets amnesia.

Scheduler - The matchmaker. When you want to run a new pod, the scheduler looks at all your servers and decides which one should run it based on resources, constraints, and a bunch of other factors.

Controller Manager - The enforcer. It constantly watches the actual state of your cluster and compares it to what you said you wanted. If something’s off, it fixes it.

The Worker Nodes (The Muscle)

These are the servers where your actual applications run. Each worker node has:

kubelet - The local agent that talks to the control plane and manages containers on this specific node. It’s like a site manager who takes orders from headquarters.

kube-proxy - Handles networking so your pods can talk to each other and the outside world.

Container Runtime - Docker, containerd, or whatever actually runs your containers.

The Building Blocks You Need to Know

Pods - The Basic Unit

A pod is the smallest thing you can deploy in Kubernetes. Most of the time, it’s just one container, but sometimes you might have a few containers that need to work closely together (like a web server and a logging sidecar).

Here’s the key thing about pods: they’re ephemeral. They come and go. Don’t get attached to them. If a pod dies, Kubernetes will start a new one with a different IP address and possibly on a different server.

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  containers:
  - name: web
    image: nginx
    ports:
    - containerPort: 80

Deployments - The Reliable Way to Run Things

You almost never create pods directly. Instead, you create a Deployment, which manages pods for you. Want three copies of your app? A Deployment will make sure you always have three running, even if servers crash.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: web
        image: nginx

Services - How Things Talk to Each Other

Since pods come and go with different IP addresses, you need a stable way for them to find each other. That’s what Services do—they provide a consistent endpoint that routes traffic to healthy pods.

apiVersion: v1
kind: Service
metadata:
  name: web-service
spec:
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 80
  type: LoadBalancer

Getting Kubernetes Running Locally

Alright, enough theory. Let’s get you a Kubernetes cluster to play with. You’ve got a few options, and honestly, they all work fine—it’s more about what you’re comfortable with.

Option 1: Minikube (My Personal Favorite for Learning)

Minikube is probably the easiest way to get started. It creates a single-node cluster on your laptop, which is perfect for learning and testing.

# On macOS
brew install minikube

# On Linux
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube

# Start your cluster
minikube start

# Check if it's working
kubectl get nodes

The nice thing about Minikube is it comes with a web dashboard that makes it easy to see what’s happening:

minikube dashboard

Option 2: Docker Desktop (If You’re Already Using It)

If you’ve got Docker Desktop installed, you can just enable Kubernetes in the settings. It’s dead simple—just check a box and restart Docker Desktop. Then you can verify it’s working:

kubectl cluster-info
kubectl get nodes

Option 3: Kind (Kubernetes in Docker)

Kind is great if you want to test multi-node clusters or if you’re doing CI/CD stuff. It runs Kubernetes inside Docker containers.

# Install it
brew install kind  # macOS

# Create a cluster
kind create cluster --name learning-cluster

# Use it
kubectl cluster-info --context kind-learning-cluster

Installing kubectl (Your New Best Friend)

kubectl (pronounced “kube-control” or “kube-cuttle”—I’ve heard both) is how you talk to Kubernetes. Think of it as your remote control for the cluster.

# On macOS (easiest way)
brew install kubectl

# On Linux
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

# Check if it's working
kubectl version --client

Let’s Actually Use This Thing

Now that you’ve got Kubernetes running, let’s do something with it. I’m going to walk you through the commands I use every day.

The Commands You’ll Use Most

# See what's in your cluster
kubectl get nodes
kubectl get pods
kubectl get services

# Get more details about something
kubectl describe pod some-pod-name

# See what's happening (this one's a lifesaver)
kubectl get events --sort-by=.metadata.creationTimestamp

# Follow logs in real-time
kubectl logs -f pod-name

The get command is probably what you’ll use most. It shows you what’s running, and you can add -o wide to get more details or -w to watch things change in real-time.

Your First Real Application

Let’s skip the “hello world” stuff and build something you might actually deploy. We’ll create a simple web app with a database—nothing fancy, but it’ll show you how the pieces fit together.

Step 1: Deploy a Database

First, let’s get MySQL running. In the real world, you’d probably use a managed database, but this is good for learning:

# Save this as mysql.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:8.0
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: "password123"
        - name: MYSQL_DATABASE
          value: "myapp"
        ports:
        - containerPort: 3306
---
apiVersion: v1
kind: Service
metadata:
  name: mysql-service
spec:
  selector:
    app: mysql
  ports:
  - port: 3306
  type: ClusterIP

Deploy it:

kubectl apply -f mysql.yaml
kubectl get pods -w  # Watch it start up

Step 2: Deploy the Web App

Now let’s add a WordPress frontend that connects to our MySQL database:

# Save this as wordpress.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: wordpress
spec:
  replicas: 2
  selector:
    matchLabels:
      app: wordpress
  template:
    metadata:
      labels:
        app: wordpress
    spec:
      containers:
      - name: wordpress
        image: wordpress:latest
        env:
        - name: WORDPRESS_DB_HOST
          value: "mysql-service:3306"
        - name: WORDPRESS_DB_NAME
          value: "myapp"
        - name: WORDPRESS_DB_PASSWORD
          value: "password123"
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: wordpress-service
spec:
  selector:
    app: wordpress
  ports:
  - port: 80
    nodePort: 30080
  type: NodePort

Deploy and access it:

kubectl apply -f wordpress.yaml

# If you're using Minikube
minikube service wordpress-service --url

# If you're using Docker Desktop, just go to http://localhost:30080

What Just Happened?

You just deployed a two-tier application! Here’s what’s cool about this:

  1. The database and web app are separate deployments - they can scale independently
  2. The Service provides stable networking - even if MySQL pods restart, WordPress can still find them at mysql-service:3306
  3. Everything is declarative - you described what you wanted, and Kubernetes made it happen

When Things Go Wrong (And They Will)

Let me save you some time with the most common issues you’ll hit:

Pod Won’t Start

# This is your debugging best friend
kubectl describe pod pod-name

# Check the logs
kubectl logs pod-name

# If it's restarting, check previous logs
kubectl logs pod-name --previous

Nine times out of ten, it’s either a wrong image name, missing environment variables, or resource constraints.

Can’t Access Your Service

# Check if the service is finding your pods
kubectl get endpoints service-name

# Make sure your labels match
kubectl get pods --show-labels

Usually, this is a label selector mismatch. Your service is looking for app: web but your pods are labeled app: webapp.

Cluster Acting Weird

# Check if all the system components are happy
kubectl get pods -n kube-system

# See what's happening
kubectl get events --sort-by=.metadata.creationTimestamp

A Few Things That’ll Save You Headaches

Always Use Labels

Labels are how everything finds everything else in Kubernetes. Be consistent:

metadata:
  labels:
    app: my-app
    version: v1.0
    environment: production

Set Resource Limits

If you don’t set limits, one misbehaving pod can take down your whole node:

resources:
  requests:
    memory: "64Mi"
    cpu: "100m"
  limits:
    memory: "128Mi"
    cpu: "200m"

Use Namespaces to Stay Organized

Don’t dump everything in the default namespace. Create separate spaces for different environments:

kubectl create namespace development
kubectl create namespace staging
kubectl create namespace production

What’s Next?

You now know enough to be dangerous with Kubernetes! You understand the basic building blocks (pods, services, deployments), you can deploy applications, and you know how to troubleshoot when things go sideways.

In the next part, we’ll dive deeper into workloads—different types of controllers, how to handle stateful applications, and more advanced deployment patterns. But honestly, what you’ve learned here will get you pretty far already.

Core Concepts and Fundamentals

Workloads and Controllers

Now that you’ve got the basics down, let’s talk about the different ways to run your applications in Kubernetes. You’ve already seen Deployments, but there are several other controllers, each designed for specific use cases. Think of them as different tools in your toolbox—you wouldn’t use a hammer for everything, right?

The beauty of Kubernetes controllers is that they’re constantly watching and adjusting. You tell them what you want, and they make sure it stays that way. Server crashes? Controller starts a new one. Need to scale up? Controller handles it. It’s like having a really attentive assistant who never takes a break.

Deployments - Your Go-To for Most Apps

Deployments are what you’ll use 90% of the time. They’re perfect for stateless applications—web servers, APIs, microservices, that sort of thing. The key word here is “stateless,” meaning your app doesn’t care which specific server it’s running on or store important data locally.

Here’s a more realistic deployment than the basic nginx example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api-server
  template:
    metadata:
      labels:
        app: api-server
    spec:
      containers:
      - name: api
        image: mycompany/api:v1.2.3
        ports:
        - containerPort: 8080
        env:
        - name: DATABASE_URL
          value: "postgres://db-service:5432/myapp"
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "256Mi"
            cpu: "200m"

What’s happening here? We’re asking for three copies of our API server, each with specific resource limits. If one crashes, Kubernetes immediately starts a replacement. If the whole node dies, Kubernetes moves the pods to healthy nodes.

Rolling Updates - The Magic of Zero Downtime

The coolest thing about Deployments is how they handle updates. Let’s say you want to deploy version 1.2.4 of your API:

kubectl set image deployment/api-server api=mycompany/api:v1.2.4

Kubernetes doesn’t just kill all your old pods and start new ones (that would cause downtime). Instead, it gradually replaces them—starting new pods with the new version, waiting for them to be ready, then terminating old ones. Your users never notice a thing.

You can watch this happen in real-time:

kubectl rollout status deployment/api-server
kubectl get pods -w  # Watch pods change

And if something goes wrong with the new version? Easy rollback:

kubectl rollout undo deployment/api-server

ReplicaSets - The Behind-the-Scenes Worker

You’ll rarely create ReplicaSets directly, but it’s worth understanding them because Deployments use them under the hood. A ReplicaSet ensures a specific number of pod replicas are running at any given time. Think of it as the middle manager between your Deployment and your pods.

When you create a Deployment, it creates a ReplicaSet, which creates the pods. When you update a Deployment, it creates a new ReplicaSet with the new configuration while gradually scaling down the old one.

# See the ReplicaSets created by your Deployment
kubectl get replicasets
kubectl describe replicaset api-server-abc123

DaemonSets - One Pod Per Node

Sometimes you need exactly one pod running on every node in your cluster. Log collectors, monitoring agents, network plugins—these are perfect for DaemonSets. As you add nodes to your cluster, DaemonSets automatically deploy pods to them.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: log-collector
spec:
  selector:
    matchLabels:
      app: log-collector
  template:
    metadata:
      labels:
        app: log-collector
    spec:
      containers:
      - name: fluentd
        image: fluentd:latest
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: dockerlogs
          mountPath: /var/lib/docker/containers
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: dockerlogs
        hostPath:
          path: /var/lib/docker/containers

This DaemonSet runs a log collector on every node, mounting the host’s log directories so it can collect logs from all containers.

StatefulSets - For Apps That Care About Identity

Most web applications are stateless—they don’t care which server they’re on or what their hostname is. But some applications do care. Databases, for example, often need stable network identities and persistent storage. That’s where StatefulSets come in.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres-headless
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:13
        env:
        - name: POSTGRES_PASSWORD
          value: "secretpassword"
        volumeMounts:
        - name: postgres-storage
          mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
  - metadata:
      name: postgres-storage
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 10Gi

StatefulSets give each pod a stable name (postgres-0, postgres-1, postgres-2) and ensure they start up in order. Each pod gets its own persistent volume that survives pod restarts and rescheduling.

Jobs and CronJobs - One-Time and Scheduled Tasks

Not everything needs to run forever. Sometimes you just need to run a task once, or on a schedule. That’s what Jobs and CronJobs are for.

Jobs - Run Once and Exit

apiVersion: batch/v1
kind: Job
metadata:
  name: database-migration
spec:
  template:
    spec:
      containers:
      - name: migrate
        image: mycompany/migrator:latest
        command: ["python", "migrate.py"]
        env:
        - name: DATABASE_URL
          value: "postgres://db-service:5432/myapp"
      restartPolicy: Never
  backoffLimit: 3

This Job runs a database migration script. If it fails, Kubernetes will retry up to 3 times. Once it succeeds, the Job is complete.

CronJobs - Scheduled Tasks

apiVersion: batch/v1
kind: CronJob
metadata:
  name: backup-job
spec:
  schedule: "0 2 * * *"  # Every day at 2 AM
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: backup
            image: mycompany/backup:latest
            command: ["./backup.sh"]
          restartPolicy: OnFailure

This CronJob runs a backup script every night at 2 AM. The schedule format is the same as regular cron.

Services - Making Your Apps Accessible

You’ve seen Services before, but let’s dive deeper. Services solve the fundamental problem of networking in Kubernetes: pods come and go with different IP addresses, but your applications need stable endpoints to communicate.

ClusterIP - Internal Communication

apiVersion: v1
kind: Service
metadata:
  name: api-service
spec:
  selector:
    app: api-server
  ports:
  - port: 80
    targetPort: 8080
  type: ClusterIP

This is the default service type. It creates a stable IP address that other pods in the cluster can use to reach your API servers. The service automatically load-balances between all healthy pods.

NodePort - External Access (Development)

apiVersion: v1
kind: Service
metadata:
  name: web-service
spec:
  selector:
    app: web-app
  ports:
  - port: 80
    targetPort: 8080
    nodePort: 30080
  type: NodePort

NodePort opens a specific port on every node in your cluster. It’s great for development, but not ideal for production because you have to manage port numbers and node IPs.

LoadBalancer - External Access (Production)

apiVersion: v1
kind: Service
metadata:
  name: web-service
spec:
  selector:
    app: web-app
  ports:
  - port: 80
    targetPort: 8080
  type: LoadBalancer

If you’re running on a cloud provider (AWS, GCP, Azure), this creates an actual load balancer that routes traffic to your pods. It’s the cleanest way to expose services to the internet.

ConfigMaps and Secrets - Configuration Management

Hard-coding configuration in your containers is a bad idea. What if you need different database URLs for development and production? ConfigMaps and Secrets let you inject configuration at runtime.

ConfigMaps - Non-Sensitive Configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  database_host: "postgres-service"
  database_port: "5432"
  log_level: "info"
  feature_flags: |
    {
      "new_ui": true,
      "beta_feature": false
    }

Use it in your deployment:

spec:
  containers:
  - name: app
    image: myapp:latest
    env:
    - name: DATABASE_HOST
      valueFrom:
        configMapKeyRef:
          name: app-config
          key: database_host
    - name: LOG_LEVEL
      valueFrom:
        configMapKeyRef:
          name: app-config
          key: log_level

Secrets - Sensitive Data

apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
data:
  database_password: cGFzc3dvcmQxMjM=  # base64 encoded
  api_key: YWJjZGVmZ2hpams=

Use secrets the same way as ConfigMaps, but they’re stored more securely and not displayed in plain text when you run kubectl get.

Putting It All Together

Here’s how these pieces typically work together in a real application:

  1. Deployment manages your application pods
  2. Service provides stable networking for the deployment
  3. ConfigMap holds non-sensitive configuration
  4. Secret holds passwords and API keys
  5. Job runs database migrations during deployment
  6. CronJob handles periodic maintenance tasks

The beauty is that each piece has a single responsibility, but they work together seamlessly. You can update configuration without redeploying your app, scale your deployment independently of your database, and run maintenance tasks without affecting your main application.

In the next part, we’ll look at practical examples of deploying real applications using these building blocks, including how to handle persistent data and more complex networking scenarios.

Practical Applications and Examples

Networking and Real-World Applications

Let’s get practical. You know the basics, you understand the building blocks, now let’s build something real. I’m going to walk you through deploying a complete web application—the kind you might actually run in production.

We’ll build a typical three-tier app: a React frontend, a Node.js API, and a PostgreSQL database. Along the way, we’ll tackle the networking challenges that trip up most people when they’re starting with Kubernetes.

Understanding Kubernetes Networking (The Simple Version)

Before we dive into the deployment, let’s clear up networking. Kubernetes networking seems complicated, but the basic idea is simple:

  • Every pod gets its own IP address (like having its own computer on the network)
  • Pods can talk to each other directly using these IPs
  • But pod IPs change when pods restart, so you use Services for stable addresses
  • Services act like phone books—they keep track of which pods are healthy and route traffic to them

Think of it this way: pods are like people who might move apartments, Services are like the post office that always knows how to reach them.

Building Our Application Stack

Step 1: The Database Layer

Let’s start with PostgreSQL. In production, you’d probably use a managed database, but this shows you how persistent storage works in Kubernetes.

# postgres.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:13
        env:
        - name: POSTGRES_DB
          value: "myapp"
        - name: POSTGRES_USER
          value: "appuser"
        - name: POSTGRES_PASSWORD
          value: "secretpassword"  # Don't do this in production!
        ports:
        - containerPort: 5432
        volumeMounts:
        - name: postgres-storage
          mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
  - metadata:
      name: postgres-storage
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
  name: postgres-service
spec:
  selector:
    app: postgres
  ports:
  - port: 5432
  type: ClusterIP

The key things here: we’re using a StatefulSet because databases need stable storage, and we’re creating a persistent volume so our data survives pod restarts.

Deploy it:

kubectl apply -f postgres.yaml
kubectl get pods -w  # Watch it start up

Step 2: The API Layer

Now let’s add our Node.js API. This is a stateless service, so we’ll use a Deployment:

# api.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
      - name: api
        image: node:16-alpine
        command: ["node", "server.js"]
        ports:
        - containerPort: 3000
        env:
        - name: DATABASE_URL
          value: "postgresql://appuser:secretpassword@postgres-service:5432/myapp"
        - name: PORT
          value: "3000"
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "256Mi"
            cpu: "200m"
---
apiVersion: v1
kind: Service
metadata:
  name: api-service
spec:
  selector:
    app: api
  ports:
  - port: 80
    targetPort: 3000
  type: ClusterIP

Notice how the API connects to the database using the service name postgres-service. Kubernetes has built-in DNS that resolves service names to IP addresses.

Step 3: The Frontend

Finally, let’s add a React frontend served by nginx:

# frontend.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
spec:
  replicas: 2
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - name: frontend
        image: nginx:alpine
        ports:
        - containerPort: 80
        volumeMounts:
        - name: nginx-config
          mountPath: /etc/nginx/conf.d
      volumes:
      - name: nginx-config
        configMap:
          name: nginx-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
data:
  default.conf: |
    server {
        listen 80;
        location / {
            root /usr/share/nginx/html;
            try_files $uri $uri/ /index.html;
        }
        location /api/ {
            proxy_pass http://api-service/;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
        }
    }
---
apiVersion: v1
kind: Service
metadata:
  name: frontend-service
spec:
  selector:
    app: frontend
  ports:
  - port: 80
  type: LoadBalancer

The ConfigMap contains nginx configuration that serves static files and proxies API requests to our backend service.

Making It Accessible from the Internet

Right now, our app is only accessible from inside the cluster. To expose it to the internet, we have a few options:

Option 1: LoadBalancer (Cloud Providers)

If you’re on AWS, GCP, or Azure, the LoadBalancer service type will create an actual load balancer:

apiVersion: v1
kind: Service
metadata:
  name: frontend-service
spec:
  selector:
    app: frontend
  ports:
  - port: 80
  type: LoadBalancer

Option 2: Ingress (More Flexible)

For more control over routing, use an Ingress. First, you need an ingress controller (like nginx-ingress):

# Install nginx ingress controller
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.8.1/deploy/static/provider/cloud/deploy.yaml

Then create an Ingress resource:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: myapp.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: frontend-service
            port:
              number: 80
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 80

Option 3: NodePort (Development)

For local development, NodePort is the simplest:

apiVersion: v1
kind: Service
metadata:
  name: frontend-service
spec:
  selector:
    app: frontend
  ports:
  - port: 80
    nodePort: 30080
  type: NodePort

Then access your app at http://localhost:30080 (Docker Desktop) or minikube service frontend-service --url (Minikube).

Handling Configuration Properly

Hard-coding database passwords is obviously a bad idea. Let’s fix that with proper Secrets and ConfigMaps:

# secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
data:
  database-password: c2VjcmV0cGFzc3dvcmQ=  # base64 encoded "secretpassword"
  jwt-secret: bXlqd3RzZWNyZXQ=  # base64 encoded "myjwtsecret"
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  database-host: "postgres-service"
  database-port: "5432"
  database-name: "myapp"
  database-user: "appuser"
  api-port: "3000"

Update your API deployment to use these:

env:
- name: DATABASE_HOST
  valueFrom:
    configMapKeyRef:
      name: app-config
      key: database-host
- name: DATABASE_PASSWORD
  valueFrom:
    secretKeyRef:
      name: app-secrets
      key: database-password
- name: JWT_SECRET
  valueFrom:
    secretKeyRef:
      name: app-secrets
      key: jwt-secret

Adding Health Checks

Production applications need health checks so Kubernetes knows when pods are ready to receive traffic and when they need to be restarted:

spec:
  containers:
  - name: api
    image: mycompany/api:latest
    ports:
    - containerPort: 3000
    livenessProbe:
      httpGet:
        path: /health
        port: 3000
      initialDelaySeconds: 30
      periodSeconds: 10
    readinessProbe:
      httpGet:
        path: /ready
        port: 3000
      initialDelaySeconds: 5
      periodSeconds: 5

Your application needs to implement these endpoints:

  • /health - returns 200 if the app is running (liveness)
  • /ready - returns 200 if the app is ready to serve traffic (readiness)

Scaling and Resource Management

As your app grows, you’ll need to scale different components independently:

# Scale the API to handle more traffic
kubectl scale deployment api --replicas=5

# Scale the frontend
kubectl scale deployment frontend --replicas=3

# Check resource usage
kubectl top pods
kubectl top nodes

Set resource requests and limits to ensure fair resource allocation:

resources:
  requests:
    memory: "128Mi"
    cpu: "100m"
  limits:
    memory: "256Mi"
    cpu: "200m"

Monitoring and Debugging

When things go wrong (and they will), here’s your debugging toolkit:

# Check pod status
kubectl get pods
kubectl describe pod api-xyz123

# Check logs
kubectl logs api-xyz123
kubectl logs api-xyz123 -f  # Follow logs

# Check services and endpoints
kubectl get services
kubectl get endpoints api-service

# Test connectivity from inside the cluster
kubectl run debug --image=busybox -it --rm -- /bin/sh
# Inside the pod: wget -qO- http://api-service/health

Putting It All Together

Deploy the complete application:

# Deploy in order
kubectl apply -f postgres.yaml
kubectl apply -f secrets.yaml
kubectl apply -f api.yaml
kubectl apply -f frontend.yaml

# Watch everything come up
kubectl get pods -w

# Check that services are working
kubectl get services
kubectl get ingress  # if using ingress

This gives you a production-ready application architecture with proper separation of concerns, configuration management, and networking. Each component can be scaled, updated, and monitored independently.

In the next part, we’ll look at more advanced patterns like persistent volumes, advanced networking with network policies, and how to handle more complex deployment scenarios.

Advanced Techniques and Patterns

Storage, Configuration, and Advanced Patterns

Now we’re getting into the stuff that separates toy projects from production applications. Storage, configuration management, and deployment patterns—these are the things that’ll make or break your Kubernetes experience when you’re running real workloads.

I’ve seen too many people get excited about Kubernetes, deploy a few stateless apps, then hit a wall when they need to handle databases, file uploads, or complex configuration. Let’s fix that.

Persistent Storage - Making Data Stick Around

The thing about containers is they’re ephemeral—when they die, everything inside them disappears. That’s great for stateless apps, but terrible for databases or anything that needs to store files. Kubernetes solves this with persistent volumes.

The Storage Hierarchy

Think of Kubernetes storage like this:

  • Persistent Volume (PV) - The actual storage (like a hard drive)
  • Persistent Volume Claim (PVC) - A request for storage (like asking for a 10GB drive)
  • Volume Mount - Connecting the storage to your container (like plugging in a USB drive)

Creating Persistent Storage

Let’s say you’re running a blog and need to store uploaded images. Here’s how you’d set that up:

# storage.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: blog-images
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  storageClassName: fast-ssd  # Optional: specify storage type

Then use it in your deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: blog-app
spec:
  replicas: 1  # Note: ReadWriteOnce means only one pod can use this
  selector:
    matchLabels:
      app: blog-app
  template:
    metadata:
      labels:
        app: blog-app
    spec:
      containers:
      - name: blog
        image: wordpress:latest
        volumeMounts:
        - name: image-storage
          mountPath: /var/www/html/wp-content/uploads
        env:
        - name: WORDPRESS_DB_HOST
          value: mysql-service
      volumes:
      - name: image-storage
        persistentVolumeClaim:
          claimName: blog-images

Storage Classes - Different Types of Storage

Not all storage is created equal. You might want fast SSD storage for databases and cheaper storage for backups:

# Check what storage classes are available
kubectl get storageclass

# Create a custom storage class (cloud provider specific)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/aws-ebs  # AWS example
parameters:
  type: gp3
  iops: "3000"
allowVolumeExpansion: true

Shared Storage for Multiple Pods

Sometimes you need multiple pods to access the same storage. Use ReadWriteMany access mode:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-files
spec:
  accessModes:
    - ReadWriteMany  # Multiple pods can read/write
  resources:
    requests:
      storage: 100Gi

Note: Not all storage providers support ReadWriteMany. Check your cloud provider’s documentation.

Configuration Management Done Right

Hard-coding configuration is the enemy of flexibility. Let’s look at better patterns for managing configuration in Kubernetes.

Environment-Specific Configuration

Create different ConfigMaps for different environments:

# config-dev.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: development
data:
  database_host: "dev-postgres-service"
  log_level: "debug"
  cache_ttl: "60"
  feature_flags: |
    {
      "new_ui": true,
      "beta_features": true,
      "analytics": false
    }
---
# config-prod.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: production
data:
  database_host: "prod-postgres-service"
  log_level: "warn"
  cache_ttl: "3600"
  feature_flags: |
    {
      "new_ui": true,
      "beta_features": false,
      "analytics": true
    }

Configuration Files as Volumes

Sometimes you need entire configuration files, not just environment variables:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
data:
  nginx.conf: |
    events {
        worker_connections 1024;
    }
    http {
        upstream backend {
            server api-service:80;
        }
        server {
            listen 80;
            location / {
                root /usr/share/nginx/html;
            }
            location /api/ {
                proxy_pass http://backend/;
            }
        }
    }

Mount it as a file:

spec:
  containers:
  - name: nginx
    image: nginx:alpine
    volumeMounts:
    - name: config
      mountPath: /etc/nginx/nginx.conf
      subPath: nginx.conf
  volumes:
  - name: config
    configMap:
      name: nginx-config

Secrets Management - Keeping Things Secure

Secrets are like ConfigMaps but for sensitive data. They’re base64 encoded (not encrypted!) and have some additional security features.

Creating Secrets Properly

Don’t put secrets in your YAML files. Create them from the command line:

# From literal values
kubectl create secret generic app-secrets \
  --from-literal=database-password=supersecret \
  --from-literal=api-key=abc123xyz

# From files
kubectl create secret generic tls-certs \
  --from-file=tls.crt=./server.crt \
  --from-file=tls.key=./server.key

# From environment file
kubectl create secret generic env-secrets \
  --from-env-file=.env

Using Secrets Securely

Mount secrets as files, not environment variables (they’re more secure that way):

spec:
  containers:
  - name: app
    image: myapp:latest
    volumeMounts:
    - name: secrets
      mountPath: /etc/secrets
      readOnly: true
    env:
    - name: SECRET_PATH
      value: "/etc/secrets"
  volumes:
  - name: secrets
    secret:
      secretName: app-secrets
      defaultMode: 0400  # Read-only for owner

Your application reads the secrets from files:

// In your Node.js app
const fs = require('fs');
const dbPassword = fs.readFileSync('/etc/secrets/database-password', 'utf8');
const apiKey = fs.readFileSync('/etc/secrets/api-key', 'utf8');

Advanced Deployment Patterns

Blue-Green Deployments

Deploy a new version alongside the old one, then switch traffic:

# Deploy green version
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-green
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
      version: green
  template:
    metadata:
      labels:
        app: myapp
        version: green
    spec:
      containers:
      - name: app
        image: myapp:v2.0.0

Switch the service to point to the new version:

apiVersion: v1
kind: Service
metadata:
  name: app-service
spec:
  selector:
    app: myapp
    version: green  # Change from blue to green
  ports:
  - port: 80

Canary Deployments

Gradually roll out to a percentage of users:

# 90% of traffic goes to stable version
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-stable
spec:
  replicas: 9
  selector:
    matchLabels:
      app: myapp
      version: stable
---
# 10% of traffic goes to canary version
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-canary
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
      version: canary

The service selects both versions:

apiVersion: v1
kind: Service
metadata:
  name: app-service
spec:
  selector:
    app: myapp  # Selects both stable and canary
  ports:
  - port: 80

Resource Management and Limits

Setting Appropriate Limits

Don’t just guess at resource limits. Monitor your applications and set realistic values:

spec:
  containers:
  - name: app
    image: myapp:latest
    resources:
      requests:
        memory: "256Mi"  # Guaranteed memory
        cpu: "200m"      # Guaranteed CPU (0.2 cores)
      limits:
        memory: "512Mi"  # Maximum memory
        cpu: "500m"      # Maximum CPU (0.5 cores)

Quality of Service Classes

Kubernetes assigns QoS classes based on your resource configuration:

  • Guaranteed - requests = limits (highest priority)
  • Burstable - requests < limits (medium priority)
  • BestEffort - no requests or limits (lowest priority)

For critical applications, use Guaranteed QoS:

resources:
  requests:
    memory: "256Mi"
    cpu: "200m"
  limits:
    memory: "256Mi"  # Same as requests
    cpu: "200m"      # Same as requests

Horizontal Pod Autoscaling

Automatically scale based on CPU or memory usage:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app-deployment
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

Init Containers - Setup Before Main Containers

Sometimes you need to do setup work before your main application starts:

spec:
  initContainers:
  - name: migration
    image: myapp:latest
    command: ['python', 'migrate.py']
    env:
    - name: DATABASE_URL
      valueFrom:
        secretKeyRef:
          name: app-secrets
          key: database-url
  - name: wait-for-db
    image: busybox
    command: ['sh', '-c', 'until nc -z postgres-service 5432; do sleep 1; done']
  containers:
  - name: app
    image: myapp:latest

Init containers run to completion before the main containers start. They’re perfect for database migrations, downloading files, or waiting for dependencies.

Putting Advanced Patterns Together

Here’s how you might combine these patterns in a real application:

# Complete application with advanced patterns
apiVersion: apps/v1
kind: Deployment
metadata:
  name: blog-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: blog-app
  template:
    metadata:
      labels:
        app: blog-app
    spec:
      initContainers:
      - name: wait-for-db
        image: busybox
        command: ['sh', '-c', 'until nc -z postgres-service 5432; do sleep 1; done']
      containers:
      - name: app
        image: wordpress:latest
        resources:
          requests:
            memory: "256Mi"
            cpu: "200m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        livenessProbe:
          httpGet:
            path: /wp-admin/install.php
            port: 80
          initialDelaySeconds: 60
          periodSeconds: 30
        readinessProbe:
          httpGet:
            path: /wp-admin/install.php
            port: 80
          initialDelaySeconds: 30
          periodSeconds: 10
        volumeMounts:
        - name: uploads
          mountPath: /var/www/html/wp-content/uploads
        - name: config
          mountPath: /etc/wordpress-config
          readOnly: true
        env:
        - name: WORDPRESS_DB_HOST
          valueFrom:
            configMapKeyRef:
              name: app-config
              key: database-host
        - name: WORDPRESS_DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: app-secrets
              key: database-password
      volumes:
      - name: uploads
        persistentVolumeClaim:
          claimName: blog-uploads
      - name: config
        configMap:
          name: wordpress-config

This deployment includes persistent storage, proper configuration management, health checks, resource limits, and init containers. It’s production-ready.

In the next part, we’ll look at best practices for production deployments, monitoring, and troubleshooting—the operational side of running Kubernetes in the real world.

Best Practices and Optimization

Production Best Practices and Optimization

Alright, you’ve learned the fundamentals, built some applications, and now you’re thinking about production. This is where things get real. Running Kubernetes in production isn’t just about getting your apps to work—it’s about making them reliable, secure, and maintainable.

I’ve seen plenty of teams rush to production with Kubernetes and then spend months fixing issues they could have avoided. Let’s make sure you’re not one of them.

Security - Don’t Be That Company in the News

Security in Kubernetes isn’t optional. It’s not something you add later. It needs to be baked in from the start, because once you’re compromised, you’re in for a world of hurt.

The Principle of Least Privilege

Never give more permissions than absolutely necessary. This applies to everything—users, service accounts, network access, you name it.

# Bad: Running as root
apiVersion: v1
kind: Pod
metadata:
  name: bad-pod
spec:
  containers:
  - name: app
    image: myapp:latest
    # Runs as root by default - dangerous!

# Good: Running as non-root user
apiVersion: v1
kind: Pod
metadata:
  name: good-pod
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 1000
  containers:
  - name: app
    image: myapp:latest
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop:
        - ALL

Network Policies - Lock Down Your Traffic

By default, any pod can talk to any other pod. That’s convenient for development, but terrible for security. Network policies let you control traffic flow:

# Deny all traffic by default
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
---
# Allow specific traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-api
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080

RBAC - Control Who Can Do What

Role-Based Access Control (RBAC) controls what users and service accounts can do in your cluster:

# Create a role for developers
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: development
  name: developer
rules:
- apiGroups: [""]
  resources: ["pods", "services", "configmaps", "secrets"]
  verbs: ["get", "list", "create", "update", "patch", "delete"]
- apiGroups: ["apps"]
  resources: ["deployments", "replicasets"]
  verbs: ["get", "list", "create", "update", "patch", "delete"]
---
# Bind the role to users
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: developer-binding
  namespace: development
subjects:
- kind: User
  name: [email protected]
  apiGroup: rbac.authorization.k8s.io
- kind: User
  name: [email protected]
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: developer
  apiGroup: rbac.authorization.k8s.io

Resource Management - Don’t Let One App Kill Your Cluster

Always Set Resource Limits

I can’t stress this enough: always set resource requests and limits. Without them, one misbehaving pod can consume all resources on a node and bring down other applications.

spec:
  containers:
  - name: app
    image: myapp:latest
    resources:
      requests:
        memory: "256Mi"
        cpu: "200m"
      limits:
        memory: "512Mi"
        cpu: "500m"

Use ResourceQuotas for Namespaces

Prevent teams from consuming all cluster resources:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
  namespace: team-alpha
spec:
  hard:
    requests.cpu: "10"
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi
    pods: "50"
    persistentvolumeclaims: "10"

LimitRanges for Default Limits

Set default resource limits so developers don’t have to remember:

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: production
spec:
  limits:
  - default:
      memory: "256Mi"
      cpu: "200m"
    defaultRequest:
      memory: "128Mi"
      cpu: "100m"
    type: Container

Monitoring and Observability - Know What’s Happening

You can’t manage what you can’t see. Proper monitoring is essential for production Kubernetes.

Health Checks Are Non-Negotiable

Every container should have health checks:

spec:
  containers:
  - name: app
    image: myapp:latest
    livenessProbe:
      httpGet:
        path: /health
        port: 8080
      initialDelaySeconds: 30
      periodSeconds: 10
      timeoutSeconds: 5
      failureThreshold: 3
    readinessProbe:
      httpGet:
        path: /ready
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 5
      timeoutSeconds: 3
      failureThreshold: 3

Your application needs to implement these endpoints:

  • /health - Is the app running? (liveness)
  • /ready - Is the app ready to serve traffic? (readiness)

Logging Strategy

Centralize your logs. Don’t rely on kubectl logs in production:

# Use a logging sidecar or DaemonSet
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
spec:
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      containers:
      - name: fluentd
        image: fluent/fluentd:latest
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: dockerlogs
          mountPath: /var/lib/docker/containers
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: dockerlogs
        hostPath:
          path: /var/lib/docker/containers

Metrics Collection

Use Prometheus for metrics collection:

# Add Prometheus annotations to your pods
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/metrics"
    spec:
      containers:
      - name: app
        image: myapp:latest
        ports:
        - containerPort: 8080

Deployment Strategies - Rolling Out Changes Safely

Rolling Updates with Proper Configuration

Configure rolling updates to minimize disruption:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 6
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1      # Never take down more than 1 pod
      maxSurge: 2           # Can create up to 2 extra pods during update
  template:
    spec:
      containers:
      - name: app
        image: myapp:v1.2.3
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5

Pod Disruption Budgets

Protect your applications during cluster maintenance:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: app-pdb
spec:
  minAvailable: 2  # Always keep at least 2 pods running
  selector:
    matchLabels:
      app: myapp

Configuration Management Best Practices

Environment-Specific Configurations

Use Kustomize or Helm for environment-specific configurations:

# base/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 1  # Will be overridden per environment
  template:
    spec:
      containers:
      - name: app
        image: myapp:latest
        env:
        - name: LOG_LEVEL
          value: "info"  # Will be overridden per environment

# overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
patchesStrategicMerge:
- deployment-patch.yaml

# overlays/production/deployment-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 5
  template:
    spec:
      containers:
      - name: app
        env:
        - name: LOG_LEVEL
          value: "warn"

Secret Management

Never commit secrets to Git. Use external secret management:

# Use sealed-secrets or external-secrets operator
kubectl create secret generic app-secrets \
  --from-literal=database-password="$(openssl rand -base64 32)" \
  --dry-run=client -o yaml | \
  kubeseal -o yaml > sealed-secret.yaml

Performance Optimization

Node Affinity and Anti-Affinity

Control where your pods run:

spec:
  affinity:
    # Prefer to run on nodes with SSD storage
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        preference:
          matchExpressions:
          - key: storage-type
            operator: In
            values:
            - ssd
    # Don't run multiple replicas on the same node
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - myapp
        topologyKey: kubernetes.io/hostname

Horizontal Pod Autoscaling

Scale automatically based on metrics:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app
  minReplicas: 3
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 10
        periodSeconds: 60

Backup and Disaster Recovery

Backup Strategies

Don’t forget to backup your persistent data:

# Use Velero for cluster backups
apiVersion: velero.io/v1
kind: Backup
metadata:
  name: daily-backup
spec:
  includedNamespaces:
  - production
  - staging
  storageLocation: aws-s3
  ttl: 720h0m0s  # 30 days

Database Backups

For databases, use application-specific backup tools:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: postgres-backup
spec:
  schedule: "0 2 * * *"  # Daily at 2 AM
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: backup
            image: postgres:13
            command:
            - /bin/bash
            - -c
            - |
              pg_dump -h postgres-service -U postgres myapp > /backup/backup-$(date +%Y%m%d).sql
              aws s3 cp /backup/backup-$(date +%Y%m%d).sql s3://my-backups/
            env:
            - name: PGPASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-secret
                  key: password
            volumeMounts:
            - name: backup-storage
              mountPath: /backup
          volumes:
          - name: backup-storage
            emptyDir: {}
          restartPolicy: OnFailure

Troubleshooting Production Issues

Essential Debugging Commands

Keep these handy for when things go wrong:

# Check cluster health
kubectl get nodes
kubectl get pods --all-namespaces
kubectl top nodes
kubectl top pods --all-namespaces

# Investigate specific issues
kubectl describe pod problematic-pod
kubectl logs problematic-pod --previous
kubectl get events --sort-by=.metadata.creationTimestamp

# Check resource usage
kubectl describe node node-name
kubectl get pods -o wide

# Network debugging
kubectl run debug --image=busybox -it --rm -- /bin/sh
# Inside: nslookup service-name, wget -qO- http://service-name/health

Common Production Issues

Pods stuck in Pending: Usually resource constraints or node selector issues CrashLoopBackOff: Application startup issues, check logs and health checks Service not accessible: Check endpoints, labels, and network policies High resource usage: Check for memory leaks, inefficient queries, or missing limits

The key to production success is preparation. Set up proper monitoring, have runbooks for common issues, and practice your incident response procedures. Kubernetes is powerful, but with great power comes great responsibility.

In the final part, we’ll put everything together with real-world examples and discuss advanced topics like GitOps, service mesh, and scaling strategies.

Real-World Projects and Implementation

Real-World Implementation and Next Steps

We’ve covered a lot of ground together. You started knowing nothing about Kubernetes, and now you understand pods, services, deployments, storage, security, and production best practices. But here’s the thing—reading about Kubernetes and actually running it in production are two very different experiences.

Let me share some real-world scenarios and advanced patterns that’ll help you bridge that gap. These are the kinds of challenges you’ll face when you’re responsible for keeping applications running 24/7.

A Complete E-commerce Platform

Let’s build something realistic—an e-commerce platform with multiple services, databases, caching, and all the complexity that comes with real applications.

The Architecture

We’ll need:

  • Frontend (React app served by nginx)
  • API Gateway (nginx with routing rules)
  • User Service (handles authentication)
  • Product Service (manages catalog)
  • Order Service (processes orders)
  • Payment Service (handles payments)
  • Redis (for caching and sessions)
  • PostgreSQL (for persistent data)
  • Background workers (for email, notifications)

Starting with the Data Layer

# postgres.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:13
        env:
        - name: POSTGRES_DB
          value: ecommerce
        - name: POSTGRES_USER
          value: app
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: password
        ports:
        - containerPort: 5432
        volumeMounts:
        - name: postgres-storage
          mountPath: /var/lib/postgresql/data
        resources:
          requests:
            memory: "512Mi"
            cpu: "200m"
          limits:
            memory: "1Gi"
            cpu: "500m"
  volumeClaimTemplates:
  - metadata:
      name: postgres-storage
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 50Gi
---
apiVersion: v1
kind: Service
metadata:
  name: postgres-service
spec:
  selector:
    app: postgres
  ports:
  - port: 5432
  type: ClusterIP

Redis for Caching

# redis.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: redis:6-alpine
        ports:
        - containerPort: 6379
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "256Mi"
            cpu: "200m"
---
apiVersion: v1
kind: Service
metadata:
  name: redis-service
spec:
  selector:
    app: redis
  ports:
  - port: 6379
  type: ClusterIP

Microservices with Shared Configuration

# shared-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  database_host: "postgres-service"
  database_port: "5432"
  database_name: "ecommerce"
  redis_host: "redis-service"
  redis_port: "6379"
  log_level: "info"
---
apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
data:
  database_password: <base64-encoded-password>
  jwt_secret: <base64-encoded-jwt-secret>
  stripe_api_key: <base64-encoded-stripe-key>

User Service

# user-service.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: user-service
  template:
    metadata:
      labels:
        app: user-service
    spec:
      containers:
      - name: user-service
        image: mycompany/user-service:v1.2.3
        ports:
        - containerPort: 8080
        env:
        - name: DATABASE_HOST
          valueFrom:
            configMapKeyRef:
              name: app-config
              key: database_host
        - name: DATABASE_PASSWORD
          valueFrom:
            secretKeyRef:
              name: app-secrets
              key: database_password
        - name: JWT_SECRET
          valueFrom:
            secretKeyRef:
              name: app-secrets
              key: jwt_secret
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
        resources:
          requests:
            memory: "256Mi"
            cpu: "200m"
          limits:
            memory: "512Mi"
            cpu: "500m"
---
apiVersion: v1
kind: Service
metadata:
  name: user-service
spec:
  selector:
    app: user-service
  ports:
  - port: 80
    targetPort: 8080
  type: ClusterIP

API Gateway with Ingress

# ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-gateway
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/rate-limit: "100"
    nginx.ingress.kubernetes.io/rate-limit-window: "1m"
spec:
  rules:
  - host: api.mystore.com
    http:
      paths:
      - path: /users(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: user-service
            port:
              number: 80
      - path: /products(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: product-service
            port:
              number: 80
      - path: /orders(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: order-service
            port:
              number: 80
  tls:
  - hosts:
    - api.mystore.com
    secretName: api-tls-secret

GitOps Workflow - The Modern Way to Deploy

Instead of running kubectl apply manually, use GitOps for automated, auditable deployments.

ArgoCD Setup

# argocd-application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: ecommerce-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/mycompany/k8s-manifests
    targetRevision: HEAD
    path: environments/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

Directory Structure for GitOps

k8s-manifests/
├── base/
│   ├── user-service/
│   ├── product-service/
│   ├── order-service/
│   └── kustomization.yaml
├── environments/
│   ├── development/
│   │   ├── kustomization.yaml
│   │   └── patches/
│   ├── staging/
│   │   ├── kustomization.yaml
│   │   └── patches/
│   └── production/
│       ├── kustomization.yaml
│       └── patches/

Monitoring and Observability Stack

Prometheus and Grafana

# monitoring.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
---
# Prometheus deployment (simplified)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
      - name: prometheus
        image: prom/prometheus:latest
        ports:
        - containerPort: 9090
        volumeMounts:
        - name: config
          mountPath: /etc/prometheus
        - name: storage
          mountPath: /prometheus
      volumes:
      - name: config
        configMap:
          name: prometheus-config
      - name: storage
        persistentVolumeClaim:
          claimName: prometheus-storage

Application Metrics

Add metrics to your applications:

// In your Node.js service
const prometheus = require('prom-client');

// Create metrics
const httpRequestDuration = new prometheus.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status']
});

const httpRequestsTotal = new prometheus.Counter({
  name: 'http_requests_total',
  help: 'Total number of HTTP requests',
  labelNames: ['method', 'route', 'status']
});

// Middleware to collect metrics
app.use((req, res, next) => {
  const start = Date.now();
  
  res.on('finish', () => {
    const duration = (Date.now() - start) / 1000;
    httpRequestDuration
      .labels(req.method, req.route?.path || req.path, res.statusCode)
      .observe(duration);
    httpRequestsTotal
      .labels(req.method, req.route?.path || req.path, res.statusCode)
      .inc();
  });
  
  next();
});

// Metrics endpoint
app.get('/metrics', (req, res) => {
  res.set('Content-Type', prometheus.register.contentType);
  res.end(prometheus.register.metrics());
});

Scaling Strategies

Horizontal Pod Autoscaling with Custom Metrics

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: user-service-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: user-service
  minReplicas: 3
  maxReplicas: 50
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "100"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
      - type: Percent
        value: 100
        periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 10
        periodSeconds: 60

Vertical Pod Autoscaling

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: user-service-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: user-service
  updatePolicy:
    updateMode: "Auto"
  resourcePolicy:
    containerPolicies:
    - containerName: user-service
      maxAllowed:
        cpu: "2"
        memory: "4Gi"
      minAllowed:
        cpu: "100m"
        memory: "128Mi"

Disaster Recovery and Business Continuity

Multi-Region Setup

# Cross-region replication for critical services
apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-service-replica
  namespace: disaster-recovery
spec:
  replicas: 2
  selector:
    matchLabels:
      app: user-service-replica
  template:
    metadata:
      labels:
        app: user-service-replica
    spec:
      containers:
      - name: user-service
        image: mycompany/user-service:v1.2.3
        env:
        - name: DATABASE_HOST
          value: "replica-postgres-service"
        - name: READ_ONLY_MODE
          value: "true"

Backup Automation

apiVersion: batch/v1
kind: CronJob
metadata:
  name: database-backup
spec:
  schedule: "0 2 * * *"  # Daily at 2 AM
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: backup
            image: postgres:13
            command:
            - /bin/bash
            - -c
            - |
              pg_dump -h postgres-service -U app ecommerce | gzip > /backup/backup-$(date +%Y%m%d-%H%M%S).sql.gz
              aws s3 cp /backup/backup-$(date +%Y%m%d-%H%M%S).sql.gz s3://my-backups/postgres/
              # Keep only last 30 days of backups
              find /backup -name "*.sql.gz" -mtime +30 -delete
            env:
            - name: PGPASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-secret
                  key: password
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  name: aws-credentials
                  key: access-key-id
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: aws-credentials
                  key: secret-access-key
            volumeMounts:
            - name: backup-storage
              mountPath: /backup
          volumes:
          - name: backup-storage
            persistentVolumeClaim:
              claimName: backup-pvc
          restartPolicy: OnFailure

Performance Optimization

Database Connection Pooling

# PgBouncer for connection pooling
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pgbouncer
spec:
  replicas: 2
  selector:
    matchLabels:
      app: pgbouncer
  template:
    metadata:
      labels:
        app: pgbouncer
    spec:
      containers:
      - name: pgbouncer
        image: pgbouncer/pgbouncer:latest
        ports:
        - containerPort: 5432
        env:
        - name: DATABASES_HOST
          value: postgres-service
        - name: DATABASES_PORT
          value: "5432"
        - name: DATABASES_USER
          value: app
        - name: DATABASES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: password
        - name: POOL_MODE
          value: transaction
        - name: MAX_CLIENT_CONN
          value: "1000"
        - name: DEFAULT_POOL_SIZE
          value: "25"

Caching Strategies

# Redis Cluster for high availability caching
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-cluster
spec:
  serviceName: redis-cluster
  replicas: 6
  selector:
    matchLabels:
      app: redis-cluster
  template:
    metadata:
      labels:
        app: redis-cluster
    spec:
      containers:
      - name: redis
        image: redis:6-alpine
        command:
        - redis-server
        - /etc/redis/redis.conf
        - --cluster-enabled
        - "yes"
        - --cluster-config-file
        - /data/nodes.conf
        ports:
        - containerPort: 6379
        - containerPort: 16379
        volumeMounts:
        - name: redis-data
          mountPath: /data
        - name: redis-config
          mountPath: /etc/redis
      volumes:
      - name: redis-config
        configMap:
          name: redis-config
  volumeClaimTemplates:
  - metadata:
      name: redis-data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 10Gi

What’s Next?

You’ve now got a solid foundation in Kubernetes. Here’s what I’d recommend for your continued learning:

Advanced Topics to Explore

  1. Service Mesh (Istio, Linkerd) - For advanced traffic management and security
  2. Custom Resources and Operators - Extend Kubernetes for your specific needs
  3. Multi-cluster Management - Tools like Rancher, Anthos, or OpenShift
  4. Advanced Networking - CNI plugins, network policies, ingress controllers
  5. Security Hardening - Pod Security Standards, OPA Gatekeeper, Falco

Hands-On Practice

The best way to learn Kubernetes is to use it. Set up a homelab, contribute to open source projects, or volunteer to help with your company’s Kubernetes migration. There’s no substitute for real experience.

Community and Resources

  • Join the Kubernetes Slack community
  • Attend local Kubernetes meetups
  • Follow the Kubernetes blog and release notes
  • Practice with platforms like Katacoda or Play with Kubernetes

Remember, Kubernetes is a journey, not a destination. The ecosystem is constantly evolving, and there’s always something new to learn. But with the foundation you’ve built here, you’re well-equipped to tackle whatever challenges come your way.

Good luck, and welcome to the Kubernetes community!