Container Orchestration and Docker Compose

Managing multiple containers manually becomes unwieldy fast. I learned this the hard way when I had to coordinate a web server, database, cache, and background workers across different environments. Docker Compose solved this by letting me define entire application stacks in a single file.

Understanding Multi-Container Applications

Modern applications rarely run in isolation. A typical web application might include:

Web server (frontend)
API server (backend)
Database (PostgreSQL, MySQL)
Cache (Redis, Memcached)
Message queue (RabbitMQ, Apache Kafka)

Coordinating these services manually means remembering port mappings, network configurations, environment variables, and startup order. Docker Compose eliminates this complexity.

Docker Compose Fundamentals

Docker Compose uses YAML files to define services, networks, and volumes:

# docker-compose.yml
version: '3.8'

services:
  web:
    build: .
    ports:
      - "8000:8000"
    environment:
      - DATABASE_URL=postgresql://user:password@db:5432/myapp
    depends_on:
      - db
      - redis

  db:
    image: postgres:15-alpine
    environment:
      - POSTGRES_DB=myapp
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=password
    volumes:
      - postgres_data:/var/lib/postgresql/data

  redis:
    image: redis:7-alpine
    command: redis-server --appendonly yes
    volumes:
      - redis_data:/data

volumes:
  postgres_data:
  redis_data:

This defines three services that can communicate with each other using service names as hostnames. The web service can connect to the database using db:5432 and Redis using redis:6379.

Service Configuration and Dependencies

The depends_on directive controls startup order, but it doesn’t wait for services to be ready. For applications that need the database to be fully initialized:

version: '3.8'

services:
  web:
    build: .
    ports:
      - "8000:8000"
    depends_on:
      db:
        condition: service_healthy

  db:
    image: postgres:15-alpine
    environment:
      - POSTGRES_DB=myapp
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=password
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U user -d myapp"]
      interval: 10s
      timeout: 5s
      retries: 5

volumes:
  postgres_data:

The healthcheck ensures the database is accepting connections before starting the web service. I always include health checks for critical services like databases.

Networking in Docker Compose

Docker Compose automatically creates a network for your services. You can also define custom networks for better isolation:

version: '3.8'

services:
  frontend:
    build: ./frontend
    networks:
      - frontend-net
    ports:
      - "3000:3000"

  api:
    build: ./api
    networks:
      - frontend-net
      - backend-net

  db:
    image: postgres:15-alpine
    networks:
      - backend-net
    volumes:
      - postgres_data:/var/lib/postgresql/data

networks:
  frontend-net:
  backend-net:

volumes:
  postgres_data:

This setup isolates the database on the backend network, preventing the frontend from directly accessing it.

Environment Configuration

Managing configuration across different environments is crucial. I use multiple compose files and environment files:

# docker-compose.yml (base configuration)
version: '3.8'

services:
  web:
    build: .
    environment:
      - NODE_ENV=${NODE_ENV:-development}
    env_file:
      - .env

  db:
    image: postgres:15-alpine
    environment:
      - POSTGRES_DB=${POSTGRES_DB}
      - POSTGRES_USER=${POSTGRES_USER}
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}

# docker-compose.override.yml (development overrides)
version: '3.8'

services:
  web:
    ports:
      - "8000:8000"
    volumes:
      - ./src:/app/src
    command: npm run dev

  db:
    ports:
      - "5432:5432"

Use different configurations with:

# Development (uses docker-compose.override.yml automatically)
docker-compose up

# Production
docker-compose -f docker-compose.yml -f docker-compose.prod.yml up

Development Workflow Optimization

For development, I optimize for fast feedback loops and easy debugging:

version: '3.8'

services:
  web:
    build:
      context: .
      dockerfile: Dockerfile.dev
    ports:
      - "8000:8000"
    volumes:
      - ./src:/app/src
      - /app/node_modules  # Anonymous volume to preserve node_modules
    environment:
      - NODE_ENV=development
      - CHOKIDAR_USEPOLLING=true  # For file watching in containers
    command: npm run dev

  db:
    image: postgres:15-alpine
    ports:
      - "5432:5432"  # Expose for external tools
    environment:
      - POSTGRES_DB=myapp_dev
      - POSTGRES_USER=dev
      - POSTGRES_PASSWORD=devpass
    volumes:
      - postgres_dev_data:/var/lib/postgresql/data

volumes:
  postgres_dev_data:

Common Patterns and Troubleshooting

I’ve encountered these issues repeatedly and learned to avoid them:

Problem: Services can’t communicate

# Check if services are on the same network
docker-compose ps
docker network ls

Problem: Database connection refused

# Add health checks and proper depends_on
services:
  web:
    depends_on:
      db:
        condition: service_healthy
  
  db:
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U user"]
      interval: 10s
      retries: 5

Problem: File changes not reflected in development

# Ensure proper volume mounting and file watching
services:
  web:
    volumes:
      - ./src:/app/src
    environment:
      - CHOKIDAR_USEPOLLING=true

Docker Compose provides an excellent foundation for understanding container orchestration concepts. The skills you learn here translate directly to Kubernetes, which I’ll cover in the next part along with production-grade orchestration strategies.