Graceful Shutdown and Resource Cleanup

Proper shutdown handling ensures your async applications clean up resources gracefully and don’t lose data. Let’s explore shutdown patterns and cleanup strategies.

Signal Handling

Handle shutdown signals properly:

import asyncio
import signal
import logging
from typing import Set

class GracefulShutdown:
    def __init__(self):
        self.shutdown_event = asyncio.Event()
        self.running_tasks: Set[asyncio.Task] = set()
        self.cleanup_callbacks = []

The shutdown handler tracks running tasks and cleanup callbacks. The event signals when shutdown should begin.

Set up signal handlers to catch termination signals:

    def setup_signal_handlers(self):
        """Setup signal handlers for graceful shutdown"""
        for sig in (signal.SIGTERM, signal.SIGINT):
            signal.signal(sig, self._signal_handler)
    
    def _signal_handler(self, signum, frame):
        """Handle shutdown signals"""
        logging.info(f"Received signal {signum}, initiating graceful shutdown...")
        self.shutdown_event.set()
    
    async def wait_for_shutdown(self):
        """Wait for shutdown signal"""
        await self.shutdown_event.wait()

SIGTERM comes from process managers like systemd, while SIGINT comes from Ctrl+C. Both trigger the same graceful shutdown process.

Add cleanup callbacks and manage tasks:

    def add_cleanup_callback(self, callback):
        """Add cleanup callback to be called during shutdown"""
        self.cleanup_callbacks.append(callback)
    
    async def cleanup(self):
        """Perform cleanup operations"""
        logging.info("Starting cleanup process...")
        
        # Cancel running tasks
        if self.running_tasks:
            logging.info(f"Cancelling {len(self.running_tasks)} running tasks...")
            for task in self.running_tasks:
                task.cancel()
            
            # Wait for tasks to complete cancellation
            await asyncio.gather(*self.running_tasks, return_exceptions=True)
        
        # Run cleanup callbacks
        for callback in self.cleanup_callbacks:
            try:
                if asyncio.iscoroutinefunction(callback):
                    await callback()
                else:
                    callback()
            except Exception as e:
                logging.error(f"Error in cleanup callback: {e}")
        
        logging.info("Cleanup completed")

# Global shutdown handler
shutdown_handler = GracefulShutdown()

FastAPI Graceful Shutdown

Implement graceful shutdown in FastAPI applications:

from fastapi import FastAPI
from contextlib import asynccontextmanager
import asyncio
import logging

# Application state
app_state = {
    "database_pool": None,
    "redis_connection": None,
    "background_tasks": set()
}

FastAPI’s lifespan context manager handles startup and shutdown events. We track resources in app_state for proper cleanup.

The lifespan function manages the application lifecycle:

@asynccontextmanager
async def lifespan(app: FastAPI):
    # Startup
    logging.info("Starting application...")
    
    # Initialize resources
    app_state["database_pool"] = await create_database_pool()
    app_state["redis_connection"] = await create_redis_connection()
    
    # Start background tasks
    task = asyncio.create_task(background_worker())
    app_state["background_tasks"].add(task)
    
    yield
    
    # Shutdown
    logging.info("Shutting down application...")
    
    # Cancel background tasks
    for task in app_state["background_tasks"]:
        task.cancel()
    
    await asyncio.gather(*app_state["background_tasks"], return_exceptions=True)
    
    # Close connections
    if app_state["database_pool"]:
        await app_state["database_pool"].close()
    
    if app_state["redis_connection"]:
        await app_state["redis_connection"].close()

app = FastAPI(lifespan=lifespan)

Everything before yield runs at startup, everything after runs at shutdown. This ensures proper resource management throughout the application lifecycle.

Background tasks need to handle cancellation gracefully:

async def background_worker():
    """Background task that needs graceful shutdown"""
    try:
        while True:
            await asyncio.sleep(1)
            logging.info("Background worker running...")
    except asyncio.CancelledError:
        logging.info("Background worker cancelled, cleaning up...")
        raise

Database Connection Cleanup

Handle database connections properly:

import asyncio
import asyncpg
from contextlib import asynccontextmanager

class DatabaseManager:
    def __init__(self, database_url: str):
        self.database_url = database_url
        self.pool = None
    
    async def initialize(self):
        """Initialize database pool"""
        self.pool = await asyncpg.create_pool(
            self.database_url,
            min_size=5,
            max_size=20
        )

Connection pools are essential for async applications. They manage multiple database connections efficiently and handle connection lifecycle automatically.

Provide safe connection access with automatic cleanup:

    @asynccontextmanager
    async def get_connection(self):
        """Get database connection with automatic cleanup"""
        connection = await self.pool.acquire()
        try:
            yield connection
        finally:
            await self.pool.release(connection)

The context manager ensures connections are always returned to the pool, even if exceptions occur during database operations.

Handle shutdown cleanup:

    async def close_all_connections(self):
        """Close all database connections"""
        if self.pool:
            await self.pool.close()
            logging.info("Database pool closed")

# Usage
db_manager = DatabaseManager("postgresql://user:pass@localhost/db")
shutdown_handler.add_cleanup_callback(db_manager.close_all_connections)

File and Resource Cleanup

Clean up files and other resources:

import asyncio
import aiofiles
import tempfile
import os

class ResourceManager:
    def __init__(self):
        self.temp_files = set()
        self.open_files = set()

Track temporary files and open file handles to ensure they’re cleaned up during shutdown.

Create and track temporary files:

    async def create_temp_file(self, suffix=".tmp"):
        """Create temporary file with automatic cleanup"""
        temp_file = tempfile.NamedTemporaryFile(delete=False, suffix=suffix)
        temp_file.close()
        self.temp_files.add(temp_file.name)
        return temp_file.name
    
    async def open_file(self, filename: str, mode: str = 'r'):
        """Open file with tracking for cleanup"""
        file_handle = await aiofiles.open(filename, mode)
        self.open_files.add(file_handle)
        return file_handle

By tracking file operations, we can ensure proper cleanup even if the application terminates unexpectedly.

Perform comprehensive resource cleanup:

    async def cleanup_resources(self):
        """Clean up all managed resources"""
        # Close open files
        for file_handle in list(self.open_files):
            try:
                await file_handle.close()
            except Exception as e:
                logging.error(f"Error closing file: {e}")
        
        # Remove temporary files
        for temp_file in self.temp_files:
            try:
                if os.path.exists(temp_file):
                    os.unlink(temp_file)
            except Exception as e:
                logging.error(f"Error removing temp file: {e}")

# Global resource manager
resource_manager = ResourceManager()
shutdown_handler.add_cleanup_callback(resource_manager.cleanup_resources)

Best Practices

Key principles for graceful shutdown:

Signal Handling:

Handle SIGTERM and SIGINT signals
Set shutdown flags instead of immediate exit
Log shutdown initiation

Task Management:

Track all running tasks
Cancel tasks gracefully
Wait for task completion with timeout

Resource Cleanup:

Close database connections
Clean up temporary files
Release network connections
Save application state if needed

Error Handling:

Handle cleanup errors gracefully
Log cleanup progress
Don’t let cleanup errors prevent shutdown

Summary

Graceful shutdown essentials:

Implement proper signal handling for shutdown events
Track and cancel running tasks during shutdown
Clean up resources (databases, files, connections) properly
Use FastAPI lifespan events for web applications
Log shutdown progress for debugging
Handle cleanup errors without blocking shutdown

Proper shutdown handling ensures data integrity and resource cleanup in production async applications.

In Part 21, we’ll explore containerization and deployment strategies.