State Migration and Refactoring

State migration is one of the most nerve-wracking operations in Terraform. Whether you’re moving resources between configurations, changing backend types, or refactoring module structures, state migration requires careful planning and execution. A mistake can leave you with orphaned resources, corrupted state, or worse—accidentally destroyed infrastructure.

This part covers safe migration strategies, refactoring techniques, and recovery procedures that let you evolve your Terraform configurations without risking your infrastructure.

Backend Migration Strategies

Moving state between different backend types requires careful coordination:

#!/bin/bash
# scripts/migrate-backend.sh

set -e

SOURCE_BACKEND=${1:-"local"}
TARGET_BACKEND=${2:-"s3"}
BACKUP_DIR=${3:-"state-backups"}

echo "Migrating Terraform backend from $SOURCE_BACKEND to $TARGET_BACKEND"

# Create backup directory
mkdir -p "$BACKUP_DIR"

# Step 1: Backup current state
echo "Creating state backup..."
BACKUP_FILE="$BACKUP_DIR/terraform-state-backup-$(date +%Y%m%d-%H%M%S).tfstate"
terraform state pull > "$BACKUP_FILE"
echo "State backed up to: $BACKUP_FILE"

# Step 2: Verify current state
echo "Verifying current state..."
terraform plan -detailed-exitcode
if [ $? -eq 2 ]; then
    echo "WARNING: Current state has pending changes. Consider applying them first."
    read -p "Continue with migration? (y/N): " -n 1 -r
    echo
    if [[ ! $REPLY =~ ^[Yy]$ ]]; then
        exit 1
    fi
fi

# Step 3: Update backend configuration
echo "Please update your backend configuration in your Terraform files."
echo "Press Enter when ready to continue..."
read

# Step 4: Initialize with new backend
echo "Initializing new backend..."
terraform init -migrate-state

# Step 5: Verify migration
echo "Verifying migration..."
terraform plan -detailed-exitcode
if [ $? -eq 0 ]; then
    echo "✅ Migration successful - no changes detected"
else
    echo "⚠️  Migration may have issues - please review the plan output"
    exit 1
fi

echo "Backend migration completed successfully!"
echo "Backup saved at: $BACKUP_FILE"

Resource Refactoring

Move resources between configurations or modules safely:

# Before refactoring - monolithic configuration
resource "aws_vpc" "main" {
  cidr_block = "10.0.0.0/16"
  
  tags = {
    Name = "main-vpc"
  }
}

resource "aws_subnet" "public" {
  count             = 2
  vpc_id            = aws_vpc.main.id
  cidr_block        = "10.0.${count.index + 1}.0/24"
  availability_zone = data.aws_availability_zones.available.names[count.index]
  
  tags = {
    Name = "public-subnet-${count.index + 1}"
  }
}

resource "aws_internet_gateway" "main" {
  vpc_id = aws_vpc.main.id
  
  tags = {
    Name = "main-igw"
  }
}

#!/bin/bash
# scripts/refactor-to-module.sh

set -e

echo "Refactoring resources to use VPC module..."

# Step 1: Backup state
terraform state pull > "state-backup-$(date +%Y%m%d-%H%M%S).tfstate"

# Step 2: Remove resources from current state
echo "Removing resources from current state..."
terraform state rm aws_vpc.main
terraform state rm 'aws_subnet.public[0]'
terraform state rm 'aws_subnet.public[1]'
terraform state rm aws_internet_gateway.main

# Step 3: Update configuration to use module
cat > main.tf << 'EOF'
module "vpc" {
  source = "./modules/vpc"
  
  name               = "main"
  cidr_block         = "10.0.0.0/16"
  availability_zones = ["us-west-2a", "us-west-2b"]
  
  public_subnet_cidrs = ["10.0.1.0/24", "10.0.2.0/24"]
}
EOF

# Step 4: Initialize and import resources into module
echo "Importing resources into module..."
terraform init
terraform import 'module.vpc.aws_vpc.main' vpc-12345678
terraform import 'module.vpc.aws_subnet.public[0]' subnet-12345678
terraform import 'module.vpc.aws_subnet.public[1]' subnet-87654321
terraform import 'module.vpc.aws_internet_gateway.main' igw-12345678

# Step 5: Verify refactoring
echo "Verifying refactoring..."
terraform plan
echo "If the plan shows no changes, refactoring was successful!"

State Splitting and Merging

Split large state files or merge related configurations:

#!/bin/bash
# scripts/split-state.sh

set -e

SOURCE_STATE_DIR=${1:-"."}
TARGET_STATE_DIR=${2:-"../networking"}
RESOURCES_TO_MOVE=${3:-"aws_vpc.main aws_subnet.public aws_internet_gateway.main"}

echo "Splitting state: moving networking resources to separate configuration"

# Step 1: Backup both state files
echo "Creating backups..."
cd "$SOURCE_STATE_DIR"
terraform state pull > "state-backup-source-$(date +%Y%m%d-%H%M%S).tfstate"

cd "$TARGET_STATE_DIR"
if [ -f "terraform.tfstate" ]; then
    terraform state pull > "state-backup-target-$(date +%Y%m%d-%H%M%S).tfstate"
fi

# Step 2: Export resources from source
echo "Exporting resources from source state..."
cd "$SOURCE_STATE_DIR"

for resource in $RESOURCES_TO_MOVE; do
    echo "Exporting $resource..."
    
    # Get resource configuration
    terraform state show "$resource" > "/tmp/${resource//[.\/]/_}.tf"
    
    # Remove from source state
    terraform state rm "$resource"
done

# Step 3: Import resources into target
echo "Importing resources into target state..."
cd "$TARGET_STATE_DIR"

# Initialize target if needed
if [ ! -d ".terraform" ]; then
    terraform init
fi

for resource in $RESOURCES_TO_MOVE; do
    echo "Importing $resource..."
    
    # Get resource ID from exported configuration
    RESOURCE_ID=$(grep -E "^# " "/tmp/${resource//[.\/]/_}.tf" | head -1 | awk '{print $NF}')
    
    if [ -n "$RESOURCE_ID" ]; then
        terraform import "$resource" "$RESOURCE_ID"
    else
        echo "Warning: Could not determine resource ID for $resource"
    fi
done

# Step 4: Verify both configurations
echo "Verifying source configuration..."
cd "$SOURCE_STATE_DIR"
terraform plan

echo "Verifying target configuration..."
cd "$TARGET_STATE_DIR"
terraform plan

echo "State splitting completed!"

Cross-Account State Migration

Migrate state between different AWS accounts:

# Source account backend configuration
terraform {
  backend "s3" {
    bucket         = "source-account-terraform-state"
    key            = "infrastructure/terraform.tfstate"
    region         = "us-west-2"
    dynamodb_table = "terraform-locks"
    encrypt        = true
    
    # Source account credentials
    profile = "source-account"
  }
}

# Target account backend configuration
terraform {
  backend "s3" {
    bucket         = "target-account-terraform-state"
    key            = "infrastructure/terraform.tfstate"
    region         = "us-west-2"
    dynamodb_table = "terraform-locks"
    encrypt        = true
    
    # Target account credentials
    profile = "target-account"
  }
}

#!/bin/bash
# scripts/cross-account-migration.sh

set -e

SOURCE_PROFILE=${1:-"source-account"}
TARGET_PROFILE=${2:-"target-account"}
RESOURCES_TO_MIGRATE=${3:-"aws_s3_bucket.shared_data"}

echo "Migrating resources between AWS accounts..."

# Step 1: Export from source account
echo "Exporting resources from source account..."
export AWS_PROFILE="$SOURCE_PROFILE"

# Backup source state
terraform state pull > "source-state-backup-$(date +%Y%m%d-%H%M%S).tfstate"

# Get resource details
for resource in $RESOURCES_TO_MIGRATE; do
    echo "Getting details for $resource..."
    terraform state show "$resource" > "/tmp/${resource//[.\/]/_}-config.txt"
    
    # Extract resource ID
    RESOURCE_ID=$(terraform state show "$resource" | grep "^# " | head -1 | awk '{print $NF}')
    echo "$resource:$RESOURCE_ID" >> "/tmp/resource-mappings.txt"
done

# Step 2: Remove from source state
for resource in $RESOURCES_TO_MIGRATE; do
    terraform state rm "$resource"
done

# Step 3: Import into target account
echo "Importing resources into target account..."
export AWS_PROFILE="$TARGET_PROFILE"

# Initialize target configuration
terraform init

# Import resources
while IFS=':' read -r resource resource_id; do
    echo "Importing $resource with ID $resource_id..."
    terraform import "$resource" "$resource_id"
done < "/tmp/resource-mappings.txt"

# Step 4: Verify both accounts
echo "Verifying source account..."
export AWS_PROFILE="$SOURCE_PROFILE"
terraform plan

echo "Verifying target account..."
export AWS_PROFILE="$TARGET_PROFILE"
terraform plan

echo "Cross-account migration completed!"

Module Refactoring

Refactor resources into modules without losing state:

#!/bin/bash
# scripts/refactor-to-modules.sh

set -e

MODULE_NAME=${1:-"vpc"}
RESOURCES_TO_REFACTOR=${2:-"aws_vpc.main aws_subnet.public aws_internet_gateway.main"}

echo "Refactoring resources into $MODULE_NAME module..."

# Step 1: Backup current state
terraform state pull > "state-backup-$(date +%Y%m%d-%H%M%S).tfstate"

# Step 2: Create module directory structure
mkdir -p "modules/$MODULE_NAME"

# Step 3: Move resource configurations to module
echo "Creating module configuration..."
cat > "modules/$MODULE_NAME/main.tf" << 'EOF'
resource "aws_vpc" "main" {
  cidr_block           = var.cidr_block
  enable_dns_hostnames = var.enable_dns_hostnames
  enable_dns_support   = var.enable_dns_support
  
  tags = merge(var.tags, {
    Name = "${var.name}-vpc"
  })
}

resource "aws_subnet" "public" {
  count             = length(var.public_subnet_cidrs)
  vpc_id            = aws_vpc.main.id
  cidr_block        = var.public_subnet_cidrs[count.index]
  availability_zone = var.availability_zones[count.index]
  
  map_public_ip_on_launch = true
  
  tags = merge(var.tags, {
    Name = "${var.name}-public-${count.index + 1}"
  })
}

resource "aws_internet_gateway" "main" {
  vpc_id = aws_vpc.main.id
  
  tags = merge(var.tags, {
    Name = "${var.name}-igw"
  })
}
EOF

cat > "modules/$MODULE_NAME/variables.tf" << 'EOF'
variable "name" {
  description = "Name prefix for resources"
  type        = string
}

variable "cidr_block" {
  description = "CIDR block for VPC"
  type        = string
}

variable "availability_zones" {
  description = "List of availability zones"
  type        = list(string)
}

variable "public_subnet_cidrs" {
  description = "CIDR blocks for public subnets"
  type        = list(string)
}

variable "enable_dns_hostnames" {
  description = "Enable DNS hostnames"
  type        = bool
  default     = true
}

variable "enable_dns_support" {
  description = "Enable DNS support"
  type        = bool
  default     = true
}

variable "tags" {
  description = "Additional tags"
  type        = map(string)
  default     = {}
}
EOF

cat > "modules/$MODULE_NAME/outputs.tf" << 'EOF'
output "vpc_id" {
  description = "VPC ID"
  value       = aws_vpc.main.id
}

output "public_subnet_ids" {
  description = "Public subnet IDs"
  value       = aws_subnet.public[*].id
}

output "internet_gateway_id" {
  description = "Internet Gateway ID"
  value       = aws_internet_gateway.main.id
}
EOF

# Step 4: Move resources in state
echo "Moving resources to module namespace..."
for resource in $RESOURCES_TO_REFACTOR; do
    NEW_ADDRESS="module.$MODULE_NAME.$resource"
    echo "Moving $resource to $NEW_ADDRESS"
    terraform state mv "$resource" "$NEW_ADDRESS"
done

# Step 5: Update main configuration
echo "Updating main configuration to use module..."
cat > main.tf << EOF
module "$MODULE_NAME" {
  source = "./modules/$MODULE_NAME"
  
  name               = "main"
  cidr_block         = "10.0.0.0/16"
  availability_zones = ["us-west-2a", "us-west-2b"]
  public_subnet_cidrs = ["10.0.1.0/24", "10.0.2.0/24"]
  
  tags = {
    Environment = "production"
    ManagedBy   = "terraform"
  }
}

# Update any references to the old resource names
output "vpc_id" {
  value = module.$MODULE_NAME.vpc_id
}

output "public_subnet_ids" {
  value = module.$MODULE_NAME.public_subnet_ids
}
EOF

# Step 6: Verify refactoring
echo "Verifying refactoring..."
terraform init
terraform plan

echo "If the plan shows no changes, refactoring was successful!"
echo "Backup saved at: $BACKUP_FILE"

State Import Strategies

Import existing resources into Terraform management:

#!/usr/bin/env python3
# scripts/bulk_import.py

import boto3
import subprocess
import json
from typing import List, Dict, Tuple

class TerraformImporter:
    def __init__(self, aws_region: str = "us-west-2"):
        self.aws_region = aws_region
        self.ec2 = boto3.client('ec2', region_name=aws_region)
        self.rds = boto3.client('rds', region_name=aws_region)
        self.s3 = boto3.client('s3')
    
    def discover_ec2_instances(self) -> List[Tuple[str, str]]:
        """Discover EC2 instances for import"""
        instances = []
        
        response = self.ec2.describe_instances()
        for reservation in response['Reservations']:
            for instance in reservation['Instances']:
                if instance['State']['Name'] != 'terminated':
                    instance_id = instance['InstanceId']
                    
                    # Generate Terraform resource name from tags
                    name_tag = next(
                        (tag['Value'] for tag in instance.get('Tags', []) if tag['Key'] == 'Name'),
                        instance_id
                    )
                    
                    # Clean name for Terraform resource
                    resource_name = name_tag.lower().replace(' ', '_').replace('-', '_')
                    terraform_address = f"aws_instance.{resource_name}"
                    
                    instances.append((terraform_address, instance_id))
        
        return instances
    
    def discover_s3_buckets(self) -> List[Tuple[str, str]]:
        """Discover S3 buckets for import"""
        buckets = []
        
        response = self.s3.list_buckets()
        for bucket in response['Buckets']:
            bucket_name = bucket['Name']
            
            # Generate Terraform resource name
            resource_name = bucket_name.replace('-', '_').replace('.', '_')
            terraform_address = f"aws_s3_bucket.{resource_name}"
            
            buckets.append((terraform_address, bucket_name))
        
        return buckets
    
    def discover_rds_instances(self) -> List[Tuple[str, str]]:
        """Discover RDS instances for import"""
        instances = []
        
        response = self.rds.describe_db_instances()
        for db_instance in response['DBInstances']:
            if db_instance['DBInstanceStatus'] != 'deleting':
                db_identifier = db_instance['DBInstanceIdentifier']
                
                # Generate Terraform resource name
                resource_name = db_identifier.replace('-', '_')
                terraform_address = f"aws_db_instance.{resource_name}"
                
                instances.append((terraform_address, db_identifier))
        
        return instances
    
    def generate_terraform_config(self, resources: List[Tuple[str, str]], resource_type: str) -> str:
        """Generate Terraform configuration for discovered resources"""
        config_lines = []
        
        for terraform_address, resource_id in resources:
            resource_name = terraform_address.split('.')[1]
            
            if resource_type == "aws_instance":
                config_lines.append(f'''
resource "aws_instance" "{resource_name}" {{
  # Configuration will be populated after import
  # Run 'terraform plan' to see the current configuration
  
  lifecycle {{
    ignore_changes = [
      ami,  # Prevent replacement due to AMI updates
      user_data,  # Ignore user data changes
    ]
  }}
  
  tags = {{
    Name      = "{resource_name}"
    ManagedBy = "terraform"
    Imported  = "true"
  }}
}}
''')
            
            elif resource_type == "aws_s3_bucket":
                config_lines.append(f'''
resource "aws_s3_bucket" "{resource_name}" {{
  bucket = "{resource_id}"
  
  tags = {{
    Name      = "{resource_name}"
    ManagedBy = "terraform"
    Imported  = "true"
  }}
}}
''')
            
            elif resource_type == "aws_db_instance":
                config_lines.append(f'''
resource "aws_db_instance" "{resource_name}" {{
  identifier = "{resource_id}"
  
  # Configuration will be populated after import
  skip_final_snapshot = true
  
  tags = {{
    Name      = "{resource_name}"
    ManagedBy = "terraform"
    Imported  = "true"
  }}
}}
''')
        
        return '\n'.join(config_lines)
    
    def import_resources(self, resources: List[Tuple[str, str]]) -> Dict[str, bool]:
        """Import resources into Terraform state"""
        results = {}
        
        for terraform_address, resource_id in resources:
            try:
                print(f"Importing {terraform_address} with ID {resource_id}...")
                
                result = subprocess.run(
                    ["terraform", "import", terraform_address, resource_id],
                    capture_output=True,
                    text=True,
                    check=True
                )
                
                results[terraform_address] = True
                print(f"✅ Successfully imported {terraform_address}")
                
            except subprocess.CalledProcessError as e:
                results[terraform_address] = False
                print(f"❌ Failed to import {terraform_address}: {e.stderr}")
        
        return results
    
    def run_bulk_import(self, resource_types: List[str]) -> Dict[str, any]:
        """Run bulk import for specified resource types"""
        all_resources = []
        generated_configs = []
        
        for resource_type in resource_types:
            if resource_type == "aws_instance":
                resources = self.discover_ec2_instances()
                config = self.generate_terraform_config(resources, resource_type)
            elif resource_type == "aws_s3_bucket":
                resources = self.discover_s3_buckets()
                config = self.generate_terraform_config(resources, resource_type)
            elif resource_type == "aws_db_instance":
                resources = self.discover_rds_instances()
                config = self.generate_terraform_config(resources, resource_type)
            else:
                continue
            
            all_resources.extend(resources)
            generated_configs.append(config)
        
        # Write generated configuration
        with open('imported_resources.tf', 'w') as f:
            f.write('\n'.join(generated_configs))
        
        print(f"Generated configuration for {len(all_resources)} resources")
        print("Configuration written to imported_resources.tf")
        
        # Import resources
        import_results = self.import_resources(all_resources)
        
        successful_imports = sum(1 for success in import_results.values() if success)
        total_imports = len(import_results)
        
        return {
            'total_resources_discovered': len(all_resources),
            'total_imports_attempted': total_imports,
            'successful_imports': successful_imports,
            'failed_imports': total_imports - successful_imports,
            'import_results': import_results
        }

def main():
    import argparse
    
    parser = argparse.ArgumentParser(description='Bulk import AWS resources into Terraform')
    parser.add_argument('--resource-types', nargs='+', 
                       choices=['aws_instance', 'aws_s3_bucket', 'aws_db_instance'],
                       default=['aws_instance', 'aws_s3_bucket'],
                       help='Resource types to discover and import')
    parser.add_argument('--aws-region', default='us-west-2', help='AWS region')
    parser.add_argument('--output', help='Output file for import results')
    
    args = parser.parse_args()
    
    importer = TerraformImporter(args.aws_region)
    results = importer.run_bulk_import(args.resource_types)
    
    if args.output:
        with open(args.output, 'w') as f:
            json.dump(results, f, indent=2)
    
    print(f"\nBulk import completed:")
    print(f"  Discovered: {results['total_resources_discovered']} resources")
    print(f"  Imported: {results['successful_imports']}/{results['total_imports_attempted']}")
    
    if results['failed_imports'] > 0:
        print(f"  Failed: {results['failed_imports']} imports")
        exit(1)

if __name__ == "__main__":
    main()

What’s Next

State migration and refactoring techniques enable you to evolve your Terraform configurations safely while preserving your infrastructure. These patterns are essential for maintaining long-term infrastructure projects that need to adapt to changing requirements and organizational structures.

In the next part, we’ll explore locking and concurrency control mechanisms that prevent state corruption and enable safe collaboration in team environments where multiple people need to make infrastructure changes.

Continue Your Learning

This is part 2 of 6 in the comprehensive guide.

← Previous Remote Backend Patterns Guide Overview See all 6 parts Next → Locking and Concurrency