Source

This page is generated from skills/terraform-skill/references/state-management.md. Edit the source, not this page.

Third-party skill

This skill is maintained by Anton Babenko (terraform-best-practices.com, Compliance.tf) under the Apache-2.0 license. Upstream: https://github.com/antonbabenko/terraform-skill

State Management

Part of: terraform-skill Purpose: Comprehensive state management patterns and best practices for Terraform/OpenTofu

This document provides detailed guidance on state management, from remote backend configuration to recovery strategies and multi-team isolation patterns.

---

Table of Contents

1. Remote State Configuration
2. State Locking
3. State Security
4. State Migration
5. Multi-Team State Isolation
6. State Recovery & Troubleshooting
7. State Best Practices Summary
   • Safe Destroy Protocol

---

Remote State Configuration

Why Remote State?

Never use local state in teams or production:

• ❌ No locking → concurrent operations → state corruption
• ❌ No backup → accidental deletion → infrastructure loss
• ❌ No versioning → rollback impossible
• ❌ No collaboration → single point of failure
• ❌ Secrets in plaintext → security risk

Use remote backends for:

• ✅ Automatic state locking
• ✅ Encryption at rest and in transit
• ✅ State versioning and backup
• ✅ Team collaboration
• ✅ Audit logging

Choosing a Remote Backend

Backend	Use when
s3	AWS workloads, existing AWS state
azurerm	Azure workloads
gcs	GCP workloads
cloud / TF Cloud / HCP	hosted state, run management, policy enforcement

Locking mechanism per backend: see Backend Locking Support below.

Cross-cloud equivalents

Concern	AWS	Azure	GCP
Backend block	backend "s3" { bucket, key, region, encrypt, use_lockfile }	backend "azurerm" { resource_group_name, storage_account_name, container_name, key }	backend "gcs" { bucket, prefix }
Access control	IAM policy on bucket/role	RBAC role assignment on storage account/container	IAM binding on the bucket
Remote-state data source	terraform_remote_state (backend s3)	terraform_remote_state (backend azurerm)	terraform_remote_state (backend gcs)

Bootstrap parity

Concern	AWS	Azure	GCP
Versioning	S3 bucket versioning	storage account / blob versioning	GCS object versioning
Encryption at rest	explicit: SSE / KMS (aws_s3_bucket_server_side_encryption_configuration)	default-on (SSE; optional CMK)	default-on (Google-managed; optional CMEK)
Public-access block	aws_s3_bucket_public_access_block	allow_nested_items_to_be_public = false + private container	uniform bucket-level access + public access prevention
Bootstrap auth	IAM / OIDC	RBAC / federated credentials	IAM / Workload Identity Federation

AWS S3 Backend (Recommended)

S3 with Native Lock-File (Terraform 1.10+, Recommended)

Simplest setup - no DynamoDB required:

# backend.tf
terraform {
  backend "s3" {
    bucket       = "my-terraform-state"
    key          = "prod/vpc/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true  # Native S3 locking (Terraform 1.10+)
    
    # Optional but recommended
    kms_key_id = "arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012"
  }
}

Tradeoffs vs DynamoDB locking: no separate table, no DynamoDB charges, state and locks co-located in one bucket.

S3 with DynamoDB Locking (Pre-1.10 or Legacy)

Complete setup with DynamoDB:

# backend.tf
terraform {
  backend "s3" {
    bucket         = "my-terraform-state"
    key            = "prod/vpc/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-state-lock"
    
    # Optional but recommended
    kms_key_id = "arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012"
  }
}

When to use DynamoDB locking:

• Terraform versions < 1.10
• Existing infrastructure already using DynamoDB
• Need DynamoDB for other purposes

Migration note: Existing setups using DynamoDB will continue to work. The use_lockfile option is opt-in.

Backend infrastructure setup (Terraform 1.10+ with lock-file):

# bootstrap/main.tf - Run this ONCE to create state backend
resource "aws_s3_bucket" "terraform_state" {
  bucket = "my-terraform-state"

  lifecycle {
    prevent_destroy = true
  }
}

resource "aws_s3_bucket_versioning" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.terraform_state.arn
    }
  }
}

resource "aws_s3_bucket_public_access_block" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

# MFA Delete for production
# Note: Terraform cannot enable S3 MFA Delete. This must be configured
# outside of Terraform using the AWS CLI or an SDK with the root account.
#
# Example (run once, after bucket creation and versioning are enabled):
#
# aws s3api put-bucket-versioning \
#   --bucket my-terraform-state \
#   --versioning-configuration Status=Enabled,MFADelete=Enabled \
#   --mfa "arn-of-mfa-device mfa-code"

# KMS key for encryption
resource "aws_kms_key" "terraform_state" {
  description             = "KMS key for Terraform state encryption"
  deletion_window_in_days = 30
  enable_key_rotation     = true

  tags = {
    Name = "terraform-state-encryption"
  }
}

resource "aws_kms_alias" "terraform_state" {
  name          = "alias/terraform-state"
  target_key_id = aws_kms_key.terraform_state.key_id
}

Backend infrastructure setup (Pre-1.10 with DynamoDB):

# If using DynamoDB locking, add this resource to the above configuration:

# DynamoDB table for state locking
resource "aws_dynamodb_table" "terraform_state_lock" {
  name         = "terraform-state-lock"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }

  lifecycle {
    prevent_destroy = true
  }

  tags = {
    Name        = "Terraform State Lock Table"
    Environment = "shared"
  }
}

Key organization pattern:

s3://my-terraform-state/
├── prod/
│   ├── vpc/terraform.tfstate
│   ├── eks/terraform.tfstate
│   └── rds/terraform.tfstate
├── staging/
│   ├── vpc/terraform.tfstate
│   └── eks/terraform.tfstate
└── dev/
    └── vpc/terraform.tfstate

Azure Storage Backend

# backend.tf
terraform {
  backend "azurerm" {
    resource_group_name  = "terraform-state-rg"
    storage_account_name = "tfstatestorage"
    container_name       = "tfstate"
    key                  = "prod.terraform.tfstate"
    
    # Optional: Use service principal or managed identity
    use_azuread_auth = true
  }
}

Backend setup:

# bootstrap/main.tf
resource "azurerm_resource_group" "terraform_state" {
  name     = "terraform-state-rg"
  location = "East US"
}

resource "azurerm_storage_account" "terraform_state" {
  name                     = "tfstatestorage"
  resource_group_name      = azurerm_resource_group.terraform_state.name
  location                 = azurerm_resource_group.terraform_state.location
  account_tier             = "Standard"
  account_replication_type = "GRS"  # Geo-redundant
  
  # Security settings
  min_tls_version                 = "TLS1_2"
  allow_nested_items_to_be_public = false
  
  blob_properties {
    versioning_enabled = true
  }
}

resource "azurerm_storage_container" "terraform_state" {
  name                  = "tfstate"
  storage_account_name  = azurerm_storage_account.terraform_state.name
  container_access_type = "private"
}

Google Cloud Storage Backend

# backend.tf
terraform {
  backend "gcs" {
    bucket = "my-terraform-state"
    prefix = "prod/vpc"

    # For customer-managed encryption, configure the bucket itself with
    # `default_kms_key_name` (see bootstrap below) rather than the backend.
    # The backend's `encryption_key` attribute is for CSEK (a base64-encoded
    # 32-byte AES-256 key), NOT a Cloud KMS resource name.
  }
}

Backend setup:

# bootstrap/main.tf
resource "google_storage_bucket" "terraform_state" {
  name          = "my-terraform-state"
  location      = "US"
  force_destroy = false

  versioning {
    enabled = true
  }

  encryption {
    default_kms_key_name = google_kms_crypto_key.terraform_state.id
  }

  uniform_bucket_level_access = true

  lifecycle_rule {
    condition {
      num_newer_versions = 10
    }
    action {
      type = "Delete"
    }
  }
}

resource "google_kms_key_ring" "terraform_state" {
  name     = "terraform-state"
  location = "us-central1"
}

resource "google_kms_crypto_key" "terraform_state" {
  name     = "terraform-state-key"
  key_ring = google_kms_key_ring.terraform_state.id

  rotation_period = "7776000s"  # 90 days
}

Terraform Cloud/Enterprise Backend

# backend.tf
terraform {
  cloud {
    organization = "my-org"
    
    workspaces {
      name = "prod-infrastructure"
      # Or use tags for dynamic workspace selection
      # tags = ["prod", "networking"]
    }
  }
}

# Alternative: Terraform Enterprise with custom hostname
terraform {
  cloud {
    hostname     = "terraform.company.com"
    organization = "my-org"
    
    workspaces {
      name = "prod-infrastructure"
    }
  }
}

Terraform Cloud provides: built-in state management and locking, remote execution, Sentinel policy enforcement, cost estimation, private module registry, VCS integration — no backend infra to manage.

Backend Configuration Best Practices

✅ DO:

• Use separate state files per logical component
• Enable versioning on state storage
• Use encryption at rest (KMS)
• Configure state locking
• Use separate backends per environment
• Store backend config in version control
• Use partial configuration for sensitive values

❌ DON'T:

• Use local state for teams or production
• Share state files across unrelated resources
• Hardcode credentials in backend config
• Disable versioning
• Skip encryption
• Use same state for all environments

Partial backend configuration:

# backend.tf - No sensitive values
terraform {
  backend "s3" {
    bucket = "my-terraform-state"
    region = "us-east-1"
    # key, dynamodb_table specified via -backend-config
  }
}

# Pass sensitive config at init time
terraform init \
  -backend-config="key=prod/vpc/terraform.tfstate" \
  -backend-config="dynamodb_table=terraform-state-lock"

# Or use a file
terraform init -backend-config=backend-prod.hcl

---

State Locking

Why Locking Matters

Without locking:

User A: terraform apply  (starts)
User B: terraform apply  (starts at same time)
Result: Both read same state, make conflicting changes
        → State corruption
        → Infrastructure drift
        → Potential outages

With locking:

User A: terraform apply  (acquires lock)
User B: terraform apply  (waits for lock)
Result: Operations are serialized
        → State consistency maintained

Backend Locking Support

Backend	Locking	Lock Mechanism
S3 (Terraform 1.10+)	✅ Native	Lock files
S3 (Pre-1.10)	✅ With DynamoDB	DynamoDB table
Azure Storage	✅ Native	Blob lease
GCS	✅ Native	Object metadata
Terraform Cloud	✅ Native	Built-in
Consul	✅ Native	Consul KV
Postgres	✅ Native	Row locking
Local	❌ None	N/A

S3 Native Lock-File (Terraform 1.10+)

How it works:

• Uses regular S3 objects as lock files
• Lock files stored in the same bucket as state files
• No additional AWS services required
• Automatically deleted when operations complete

Configuration:

terraform {
  backend "s3" {
    bucket       = "my-terraform-state"
    key          = "prod/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true  # Enable native S3 locking
  }
}

Migration from DynamoDB: Set both dynamodb_table and use_lockfile = true during Terraform 1.10+ migration — locks acquire via both mechanisms. Once every workflow runs on 1.10+, remove dynamodb_table.

DynamoDB Locking for S3 (Pre-1.10 or Legacy)

Lock table attributes:

• LockID (String, Hash Key) - Must be exactly "LockID"
• No other attributes needed
• Pay-per-request billing recommended

Lock behavior:

# Terraform acquires lock
terraform plan
# Creates lock: LockID = "bucket/path/to/state"

# Another user attempts operation
terraform apply
# Sees: Error acquiring the state lock
# Default: `-lock-timeout=0s` — fail immediately on lock contention.
# Set `-lock-timeout=<duration>` (e.g. `5m`) to retry with backoff for the specified window.
#   terraform apply -lock-timeout=5m

View current locks:

# Check DynamoDB for active locks
aws dynamodb scan \
  --table-name terraform-state-lock \
  --projection-expression "LockID,Info"

Handling Lock Conflicts

Scenario 1: Lock Already Held

Symptom:

Error: Error acquiring the state lock

Lock Info:
  ID:        a1b2c3d4-e5f6-7890-abcd-ef1234567890
  Path:      bucket/prod/terraform.tfstate
  Operation: OperationTypeApply
  Who:       user@host
  Created:   2026-01-20 12:00:00.123456789 +0000 UTC

Solutions:

1. Wait for operation to complete (recommended)

   # Just wait - the other operation will release the lock

2. Check if operation is actually running

   # If user@host is accessible, check if terraform is running
   ssh user@host "ps aux | grep terraform"

3. Force unlock if operation crashed (DANGEROUS)

   # Only if you're CERTAIN the lock is stale
   terraform force-unlock a1b2c3d4-e5f6-7890-abcd-ef1234567890

Scenario 2: Stale Lock (Operation Crashed)

When to force-unlock:

• ✅ Process crashed/killed
• ✅ Network interruption
• ✅ CI/CD job terminated
• ✅ You verified no operation is running

When NOT to force-unlock:

• ❌ Just because you're impatient
• ❌ Without checking if operation is running
• ❌ In automation (should fail instead)

Safe force-unlock workflow:

# 1. Verify lock exists
terraform plan
# Note the Lock ID from error message

# 2. Check if operation is actually running
# - SSH to the host if accessible
# - Check CI/CD job status
# - Ask team members

# 3. Only if confirmed stale, force unlock
terraform force-unlock LOCK_ID

# 4. Document why you force-unlocked
git commit -m "Force-unlocked state after CI job termination"

Automatic Lock Timeout

Terraform Cloud lock timeout:

Lock timeout in Terraform Cloud is configured through workspace settings in the Terraform Cloud UI under "General Settings" → "Remote Operations" → "Lock Timeout". It cannot be configured through the cloud block in Terraform code.

For other backends, implement timeout in automation:

#!/bin/bash
# wrapper-script.sh
LOCK_TIMEOUT=300  # 5 minutes

timeout $LOCK_TIMEOUT terraform apply -auto-approve

if [ $? -eq 124 ]; then
  echo "Terraform timed out - likely lock held"
  exit 1
fi

State Locking in CI/CD

Prevent concurrent runs:

# GitHub Actions - Use concurrency control
concurrency:
  group: terraform-${{ github.ref }}
  cancel-in-progress: false  # Don't cancel, wait instead

jobs:
  terraform:
    runs-on: ubuntu-latest
    steps:
      - name: Terraform Apply
        run: terraform apply -auto-approve

GitLab CI:

# .gitlab-ci.yml
terraform-apply:
  script:
    - terraform apply -auto-approve
  resource_group: terraform-prod  # Only one job at a time

---

State Security

Encryption at Rest

S3 Backend:

terraform {
  backend "s3" {
    bucket     = "my-terraform-state"
    key        = "prod/terraform.tfstate"
    region     = "us-east-1"
    encrypt    = true  # ✅ Always enable
    kms_key_id = "arn:aws:kms:us-east-1:123456789012:key/..."  # Optional: Use customer-managed key
  }
}

Encryption options:

Method	Key Management	Cost	Use Case
SSE-S3 (AES-256)	AWS-managed	Included	Basic encryption
SSE-KMS	Customer-managed	$$$ per 10K requests	Compliance requirements
SSE-C	Client-managed	Included	Full control needed

Encryption in Transit

All backends use TLS by default:

• S3: HTTPS
• Azure Storage: HTTPS
• GCS: HTTPS
• Terraform Cloud: HTTPS

Enforce TLS-only access:

// S3 bucket policy
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyInsecureTransport",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::my-terraform-state",
        "arn:aws:s3:::my-terraform-state/*"
      ],
      "Condition": {
        "Bool": {
          "aws:SecureTransport": "false"
        }
      }
    }
  ]
}

Access Control Patterns

IAM Policy for S3 Backend (AWS)

Minimal permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::my-terraform-state"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::my-terraform-state/prod/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:DescribeTable",
        "dynamodb:GetItem",
        "dynamodb:PutItem",
        "dynamodb:DeleteItem"
      ],
      "Resource": "arn:aws:dynamodb:us-east-1:*:table/terraform-state-lock"
    },
    {
      "Effect": "Allow",
      "Action": [
        "kms:DescribeKey",
        "kms:Decrypt",
        "kms:Encrypt",
        "kms:GenerateDataKey"
      ],
      "Resource": "arn:aws:kms:us-east-1:*:key/*"
    }
  ]
}

Read-only policy (for auditing):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::my-terraform-state",
        "arn:aws:s3:::my-terraform-state/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:us-east-1:*:key/*"
    }
  ]
}

Environment Isolation with IAM

Principle: Each environment gets its own IAM role with scoped access

# prod-role can only access prod state
resource "aws_iam_role" "terraform_prod" {
  name = "terraform-prod"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          AWS = "arn:aws:iam::123456789012:root"
        }
        Action = "sts:AssumeRole"
      }
    ]
  })
}

resource "aws_iam_role_policy" "terraform_prod_state" {
  role = aws_iam_role.terraform_prod.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "s3:ListBucket",
          "s3:GetObject",
          "s3:PutObject"
        ]
        Resource = [
          "arn:aws:s3:::my-terraform-state",
          "arn:aws:s3:::my-terraform-state/prod/*"  # Only prod path
        ]
      }
    ]
  })
}

Sensitive Data in State Files

What gets stored in state:

• Resource attributes (including computed values)
• Variable values
• Output values
• Sensitive data like:
  • Database passwords
  • API keys
  • SSH keys
  • Certificate private keys

❌ DON'T: Store Secrets in Variables

# BAD: Secret visible in state
variable "database_password" {
  type      = string
  sensitive = true
  default   = "SuperSecret123!"  # ❌ Still stored in state
}

✅ DO: Use Write-Only Arguments (Terraform 1.11+)

# Good: Password never stored in state
resource "aws_db_instance" "this" {
  # ... other config ...
  
  manage_master_user_password = true  # AWS generates and stores password
  
  lifecycle {
    ignore_changes = [master_user_secret]
  }
}

# Access password via Secrets Manager
data "aws_secretsmanager_secret_version" "db_password" {
  secret_id = aws_db_instance.this.master_user_secret[0].secret_arn
}

Caveat: Reading a secret through data "aws_secretsmanager_secret_version" pulls secret_string into the state file on every refresh. If the goal is to keep the raw secret out of state, use an ephemeral resource/data source (Terraform 1.10+), manage_master_user_password, or inject the value via a CI-only environment variable instead of a data source.

✅ DO: Reference External Secrets

# Good: Fetch secret at runtime, not stored in state
data "aws_secretsmanager_secret_version" "db_password" {
  secret_id = "prod/database/master-password"
}

resource "aws_db_instance" "this" {
  password = data.aws_secretsmanager_secret_version.db_password.secret_string
  # Password still in state, but not hardcoded
}

Caveat: The data source writes secret_string into state on every refresh, so this pattern avoids hardcoding — it does not exclude the secret from state. For true state exclusion, use an ephemeral resource/data source (Terraform 1.10+), manage_master_user_password, or a CI-injected env var.

Best Practice: Reconcile State After External Secret Rotation

# After rotation (handled outside Terraform), refresh state so it reflects
# the new value. This does not rotate the secret itself.
terraform apply -refresh-only

State File Audit Logging

S3 Bucket logging:

resource "aws_s3_bucket_logging" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  target_bucket = aws_s3_bucket.logs.id
  target_prefix = "terraform-state-access/"
}

# CloudTrail for API-level logging
resource "aws_cloudtrail" "terraform_state" {
  name           = "terraform-state-trail"
  s3_bucket_name = aws_s3_bucket.cloudtrail_logs.id

  event_selector {
    read_write_type           = "All"
    include_management_events = true

    data_resource {
      type   = "AWS::S3::Object"
      values = ["arn:aws:s3:::my-terraform-state/*"]
    }
  }
}

Azure Storage logging:

azurerm_storage_account.blob_properties does NOT expose a logging sub-block. Configure blob-service audit logs via azurerm_monitor_diagnostic_setting targeting the storage account's blobServices resource.

What to monitor:

• Who accessed state files
• When state was modified
• What changes were made
• Failed access attempts

---

State Migration

Migrating Between Backends

Local → S3 Migration

Step 1: Set up S3 backend infrastructure

# In bootstrap directory
terraform init
terraform apply

Step 2: Add backend config to your Terraform code

# backend.tf - Add this file
terraform {
  backend "s3" {
    bucket         = "my-terraform-state"
    key            = "prod/vpc/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-state-lock"
  }
}

Step 3: Initialize with migration

# Backup local state first
cp terraform.tfstate terraform.tfstate.backup

# Migrate to S3
terraform init -migrate-state

# Terraform will ask: "Do you want to copy existing state to the new backend?"
# Answer: yes

# Verify migration
terraform plan  # Should show no changes

# Verify state in S3
aws s3 ls s3://my-terraform-state/prod/vpc/

Step 4: Clean up local state (AFTER verifying S3 works)

# Remove local state files
rm terraform.tfstate*

# Commit backend config
git add backend.tf
git commit -m "Migrate state to S3 backend"

S3 → Terraform Cloud Migration

Step 1: Authenticate to Terraform Cloud

# Authenticate the CLI
terraform login

The Terraform Cloud workspace is created automatically on first terraform init against the cloud {} block below (provided the org permits auto-creation). Alternatively, pre-create it in the TFC UI or with the tfe_workspace resource. Do NOT use terraform workspace new here — CLI workspaces are a different concept from Terraform Cloud workspaces.

Step 2: Update backend config

# backend.tf - Change from S3 to cloud
terraform {
  cloud {
    organization = "my-org"
    
    workspaces {
      name = "prod-infrastructure"
    }
  }
}

# Remove old S3 backend config

Step 3: Migrate state

# Initialize with migration
terraform init -migrate-state

# Confirm migration
# State will be uploaded to Terraform Cloud

Step 4: Verify and clean up

# Verify in Terraform Cloud UI or CLI
terraform state list

# Old S3 state remains as backup - don't delete immediately
# Keep for 30-90 days, then remove

Backend Change Without Migration

When recreating infrastructure is acceptable:

# Change backend config in backend.tf

# Re-initialize (will create new empty state)
terraform init -reconfigure

# Import existing resources
terraform import aws_vpc.this vpc-12345678
terraform import aws_subnet.private subnet-abcd1234
# ... import all resources ...

# Or destroy and recreate
terraform destroy  # In old backend
terraform apply    # In new backend

State Refactoring with terraform state mv

Use cases:

• Renaming resources
• Moving resources between modules
• Reorganizing state structure
• Splitting monolithic state

Renaming a Resource

# Before: aws_instance.server
# After:  aws_instance.web_server

terraform state mv aws_instance.server aws_instance.web_server

# Update code to match
# In main.tf: resource "aws_instance" "web_server" { ... }

# Verify
terraform plan  # Should show no changes

Moving Resource to Module

# Before: aws_s3_bucket.logs (in root module)
# After:  module.logging.aws_s3_bucket.logs

# Step 1: Create module with resource
# Step 2: Move state
terraform state mv aws_s3_bucket.logs module.logging.aws_s3_bucket.logs

# Step 3: Remove old resource from root module
# Step 4: Add module call
# Step 5: Verify
terraform plan  # Should show no changes

Moving Resource Between Modules

# Move from module.old to module.new
terraform state mv \
  module.old.aws_instance.app \
  module.new.aws_instance.app

Moving Resource to Different State File

Scenario: Splitting state into separate files

Note: terraform state mv only works within the same state file. To move resources between different state files, use the approach below.

Recommended approach: Use terraform state rm and import

# In source state - remove resource
terraform state rm aws_rds_cluster.main

# In destination state - import resource
terraform import aws_rds_cluster.main cluster-identifier

State Push/Pull Operations

Pull state (download):

# View current state
terraform state pull

# Save to file
terraform state pull > terraform.tfstate.backup

# View specific resource
terraform state show aws_instance.web

Push state (upload):

# Restore from backup
terraform state push terraform.tfstate.backup

# DANGEROUS: Overwrites remote state
# Only use for disaster recovery

When to use push/pull:

• ✅ Creating backups
• ✅ Disaster recovery
• ✅ Debugging state issues
• ✅ Manual state surgery (advanced)
• ❌ Regular operations (use terraform commands)
• ❌ Concurrent team access

State Backup Strategies

Automatic Backups

S3 versioning (automatic):

# List all versions
aws s3api list-object-versions \
  --bucket my-terraform-state \
  --prefix prod/vpc/terraform.tfstate

# Restore specific version
aws s3api get-object \
  --bucket my-terraform-state \
  --key prod/vpc/terraform.tfstate \
  --version-id VERSION_ID \
  terraform.tfstate.restored

Pre-operation backup:

# Manual backup before major changes
terraform state pull > backup-$(date +%Y%m%d-%H%M%S).tfstate

# Or in automation
#!/bin/bash
BACKUP_DIR="./state-backups"
mkdir -p $BACKUP_DIR

terraform state pull > "$BACKUP_DIR/terraform.tfstate.$(date +%Y%m%d-%H%M%S)"

# Keep last 30 backups
ls -t $BACKUP_DIR/terraform.tfstate.* | tail -n +31 | xargs rm -f

Disaster Recovery Plan

1. Backup checklist:

• State file backed up (versioning enabled)
• Backend config documented
• IAM policies documented
• Encryption keys accessible
• Restore procedure tested

2. Recovery procedure:

# If state corrupted
# Step 1: Download last known good version
aws s3api get-object \
  --bucket my-terraform-state \
  --key prod/vpc/terraform.tfstate \
  --version-id PREVIOUS_VERSION_ID \
  terraform.tfstate.recovered

# Step 2: Push recovered state
terraform state push terraform.tfstate.recovered

# Step 3: Verify
terraform plan

# Step 4: If resources drifted, reconcile
terraform apply -refresh-only

Provider Removal

Terraform calls the provider plugin's Destroy RPC during apply. Keep the provider installed until every resource for that provider is destroyed or removed from state.

Goal	Use	Tradeoff
Remove provider and destroy the real resource	Two-phase removal (default)	Safe; requires apply
Remove provider and keep the real resource	removed block (Terraform 1.7+, OpenTofu 1.7+)	Declarative; real resource stays but becomes unmanaged
Remove from state manually	terraform state rm <addr>	Orphans the real resource; use only when intentionally abandoning

Two-phase removal

1. Phase 1 — destroy resources, keep provider: Delete resource blocks from config (or mark for destruction). Keep the provider block and required_providers entry. Run terraform plan and confirm target resources show destroy. Run terraform apply. Run terraform state list and verify no resources remain for that provider.
2. Phase 2 — remove provider: Remove the provider block and the required_providers entry. Run terraform init. Run terraform plan and expect no changes and no errors.

removed block

removed {
  from = vault_policy.ops

  lifecycle {
    destroy = false
  }
}

Rules

• ❌ Remove the provider block first: plan cannot resolve the resource type → hard error.
• ✅ Same rule applies to provider aliases and multi-provider modules.
• ✅ Plain terraform init after removal; -upgrade is for bumping existing providers, not required here.

---

Multi-Team State Isolation

State Organization Patterns

Pattern 1: State Per Environment

Structure:

my-company-tf-state/
├── dev/
│   ├── networking/terraform.tfstate
│   ├── compute/terraform.tfstate
│   └── data/terraform.tfstate
├── staging/
│   ├── networking/terraform.tfstate
│   └── compute/terraform.tfstate
└── prod/
    ├── networking/terraform.tfstate
    ├── compute/terraform.tfstate
    └── data/terraform.tfstate

Benefits:

• ✅ Clear environment separation
• ✅ Different IAM roles per environment
• ✅ Blast radius limited to environment
• ✅ Easy to understand

Drawbacks:

• ⚠️ Duplicate code across environments
• ⚠️ Harder to keep environments in sync

Pattern 2: State Per Team/Component

Structure:

my-company-tf-state/
├── networking-team/
│   ├── prod-vpc/terraform.tfstate
│   ├── staging-vpc/terraform.tfstate
│   └── vpn/terraform.tfstate
├── platform-team/
│   ├── prod-eks/terraform.tfstate
│   └── staging-eks/terraform.tfstate
└── data-team/
    ├── prod-rds/terraform.tfstate
    └── prod-redshift/terraform.tfstate

Benefits:

• ✅ Team ownership clear
• ✅ Team-specific access control
• ✅ Independent release cycles
• ✅ Reduced coordination overhead

Drawbacks:

• ⚠️ Cross-team dependencies complex
• ⚠️ Need data sharing mechanisms

Pattern 3: Hybrid (Environment + Component)

Structure:

my-company-tf-state/
├── prod/
│   ├── 01-networking/terraform.tfstate  # VPC, subnets
│   ├── 02-platform/terraform.tfstate    # EKS, ALB
│   ├── 03-data/terraform.tfstate        # RDS, Redis
│   └── 04-applications/terraform.tfstate
├── staging/
│   ├── 01-networking/terraform.tfstate
│   └── 02-platform/terraform.tfstate

Benefits:

• ✅ Clear environment boundaries
• ✅ Component isolation within environment
• ✅ Numbered prefixes show dependencies
• ✅ Team ownership possible

Recommended: This pattern for most teams

Cross-State Data Sharing

Using terraform_remote_state

Producer module (networking):

# outputs.tf in networking module
output "vpc_id" {
  description = "VPC ID for other modules"
  value       = aws_vpc.main.id
}

output "private_subnet_ids" {
  description = "Private subnet IDs"
  value       = aws_subnet.private[*].id
}

output "database_security_group_id" {
  description = "Security group for databases"
  value       = aws_security_group.database.id
}

Consumer module (compute):

# data.tf in compute module
data "terraform_remote_state" "networking" {
  backend = "s3"
  
  config = {
    bucket = "my-terraform-state"
    key    = "prod/networking/terraform.tfstate"
    region = "us-east-1"
  }
}

# Use networking outputs
resource "aws_instance" "app" {
  subnet_id              = data.terraform_remote_state.networking.outputs.private_subnet_ids[0]
  vpc_security_group_ids = [data.terraform_remote_state.networking.outputs.database_security_group_id]
}

Best practices:

• ✅ Document what outputs are for cross-module use
• ✅ Version outputs (add v2 suffix if breaking change)
• ✅ Keep outputs stable (don't rename casually)
• ✅ Use descriptive output names

Alternative: SSM Parameter Store

Producer:

# Store values in SSM
resource "aws_ssm_parameter" "vpc_id" {
  name  = "/terraform/prod/networking/vpc_id"
  type  = "String"
  value = aws_vpc.main.id
}

Consumer:

# Read from SSM
data "aws_ssm_parameter" "vpc_id" {
  name = "/terraform/prod/networking/vpc_id"
}

resource "aws_instance" "app" {
  subnet_id = data.aws_ssm_parameter.vpc_id.value
}

Benefits over remote_state:

• ✅ No direct state dependency
• ✅ Can be read by non-Terraform tools
• ✅ Can be updated without Terraform
• ✅ Fine-grained IAM control

Drawbacks:

• ⚠️ Extra resources to manage
• ⚠️ Potential for values to be out of sync

When to Split vs Combine State

Split State When:

✅ Different lifecycles:

• VPC (rarely changes) vs EC2 instances (frequently updated)

✅ Different teams own components:

• Networking team manages VPC
• Platform team manages Kubernetes
• App teams manage applications

✅ Different risk profiles:

• Critical infrastructure vs experimental features

✅ Large state files:

• State operations becoming slow (>1000 resources — rough heuristic, depends on provider refresh time)

✅ Independent deployment cadence:

• Database needs weekly updates
• Application needs daily updates

Combine State When:

✅ Tightly coupled resources:

• EC2 instance + EBS volume + ENI

✅ Same lifecycle:

• All resources created/destroyed together

✅ Simple, small project:

• < 100 resources
• Single team ownership

✅ Resources reference each other frequently:

• Security groups that reference each other

Decision Matrix

Factor	Split State	Single State
Team size	Multiple teams	Single team
Resource count	>500 resources	<100 resources (rough heuristics — depends on provider refresh time)
Update frequency	Different cadences	Same cadence
Risk tolerance	Low (production)	High (dev/test)
Coupling	Loosely coupled	Tightly coupled
Ownership	Multiple owners	Single owner

---

State Recovery & Troubleshooting

Recovering from State Corruption

Scenario 1: State File Corrupted

Symptoms:

Error: state snapshot was created by Terraform v1.8.0,
       which is newer than current v1.6.0

Solutions:

A) Restore from backup:

# S3 versioning
aws s3api list-object-versions \
  --bucket my-terraform-state \
  --prefix prod/terraform.tfstate

aws s3api get-object \
  --bucket my-terraform-state \
  --key prod/terraform.tfstate \
  --version-id PREVIOUS_VERSION \
  terraform.tfstate.restored

terraform state push terraform.tfstate.restored

B) Upgrade Terraform:

# Download newer version
tfenv install 1.8.0
tfenv use 1.8.0

# Verify
terraform version

Scenario 2: State Completely Lost

If no backup exists:

1. Recreate state from existing infrastructure:

# List all resources to import
# (You'll need to know what was managed)

# Import resources one by one
terraform import aws_vpc.main vpc-12345678
terraform import aws_subnet.private[0] subnet-abcd1234
terraform import aws_subnet.private[1] subnet-efgh5678
# ... continue for all resources ...

# Verify
terraform plan  # Should eventually show no changes

2. Use import blocks (Terraform 1.5+):

# import.tf
import {
  to = aws_vpc.main
  id = "vpc-12345678"
}

import {
  to = aws_subnet.private[0]
  id = "subnet-abcd1234"
}

terraform plan -generate-config-out=generated.tf
# Review generated.tf and merge with existing config
terraform apply

Handling State Lock Stuck Issues

Issue: Lock Persists After Crash

Check lock status:

# DynamoDB (S3 backend)
aws dynamodb get-item \
  --table-name terraform-state-lock \
  --key '{"LockID":{"S":"my-terraform-state/prod/terraform.tfstate"}}'

# If item exists, lock is held

Force unlock:

# Get Lock ID from error message or DynamoDB
terraform force-unlock LOCK_ID

# Confirm when prompted

Prevent future issues in CI/CD:

Use concurrency controls instead of automatic force-unlock (see CI/CD section below).

Issue: Cannot Acquire Lock in CI/CD

Problem: Parallel CI jobs trying to acquire lock

Solution 1: Use concurrency control

# GitHub Actions
concurrency:
  group: terraform-${{ matrix.environment }}
  cancel-in-progress: false

jobs:
  terraform:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        environment: [dev, staging, prod]

Solution 2: Separate state per branch/PR

# backend.tf (partial configuration - key provided dynamically)
terraform {
  backend "s3" {
    bucket = "my-terraform-state"
    region = "us-east-1"
    # Note: Backend configuration does not support interpolation or env vars.
    # Set the key dynamically during init in CI/CD:
  }
}

# In CI/CD workflow (e.g., GitHub Actions):
# terraform init -backend-config="key=pr-${GITHUB_PR_NUMBER}/terraform.tfstate"

State Refresh and Reconciliation

State drift: State doesn't match reality

Detect drift:

# Terraform 0.15.4+
terraform plan -refresh-only

# Shows what's changed in infrastructure vs state

Reconcile drift:

# Update state to match reality (no infrastructure changes)
terraform apply -refresh-only

# Or during regular plan/apply
terraform plan   # Includes refresh
terraform apply  # Updates state

Common drift causes:

• Manual changes in AWS console
• Changes by other tools (aws cli, CDK)
• Resource deletion outside Terraform
• Provider API changes

Prevent drift:

• ✅ Use CloudTrail to monitor manual changes
• ✅ Implement policy to block manual changes
• ✅ Use drift detection tools (Terraform Cloud drift detection, driftctl)
• ✅ Regular terraform plan in CI/CD
• ✅ Enable termination protection on critical resources

Import for State Recovery

When to use import:

• Resources created manually, now want Terraform to manage
• Recovering from state loss
• Adopting existing infrastructure
• Migrating from other IaC tools

Import workflow:

1. Write resource configuration:

# main.tf
resource "aws_instance" "web" {
  ami           = "ami-12345678"
  instance_type = "t3.micro"
  # ... other attributes ...
}

2. Import existing resource:

# Find resource ID
aws ec2 describe-instances --filters "Name=tag:Name,Values=web-server"

# Import
terraform import aws_instance.web i-1234567890abcdef0

3. Reconcile configuration:

# Plan will show attributes that don't match
terraform plan

# Update main.tf to match actual resource
# Or update resource to match main.tf

4. Verify:

terraform plan  # Should show no changes

Bulk import:

#!/bin/bash
# import-instances.sh

# Get all instance IDs
INSTANCE_IDS=$(aws ec2 describe-instances \
  --query 'Reservations[].Instances[].InstanceId' \
  --output text)

# Import each
for instance_id in $INSTANCE_IDS; do
  terraform import "aws_instance.imported[\"$instance_id\"]" "$instance_id"
done

Terraform 1.5+ import blocks:

# Generate configuration from imports
import {
  to = aws_instance.web
  id = "i-1234567890abcdef0"
}

import {
  to = aws_security_group.web
  id = "sg-0123456789abcdef"
}

# Generate configuration
terraform plan -generate-config-out=imported.tf

# Review and merge
terraform apply

---

State Best Practices Summary

Decision Matrix: State Organization

Scenario	Recommendation	Reasoning
Single team, <100 resources	Single state file	Simple, low overhead
Multiple teams	State per team	Clear ownership, independent deploys
Mixed update frequencies	State per lifecycle	Deploy VPC separately from apps
Production environment	State per component	Limit blast radius
Shared infrastructure	Separate state + remote_state	Core infra stable, apps change often
Development/testing	Combined state OK	Less critical, faster to rebuild

Common Anti-Patterns to Avoid

❌ Anti-Pattern 1: Manual state editing

# DON'T manually edit state files
vim terraform.tfstate  # ❌ Likely to corrupt

✅ Instead: Use terraform state commands

terraform state mv
terraform state rm
terraform import

❌ Anti-Pattern 2: Sharing state files via git

git add terraform.tfstate  # ❌ No locking, merge conflicts

✅ Instead: Use remote backend

❌ Anti-Pattern 3: Bypassing locks

rm .terraform/terraform.tfstate.lock.info  # ❌ Dangerous

✅ Instead: Investigate why lock exists, then force-unlock if safe

❌ Anti-Pattern 4: No backup strategy ✅ Instead: Enable versioning, regular backups, test restore

❌ Anti-Pattern 5: Monolithic state

• 5000+ resources in one state
• Slow operations
• High blast radius ✅ Instead: Split by component/team/environment

Safe Destroy Protocol

A targeted destroy can cascade far beyond its targets via implicit dependencies. Always follow this sequence:

1. Run terraform plan -destroy [-target=...] - never skip straight to destroy
2. Read every resource under "will be destroyed", not just the targeted ones
3. ⚠️ Watch for for_each resources fed by a local that references a targeted resource - all instances become implicit dependents (e.g. targeting an aws_eip that is referenced in a locals {} block used by a for_each record resource will destroy all those records)
4. Get explicit user confirmation of the full list before proceeding
5. ❌ Never use -auto-approve on destroy in production

LLM Mistake Checklist — State Management

Common model mistakes to correct before returning state-related recommendations:

• recommends local state in team/production contexts
• proposes one monolithic root state for "convenience"
• suggests rm .terraform.tfstate.lock.info or force-unlock without investigating why the lock exists
• edits terraform.tfstate manually instead of using terraform state mv/rm/import
• commits *.tfstate to git
• mixes prod and non-prod in the same backend key
• recommends workspace-only isolation as a substitute for backend-level IAM separation
• writes DynamoDB-lock configuration on Terraform 1.10+ instead of using use_lockfile = true on the S3 backend
• runs terraform destroy -auto-approve or skips plan -destroy before a targeted destroy - missing implicit dependents pulled in via shared locals
• reads via terraform_remote_state within a single team's stack instead of using module outputs (see module-patterns.md)
• omits the rollback/recovery note for destructive state operations

State Management Checklist

Setup: remote backend, locking, encryption at rest, IAM-scoped access, versioning/backup, audit logging.

Per environment: separate backend key, scoped IAM role, documented key naming convention, documented DR plan.

Ongoing: scheduled drift detection (plan -detailed-exitcode), tested state restore, review when a state exceeds ~500 resources, documented force-unlock policy.

Security: no secrets in variables/state (use write_only or external lookup), TLS enforced, audit log reviewed, encryption keys rotated.

---

Back to: Main Skill File