3-Layer Architecture for Fetching Multiple AWS Resources with Terraform Data Source (AWS Provider v6 / awscc / EventBridge)

📅 Created: 2024/05/15, Updated: 2026/11/10 ⏱️ Estimated reading time ~3 mins (635 words)

Thorough explanation of the 3-layer architecture for dynamically fetching multiple AWS resource configurations in Terraform (AWS Provider v6 plural Data Sources, Cloud Control API awscc, and External Data Source), including EventBridge migration and safe for_each = toset() loops.

Table of Contents

Revision History
3 Costly Lessons I Learned from Dynamic AWS Resource References
1. When Should Terraform Data Sources Be Used?
2. The 3-Layer Architecture for Fetching Multiple AWS Resources
3. Practical Pattern 1: AWS Provider v6 and EventBridge Native Deployment
1. 3.1. Standard aws_cloudwatch_log_groups + for_each = toset(...) + check Block
2. 3.2. Modern Amazon EventBridge Namespace and moved Block Migration
4. Summary & Production Guidelines

Revision History

Date Update Details

2026/11/10 - Fully reflected breaking changes and latest features in Terraform AWS Provider (v6.72+).
- Added recommended migration from legacy CloudWatch Events to Amazon EventBridge ( aws_eventbridge_* ) and zero-downtime state migration pattern using moved blocks.
- Introduced safe zero-result handling for plural Data Sources ( aws_subnets , aws_cloudwatch_log_groups , aws_eventbridge_rules ) and existence assertion with Terraform 1.5+ check blocks.
- Replaced fragile count with for_each = toset(...) across all dynamic resource loops to eliminate index-shift destroy/recreate outages.
- Established 3-layer architecture: AWS Provider v6 plural Data Sources -> AWS Cloud Control API ( hashicorp/awscc ) -> sanitized data "external" as deprecated last resort.

2024/03/03 - Initial release (Data Source use cases, AWS resource single vs plural mapping table, and External Data Source workaround with AWS CLI/jq).

3 Costly Lessons I Learned from Dynamic AWS Resource References

Back in March 2024 when I published the first edition of this guide, I was designing cross-account, multi-repository Terraform modules for enterprise AWS environments. The goal was to dynamically fetch multiple existing resources (VPCs, subnets, CloudWatch Log Groups, CodePipelines) via Terraform Data Sources and bulk-deploy monitoring alarms and EventBridge failure notifications.

However, operating these modules in production revealed 3 critical traps that broke down under scale.

First, whenever a plural Data Source was missing in hashicorp/aws , I hastily created Bash scripts with data "external" ( aws cli + jq ). While it worked on local macOS, moving CI/CD to lightweight GitHub Actions or Terraform Cloud runners caused widespread terraform plan failures because runners lacked jq or subshells lost cross-account AssumeRole credentials. Later investigation revealed that AWS Provider v6 added native plural Data Sources ( aws_dynamodb_tables , aws_ecs_clusters , aws_s3_buckets , aws_eventbridge_rules ), and the official AWS Cloud Control API provider ( hashicorp/awscc ) allowed fetching resources like CodePipeline purely in HCL without writing shell scripts.

Second, using data "external" triggers strict Go type constraints: Result Error: json: cannot unmarshal array into Go value of type string . The external provider strictly demands a flat map(string) . Serializing arrays into strings with jq '@json' and decoding in HCL with jsondecode() is required, but empty query results ( [] ) crashed jsondecode("") if the script returned an empty string.

Third, iterating plural Data Source lists with count = length(local.target_ids) led to index-shift outages. In an environment with 5 pipelines where EventBridge failure rules were generated via count , removing pipeline-02 caused all subsequent rules ( index[1] through index[4] ) to be destroyed and recreated simultaneously. Dynamic loops must always bind immutable resource identifiers as state keys using for_each = toset(...) .

This guide outlines the modern 3-layer architecture for safely fetching multiple AWS resources using Terraform AWS Provider v6 (v6.72+), complete with HCL examples and EventBridge best practices.

        flowchart TD
    Start["Requirement: Fetch Multiple AWS Resources"] --> Q1{"Does standard hashicorp/aws (v6.x)<br/>provide a Plural Data Source?"}
    Q1 -- "Yes (Recommended)" --> Layer1["【Layer 1: Highest Priority】<br/>Standard Plural Data Source<br/>(aws_subnets / aws_cloudwatch_log_groups / aws_eventbridge_rules)"]
    Q1 -- "No" --> Q2{"Is it supported by AWS Cloud Control API<br/>(hashicorp/awscc)?"}
    Q2 -- "Yes" --> Layer2["【Layer 2: HCL Native Alternative】<br/>awscc_* Plural Data Source<br/>(awscc_codepipeline_pipelines etc.)"]
    Q2 -- "No / Complex JMESPath Filtering" --> Layer3["【Layer 3: Deprecated Last Resort】<br/>data external + AWS CLI + jq<br/>(Requires jq @json + jsondecode)"]

    Layer1 --> SafeLoop["Key-Pinned Loop: for_each = toset(...)<br/>+ 0-Result Guard: check block"]
    Layer2 --> SafeLoop
    Layer3 --> SafeLoop

    classDef primary fill:#e0f2fe,stroke:#0284c7,stroke-width:2px;
    classDef secondary fill:#fef3c7,stroke:#d97706,stroke-width:2px;
    classDef danger fill:#fee2e2,stroke:#dc2626,stroke-width:2px;
    classDef safe fill:#ecfdf5,stroke:#059669,stroke-width:2px;

    class Layer1 primary;
    class Layer2 secondary;
    class Layer3 danger;
    class SafeLoop safe;
       

1. When Should Terraform Data Sources Be Used?

Terraform Data Sources dynamically query read-only metadata from infrastructure managed outside the current module or in separate repositories.

Referencing Existing Foundation Resources : Querying shared VPCs, subnets, Direct Connect gateways, or security groups managed by central platform teams.
Loosely Coupled Multi-Repo Architecture : Sharing resource IDs across decoupled repositories without granting full State read permissions via terraform_remote_state .
Dynamic Bulk Expansion : Attaching unified monitoring, retention, or tagging across all discovered resources in a region.
Environment Context Discovery : Resolving dynamic account IDs ( aws_caller_identity ), current region ( aws_region ), or latest AMI IDs ( aws_ami ).

2. The 3-Layer Architecture for Fetching Multiple AWS Resources

Layer	Strategy	Advantages	Caveats & Tradeoffs	Recommendation
Layer 1	Standard `hashicorp/aws` Plural Data Source ( `aws_subnets` , `aws_eventbridge_rules` )	Native provider, zero external binary dependency, type-safe	Not all AWS services have plural sources	Top Priority
Layer 2	Official `hashicorp/awscc` Plural Data Source ( `awscc_codepipeline_pipelines` )	Pure HCL without shell scripts, broad CloudFormation coverage	Requires additional provider block, client-side HCL filtering	Native Alternative
Layer 3	`hashicorp/external` Provider ( `data "external"` + AWS CLI + `jq` )	Unlimited CLI JMESPath query filtering	Requires CLI binaries in CI/CD, string map serialization overhead	Deprecated Last Resort

3. Practical Pattern 1: AWS Provider v6 and EventBridge Native Deployment

3.1. Standard `aws_cloudwatch_log_groups` + `for_each = toset(...)` + `check` Block

terraform {
  required_version = ">= 1.5.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.72"
    }
  }
}

# 1. Fetch all log groups matching prefix
data "aws_cloudwatch_log_groups" "prd_lambda_logs" {
  log_group_name_prefix = "/aws/lambda/prd-"
}

# 2. Guard against accidental 0-result deletion using check block (Terraform 1.5+)
check "verify_prd_lambda_log_groups_exist" {
  assert {
    condition     = length(data.aws_cloudwatch_log_groups.prd_lambda_logs.log_group_names) > 0
    error_message = "No CloudWatch log groups found matching /aws/lambda/prd-. Check prefix or region."
  }
}

# 3. Bind resource names as state keys using for_each = toset(...)
resource "aws_cloudwatch_log_metric_filter" "error_count" {
  for_each = toset(data.aws_cloudwatch_log_groups.prd_lambda_logs.log_group_names)

  name           = "error-count-${replace(trimprefix(each.value, "/aws/lambda/"), "/", "-")}"
  pattern        = "?ERROR ?Exception ?Traceback"
  log_group_name = each.value

  metric_transformation {
    name      = "ErrorCount-${replace(trimprefix(each.value, "/aws/lambda/"), "/", "-")}"
    namespace = "Custom/LambdaErrors"
    value     = "1"
  }
}

3.2. Modern Amazon EventBridge Namespace and `moved` Block Migration

# Fetch EventBridge rules using native plural Data Source
data "aws_eventbridge_rules" "app_rules" {
  name_prefix    = "prd-order-"
  event_bus_name = "default"
}

# Attach SNS alert targets across all discovered rules
resource "aws_eventbridge_target" "sns_alert" {
  for_each       = toset(data.aws_eventbridge_rules.app_rules.names)
  rule           = each.value
  event_bus_name = "default"
  arn            = var.alert_sns_topic_arn
}

# Zero-downtime state migration from legacy aws_cloudwatch_event_rule
moved {
  from = aws_cloudwatch_event_rule.legacy_rule
  to   = aws_eventbridge_rule.app_rule
}

4. Summary & Production Guidelines

Prioritize Layer 1 ( hashicorp/aws v6.x) : Always inspect standard plural Data Sources before introducing workarounds.
Adopt Modern EventBridge Namespace : Transition away from legacy aws_cloudwatch_event_* and safely rename in state with moved blocks.
Enforce for_each = toset(...) : Completely eliminate count on dynamic lists to prevent cascading recreation outages.
Defend with check Blocks : Prevent accidental resource destructions when queries return empty sets.
Minimize data "external" : Restrict external scripts strictly to edge cases where client-side query pre-filtering is mandatory.

CONT-AID PRODUCTS

Production AI Applications & Official Ecosystem

AI COACH & STUDIO

CandidLens

AI Mock Calls & 5-Axis Voice Analytics

Master conversation skills with realistic AI phone training and 5-axis feedback. Includes 16-lesson interactive academy and studio AI portraits.

AI Mock Call Voice Analytics AI Studio

Explore CandidLens

SMART EXPENSE & OCR

DEALER

Zero-Friction Bill Split with Smart OCR

Effortlessly snap receipts for instant AI itemization. Integrated with LINE for seamless group expense tracking and pain-free settlement.

Smart Receipt OCR LINE Auth Smart Split

Explore DEALER

AI GAME MASTER

Mingli

Interactive AI Party Game Host

Interactive AI personas (Fortune Teller, Butler, Bartender) host real-time games, challenges, and conversation starters for unforgettable gatherings.

AI Persona Host Party Games Icebreaker

Explore Mingli

⏳ Coming Soon

Download Production-Ready Multi-Agent Rules & Hooks

Get the complete configuration pack featured in this article: universal AGENTS.md, deterministic hooks.json, Git safety guardrails, and validation scripts tested across Antigravity, Kiro, Claude Code, and Codex.

Get Starter Kit (Coming Soon) ※ Currently in preparation on Gumroad. The download link will become active upon launch.