How to get information on multiple AWS resources in Terraforms Data Source

Created date: 2024/05/15, Update date: 2024/05/15

I had the opportunity to use Terraforms Data Source in construction projects, but since there were many AWS resources that were not supported, I summarized the usage cases and correspondence table.
If Data Source is not compatible, a workaround using External Data Source is also described.

Table of Contents

What are the use cases for Terraform Data Sources?
Data Source List for Common AWS Resources
How to Retrieve Multiple AWS Resources Using the External Data Source

What are the use cases for Terraform Data Sources?

Referring to manually created AWS resources
- Use a Data Source when you need to reference an AWS resource that was created manually.
  - For example, VPC-related resources such as VPCs, subnets, and shared security groups are often created manually and rarely change, so referencing them with a Data Source is common.
  - You may also use a Data Source to reference stateful AWS resources or resources you want to manage manually.
When the Terraform template’s Git repository is split or when using modules
- Use a Data Source to reference AWS resources defined in other Git repositories.
  - For example, a KMS key might be managed in a separate repository by KMS administrators, and other systems can reference it using a Data Source in their encryption settings.
- Even if you manage all AWS resources in a single repository, you might want to split them later. It’s a good idea to establish rules for splitting, such as by AWS service, function, administrator, state, or lifecycle.
- When using modules to define resources, use a Data Source to reference AWS resources created within a module.
  - For example, if a VPC is defined in a module, use a Data Source to reference that VPC.
Dynamically deploying AWS resources based on AWS settings and configuration
- Use a Data Source to retrieve information about multiple AWS resources when you need to deploy resources dynamically for monitoring, operations, or CI/CD purposes.
  - For example, you might want to set up a metrics filter with the same pattern for multiple log groups.
Retrieving account and region information
- Use a Data Source to get information about AWS accounts and regions. This allows you to dynamically reference regions and account IDs in your Terraform code, enabling more flexible infrastructure.
  - For example, you can reference resources across accounts or deploy resources based on the region.
Retrieving AMI and image information
- Use a Data Source to get information about AWS AMIs (Amazon Machine Images) and other images. You can dynamically search for the latest AMI or images that match specific criteria, which is useful for instances and auto scaling groups.
  - For example, you can retrieve the latest AMI for a specific OS version, launch an EC2 instance, or create a Docker image for an ECS task definition.

Data Source List for Common AWS Resources

Here is a list of AWS resources I frequently use.
Sometimes a Data Source is not available, but you can achieve similar results by using the External Data Source to retrieve AWS resource information.
There are single and plural Data Sources. Plural Data Sources allow you to retrieve information about multiple AWS resources at once.
- A single Data Source will return an error if the target AWS resource does not exist, but a plural Data Source will return an empty list if no resources are found.
- This can be used to check for the existence of resources.

AWS Service Name	AWS Resource Type	Single Data Source	Multi Data Source
AWS IAM	User	aws_iam_user	aws_iam_users
〃	Group	aws_iam_group	Not available
〃	Role	aws_iam_role	aws_iam_roles
〃	Policy	aws_iam_policy	Not available
AWS CloudTrail	Trail	Not available	Not available
AWS Config	Recorder	Not available	Not available
〃	Aggregator	Not available	Not available
〃	Rule	Not available	Not available
Amazon GuardDuty	Detector	aws_guardduty_detector	Not available
Amazon VPC	VPC	aws_vpc	aws_vpcs
〃	Subnet	aws_subnet	aws_subnets
〃	Route Table	aws_route_table	aws_route_tables
〃	Route	aws_route	Not available
〃	Security Group	aws_security_group	aws_security_groups
Amazon Route 53	Zone	aws_route53_zone	Not available
〃	Record	Not available	Not available
Elastic Load Balancing	Load Balancer	aws_lb	aws_albs
〃	Listener	aws_lb_listener	Not available
〃	Target Group	aws_lb_target_group	Not available
Amazon EC2	Instance	aws_instance	aws_instances
Amazon AutoScaling	Auto Scaling Group	aws_autoscaling_group	aws_autoscaling_groups
Amazon ECS	Cluster	aws_ecs_cluster	Not available
AWS Lambda	Function	aws_lambda_function	aws_lambda_functions
Amazon SQS	Queue	aws_sqs_queue	aws_sqs_queues
Amazon SNS	Topic	aws_sns_topic	Not available
Amazon RDS	Instance	aws_db_instance	aws_db_instances
〃	Cluster	aws_rds_cluster	aws_rds_clusters
Amazon DynamoDB	Table	aws_dynamodb_table	Not available
Amazon S3	Bucket	aws_s3_bucket	Not available
Amazon ECR	Repository	aws_ecr_repository	Not available
AWS CodeCommit	Repository	aws_codecommit_repository	Not available
AWS CodePipeline	Pipeline	Not available	Not available
AWS CodeBuild	Build Project	Not available	Not available
Amazon Athena	Work Group	Not available	Not available
Amazon CloudWatch	Dashboard	Not available	Not available
〃	Log Group	aws_cloudwatch_log_group	aws_cloudwatch_log_groups
〃	Metrics Filter	Not available	Not available
〃	Subscription Filter	Not available	Not available
〃	Alarm	Not available	Not available
Amazon EventBridge	Rule	Not available	Not available
〃	Schedule	Not available	Not available
AWS Backup	Vault	aws_backup_vault	Not available
〃	Plan	aws_backup_plan	Not available
〃	Selection	aws_backup_selection	Not available
AWS Systems Manager	Parameter	aws_ssm_parameter	Not available
〃	Document	aws_ssm_document	Not available

How to Retrieve Multiple AWS Resources Using the External Data Source

If you need to define a custom Data Source, use the External Data Source.
- external_external | Data Sources | hashicorp/external | Terraform Registry
- To retrieve information about existing resources as a Data Source, use a script (such as bash or Python) that outputs a JSON-formatted string.
For example, since CodePipeline does not have a compatible Data Source, you can use a shell script to get information.
Here is an example shell script:

#!/bin/bash
eval "$(jq -r '@sh "codepipeline_name_prefix=\(.codepipeline_name_prefix)"')"

pipelines=$(\
    aws codepipeline \
    list-pipelines \
    --query "pipelines[?starts_with(name, '"${codepipeline_name_prefix}"')].name" \
    --region ap-northeast-1 \
    --output json
)

# Key point
result=$(cat << EOS
{
  "names": $(echo "${pipelines}" | jq '@json')
}
EOS
)

jq -n "$result"

The key point is the processing at the # Key point section.
If you want to store pipeline names in an array, use | jq '@json' to convert the array to a string.

# Before the change
result=$(cat << EOS
{
  "names": $(echo "${pipelines}")
}
EOS
)

# Resulting variable
{
  "names": [
    "test-pipeline01",
    "test-pipeline02",
    "test-pipeline03",
    "test-pipeline04",
    "test-pipeline05"
  ]
}

If you use this output in Terraform’s External Data Source, you will get the following error:
- The output must be a map of string keys and string values.

% terraform plan
data.external.aws_codepipelines: Reading...
╷
│ Error: Unexpected External Program Results
│
│   with data.external.aws_codepipelines,
│   on test.tf line 2, in data "external" "aws_codepipelines":
│    2:   program = ["bash", "${path.module}/ListCodePipelineNames.sh"]
│
│ The data source received unexpected results after executing the program.
│
│ Program output must be a JSON encoded map of string keys and string values.
│
│ If the error is unclear, the output can be viewed by enabling Terraform's logging at TRACE level.
│ Terraform documentation on logging: https://www.terraform.io/internals/debugging
│
│ Program: /bin/bash
│ Result Error: json: cannot unmarshal array into Go value of type string

To solve this error, have the shell script output the array as a string, and then convert it back to an array in Terraform.
The output when you use the correct approach is as follows:
- The array is converted to a string using | jq '@json' .

# After the change
result=$(cat << EOS
{
  "names": $(echo "${pipelines}" | jq '@json')
}
EOS
)

# Resulting variable
{
  "names": "[\"test-pipeline01\",\"test-pipeline02\",\"test-pipeline03\",\"test-pipeline04\",\"test-pipeline05\"]"
}

Next, here is the Terraform code:

data "external" "aws_codepipelines" {
  program = ["bash", "${path.module}/scripts/ListCodePipelineNames.sh"]
  query = {
    codepipeline_name_prefix = "test-pipeline0"
  }
}

# Key point
output "aws_codepipelines_names" {
  value = jsondecode(data.external.aws_codepipelines.result["names"])
}

Create a scripts directory in the same directory as your Terraform template and reference the ListCodePipelineNames.sh script.
The query block allows you to specify codepipeline_name_prefix as an argument, so you can filter pipeline names by prefix.
If you specify codepipeline_name_prefix , refer to only the pipeline name starting with the specified character string.
The key point is the output section. Use Terraform’s jsondecode() function to decode the string in the names key from the External Data Source output back into an array.
In practice, you can use this pattern to reference other resources as well: convert arrays to strings in your script, then decode them in Terraform for further processing.