How to get information on multiple AWS resources in Terraform's Data Source

Created date: 2024/05/15, Update date: 2024/05/15

I had the opportunity to use Terraform's Data Source in construction projects, but since there were many AWS resources that were not supported, I summarized the usage cases and correspondence table.
If Data Source is not compatible, a workaround using External Data Source is also described.

Table of Contents

In the first place, what use case is Data Source used?
Data Source list for each AWS resource
How to get multiple AWS resources information using External Data Source

In the first place, what use case is Data Source used?

If you want to refer to a manually created AWS resource
- When you refer to the manually created AWS resource, use Data Sroce.
  - Seavy seeing VPC -related AWS resources.VPCs, subnets, and commonized security groups are unlikely to be changed once they are created, so there are cases where they are manually organized.
  - In some cases, refer to the stateful AWS resource or the AWS resource you want to manually change.

If the Git repository of the Terraform template is split or using Module
- In other Git repositories, use it to refer to AWS resources defined.
- For example, the KMS key is managed by a git repository for KMS administrators to manage, and the system side refers to the KMS key using the Data Sroce in the encryption setting of AWS resources.

If you manage all AWS resources in one Git repository, you may want to separate them later, so we recommend that you decide the rules and divide them.
- When I consider splitting, I will consider division from the viewpoint of “AWS Service Name/Function Name”, “Administrator”, “State/Steel”, and “Life Cycle”.

Also, if you use a module to define resources, use Data Sroce when referring to AWS resources generated in Module.
- For example, if VPC is defined in Module, use Data Sroce when referring to the VPC.

If you want to deploy AWS resources dynamically according to the AWS settings and configuration
- If you want to develop AWS resources according to the settings and configuration of the AWS settings and compositions for the purpose of monitoring, operation, or CI/CD, repeated processing using a Data Source that can acquire information on multiple AWS resources.
  - For example, if you want to set a metrics filter of the same pattern for multiple log groups.

If you want to get accounts and region information
- Use Data Source when acquiring information about AWS accounts and regions.This allows you to dynamically referring to regions and account IDs in the Terraform code, enabling flexible infrastructure.
  - For example, if you want to refer to the resource of the cross account or develop resources according to the region.

If you want to get AMI and image information
- Use AWS AMI (Amazon Machine Image) and Data Source to get other image information.It can be dynamically searched for the latest AMI and images that match specific conditions, and can be used in instances and auto scaling groups.
  - For example, when you get the latest AMI of a specific OS version, launch an EC2 instance, or to create a Docker image to create an ECS task definition.

Data Source list for each AWS resource

I listed it to AWS resources that are often used personally.
Sometimes Data Source is not compatible, but the same definition is possible if you use AWS resource information using External Data Source.
Has a single Data Sroce and a plural Data Source, but the plural Data Source is a Data Source that allows you to get information on multiple AWS resources.
- The single DATA SROUCE causes an error if there is no target AWS resource, but in plural Data Sroce, if there is no target AWS resource, it will return an empty list.
- By applying this, it is possible to check the existence of resources.

AWS Service name ➖➖➖➖➖➖➖➖➖➖➖➖➖	AWS Resource type name ➖➖➖➖➖➖➖➖➖➖➖➖➖➖	Single Data Srouce ➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖	Multi Data Srouce ➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖
AWS IAM	User	aws_iam_user	aws_iam_users
〃	Group	aws_iam_group	Incompatible
〃	Roll	aws_iam_role	aws_iam_roles
〃	Policy	aws_iam_policy	Incompatible
AWS CloudTrail	Trail	Incompatible	Incompatible
AWS Config	Recorder	Incompatible	Incompatible
〃	Agrigator	Incompatible	Incompatible
〃	Rule	Incompatible	Incompatible
Amazon GuardDuty	Detector	aws_guardduty_detector	Incompatible
Amazon VPC	VPC	aws_vpc	aws_vpcs
〃	Subnet	aws_subnet	aws_subnets
〃	Root Table	aws_route_table	aws_route_tables
〃	Root	aws_route	Incompatible
〃	Security Group	aws_security_group	aws_security_groups
Amazon Route 53	Zone	aws_route53_zone	Incompatible
〃	Record	Incompatible	Incompatible
Elastic Load Balancing	Load Balancer	aws_lb	aws_albs
〃	Listener	aws_lb_listener	Incompatible
〃	Target group	aws_lb_target_group	Incompatible
Amazon EC2	Instance	aws_instance	aws_instances
Amazon AutoScaling	Auto scaling group	aws_autoscaling_group	aws_autoscaling_groups
Amazon ECS	Cluster	aws_ecs_cluster	Incompatible
AWS Lambda	Function	aws_lambda_function	aws_lambda_functions
Amazon SQS	Queue	aws_sqs_queue	aws_sqs_queues
Amazon SNS	Topic	aws_sns_topic	Incompatible
Amazon RDS	Instance	aws_db_instance	aws_db_instances
〃	Cluster	aws_rds_cluster	aws_rds_clusters
Amazon Dynamo DB	Table	aws_dynamodb_table	Incompatible
Amazon S3	Bucket	aws_s3_bucket	Incompatible
Amazon ECR	Repository	aws_ecr_repository	Incompatible
AWS CodeCommit	Repository	aws_codecommit_repository	Incompatible
AWS CodePipeline	Pipeline	Incompatible	Incompatible
AWS CodeBuild	Build Project	Incompatible	Incompatible
Amazon Athena	Work group	Incompatible	Incompatible
Amazon CloudWatch	Dashboard	Incompatible	Incompatible
〃	Log Group	aws_cloudwatch_log_group	aws_cloudwatch_log_groups
〃	Metrics Filter	Incompatible	Incompatible
〃	Subscription Filter	Incompatible	Incompatible
〃	Alarm	Incompatible	Incompatible
Amazon EventBridge	Rule	Incompatible	Incompatible
〃	Schedule	Incompatible	Incompatible
AWS Backup	Vault	aws_backup_vault	Incompatible
〃	Plan	aws_backup_plan	Incompatible
〃	Selection	aws_backup_selection	Incompatible
AWS Systems Manager	Parameter	aws_ssm_parameter	Incompatible
〃	Document	aws_ssm_document	Incompatible

How to get multiple AWS resources information using External Data Source

If you want to define a custom Data Source, use External Data Source.
- external_external | Data Sources | hashicorp/external | Terraform Registry
- In order to obtain information on existing resources as Data Source, Program calls scripts such as bash and python, and output JSON format strings as execution results.

Since CodePipeline’s Data Source is Incompatic, we will use Shell Script to get information.

First, I will explain the shell script.


                    # !/bin/bash
                    eval "$(jq -r '@sh "codepipeline_name_prefix=\(.codepipeline_name_prefix)"')"

                    pipelines=$(
                        aws codepipeline \
                        list-pipelines \
                        --query "pipelines[?starts_with(name, '"${codepipeline_name_prefix}"')].name" \
                        --region ap-northeast-1 \
                        --output json
                    )

                    # pointt
                    result=$(cat << EOS
                    {
                      "names": $(echo "${pipelines}" | jq '@json')
                    }
                    EOS
                    )

                    jq -n "$result"

The point is the processing of the hydrangement of the # point part.
Names If you want to store the pipeline name in the array, you will first consider defining the | jq '@json' .


                    # Before the change of points
                    result=$(cat << EOS
                    {
                      "names": $(echo "${pipelines}")
                    }
                    EOS
                    )

                    # Result variable
                    {
                      "names": [
                        "test-pipeline01",
                        "test-pipeline02",
                        "test-pipeline03",
                        "test-pipeline04",
                        "test-pipeline05"
                      ]
                    }

Calling this output result on the Terraform’s External Data Source side will result in the following errors.
- As the error, the output must be string both the key/value.



                    % terraform plan
                    data.external.aws_codepipelines: Reading...
                    ╷
                    │ Error: Unexpected External Program Results
                    │
                    │   with data.external.aws_codepipelines,
                    │   on test.tf line 2, in data "external" "aws_codepipelines":
                    │    2:   program = ["bash", "${path.module}/ListCodePipelineNames.sh"]
                    │
                    │ The data source received unexpected results after executing the program.
                    │
                    │ Program output must be a JSON encoded map of string keys and string values.
                    │
                    │ If the error is unclear, the output can be viewed by enabling Terraform's logging at TRACE level.
                    │ Terraform documentation on logging: https://www.terraform.io/internals/debugging
                    │
                    │ Program: /bin/bash
                    │ Result Error: json: cannot unmarshal array into Go value of type string

To solve this error, Shell Script outputs an array converted to a string and returns the string to the array on the terraform side.
The output when the part of #Point is returned to the first example is as follows.
- The output of the array has been converted to a character string by | jq '@json' .



                    # Before the change of points
                    result=$(cat << EOS
                    {
                      "names": $(echo "${pipelines}" | jq '@json')
                    }
                    EOS
                    )

                    # Result variable
                    {
                      "names": "[\"test-pipeline01\",\"test-pipeline02\",\"test-pipeline03\",\"test-pipeline04\",\"test-pipeline05\"]"
                    }

Next, I will explain the Terraform code.



                    data "external" "aws_codepipelines" {
                      program = ["bash", "${path.module}/scripts/ListCodePipelineNames.sh"]
                      query = {
                        codepipeline_name_prefix = "test-pipeline0"
                      }
                    }

                    # point
                    output aws_codepipelines_names {
                        value = jsondecode(data.external.aws_codepipelines.result["names"])
                    }

I created a scripts directory in the same directory as the Terraform template file and refer to the script called listCodepipelinenames.sh .
In query , codepipeline_name_prefix can be specified as an argument. If you specify codepipeline_name_prefix`, refer to only the pipeline name starting with the specified character string.
The point is the output part of # point .Terraform’s function jsondecode () is decoded to the array by referring to the names` key of the output result of External Data Source.
In practice, when referring to other resources, etc., the same definition is performed, and after converting to an array, repeated processing is implemented.