【Consejos/solución alternativa】Cómo utilizar el último Boto3 con SSM Automation (o Lambda)

📅 Creado: 2024/05/19, Actualizado: 2024/05/19 ⏱️ Tiempo estimado de lectura ~3 mins (699 palabras)

SSM Automation aws:executeScript may not be able to run the latest API.
For example, the Basseline API of Control Tower is now available on December 15, 2024, but at the time of writing this article, it is not available on aws:executeScript.
SSM Automation aws:executeScript action and Step Functions are the same, and the latest APIs may not be available.In order to respond to this, we have examined the version of Boto3 in the SSM Automation aws:executeScript action and used the latest API.
Simply upgrading Boto3 with pip did not work, so I will describe the corresponding method.

Índice

What is Automation?
Key Takeaways
1. For SSM Automation
2. For Lambda
Verification Results (for SSM Automation)
SSM Automation Terraform Code Used for Verification
Terraform code used to verify the version upgrade of Boto3 of Lambda

What is Automation?

Automation is a feature of AWS Systems Manager that allows you to automate operational tasks.
You can automate various operations by using a runbook, which defines the tasks executed by automation.
AWS provides many runbooks, so you can execute tasks by simply passing the required parameters.
Reference: Systems Manager Automation runbook reference - AWS Systems Manager Automation runbook reference
You can also define your own runbooks, implement custom processing by combining actions, or call multiple AWS-provided runbooks to create an operational workflow.
Reference: Systems Manager Automation actions reference - AWS Systems Manager

Key Takeaways

For SSM Automation

As a prerequisite, you need to specify a zip file containing the package files for boto3 and botocore in the Attachment section of SSM Automation.
You can use the latest Boto3 by following these two steps:

Set the AWS_DATA_PATH environment variable to the directory ( /botocore/data under the current directory) where the latest model files are stored.

The directory where SSM Automation runs has a random name, so you can use the latest model files by specifying /botocore/data under the current directory.

import os

os.environ["AWS_DATA_PATH"] = f"{os.getcwd()}/botocore/data"

Reload boto3 and botocore.

import os
import sys
import boto3
import botocore

sys.path.insert(0, os.getcwd())
reload(botocore)
reload(boto3)

For Lambda

For Lambda, you must attach a zip file containing the boto3 and botocore package files in a Lambda Layer.
You can use the latest Boto3 by setting the AWS_DATA_PATH environment variable to /opt/python/botocore/data . The following Python code sets the environment variable in the Lambda code, but you can also set it in the Lambda environment settings.

import os

os.environ["AWS_DATA_PATH"] = "/opt/python/botocore/data"

In Lambda, the Lambda Layer is mounted at /opt/python , and modules in the layer take precedence. Unlike SSM Automation, you do not need to reload boto3 or botocore.

Verification Results (for SSM Automation)

As shown in the following class method, botocore loads model files from ${botocore root directory}/data , ~/.aws/models , and AWS_DATA_PATH when a client is created in boto3.
Reference: How to use new AWS services with old boto3 | DevelopersIO
The directory where botocore is installed has the following file structure for model files:

# For Control Tower
botocore/data
└── controltower
     └── 2018-05-10
         ├── endpoint-rule-set-1.json.gz
         ├── paginators-1.json # List of available APIs
         └── service-2.json.gz

If you set the AWS_DATA_PATH environment variable, the latest model files are loaded with the highest priority.
Reference: Configuration - Boto3 1.34.108 documentation
However, even if you set this environment variable, boto3 and botocore themselves are not the latest versions in SSM Automation.
In Lambda, modules in the Lambda Layer take precedence, but in SSM Automation, modules in the zip file (Attachment) are not prioritized.
In SSM Automation, the Attachment file is extracted to /tmp/${random string} . The following output shows the JSON format of sys.path , where the files in the Attachment have the lowest priority.

{
    "sys path": [
        "/var/task",
        "/opt/python/lib/python3.8/site-packages",
        "/opt/python",
        "/var/runtime",
        "/var/lang/lib/python38.zip",
        "/var/lang/lib/python3.8",
        "/var/lang/lib/python3.8/lib-dynload",
        "/var/lang/lib/python3.8/site-packages",
        "/opt/python/lib/python3.8/site-packages",
        "/opt/python",
        "/tmp/4aa790cf-cb0b-48b6-80f0-3ed7f82f228c-2024-03-17-07-15-31"
    ]
}

Therefore, in SSM Automation, you need to add the current directory to sys.path and reload boto3 and botocore.

import os
import sys
import boto3
import botocore

sys.path.insert(0, os.getcwd())
reload(botocore)
reload(boto3)

SSM Automation Terraform Code Used for Verification

To upgrade the boto3 version in SSM Automation, I used the following Terraform file structure:

├── src
│   ├── customer_script.py
│   └── modules
│       └── requirements.txt
└── ssm.tf

src/customer_script.py

In SSM Automation, in addition to upgrading boto3, log output is also considered.
You can use the ListHandler class to store log output in a list and return it at the end, so the log is displayed in the SSM Automation execution screen.

import logging
import json
import os
import sys
from importlib import reload

import boto3
import botocore

AUTOMATION_NAME = "sample-automation"
logger = logging.getLogger(__name__)

class ListHandler(logging.Handler):
    def __init__(self):
        super().__init__()
        self.log_messages = []
        log_format = json.dumps(
            {
                "logLevel": "%(levelname)s",
                "time": "%(asctime)s",
                "line": "%(filename)s: %(lineno)d",
                "message": "%(message)s",
            },
            indent=2,
            ensure_ascii=False,
        )
        self.formatter = logging.Formatter(log_format)

    def emit(self, record):
        self.log_messages.append(json.loads(self.format(record)))

def handler(event, context):
    list_handler = ListHandler()
    logger.addHandler(list_handler)
    logger.setLevel(logging.DEBUG)

    os.environ["AWS_DATA_PATH"] = f"{os.getcwd()}/botocore/data"
    session = boto3.Session()
    logger.debug({"botocore paths": session._loader.search_paths})

    sys.path.insert(0, os.getcwd())
    reload(botocore)
    reload(boto3)

    logger.debug({"sys path": sys.path})
    logger.debug({"current directory": os.getcwd()})
    logger.debug({"child directory": os.listdir()})
    logger.debug({"botocore path": os.path.dirname(botocore.__file__)})
    logger.debug({"boto3 path": boto3.__file__})
    logger.debug({"boto3 version": boto3.__version__})
    logger.debug({"botocore path": botocore.__file__})
    logger.debug({"botocore version": botocore.__version__})

    try:
        client = session.client("controltower")
        logger.info(client.list_baselines())

    except Exception as e:
        logger.error(f"Failed to list_baselines: {e}")
        raise Exception(list_handler.log_messages)

    # The max length of SSM Automation output is 100 KB (102400 characters)
    return list_handler.log_messages

if __name__ == "__main__":
    handler(event={}, context={})

src/modules/requirements.txt

The versions of boto3 and botocore used for SSM Automation verification are specified here.
No specific version is set; the latest version (1.34.64 at the time of writing) is used.

boto3
botocore

ssm.tf

This is the Terraform code used to verify upgrading boto3 in SSM Automation.
Comments like #tfsec:ignore:aws-s3-encryption-customer-key are to disable TFSec checks.
Using null_resource , archive_file , and aws_s3_object , a zip file containing boto3 and botocore is created and attached to SSM Automation.
With CloudFormation, the aws cloudformation package command is not supported, so you must manually create a zip file and get a SHA256 hash. Terraform is more flexible for this purpose.

data "aws_caller_identity" "current" {}

#tfsec:ignore:aws-s3-encryption-customer-key
#tfsec:ignore:aws-s3-enable-bucket-encryption
#tfsec:ignore:aws-s3-enable-bucket-logging
resource "aws_s3_bucket" "bucket" {
  bucket = "sample-automation-${data.aws_caller_identity.current.account_id}"
}

resource "aws_s3_bucket_versioning" "bucket_versioning" {
  bucket = aws_s3_bucket.bucket.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_public_access_block" "bucket_public_access_block" {
  bucket = aws_s3_bucket.bucket.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "null_resource" "install_requirements" {
  triggers = {
    always_run = "${timestamp()}"
  }

  provisioner "local-exec" {
    command = "pip install -r ${path.root}/src/modules/requirements.txt -t ${path.root}/src/modules"
  }
}

data "archive_file" "file" {
  depends_on = [ null_resource.install_requirements ]
  type        = "zip"
  source_dir  = "${path.root}/src/modules"
  output_path = "${path.root}/src/modules.zip"
}

resource "aws_s3_object" "object" {
  bucket = aws_s3_bucket.bucket.id
  key    = "modules.zip"
  source = data.archive_file.file.output_path
}

resource "aws_ssm_document" "runbook" {
  depends_on = [ aws_s3_object.object ]

  name            = "sample-automation"
  document_type   = "Automation"
  document_format = "JSON"
  attachments_source {
    key = "S3FileUrl"
    name = "modules.zip"
    values = [
      "https://${aws_s3_bucket.bucket.bucket}.s3.amazonaws.com/modules.zip"
    ]
  }

  content = jsonencode({
    assumeRole      = aws_iam_role.role.arn
    schemaVersion = "0.3"
    description   = "TestBoto3VersionUp"
    mainSteps = [
      {
        name   = "TestBoto3VersionUp"
        action = "aws:executeScript"
        inputs = {
          Runtime = "python3.8"
          Handler = "handler"
          Script  = "${file("${path.root}/src/customer_script.py")}"
          Attachment = "modules.zip"
        }
      }
    ]
    files = {
      "modules.zip" = {
        checksums = {
          sha256 = filesha256(data.archive_file.file.output_path)
        }
      }
    }
  })
}

resource "aws_iam_role" "role" {
  name = "sample-automation-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Action = "sts:AssumeRole",
        Effect = "Allow",
        Principal = {
          Service = "ssm.amazonaws.com"
        }
      }
    ]
  })
  managed_policy_arns = [
    "arn:aws:iam::aws:policy/service-role/AmazonSSMAutomationRole",
    "arn:aws:iam::aws:policy/AdministratorAccess"
  ]
}

Terraform code used to verify the version upgrade of Boto3 of Lambda

I used the Terraform code with the following file configuration to perform Lambda’s Boto3 version upgrade verification.

├── src
│   ├── function
│   │   └── main.py
│   └── layer
│   │   ├── python # Store the package file of Boto3 / Botocore by processing Terraform under this directory
│   │   └── requirements.txt
└── lambda.tf

src/function/main.py

The Python code of Lambda is simple.

import logging
import os
import sys

import boto3
import botocore

logger = logging.getLogger(__name__)

def handler(event, context):
    logger.setLevel(logging.DEBUG)

    os.environ["AWS_DATA_PATH"] = "/opt/python/botocore/data"
    session = boto3.Session()
    logger.debug({"botocore paths": session._loader.search_paths})

    logger.debug({"sys path": sys.path})
    logger.debug({"botocore path": os.path.dirname(botocore.__file__)})
    logger.debug({"boto3 path": boto3.__file__})
    logger.debug({"boto3 version": boto3.__version__})
    logger.debug({"botocore path": botocore.__file__})
    logger.debug({"botocore version": botocore.__version__})

    client = session.client("controltower")
    logger.info(client.list_baselines())

if __name__ == "__main__":
    handler(event={}, context={})

src/layer/requirements.txt

The package file to be included in Lambda Layer is described.
It is not particularly version, but uses the latest version (1.34.64 at the time of writing).

boto3
botocore

lambda.tf

This is a Terraform code used to verify the version upgrade of Boto3 of Lambda.
Like SSM Automation, we use null_resource and data.archive_file to create a ZIP file including Boto3 / Botocore package files in Lambda Layer.

data "archive_file" "function" {
  depends_on  = [null_resource.install_requirements]
  type        = "zip"
  source_dir  = "${path.root}/src/function"
  output_path = "${path.root}/src/function.zip"
}

resource "aws_lambda_function" "function" {
  function_name    = "sample-function"
  filename         = data.archive_file.function.output_path
  source_code_hash = data.archive_file.function.output_base64sha256
  handler          = "main.handler"
  runtime          = "python3.11"
  role             = aws_iam_role.role.arn
  # os.environ["AWS_DATA_PATH"] is OK even if you set the following environment variables
  # environment {
  #   variables = {
  #     AWS_DATA_PATH = "/opt/python/botocore/data"
  #   }
  # }
  layers        = [aws_lambda_layer_version.layer_version.arn]
}

resource "null_resource" "install_requirements" {
  triggers = {
    always_run = "${timestamp()}"
  }

  provisioner "local-exec" {
    command = "pip install -r ${path.root}/src/layer/requirements.txt -t ${path.root}/src/layer/python"
  }
}

data "archive_file" "layer" {
  depends_on  = [null_resource.install_requirements]

  type        = "zip"
  source_dir  = "${path.root}/src/layer"
  output_path = "${path.root}/src/layer.zip"
}

resource "aws_lambda_layer_version" "layer_version" {
  layer_name = "sample-layer"
  filename  = data.archive_file.layer.output_path
  compatible_runtimes = ["python3.11"]
  source_code_hash = data.archive_file.layer.output_base64sha256
}

resource "aws_iam_role" "role" {
  name = "sample-lambda-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Action = "sts:AssumeRole",
        Effect = "Allow",
        Principal = {
          Service = "lambda.amazonaws.com"
        }
      }
    ]
  })
  managed_policy_arns = [
    "arn:aws:iam::aws:policy/AdministratorAccess"
  ]
}