.gif)
Creating a New Secret in AWS Secrets Manager
Before you can use AWS Secrets Manager with Mage, you need to create a new secret in the AWS Secrets Manager console. Follow these steps:- Open the AWS Secrets Manager console
- Click “Store a new secret.”
- On the “Choose Secret Type” page, select the appropriate secret type for your use case, and then click “Next.”
- On the “Configure Secret” page, provide the necessary details for your secret, such as the key-value pairs or the plaintext value, and then click “Next.”
- (Optional) On the “Configure Rotation” page, you can enable automatic rotation for your secret. Automatic rotation periodically updates the secret value, ensuring that your sensitive data remains secure. Configure the rotation settings according to your requirements, and then click “Next.”
- Review the secret details on the “Review” page, and if everything looks correct, click “Store.”
Working with Secrets in Mage
Mage works with AWS secrets in two ways: directly accessing them through Python code, and referencing them in YAML configuration files using placeholders that Mage resolves at runtime. Python: To use secrets in Python code, read these instructions. YAML:Add Secrets Through Mage UI
Add Secrets Through Mage UI
Mage allows you to add your secrets directly through the UI. Here’s how:
- From the right side of the Pipeline editor page, hover over the sidekick and select the Secrets tab
- Enter each secret by clicking the new button
- AWS_ACCESS_KEY_ID: A unique identifier that forms part of the credentials required to authenticate with AWS services. (not needed if using IAM role to authenticate)
- AWS_SECRET_ACCESS_KEY: The secret key that, combined with the access key ID, is used to cryptographically sign requests to AWS services. (not needed if using IAM role to authenticate)
- AWS_DEFAULT_REGION: The default AWS region to use for service API calls when not explicitly specified.(or AWS_REGION_NAME)
- Make the changes as pictured below to represent your secrets in the io_config.yml file
- Each secret is now stored and referenced for configuration to AWS
Add Secrets Through Environment Variables
Add Secrets Through Environment Variables
Add Secrets directly into your environment variables through the io_config.yml file or your docker compose file. You can also add secrets in a .env file using a similar strategy.To use secrets in YAML files (e.g. in data integration pipelines):For Example
- Add the following keys and values to your environment variables
- AWS_ACCESS_KEY_ID: A unique identifier that forms part of the credentials required to authenticate with AWS services. (not needed if using IAM role to authenticate)
- AWS_SECRET_ACCESS_KEY: The secret key that, combined with the access key ID, is used to cryptographically sign requests to AWS services. (not needed if using IAM role to authenticate)
- AWS_DEFAULT_REGION: The default AWS region to use for service API calls when not explicitly specified.(or AWS_REGION_NAME)
- Store a secret in AWS Secrets Manager.
- Use the following syntax in your YAML file to interpolate secret values from AWS Secrets Manager:
- Make the changes as pictured below to represent your secrets in the io_config.yml file
- Each secret is now stored and referenced for configuration to AWS
Working with AWS Secrets Manager
.gif)
- AWS_SECRET_ACCESS_KEY: The secret value you want get from AWS Secrets Manager
- arn: This is a prefix that identifies the resource as an ARN.
- aws: This is the partition that the resource is in. In this case, it’s the AWS partition.
- secretsmanager: This is the service namespace, which identifies the AWS service. In this case, it’s the AWS Secrets Manager service.
- AWS Region: This is the AWS region where the secret is stored (i.e. us-west-2).
- AWS Account ID: This is the 12-digit AWS account ID that the secret belongs to.
- secret This is the resource type, which indicates that the resource is a secret.
- AWS_SECRET_ACCESS_KEY Name: This is the friendly name or ID of the specific secret you want to reference. It can include alphanumeric characters, hyphens, and underscores.