Kinesis

​

Config

connector_type: kinesis
stream_name: stream_name
partition_key: key

• partition_key: A partition key is used to group data by shard within a stream. Kinesis Data Streams segregates the data records belonging to a stream into multiple shards. It uses the partition key that is associated with each data record to determine which shard a given data record belongs to.

​

Authentication

Here are the options to authenticate with the AWS Kinesis.

1. Add the following keys and values to your environment variables

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_REGION

2. If you deploy Mage on AWS ECS cluster, you can use ECS execution task role to authenticate. You can grant the ECS task permissions to access other AWS services by attaching IAM policies to this ECS task execution role.

If you get the botocore.exceptions.NoRegionError error, try setting the AWS_DEFAULT_REGION environment variable.