DynamoDB Collect Slice
Overview
Datacoral’s DynamoDB slice is a collect (datasource) slice that collects data from DynamoDB tables in a given AWS account.
Steps to add this slice to your installation
The steps to add your slice are:
- Set up the necessary roles and permissions
- Add the DynamoDB slice through UI/CLI
1. Set up the necessary roles and permissions
Setting up the required roles and permissions depends on whether the DynamoDB tables reside in the same AWS account as the Datacoral installation or a different one.
DynamoDB tables exist in the same AWS account as that of Datacoral installation
In the LambdaExec role, add the DynamoDB tables that we need to collect data from in the resources section of DynamoDB actions.
Navigate to IAM console in AWS
- In the IAM console, click on "Roles" in the left menu. Search for LambdaExec role with the name starting with datacoral-
<installation>
-*
- Click on the edit policy button on the LambdaExec role screen
- Go to the json editor for the role policy
- In the Action block that has DynamoDB permissions, add a new resource to the resource list that has the table name pattern of the DynamoDB tables to pull
- Click on "Review Policy" and save the changes
DynamoDB tables exist in a different AWS account
a. Create a cross-account role in the external AWS account to allow access to the DynamoDB tables present in the external AWS account. Follow the steps below to create the cross-account role
Navigate to CloudFormation console in AWS
In the CloudFormation console, click on "Create Stack".
Then, click on "With new resources"
Create CF Stack from S3 link
In the Create Stack screen, pick "Template is Ready" and add the following S3 URL:
Configure input parameters for CF Stack
The following input parameters have to be set:
- AccountID: Account id for the AWS account with Datacoral installation
- AllowedIPAddress (optional): vertical bar (|) delimited list of AllowedIPAddress to access dynamodb access (NatGateway EIP)
- DynamoDBTable: DynamoDB Table name or Table name pattern or *
- ExternalId: Unique ExternalId string for the account with the DynamoDB tables.
Click Next to progress to the next screen.
Configure Stack Options, Review and Create Stack
Click Next to proceed through this screen, review your selections and create the stack.
Wait for CF Stack to be created successfully
In the CloudFormation AWS console, wait for the CloudFormation stack to successfully create all resources.
Retrieving information from the successful stack
In the CloudFormation console, click on the DynamoDB cross-account role stack that was successfully created, and click on Outputs to retrieve the following information:
- RoleARN
b. Add policy to allow assume role in the LambdaExec role for that cross-account role
Navigate to IAM console in AWS
- In the IAM console, click on "Roles" in the left menu. Search for LambdaExec role with the name starting with datacoral-
<installation>
-*
- Click on the edit policy button on the LambdaExec role screen
- Go to the json editor for the role policy
- Add a new "Action" block at the end of the policy which is as below
- Click on "Review Policy" and save the changes
Steps to add this slice to your installation via UI
- If the DynamoDB tables exist in a different AWS account than the one with Datacoral installation, specify the cross-account role that is created, external id that was specified while creating cross account role and the region where the DynamoDB tables are.
- Specify the name of the DynamoDB tables that you want to pull data from by clicking "Add a loadunit" button.
- In the loadunit options, please note that you do not need to fill in the DynamoDB Table Name if it is the same as Loadunit name.
- Click on Add Slice
Steps to add this slice to your installation using the CLI
The steps to launch your slice are:
- Specify the slice config
- Add the DynamoDB slice
1. Specify the slice config
To get a template for the DynamoDB slice configuration save the output of the describe --input-parameters
command as follows:
Necessary input parameters:
loadunits
- loadunits configuration
Optional input parameters:
schedule
- in cron format
Loadunits:
Multiple loadunits can be specified in a single slice to pull different tables data. Below is the sample format of the loadunits section of the input parameters file for the DynamoDB Collect Slice.
2. Add the Slice
slice-name
Name of your slice. A schema with your slice-name is automatically created in your warehouseparams-file
File path to your input parameters file. Ex. dynamodb_parameters_file.json
Slice output
Data stored in AWS S3 is partitioned by date and time in the following bucket
s3//<customer_installation>.datacoral/<sliceName>
Destination Warehouse Schema - schema name will be same as a slice-name. Tables produced by the slice are:
Questions? Interested?
If you have questions or feedback, feel free to reach out at hello@datacoral.co or Request a demo