DynamoDB Collect Slice

Overview

Datacoral’s DynamoDB slice is a collect (datasource) slice that collects data from DynamoDB tables in a given AWS account.

Steps to add this slice to your installation

The steps to add your slice are:

  1. Set up the necessary roles and permissions
  2. Add the DynamoDB slice through UI/CLI

1. Set up the necessary roles and permissions

Setting up the required roles and permissions depends on whether the DynamoDB tables reside in the same AWS account as the Datacoral installation or a different one.

DynamoDB tables exist in the same AWS account as that of Datacoral installation

In the LambdaExec role, add the DynamoDB tables that we need to collect data from in the resources section of DynamoDB actions.

Navigate to IAM console in AWS
  • In the IAM console, click on "Roles" in the left menu. Search for LambdaExec role with the name starting with datacoral-<installation>-*

  • Click on the edit policy button on the LambdaExec role screen
  • Go to the json editor for the role policy
  • In the Action block that has DynamoDB permissions, add a new resource to the resource list that has the table name pattern of the DynamoDB tables to pull
{
"Action": [
"dynamodb:BatchGetItem",
"dynamodb:BatchWriteItem",
"dynamodb:CreateBackup",
"dynamodb:DeleteItem",
"dynamodb:DescribeTable",
"dynamodb:GetItem",
"dynamodb:PutItem",
"dynamodb:Query",
"dynamodb:Scan",
"dynamodb:UpdateItem"
],
"Resource": [
"arn:aws:dynamodb:eu-west-1:885781319000:table/existingredshift1_*", <== table name pattern
"arn:aws:dynamodb:eu-west-1:885781319000:table/datacoral_existingredshift1_*",
"arn:aws:dynamodb:eu-west-1:885781319000:table/Datacoral_Installations"
],
"Effect": "Allow"
},
  • Click on "Review Policy" and save the changes

DynamoDB tables exist in a different AWS account

a. Create a cross-account role in the external AWS account to allow access to the DynamoDB tables present in the external AWS account. Follow the steps below to create the cross-account role

Navigate to CloudFormation console in AWS

In the CloudFormation console, click on "Create Stack".

Then, click on "With new resources"

Create CF Stack from S3 link

In the Create Stack screen, pick "Template is Ready" and add the following S3 URL:

s3://static.datacoral.io/prod/cftemplates/latest/datacoral-dynamodb-cross-account-role.json

Configure input parameters for CF Stack

The following input parameters have to be set:

  • AccountID: Account id for the AWS account with Datacoral installation
  • AllowedIPAddress (optional): vertical bar (|) delimited list of AllowedIPAddress to access dynamodb access (NatGateway EIP)
  • DynamoDBTable: DynamoDB Table name or Table name pattern or *
  • ExternalId: Unique ExternalId string for the account with the DynamoDB tables.

Click Next to progress to the next screen.

Configure Stack Options, Review and Create Stack

Click Next to proceed through this screen, review your selections and create the stack.

Wait for CF Stack to be created successfully

In the CloudFormation AWS console, wait for the CloudFormation stack to successfully create all resources.

Retrieving information from the successful stack

In the CloudFormation console, click on the DynamoDB cross-account role stack that was successfully created, and click on Outputs to retrieve the following information:

  • RoleARN

b. Add policy to allow assume role in the LambdaExec role for that cross-account role

Navigate to IAM console in AWS
  • In the IAM console, click on "Roles" in the left menu. Search for LambdaExec role with the name starting with datacoral-<installation>-*

  • Click on the edit policy button on the LambdaExec role screen
  • Go to the json editor for the role policy
  • Add a new "Action" block at the end of the policy which is as below
{
"Action": [
"sts:AssumeRole"
],
"Resource": [
"<ARN of the cross account role created in the step above>"
],
"Effect": "Allow"
}
  • Click on "Review Policy" and save the changes

Steps to add this slice to your installation via UI

  1. If the DynamoDB tables exist in a different AWS account than the one with Datacoral installation, specify the cross-account role that is created, external id that was specified while creating cross account role and the region where the DynamoDB tables are.
  2. Specify the name of the DynamoDB tables that you want to pull data from by clicking "Add a loadunit" button.
  3. In the loadunit options, please note that you do not need to fill in the DynamoDB Table Name if it is the same as Loadunit name.

  1. Click on Add Slice

Steps to add this slice to your installation using the CLI

The steps to launch your slice are:

  1. Specify the slice config
  2. Add the DynamoDB slice

1. Specify the slice config

To get a template for the DynamoDB slice configuration save the output of the describe --input-parameters command as follows:

datacoral collect describe --slice-type dynamodb \
--input-parameters > dynamodb_parameters_file.json

Necessary input parameters:

  • loadunits - loadunits configuration

Optional input parameters:

Loadunits:

Multiple loadunits can be specified in a single slice to pull different tables data. Below is the sample format of the loadunits section of the input parameters file for the DynamoDB Collect Slice.

loadunits: {
"loadunit1": {
"table": "dynamodb_table1",
"region": "us-west-2",
"executionMode" : "snapshot",
"paginate" : true,
"schedule" : "0 7 * * *"
},
"loadunit2”: {
..
}
}

2. Add the Slice

datacoral collect add --slice-type dynamodb --slice-name <slice-name> --parameters-file <params-file>
  • slice-name Name of your slice. A schema with your slice-name is automatically created in your warehouse
  • params-file File path to your input parameters file. Ex. dynamodb_parameters_file.json

Slice output

Data stored in AWS S3 is partitioned by date and time in the following bucket s3//<customer_installation>.datacoral/<sliceName>

Destination Warehouse Schema - schema name will be same as a slice-name. Tables produced by the slice are:

- schema.loadunitName

Questions? Interested?

If you have questions or feedback, feel free to reach out at hello@datacoral.co or Request a demo