Receive Datacoral Events through SQS

Internally, Datacoral services communicate with each other by sending or receiving data or metadata events. If you wish to get access to these events, you can specify a Simple Queue Service (SQS) endpoint which Datacoral can send data event messages to.

Step 1: Setup an SQS Queue

Step 1.1: Create the SQS Queue

If you don't already have an SQS queue set up, create one. You can follow this tutorial to create an SQS queue (Datacoral supports both standard and FIFO queues).

Step 1.2: (Optional) Update SQS Queue Permissions

If the SQS queue was created in a separate AWS account, the permissions of the SQS queue will have to be updated to allow actions from a separate AWS account to occur on the SQS queue. Go to the SQS AWS console, select your SQS queue, click on the Permissions tab at the bottom and add a permission that allows the AWS account which houses the Datacoral installation to perform actions on the SQS queue. See image below for an example:

drawing

Alternatively, you can also run the following AWS CLI command to set the same permissions on your SQS queue:

aws sqs add-permission --queue-url <queue-url> --label datacoralSQSPermission --aws-account-ids <aws-account-id-containing-Datacoral-installation> --actions SendMessage

Our notification manager lambda will have very limited permissions on the SQS queue, as can be seen in the CloudFormation template for it here.

Step 2: Go To Installation Settings

In the top-right corner of the Datacoral webapp, click on Installation Settings.

drawing

Step 3: Click on Add New Endpoint

drawing

Step 4: Configure the SQS Endpoint

Step 4.1: Add the webhook endpoint

Select SQS notification endpoint, give it a name and provide the URL for the SQS queue you created earlier.

Now, add the details of your SQS queue in the form.

drawing

Step 4.2: (Optional) Specify filters

By default datacoral sends notifications for all metadata events and failure data events. If you wish to subscribe to a subset of events, you can do so by specifying the filters.

Below are the configurable filters:

  1. Event Types - data/metadata
  2. Schema Whitelist - Notifications for only the provided list of warehouse schemas will be sent.
  3. Schema Blacklist - Events for the list of warehouse schemas will not be notified.

The schema whitelist and blacklist names could also be regex expressions. Also, they work together in conjunction. For example, if we wish to receive notifications for all but one schema starting with datacoral, we can specify [ "datacoral*" ] as the Schema Whitelist and [ "datacoral_blacklist" ] as the Schema Blacklist.