How to Add Ingest Connectors

Please be aware that ingest is a recent change to the Datacoral CLI commands for these connectors. However, the original collect command will continue to be supported for a transition period.

Add a Connector

Step 1. Download the Parameters Template

A template of the parameters for a connector can be saved to a file by running the following command.

datacoral collect describe --slice-type <slice-type> --input-parameters > parameter-file.json
  • slice-type: The type of ingest connector whose parameters are being saved
  • parameter-file: The path to the file in which the parameters will be saved

Example

This example will use a MySQL slice to walk through the process.

datacoral collect describe --slice-type mysql --input-paramters > mysql_params.json

Step 2. Populate the Configuration

Each connector will have a different set of parameters you must supply in order to add the slice. The one that is common accross all connectors is the schedule.

Example

The template for the MySQL slice has been filled below with fake values.

{
"database": "my_mysql_database",
"host": "rds_...aws.com",
"port": 3306,
"username": "martha",
"password": "marthas_password_123",
"schedule": "0 0 * * *",
"tableWhiteList": ["table1", "table2"],
"tableBlackList": ["table3", "table4"],
}

Please note that the schedule parameter is given in the cron format. Further, you are limited to a schedule granularity of at least 5 minutes. If you would like a more frequent schedule, please contact support@datacoral.co.

Step 3. Add the Connector

The new ingest connector will be added to your installation by running the following command.

datacoral collect add --slice-type <slice-type> --slice-name <slice-name> \
--parameters-file <parameter-file>.json
  • slice-name: Name given to uniquely identify this connector
  • parameter-file: The path to the file in which the parameters were specified during step 2

Example

Using the paramters saved in the example above,

datacoral collect add --slice-type mysql --slice-name mysql_test \
--parameters-file mysql_params.json

Update a Connector

After a connector has been created in an installation, there are a few simple steps to later perform updates.

Step 1. Fetch the Connector Configuration

The existing parameters for a connector can be saved to a file by running the following command.

datacoral describe --slice-name slice-name > parameter-file.json
  • slice-name: Name given that uniquely identifies the connector
  • parameter-file: The path to the file in which the parameters will be saved

Example

Again, the MySQL slice from earlier examples will be used.

datacoral describe --slice-name mysql_test > mysql_test.json

Step 2. Modify the Configuration

Similar to adding a connector, the parameter values can be edited and manipulated as needed. Bear in mind that parameters in the original saved file are those required for the connector to run.

Example

Below are modifications to the configuration of the original mysql_test connector. Notice the tableWhitelist has been switched to a SQL LIKE-compatible pattern.

{
"sliceName": "mysql",
"database": "my_mysql_database",
"host": "rds_...aws.com",
"port": 3306,
"username": "martha",
"secretKey": "marthas_password_123",
"schedule": "0 0 * * *",
"tableWhiteList": ["tablepatternschema.%"],
"tableBlackList": ["table3", "table4"],
}

Step 3. Update the Connector

The ingest connector configuration will be updated in your installation by running the following command.

datacoral collect update --slice-name <slice-name> --configuration-only \
--parameters-file <parameter-file>.json
  • slice-name: Name given that uniquely identifies the connector
  • parameter-file: The path to the file in which the parameters were saved during step 2

Example

Updating the MySQL example slice with the parameters from above,

datacoral collect update --slice-name mysql_test --configuration-only \
--parameters-file mysql_test.json

Remove a Connector

An ingest connector can be removed from an installation by running the following command.

datacoral remove --slice-name <slice-name>
  • slice-name: Name given that uniquely identifies the connector

Example

To remove the example slice used through this guide follows as below.

datacoral remove --slice-name mysql_test