Historical Sync UI/CLI Guide

Prerequisites

Setup connector in UI

A full table historical sync and the status of syncs initiated by UI, CLI and auto historic sync is visible in UI.

UI Guide

Trigger historical sync

Go to the individual table and click on Trigger Historical Sync button. To initiate syncs for multiple loadunits at once please use the CLI

Check status

The status of historical sync is available in individual loadunit page and historical sync tab where there is a consolidated status of all historical syncs.

Loadunit page

Navigate to the loadunit and hover on the icon next to the Trigger Historical Sync button

The status will be updated at the bottom of the page

Connector level

The connector page will now have two tabs

  • Runtime - Shows the sync status of all loadunits

  • Historical Syncs - Shows the status of the automatic and manual historical syncs

    • Loadunit Name - Displays the name of the loadunit where the the sync has begun
    • Operations - Operations can be table add and column add
      note

      We support automatic column backfills when changes (column add/rename) are detected at source. Column rename operation is executed by deleting the column and adding it again with the new name

    • Columns - Displays the name of the column within the loadunit where the schema change is detected
    • Status - Can be ,NOT-STARTED,INPROGRESS,SUCCESS or FAILURE
    • Timelabel - The Timelabel when the schema change is detected
    • Started At and Updated At - Marks the time of commencement and the latest status update respectively

    Clicking on See Details will show the data flow

CLI Guide

Using Datacoral CLI historical syncs can be triggered. Here is the list of commands that can be used for it.

Historical sync command

$ datacoral collect historical-sync -h
Usage: historical-sync [options]
carry out various operations around historical-sync
Options:
--slice-name <slice-name> name of the deployed slice
--loadunits [loadunits] comma seperated list of one or more loadunits. Default is all loadunits of the slice
--status get the historical sync status of loadunit(s)
--detailed get detailed historical sync status of loadunit(s). Applicable only with '--status'
-h, --help output usage information

Trigger historical syncs for the connector

Historical syncs for all the tables in a connector

$ datacoral collect historical-sync --slice-name <slice-name>

Example

$ datacoral collect historical-sync --slice-name cdc_aaa01

Output

2020-11-06T17:43:05-08:00 - info: 'historical-sync' request for loadunit 'table_110' has been submitted
2020-11-06T17:43:08-08:00 - info: 'historical-sync' request for loadunit 'table_119' has been submitted
...
2020-11-06T17:43:16-08:00 - info: 'historical-sync' request for loadunit 'table_112' has been submitted
2020-11-06T17:43:18-08:00 - info: 'historical-sync' request for loadunit 'table_111' has been submitted
2020-11-06T17:43:20-08:00 - info: 'historical-sync' request for loadunit 'table_114' has been submitted
2020-11-06T17:43:21-08:00 - info: 'historical-sync' request for loadunit 'table_113' has been submitted

Trigger historical syncs for loadunits

Multiple loadunits can be added as a comma separated string as given in the example

$ datacoral collect historical-sync --slice-name <slice-name> --loadunits <loadunits>

Example

$ datacoral collect historical-sync --slice-name cdc_aaa01 --loadunits 'table_11, table_114, table_119'

Output

2020-11-06T17:53:26-08:00 - info: 'historical-sync' request for loadunit 'table_11' has been submitted
2020-11-06T17:53:28-08:00 - info: 'historical-sync' request for loadunit 'table_114' has been submitted
2020-11-06T17:53:31-08:00 - info: 'historical-sync' request for loadunit 'table_119' has been submitted

Get status for all loadunits in a connector

$ datacoral collect historical-sync --slice-name <slice-name> --status

Example

$ datacoral collect historical-sync --slice-name cdc_aaa01 --status

Output

Obtained historical sync status for 'table_11'
...
Obtained historical sync status for 'table_113'
Writing out status object
[
{
"loadunit": "table_11",
"status": "SUCCESS"
},
...
{
"loadunit": "table_113",
"status": "INPROGRESS"
}
]
Done writing status object

Get status for specific loadunit

$ datacoral collect historical-sync --slice-name <slice-name> --loadunits <loadunits> --status

Example

$ datacoral collect historical-sync --slice-name cdc_aaa01 --loadunits 'table_11, table_114, table_119'
--status

Output

Obtained historical sync status for 'table_11'
Obtained historical sync status for 'table_114'
Obtained historical sync status for 'table_119'
Writing out status object
[
{
"loadunit": "table_11",
"status": "SUCCESS"
},
{
"loadunit": "table_114",
"status": "INPROGRESS"
},
{
"loadunit": "table_119",
"status": "SUCCESS"
}
]
Done writing status object

Get detailed historical sync status of all loadunits

$ datacoral collect historical-sync --slice-name <slice-name> --status --detailed

Example

$ datacoral collect historical-sync --slice-name cdc_aaa01 --status --detailed

Output

Obtained historical sync status for 'table_11'
Obtained historical sync status for 'table_110'
…
[
{
"loadunit": "table_11",
"status": "SUCCESS",
"targetWarehouses": [
"redshift"
],
"timelabels": [
{
"startTime": "2020-11-07 01:43:02:740 +00:00",
"version": null,
"timelabel": "20201107011500",
"executionContext": {},
"endTime": "2020-11-07 01:43 +00:00",
"sliceName": "cdc_aaa01",
"durationInMillis": 0,
"updateTime": "2020-11-07 01:43 +00:00",
"status": "SUCCESS",
"key": "backfillmanager|cdc_aaa01|table_11",
"reason": null,
"operation": "BACKFILLMANAGER"
},
...
]
},
...
{
"loadunit": "table_114",
"status": "INPROGRESS",
"targetWarehouses": [
"redshift"
],
"timelabels": [
{
…
}
]
}
]
Done writing status object

Get detailed sync status of specific loadunits

$ datacoral collect historical-sync --slice-name <slice-name> --loadunits <loadunits> --status --detailed

Example

$ datacoral collect historical-sync --slice-name cdc_aaa01 --loadunits 'table_11, table_114, table_119' --status --detailed

Output

Obtained historical sync status for 'table_11'
Obtained historical sync status for 'table_114'
Writing out status object
[
{
"loadunit": "table_11",
"status": "SUCCESS",
"targetWarehouses": [
"redshift"
],
"timelabels": [
{
"startTime": "2020-11-07 01:43:02:740 +00:00",
"version": null,
"timelabel": "20201107011500",
"executionContext": {},
"endTime": "2020-11-07 01:43 +00:00",
"sliceName": "cdc_aaa01",
"durationInMillis": 0,
"updateTime": "2020-11-07 01:43 +00:00",
"status": "SUCCESS",
"key": "backfillmanager|cdc_aaa01|table_11",
"reason": null,
"operation": "BACKFILLMANAGER"
},
...
]
},
...
{
"loadunit": "table_114",
"status": "INPROGRESS",
"targetWarehouses": [
"redshift"
],
"timelabels": [
{
…
}
]
}
]
Done writing status object