Historical Sync UI/CLI Guide
Prerequisites
Setup connector in UI
A partial or full table historical sync and the status of syncs initiated by UI, CLI and auto historic sync is visible in UI.
Concepts
- Partial Backfill : Full historical sync of data based on a specific condition. Based on these conditions there are multiple types of partial backfill
- There are multiple types of partial backfill
- FULL : Complete backfill for a specified loadunit
- COLUMN : Complete backfill for specified columns
- PARTIAL_TIME_WINDOW : Backfill data for a specific time window, this takes
startTimestamp
,endTimestamp
andtimestampCol
as an input.timestampCol
should increase incrementally - PARTIAL_PK_LIST : Backfill of data for records containing a primary key (
pk
)Capability to add multiple primary keys will be added in subsequent release
note
- If a PGCDC slot is dropped and recreated please choose the partial backfill of type
PARTIAL_TIME_WINDOW
- In the above case, there is a chance that deletes will not propagate in the time period the slot was down. To take care of this, apart from getting
updated_at
data we will capture the full-snapshot of the primary keys to identify the deletes
UI Guide
note
- UI supports triggering full historical sync for one loadunit at a time only.
- To initiate partial backfill or historical syncs for multiple loadunits at once please use the CLI
Trigger historical sync
Go to the individual table and click on Trigger Historical Sync button.
Check status
The status of historical sync is available in individual loadunit page and historical sync tab where there is a consolidated status of all historical syncs.
Loadunit page
Navigate to the loadunit and hover on the icon next to the Trigger Historical Sync button
The status will be updated at the bottom of the page, the backfill type will be FULL
Connector level
The connector page will now have two tabs
Runtime - Shows the sync status of all loadunits
Historical Syncs - Shows the status of the automatic and manual historical syncs
- Loadunit Name - Displays the name of the loadunit where the the sync has begun
- Type - Displays the type of partial backfill
- Status - Can be ,
NOT-STARTED
,INPROGRESS
,SUCCESS
orFAILURE
- Timelabel - The timelabel when the schema change is detected
- Started At and Updated At - Marks the time of commencement and the latest status update respectively
Clicking on See Details will show the data flow, the below is an example for the type PARTIAL_TIME_WINDOW
CLI Guide
Instructions on how to trigger historical syncs using Datacoral CLI can we found here.