Setup connector in UI
A partial or full table historical sync and the status of syncs initiated by UI, CLI and auto historic sync is visible in UI.
- Partial Backfill : Full historical sync of data based on a specific condition. Based on these conditions there are multiple types of partial backfill
- There are multiple types of partial backfill
- FULL : Complete backfill for a specified loadunit
- COLUMN : Complete backfill for specified columns
- PARTIAL_TIME_WINDOW : Backfill data for a specific time window, this takes
timestampColas an input.
timestampColshould increase incrementally
- PARTIAL_PK_LIST : Backfill of data for records containing a primary key (
Capability to add multiple primary keys will be added in subsequent release
- If a PGCDC slot is dropped and recreated please choose the partial backfill of type
- In the above case, there is a chance that deletes will not propagate in the time period the slot was down. To take care of this, apart from getting
updated_atdata we will capture the full-snapshot of the primary keys to identify the deletes
- UI supports triggering full historical sync for one loadunit at a time only.
- To initiate partial backfill or historical syncs for multiple loadunits at once please use the CLI
Trigger historical sync
Go to the individual table and click on Trigger Historical Sync button.
The status of historical sync is available in individual loadunit page and historical sync tab where there is a consolidated status of all historical syncs.
Navigate to the loadunit and hover on the icon next to the Trigger Historical Sync button
The status will be updated at the bottom of the page, the backfill type will be FULL
The connector page will now have two tabs
Runtime - Shows the sync status of all loadunits
Historical Syncs - Shows the status of the automatic and manual historical syncs
- Loadunit Name - Displays the name of the loadunit where the the sync has begun
- Type - Displays the type of partial backfill
- Status - Can be ,
- Timelabel - The timelabel when the schema change is detected
- Started At and Updated At - Marks the time of commencement and the latest status update respectively
Clicking on See Details will show the data flow, the below is an example for the type PARTIAL_TIME_WINDOW