Historical Sync UI/CLI Guide

Prerequisites

Setup connector in UI

A partial or full table historical sync and the status of syncs initiated by UI, CLI and auto historic sync is visible in UI.

Concepts

  • Partial Backfill : Full historical sync of data based on a specific condition. Based on these conditions there are multiple types of partial backfill
  • There are multiple types of partial backfill
    • FULL : Complete backfill for a specified loadunit
    • COLUMN : Complete backfill for specified columns
    • PARTIAL_TIME_WINDOW : Backfill data for a specific time window, this takes startTimestamp,endTimestamp and timestampCol as an input.

      timestampCol should increase incrementally

    • PARTIAL_PK_LIST : Backfill of data for records containing a primary key (pk)

      Capability to add multiple primary keys will be added in subsequent release

note
  • If a PGCDC slot is dropped and recreated please choose the partial backfill of type PARTIAL_TIME_WINDOW
  • In the above case, there is a chance that deletes will not propagate in the time period the slot was down. To take care of this, apart from getting updated_at data we will capture the full-snapshot of the primary keys to identify the deletes

UI Guide

note
  • UI supports triggering full historical sync for one loadunit at a time only.
  • To initiate partial backfill or historical syncs for multiple loadunits at once please use the CLI

Trigger historical sync

Go to the individual table and click on Trigger Historical Sync button.

Check status

The status of historical sync is available in individual loadunit page and historical sync tab where there is a consolidated status of all historical syncs.

Loadunit page

Navigate to the loadunit and hover on the icon next to the Trigger Historical Sync button

The status will be updated at the bottom of the page, the backfill type will be FULL

Connector level

The connector page will now have two tabs

  • Runtime - Shows the sync status of all loadunits

  • Historical Syncs - Shows the status of the automatic and manual historical syncs

    • Loadunit Name - Displays the name of the loadunit where the the sync has begun
    • Type - Displays the type of partial backfill
    • Status - Can be ,NOT-STARTED,INPROGRESS,SUCCESS or FAILURE
    • Timelabel - The timelabel when the schema change is detected
    • Started At and Updated At - Marks the time of commencement and the latest status update respectively

    Clicking on See Details will show the data flow, the below is an example for the type PARTIAL_TIME_WINDOW

CLI Guide

Instructions on how to trigger historical syncs using Datacoral CLI can we found here.