PostgreSQL CDC Slot Metrics

The PostgreSQL CDC connector captures several metrics and values that give you full visibility into every stage in the CDC pipeline. These metrics can be seen in the UI when you navigate to the datacoral_slot_table loadunit detail page in your PostgreSQL CDC connector. The metrics are described below:

1. Peaking and draining the slot

MetricDescriptionSample Value
Peek DurationTime taken to start reading the WAL log in milliseconds. We want this number to be fairly low. If it increases, that means that the lag in the sync will increase1452
Read DurationTime taken to acknowledge reading all the rows204

2. Reading the WAL logs, parsing them, and uploading to S3

MetricDescriptionSample Value
Rows ReadNumber of WAL log rows read after filtering only the tables that need to be sync'ed12571
Bytes ReadNumber of WAL log row bytes read203877632
Rows ParsedNumber of WAL entries parsed14577
Parse ErrorsNumber of errors parsing the change logs2
Parse DurationTime taken to parse the relevant rows in milliseconds211
Records CountNumber of records written out153
Bytes UploadedNumber of uncompressed bytes of change logs uploaded to S3110773
Compressed Bytes UploadedNumber of bytes written out to S318447
Upload DurationTime taken to upload the parsed rows to S3 in milliseconds3867

3. Position in WAL Log for current batch

MetricDescriptionSample Value
First Read LSNFirst record LSN(Log Sequence Number) read by the connector in this batch1ECF/B0A282A0
Last Read LSNLast record LSN(Log Sequence Number) read by the connector in this batch. Can be used to figure out which specific transaction was last read1ECF/BCCC3270
First Read TxnFirst Transaction ID read by the connector in this batch2211560682
Last Read TxnLast Transaction ID read by the connector in this batch2211593075
First Record TimestampFirst commit Timestamp seen in the batch of records read2021-03-18 23:45:05.05275+00
Last Record TimestampLast commit Timestamp seen from the WAL log in the batch of records read - used to perform close of books, if enabled. Also indicates the lag in replication2021-03-19 00:00:51.445482+00

4. System status

MetricDescriptionSample Value
Wal SizeCurrent WAL size in bytes that is left to be consumed by the connector. It can be different from the actual WAL size on the server. Need to make sure that this number does not get too high since that will increase the disk space consumed3682456