PostgreSQL CDC Slot Metrics
The PostgreSQL CDC connector captures several metrics and values that give you full visibility into every stage in the CDC pipeline. These metrics can be seen in the UI when you navigate to the datacoral_slot_table
loadunit detail page in your PostgreSQL CDC connector. The metrics are described below:
1. Peaking and draining the slot
Metric | Description | Sample Value |
---|---|---|
Peek Duration | Time taken to start reading the WAL log in milliseconds. We want this number to be fairly low. If it increases, that means that the lag in the sync will increase | 1452 |
Read Duration | Time taken to acknowledge reading all the rows | 204 |
2. Reading the WAL logs, parsing them, and uploading to S3
Metric | Description | Sample Value |
---|---|---|
Rows Read | Number of WAL log rows read after filtering only the tables that need to be sync'ed | 12571 |
Bytes Read | Number of WAL log row bytes read | 203877632 |
Rows Parsed | Number of WAL entries parsed | 14577 |
Parse Errors | Number of errors parsing the change logs | 2 |
Parse Duration | Time taken to parse the relevant rows in milliseconds | 211 |
Records Count | Number of records written out | 153 |
Bytes Uploaded | Number of uncompressed bytes of change logs uploaded to S3 | 110773 |
Compressed Bytes Uploaded | Number of bytes written out to S3 | 18447 |
Upload Duration | Time taken to upload the parsed rows to S3 in milliseconds | 3867 |
3. Position in WAL Log for current batch
Metric | Description | Sample Value |
---|---|---|
First Read LSN | First record LSN(Log Sequence Number) read by the connector in this batch | 1ECF/B0A282A0 |
Last Read LSN | Last record LSN(Log Sequence Number) read by the connector in this batch. Can be used to figure out which specific transaction was last read | 1ECF/BCCC3270 |
First Read Txn | First Transaction ID read by the connector in this batch | 2211560682 |
Last Read Txn | Last Transaction ID read by the connector in this batch | 2211593075 |
First Record Timestamp | First commit Timestamp seen in the batch of records read | 2021-03-18 23:45:05.05275+00 |
Last Record Timestamp | Last commit Timestamp seen from the WAL log in the batch of records read - used to perform close of books, if enabled. Also indicates the lag in replication | 2021-03-19 00:00:51.445482+00 |
4. System status
Metric | Description | Sample Value |
---|---|---|
Wal Size | Current WAL size in bytes that is left to be consumed by the connector. It can be different from the actual WAL size on the server. Need to make sure that this number does not get too high since that will increase the disk space consumed | 3682456 |