Datacoral Version 1.43.0

Connectors :

  • S3 Ingest connector:

    • S3 path prefix now supports regex inputs
    • Now supports adding multiple s3 files in a single page
    • Bootstrap fix added enabling the connected to handle over 1000 files without failure

  • Google Adwords:

    • Added support for core loadunits - ads, ad groups, campaign, accounts
    • Added loadunits for reports: Keyword performance, Search Query Performance, Ad Performance and Campaign Performance

  • PostgreSQL CDC:

    • Handles columns with keywords as part of backfill
    • Supports column level backfill with tables with no primary keys
    • Supports column level backfill with tables with no data at source
    • Backfill Notifications have additional details and metrics per stage of backfill. The notification structure will be as follows
      {
      "type": "<warehouse-name>",
      "warehouseName": "<warehouse-name>",
      "timelabel": "<timelabel>",
      "sliceName": "<slice-name>",
      "schema": "<schema-name>",
      "eventType": "data",
      "eventSubtype": "backfill",
      "status": "[SUCCESS/FAILURE]",
      "reason": null,
      "durationInMillis": "<duration-in-millis>",
      "startTime": "2020-11-13 00:47:00:000 +00:00",
      "endTime": "2020-11-13 00:55:38:905 +00:00",
      "updateTime": "2020-11-13 00:55:38:905 +00:00",
      "executionContext": {
      "workflowInstanceId": "<workflow-instance-id>",
      "recordsCount": "<number-of-records-backfilled>",
      "operation": "[TABLE_ADD/COLUMNS_ADD]",
      "operationSource": "[user_initiated/automatic]",
      "columns": [
      "new column"
      ],
      "stages": [
      {
      "operation": "S3",
      "status": "[SUCCESS/FAILURE/TIMEOUT]"
      },
      {
      "operation": "LOAD",
      "status": "[NOT_STARTED/SUCCESS/FAILURE]"
      },
      {
      "operation": "MERGE",
      "status": "[NOT_STARTED/SUCCESS/FAILURE]"
      }
      ],
      "metrics": {
      "recordsCount": "<number-of-records-backfilled>",
      "runtimeDurationInMillis": "<backfill-runtime-duration-in-millis>",
      "delayBeforeBackfillStart": "<delay-before-backfill-start-in-millis>",
      "delayBeforeLoadStart": "<delay-between-source-pull-and-load-start-in-millis>",
      "delayBeforeMergeStart": "<delay-between-load-and-merging-new-changes-in-millis>",
      "S3": {
      "durationInMillis": "<time-fetching-data-from-source-in-millis>"
      },
      "LOAD": {
      "durationInMillis": "<time-to-load-source-date-to-warehouse-in-millis>"
      },
      "MERGE": {
      "durationInMillis": "<time-to-merge-backfill-data-with-new-changes-in-millis>"
      }
      }
      }
      }

CLI :

  • Added commands in CLI to list the connector types and to describe the connector configuration.
  • Dq-pipeline will create dags with full path for easier portability
  • Organize matview-download command supports additional parameter '--include-dag' which will also create a dag file

System Improvements :

  • Fix Glue table creation for parquet format to have serialization format
  • Bugfixes:
    • Handling connector deploy failure if any slice that contains slice as a keyword
    • Handling error case in the OAuth based Connector configure flow