Datacoral Version 1.42.0

Onboarding:

Datacoral now supports free trial on sign-up. Users can choose one of the two ways to sign-up

  • Hosted by Datacoral:
    • A pre-created installation in the Datacoral cloud is assigned for the trial
    • Supports existing redshift or snowflake data warehouse
    • No infrastructure cloud costs during trial period
  • Hosted by User:
    • Full new installation will be deployed in user’s AWS cloud, the data will remain in the user’s cloud throughout
    • Supports customizations to fit the user's networking and security needs.
    • Supports existing redshift/snowflake/athena warehouses or creation of a new one as part of the datacoral installation
    • Supports testing production workloads with full security

With all installation, Athena and batchcompute will be set up as a preconfigured warehouse. To learn more click here

Connectors:

  • Marketo - Added support for loadunit List Membership
  • Mailgun - Added a new loadunit domain_stats
  • Facebook - Now supports API version 8
  • PG-CDC -
    • Table/column backfill supports SQS notifications. The notification structure will be as follows.
      {
      "key": "<warehousen-name>|<slice-name>|<loadunit-name>+backfill",
      "timelabel": "<timelabel>",
      "sliceName": "<slice-name>",
      "status": "[SUCCESS/FAILURE]",
      "reason": null,
      "durationInMillis": "<duration-in-millis>",
      "startTime": "2020-11-13 00:47:00:000 +00:00",
      "endTime": "2020-11-13 00:55:38:905 +00:00",
      "updateTime": "2020-11-13 00:55:38:905 +00:00",
      "executionContext": {
      "workflowInstanceId": "<workflow-instance-id>"}
      }
    • No additional PG backfill connector needed as backfills are enabled from the PG-CDC slide
    • Backfills can also we triggered manually
    • UI support to see the backfill status at the connector and loadunit level
    • Slot table will have a new component in the UI to indicate connector metrics

Transformations

  • ROUNDEC is added by default to all the collect slices which have redshift as target warehouse
  • Improved performance of the upsert query for Redshift loads

CLI

  • TargetWarehouses will be a required field for the connector add/update
  • Data quality check commands enhanced
  • Connectors in new ingest flow can be added/updated in CLI using the new connector commands

System improvements:

  • Connector add/update flow now supports user-defined loadunits
  • AWS Cross account role is now divided into three roles for monitoring, administration and software upgrade