Datacoral Version 1.10.0

  • Datacoral policy updates
    • Cross account role changes
      • Removed resource tags for ec2:RunInstances to be able to bring up new elastic beanstalk instances for jupyterhub/metabase slices
      • Added permissions to enable cross region snapshot for redshift
      • Added permissions to request acm certificate
      • Added permissions to enable elastic resize for redshift clusters
      • Added permissions to create/modify Network ACLs in order to lock down access to the private subnets
    • datacoral-roles changes
      • Changes to ECSInstanceProfile in datacoral-batch-roles to make it installation specific (This is to support multiple installations in different regions)
      • Changes to lambda role to include iam PassRole permissions to ensure successful creation of firehose streams as part of events slice.
    • Console user role changes
      • Added permissions to support MFA
  • Webapp
    • Added a link to helpdesk to the sidebar menu
    • Added smarter filters for materialized view and loadunit grid view
    • Added tabs for different organize slices on the organize page
    • Added an option to download MV definition as a DPL file from Materialized View summary page
  • CLI
    • prepare-aws-account command supports --update option to enable updates to datacoral related roles from v1.11.0 (instead of running cloudformation stack update commands).
    • Fixes in describe command to list out the correct loader config for the loadunits in the collect slices
    • Added describe command for GDPR organize slice
    • matview-describe provides options to list materialized view dependencies, dependents, dependents-graph
    • matview-download provides option to overwrite existing DPL files
    • Added matview-update-local to stage changes to a batch of materialized views in a local directory
    • Added matview-update-stage (replaces matview-update command) to consistently update a batch of materialized views in production through update sets
    • Added matview-list-versions and matview-list-update-sets to list the versions of materialized views and materialized view update sets
  • Collect Slices
    • MongoDB - New Slice to extract changes from MongoDB
    • Zendesk - support for new loadunit ticket_audits
    • Pglogicaldecoding
      • Recovery slice deployment fix (ie recovery slice should deploy on same subnet).
      • Handling deletes & update within same transactions.
      • Handling tables with columns name containing reserved words.
      • Parser fixes to handle DELETE from <table> operations
  • Organize Slices
    • Fix for s3 collect slice to be able to work with Athena in the case of csv file formats
    • Fixed automated recovery of failed loads in managed redshift slice for certain error types
    • Materialized view update now supports new column additions to existing materialized views
    • Multiple materialized views can be updated using update sets
  • System Improvements
    • Move to using ACM certificates so that they get renewed automatically!
    • Handle duplicate messages coming out of kinesis during collect slice pagination