Salesforce Connector Overview


Salesforce is a cloud-based customer relationship management (CRM) platform for supercharging every part of your company that interacts with customers — including marketing, sales, commerce, service, and building apps.

The Salesforce collect slice supports continuous ingest of data from a Salesforce account into a data warehouse, such as Redshift. You can ingest any standard or custom object stored in Salesforce.

Features & Capabilities

  • Backfill: Full historical sync of your entire data
  • Data Extraction Modes: snapshot, incremental with pagination
  • Data Load Modes: replace, append and merge
  • Tables and Columns selection: The ability to select individual schemas, tables and columns for replication in the Datacoral's UI.
  • Data-layout: changing the data type of your columns
  • Customizations: Update the configurations easily using the UI
  • Scheduling: Highly flexible scheduling system

Connector output

Output of this connector is stored in S3 and chosen warehouse(s).

AWS S3 Data stored in AWS S3 is partitioned by date and time in the following bucket s3//:datacoral_installation_name.datacoral/<connector-name>

AWS Redshift: Schema - Schema name is same as the name of the connector.

  • Salesforce data extract API limits the number of columns we can extract, especially with long field names. For objects with hundreds of custom fields, it is advisable to specify the exact fields that should be replicated by Datacoral
  • Formula fields replication is not advisable. Formula field values are not recalculated when the underlying attributes change in Salesforce. This could lead to stale data residing in your warehouse. You could replicate the calculation logic within your warehouse to utilize calculated fields.
  • Please note the Request Size limits for Salesforce, which limits the maximum length for the combined URI and headers to 16,384 bytes. The implication of this is load failure for loadunits with large number of columns such as accounts. One work around is to use the columnList extraction parameter to limit the number of columns in a table.
  • Some standard objects cannot be fetched with simple SOQL statements without 'where' clauses. For such objects, it is a good idea to provide extraction queries. Some known objects that need filtration criteria specified are: DataStatistics, FlexQueueItem, FieldDefinition, EntityParticle
  • FeedItem and FeedAttachment objects may run into "exceeded 100000 distinct ids" error depending on the number of rows present at source. This help article may provide some pointers to effectively replicate these objects.
  • DataType object may run into "EXCEEDED_ID_LIMIT: DataType does not support queryMore()". The workaround is to use extractionQuery with LIMIT to restrict the results to a single batch.

Next Steps

Additional Information

Got a question?

Please contact Datacoral's Support Team, we'd be more than happy to answer any of your questions.