Marketo develops and sells marketing automation software for account-based marketing and other marketing services and products including SEO and content creation. The Datacoral Marketo connector collects data from a Marketo API and writes it to an AWS S3 bucket and to the destination data warehouse
- Features and capabilities
- Supported load units
- Connector output
- Next Step
- Additional Information
Features & Capabilities
- Backfill: Full historical sync of your entire data (With an exception for Activities: 12 types of Activities have retention of only 90 days - https://nation.marketo.com/t5/Knowledgebase/Marketo-Activities-Data-Retention-Policy-Under-the-Hood/ta-p/251191#)
- Data Extraction Modes: snapshot, incremental with pagination
- Data Load Modes: replace, append and merge
- Tables and Columns selection: Ability to select tables in all loadunits and individual columns in
- Data-layout: changing the data type of your columns
- Customizations: Update the configurations easily using the UI
- Scheduling: Highly flexible scheduling system
Read more about our Features and Capabilities in the next tab.
Click here to learn about the limitations for API calls in Marketo connector
Supported load units
The Marketo slice automatically collects the following loadunits from the Marketo API and makes them available in your warehouse for analysis.
|activities||Supported through bulk export API||Data will be fetched with createdAt filter from API|
|activity types||The REST API endpoint supports snapshot pull through pagination||Not Supported|
|activity types attributes||Supported||Not supported|
|campaigns||supports snapshot pull through pagination using pagingToken||Not supported|
|deleted leads||Not Supported||The REST API endpoint supports incremental pull through pagination using pagingToken|
|leads||Through Bulk API||We fetch leads through an |
|lists||The REST API endpoint supports snapshot pull through pagination using pagingToken||Not Supported|
|list membership||The REST API endpoint supports snapshot pull through pagination using pagingToken||Not Supported|
|program membership||Supported for backfill||Data fetched via the REST API endpoint|
|programs||The REST API endpoint supports snapshot pull through pagination||Supported, we will filter programs list through |
program_membershiploadunit, we obtain the program IDs from the
activitiesand then sync all the leads that belong to a specific program into the loadunit.
Output of this connector is stored in S3 and destination warehouse.
Data stored in AWS S3 is partitioned by date and time
Warehouse: Schema - schema name will be same as a connector-name. Tables produced by the connector are:
Got a question?
Please contact Datacoral's Support Team, we'd be more than happy to answer any of your questions.
Features and Capabilities
Datacoral is a cloud-based data pipeline platform. It provides an infrastructure for ingesting and integrating data from a variety of data sources as the data gets generated by different operational systems. Various transformations can be defined that will combine or aggregate the data from the different sources and publish it to different target systems that are dedicated to Analytics, Machine Learning, or Data Warehousing.
- Data Extract/Ingest
- Data Loading Modes
- Data Extract and Load Combinations
- Data Transformation
- Data Quality Checks
Datacoral’s connectors extracts/ingest data in multiple ways; traditional pooling (extracting data on a predefined schedule) or by enabling webhooks (ingesting data on-the-fly) for data to be pushed to our connectors. For Marketo connector, the extraction configuration features are as follows.
- Selection: set dynamic rules for inclusion or exclusion at schema/table/column level
- Extraction modes:
- Extract full snapshots from source (snapshot mode)
- Extract only updated records in database sources (incremental mode). There are two types of incremental modes, incrementalappend and incrementalupdate.
- Schedule extraction: Set and update extraction frequency of each of the tables. We support historic sync as well.
- Data visibility: Complete visibility of source metadata and the data layout of all tables.
Data Loading Modes
- Replace - “wipe-and-load” operation, DELETE existing records and INSERT new ones
- Merge - updates are merged into the destination table. DELETE operations will result in a SOFT DELETE (records are marked-as-deleted)
- Append - data is appended to the destination warehouse table, so that there is a full audit of all the changes
- Configure warehouse tables based on size into regular or partitioned tables
- Easily add new tables at the destination and update exiting through UI/CLI
- Datacoral also supports movement of data from one warehouse to another
Data Extract and Load Combinations
Supported data extract/load combinations
|Extract mode||Load Mode Applicable||Warehouse table support|
|Append||Regular and Partitioned|
|Incremental||Append||Regular and Partitioned|
- Transformation methods- create declarative transformations through SQL and highly customised transformations through Datacoral’s batch compute feature.
- Materialised Views - storing transformations that can auto detect data changes
- Auto triggers multi level (dependencies) transformation upon data changes
- Data visibility see the table lineage graph (Directed acyclic graph) of data dependencies
- After the data has gone through the relevant transformations, the publishing phase will push the data to a target of your choice.
- Check our complete list of Supported Publishers
Data Quality Checks
- Quality check : The connectors copy data without missing or duplicating any data with the help of built in checks.
- Full timestamp visibility for every step of the data pipeline and for every batch of processed data to ensure freshness audits
- Datacoral detects and notifies schema changes
- Datacoral can detect and collect deleted issues in the connector
UI Installation Overview
- Step 1: Select Marketo connector
- Step 2: Configure connection parameters
- Step 3: Configure source information
- Step 4: Configure loadunits information
- Step 5: Edit data layouts
- Step 6: Configure warehouse
- Step 7: Confirm the configurations
Before adding the connector, please complete the below steps.
- Get API endpoint
- Generate Client ID and Client Secret
1. Get API endpoint
- Click on Admin and Web Services
- Copy the endpoint under REST API as shown below
2. Generate Client ID and Client Secret
- Click on Launch Point unser Admin > Integration
- If Service not available, click on New Service
- Fill the necessary fields of Display Name, Description and choose Service as Custom and API only user as the login user email id from the dropdowns respectively.
- Click create
- Once created, click on View Details and copy the Client ID and Client Secret
Step 1: Select Marketo connector
- From the main menu, click on Add connector
- In the drop down list, find and select Marketo connector
Step 2: Configure connection parameters
- Input the connector name and warehouse and click Next
Please note that the connector name once set cannot be edited later
- Fill in the credentials to connect to your Marketo account, click on Check Connection and Next
Step 3: Configure source information
- Interval : Set the frequency of data extraction
- Sync Historical data : It will load the entire past database as a one-time activity.
- Click on Fetch Source Metadata to see all the load units and click Next
Step 4: Configure load units information
The list of loadunits with extraction mode and schedule is displayed.
Extraction mode is auto detected based on the table size and availability of primary key and timestamp column at the source table. Click on Edit to update edit configuration per loadunit.
- Extraction mode: Can be snapshot, incrementalappend or incrementalupdate
- Interval: The frequency of the extraction mode ranges in discrete interval from 5 minutes
- Timestampcol: Its auto-detected for 'incrementalupdate' extraction mode
Step 5: Edit data layouts
Update data type as needed and click on Next to add the connector.
Step 6: Configure warehouse
For each of the load units on the left, you can decide the load mode
Load Mode: Datacoral supports the below load modes
- Replace : This is a wipe and load operation replacing all the rows of the destination table with the results of the transformation query
- Append: Insert operation where, the result of the transformation query are inserted into the destination table, rows already in the destination table are not updated
- Merge: Upsert operation where the transformation query results in rows that indicate that the destination table rows have to be inserted, updated, or even deleted. This mode allows for efficient incremental updates to destination tables.
Primary Key: This is a mandatory key for Incrementalappend extraction load mode.
When done with the configuration changes, please click on Update and Next on the top right.
Step 7: Confirm the configuration
You will see a pop-up dialog box, click Next to confirm addition of the connector.The connecter will be added once the tables are updated in the warehouse.
You have successfully added the connector once you have landed on the below page.
CLI Installation Overview
- Step 1 : Download an existing configuration
- Step 2: Update the parameters file
- Step 3: Add the connector
Step 1 : Download an existing configuration
Download an existing connector.
datacoral connector download --connector-name <connector-name> --download-directory <download-dir>
connector-name- Name of the existing connector that is getting
You can also copy the command directly from the UI by clicking on the download icon against the existing connector
Step 2: Update the Parameters File
Update the "slice name" in the parameters json file in download directory.
Step 3: Add the connector
datacoral connector add --connector-name <connector-name> --config-directory <download-dir>
connector-nameName of your connector set in Step 2. A schema with the connector name will be automatically created in your warehouse
<download-dir>File path to the input parameters file must be in this directory