CloudWatch Collect Slice

Overview

Amazon CloudWatch is a monitoring service for AWS cloud resources and the applications you run on AWS. You can use Amazon CloudWatch to collect and track metrics, collect and monitor log files, set alarms, and automatically react to changes in your AWS resources.

Datacoral’s CloudWatch slice is a collect (datasource) slice that collects data of CloudWatch metric statistics. With the slice, customers can conveniently separate out the different types of metrics and granularities into multiple tables in redshift to make them easy to analyze.

Steps to add this slice to your installation

The steps to launch your slice are:

  1. Generate AWS API keys
  2. Specify the slice config
  3. Add the CloudWatch slice

1. Generate AWS API keys

Setup requirements

Before getting started please make sure to have the following information:

  • Access to an active AWS account

Setup instructions

  1. Generate a new AWS keys a. Go to URL: https://console.aws.amazon.com/iam/home?region=us-east-1#/users b. Select or create new user c. On user details, open Security Credentials d. Click Create access key button, and copy credentials

2. Specify the slice config

To get a template for the CloudWatch slice configuration save the output of the describe --input-parameters command as follows:

datacoral collect describe --slice-type cloudwatch \
--input-parameters > cloudwatch_parameters_file.json

Necessary input parameters:

  • accessKeyId - your AWS Access Key Id
  • secretAccessKey - your AWS Secret Key
  • region - your AWS region

Optional input parameters:

Loadunits:

Multiple loadunits can be specified in a single slice to pull different metrics and statistics into separate tables. Below is the sample format of the loadunits section of the input parameters file for the CloudWatch Metrics Collect Slice.

loadunits: {
"loadunit1": {
"namespace": "XXXX",
"dimensions": ["dimension1", "dimension2"]
"dimension_filter": [
{
"XXXX1": "YYYY",
"XXXX2": "ZZZZ"
},
{
"XXXX1": "YYYY1"
"XXXX2": "ZZZZ1"
}
],
"metrics": [
{
"name":"M1",
"statistics": [ "Minimum", "Maximum", "Average", "SampleCount", "Sum", "pXX.XX", "pYY.YY" ],
"unit": "UU"
},
{
"name":"M2",
"statistics": [ "Minimum", "Maximum", "Average", "SampleCount", "Sum", "pXX.XX", "pYY.YY" ],
"unit": "VV"
}
],
"period": YYYY
},
"loadunit2”: {
..
}
}

Note :

  • Start time and End time needed to get statistics will be based on schedule of the load unit
  • dimension_filter is a non-mandatory list of dimension_name, dimension values combination, if provided, then the data extracted will be only for the specified dimension values for that dimension. When not provided, data will be extracted for all dimension values, example:
    dimension_filter: [
    {
    FunctionName: 'datacoral-frontprod-frontus'
    },
    {
    FunctionName: 'datacoral-frontprod-frontuswest'
    },
    {
    FunctionName: 'datacoral-frontprod-frontus3'
    }
    ]
  • Period is in seconds - 5, 10, 30, 60, or any multiple of 60, default is 3600

Restrictions:

you can use up to 10 percentile statistics.

PREREQUISITES

n\a

INPUT PARAMETERS

To get a the starting template save the output of the describe --input-parameters command as follows:

datacoral collect describe --slice-type cloudwatch \
--input-parameters > cloudwatch_parameters_file.json

Necessary input parameters:

  • namespace
  • dimensions

for metrics item:

  • name
  • statistics

Example template:

{
"loadunits": {
"lambda_usage": {
"namespace": "AWS/Lambda",
"dimensions": ["FunctionName"],
"metrics": [
{
"name": "Duration",
"statistics": ["Sum"],
"unit": "Milliseconds"
}
],
"period": 3600
}
}
}

3. Add the Slice

datacoral collect add --slice-type cloudwatch --slice-name <slice-name> --parameters-file <params-file>
  • slice-name Name of your slice. A schema with your slice-name is automatically created in your warehouse
  • params-file File path to your input parameters file. Ex. cloudwatch_parameters_file.json

Supported load units

  • cloudwatch

Slice output

Output of this slice is stored in S3 and Redshift.

AWS S3 Data stored in AWS S3 is partitioned by date and time in the following bucket s3://datacoral-data-bucket/<sliceName>

AWS Redshift: Schema - schema name will be same as a slice-name. Tables produced by the slice are:

- schema.<sliceName>

Questions? Interested?

If you have questions or feedback, feel free to reach out at hello@datacoral.co or Request a demo