Insights Batch API

cURL Request

curl -u [API_Token]: -X POST -d 'dataSource=[DataSource_APIName]&recordType=[RecordType_APIName]&batchDate=2015-12-15T11:12:00Z' "https://nw1.files.insights.zuora.com/api/files/upload"

JSON Response

{
  "signedUrl": "https:\/\/zinsights-zin-prod-fileupload.s3-us-west-2.amazonaws.com\/3\/31\/29\/2015\/12\/15\/111200.000\/42.0.txt.gz?AWSAccessKeyId=Some_Key_123&Expires=1450393826&Signature=SomeUniqueString3w%3D",
  "dataSource": "DataSource_APIName",
  "recordType": "RecordType_APIName",
  "batchDate": "2015-12-15T11:12:00Z"
}

Sample Shell Script

Shell Script Batch API Invocation.  This script can be customized with your request parameters to push your files from a specified directory to the signedUrl created for the record types in your Batch API data source.

#!/bin/bash
# This script depends on the command-line utilities 'curl' and 'jq'
#
# To upload a batch:
#
# 1) Create a new directory for the batch,
#    For example: /data/batches/2015-12-01
# 2) Create a file for each record type in the data source,
#    For example: /data/batches/2015-12-01/accounts.csv
# 3) Call the script with the path to the batch directory,
#    For example: /data/scripts/upload-file.sh /data/batches/2015-12-01
#
# Use the current time as the 'batch date'
BATCH_DATE=$(date +'%Y-%m-%dT%H:%M:%S%z')
#
# Insert your API token
TOKEN=[YOUR_TOKEN]
#
# Insert your data source API name
DATA_SOURCE=[YOUR_DATA_SOURCE]
#
# Insert the API names of the record types in this data source
RECORD_TYPES=[RECORD_TYPE1 RECORD_TYPE2 RECORD_TYPEX)
#
# The directory of the batch to upload
if [ -z "$1" ]; then
    echo "Usage: $0 batch_directory"
    exit 1
fi
BATCH_DIRECTORY=$1
if [ ! -d "$BATCH_DIRECTORY" ]; then
    echo "Failed to upload batch: directory $BATCH_DIRECTORY not found."
    exit 1
fi
for record_type in $RECORD_TYPES; do
    file="$BATCH_DIRECTORY/${record_type}.csv"
    if [ ! -f "$file" ]; then
        echo "Failed to upload batch: file $file not found."
        exit 1
    fi
done
for record_type in $RECORD_TYPES; do
    file="$BATCH_DIRECTORY/${record_type}.csv"
    echo "Uploading $file..."
    response=$(curl -s -u ${TOKEN}: -d "dataSource=${DATA_SOURCE}&recordType=${record_type}&batchDate=${BATCH_DATE}" https://files.dev.insights.zuora.io/api/files/upload)
    if [ $? -ne 0 ]; then
        echo "Failed to retrieve file upload URL:"
        curl -u ${TOKEN}: -d "dataSource=${DATA_SOURCE}&recordType=${record_type}&batchDate=${BATCH_DATE}" https://files.dev.insights.zuora.io/api/files/upload
        exit 1
    fi
    signed_url=$(echo "$response" | jq -r .signedUrl)
    if [ $? -ne 0 ]; then
        echo "Failed to parse API response: $response"
        exit 1
    fi
    curl --upload-file "$file" "$signed_url"
done

Example Python Script

Python Script Batch API Invocation.  This Python script can be customized with your request parameters to push your files from a specified directory to the signedUrl created for the record types in your Batch API data source.

#!/usr/local/bin/python3
# This script depends on the python packages 'requests' and 'pytz'
#
# To upload a batch:
#
# 1) Create a new directory for the batch,
#    For example: /data/batches/2015-12-01
# 2) Create a file for each record type in the data source,
#    For example: /data/batches/2015-12-01/accounts.csv
# 3) Call the script with the path to the batch directory,
#    For example: /data/scripts/upload-file.py /data/batches/2015-12-01
#
from datetime import datetime
import pytz
import requests
import sys
import os
#
# Use the current time as the 'batch date'
batch_date=datetime.now(pytz.utc).strftime('%Y-%m-%dT%H:%M:%S%z')
#
# Your API token
token="QJXNZxMiwORqjyKArmYexHLhgWuBekjJ"
#
# The data source API name
data_source="fileupload"
#
# The API names of the record types in this data source
record_types=["account", "accountsubscriptions"]
#
# The directory of the batch to upload
if len(sys.argv) != 2:
    print("Usage: %s batch_directory" % sys.argv[0])
    sys.exit(1)
batch_directory=sys.argv[1]
if not os.path.isdir(batch_directory):
    print("Failed to upload batch: directory %s not found or is not a directory" % batch_directory)
    sys.exit(1)
for record_type in record_types:
    file = os.path.join(batch_directory, "%s.csv" % record_type)
    if not os.path.exists(file):
        print("Failed to upload batch: file $file not found.")
        sys.exit(1)
for record_type in record_types:
    file = os.path.join(batch_directory, "%s.csv" % record_type)
    print("Uploading %s..." % file)
    params = {
        'dataSource': data_source,
        'recordType': record_type,
        'batchDate': batch_date
    }
    
    response = requests.post('https://files.dev.insights.zuora.io/api/files/upload', auth=(token, ''), data=params)
    if response.status_code != requests.codes.ok:
        print("Failed to retrieve file upload URL for dataSource=%s recordType=%s batchDate=%s: %s" % (data_source, record_type, batch_date, response.text))
        sys.exit(1)
    signed_url=response.json()['signedUrl']
    with open(file, 'rb') as f:
        response = requests.put(signed_url, data=f)
        if response.status_code != requests.codes.ok:
            print("Failed to upload file for dataSource=%s recordType=%s batchDate=%s: %s" % (data_source, record_type, batch_date, response.text))
            sys.exit(1)