Setup Zuora Connector for AWS S3
Configure AWS S3 destination within the Zuora Connector
To set up your AWS S3 connector, you must first configure your AWS S3 destination within the Zuora Connector. Follow the steps below to set up AWS IAM credentials, manage permissions, configure bucket access, and make other necessary configurations to ensure a seamless data transfer process from Zuora to AWS S3.
Prerequisites
By default, S3 authentication uses role-based access. You will need the trust policy prepopulated with the data-syncing service's identifier to grant access. It should look similar to the following JSON object with a proper service account identifier:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::123456789012:sample_role" }, "Action": "sts:AssumeRole", } ] }
Step 1: Set up destination S3 bucket
Create Bucket
- Navigate to the S3 service page.
- Click Create bucket.
- Enter a Bucket name and modify any of the default settings as desired.
Object Ownership can be set to "ACLs disabled" and Block Public Access settings for this bucket can be set to "Block all public access" as recommended by AWS. Make note of the Bucket name and AWS Region.
- Click Create bucket.
Step 2: Create policy and IAM role
Create policy
- Navigate to the IAM service page.
- Navigate to the Policies navigation tab, and click Create policy.
- Click the JSON tab, and paste the following policy, being sure to replace
BUCKET_NAME
with the name of the bucket chosen in Step 1.-
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["s3:PutObject", "s3:DeleteObject"], "Resource": "arn:aws:s3:::BUCKET_NAME/*" } ] }
-
- Click Next: Tags > Next: Review.
- Name the policy, add a description, and click Create policy.
Understanding the s3:DeleteObject requirement
By default, a connection test is performed against the destination during initial configuration and s3:DeleteObject
is required to clean up test artifacts. Once the test has been performed successfully and the destination added, this action can be safely removed, as S3 destinations are append-only by default
Create role
- Navigate to the IAM service page.
- Navigate to the Roles navigation tab, and click Create role.
- Select Custom trust policy and paste the provided trust policy to allow AssumeRole access to the new role > click Next.
- Add the permissions policy created above, and click Next.
- Enter a Role name, for example,
transfer-role,
and click Create role. - Once successfully created, search for the created role in the Roles list, click the role name, and make a note of the ARN value.
Alternative authentication method: AWS User with HMAC Access Key ID & Secret Access Key
Role-based authentication is the preferred authentication mode for S3 based on AWS recommendations. However, HMAC Access Key ID and Secret Access Key is an alternative authentication method that can be used if preferred.
- Navigate to the IAM service page.
- Navigate to the Users navigation tab and click Add users.
- Enter a User name for the service, for example,
transfer-service
and click Next. Under Select AWS access type, select the Access key - Programatic access option. - Click Next: Permissions.
- Click the Attach existing policies directly option, and search for the name of the policy created in the previous step. Select the policy > click Next: Tags.
- Click Next: Review and click Create user.
- In the Success screen, record the Access key ID and the Secret access key.
Step 3: Add Your Destination
- After completing the initial setup, share your
bucket name
with a Zuora representative who will create a connection link for you. - Using the connection link shared with you by Zuora, you can securely input your AWS S3 details, including the
Bucket name,
bucket region
,folder name,
choose the auth method and input the required credentials for finalizing the setup for the connection. - After you fill in all the required AWS S3 details through the provided link and test the connection, saving the destination will kickstart the onboarding process and begin transferring data.
Verification and Data Transfer
For the AWS S3 connector, once data synchronization is configured, the data is automatically transferred to the specified S3 bucket. This allows for direct access and querying of the data within AWS services or through compatible data analysis tools. You can manage and analyze this data seamlessly across your AWS ecosystem.
Format of Transferred Data
When transferring data to AWS S3, the files are saved in Apache Parquet format, following an Apache Hive partitioning style.
The data organization structure is detailed below:
<bucket_name>/<folder_name>/<model_name>/dt=<transfer_date>/<file_part>_<transfer_timestamp>.parquet
where;
<bucket_name>
and<folder_name>
are provided during destination configuration.<model_name>
is the name of the data model being transferred (this is equivalent to a table name in relational data destinations).<transfer_date>
and<transfer_timestamp>
are generated at transfer time and based on the transfer's start time.<transfer_date>
is of the form2006-01-01
, while<transfer_timestamp>
is of the form20060102150405
.<file_part>
is a monotonically increasing integer for a given timestamp, and does not carry any special meaning.
What are Apache Hive style partitions and Apache Parquet file format?
|