Skip to main content

Setup Zuora Connector for Google Cloud Storage

Zuora

Setup Zuora Connector for Google Cloud Storage

Provides detailed instructions for setting up the Zuora Connector for Google Cloud Storage.

Configure your Google Cloud Storage destination

To set up your Google Cloud Storage connector, you must first configure your Google Cloud Storage destination within the Zuora Connector. Follow the steps below to set up Google Cloud service account credentials, manage permissions, configure bucket access, and other necessary configurations to ensure a seamless data transfer process from Zuora to Google Cloud Storage.

Prerequisites

By default, Google Cloud Storage (GCS) authentication uses role-based access. You will need the data-syncing service's service account name available to grant access. It should look like some-name@some-project.iam.gserviceaccount.com.

Step 1: Create a service account

  1. In the GCP console, navigate to the IAM and Admin menu, click into the Service Accounts tab, and click Create service account at the top of the menu.
  2. In the first step, name the service account that will be used to transfer data into Cloud Storage and click Create and Continue. Click Continue in the following optional step without assigning any roles.
  3. In the Grant users access to this service account step, within the Service account users role field, enter the provided Service account (see prerequisite) and click Done.
  4. Once successfully created, search for the created service account in the service accounts list, click the Service account name to view the details, and make a note of the email (note: this is a different email than the service's service account).
  5. Select the permissions tab, find the provided principal name (Service account from the prerequisite), click the Edit principal button (pencil icon), click Add another role, select the Service Account Token Creator role, and click Save.

gcp-grant-role-1.png

Alternative authentication method: HMAC Access Key and Secret

Role based authentication is the preferred authentication mode for Google Cloud Storage based on GCP recommendations, however, HMAC Access Key ID & Secret Access Key is an alternative authentication method that can be used if preferred. An HMAC key is a type of credential and can be associated with a service account or a user account to access Google Cloud Storage.

  1. Navigate to the Cloud Storage page.
  2. Click the Settings tab on the left side menu.
  3. Navigate to the Interoperability tab and click the Create a key for a Service Account button.
    • gcs-bucket-settings-2.png
  4. Select the Service Account created in Step 1, and click Create key.
    • gcs-select-service-account-hmac-3.png
  5. Make a note of the Access key and Secret.

Step 2: Create destination GCS bucket

  1. Navigate to the Cloud Storage page.
  2. Click Create.
  3. Enter a bucket name > choose a region. Note: at the Choose how to control access to objects step, we recommend selecting Enforce public access prevention on this bucket.
    • gcs-prevent-public-access-4.png
  4. After choosing your preferences for the remaining steps, click Create.
  5. On the Bucket details page for the bucket you created, select the Permissions tab and click Grant access.
  6. Grant access to the principal (Service Account) you created in Step 1 (this is the service account you created, not the service account from the prerequisite), and assign the Role: Storage Legacy Bucket Writer > click Save.
    • gcp-storage-legacy-bucket-writer-5.png

Step 3: Add your destination

  1. After completing the initial setup, share your bucket name with a Zuora representative who will create a connection link for you.
  2. Using the connection link shared with you by Zuora, you can securely input your Google Cloud Storage details, including the folder and bucket name, choose the auth method, and input the required credentials, finalizing the setup for the connection.
  3. After you fill in all the required Google Cloud Storage details through the provided link and test the connection, saving the destination will kickstart the onboarding process and begin transferring data.

Verification and Data Transfer 

Once data synchronization is configured with the Google Cloud Storage connector, the data is automatically transferred to the specified Google Cloud Storage bucket. This enables direct access and querying of the data within Google Cloud services or through compatible data analysis tools. You can seamlessly manage and analyze this data across your Google Cloud ecosystem.

Format of Transferred Data

When transferring data to Google Cloud Storage, the files are saved in Apache Parquet format, following an Apache Hive partitioning style. The data organization structure is detailed below:

<bucket_name>/<folder_name>/<model_name>/dt=<transfer_date>/<file_part>_<transfer_timestamp>.parquet

where:

  • <bucket_name> and <folder_name> are provided during destination configuration.
  • <model_name> is the name of the data model being transferred (this is equivalent to a table name in relational data destinations).
  • <transfer_date> and <transfer_timestamp> are generated at transfer time and based on the transfer's start time. <transfer_date> is of the form 2006-01-01, while <transfer_timestamp> is of the form 20060102150405.
  • <file_part> is a monotonically increasing integer for a given timestamp, and does not carry any special meaning.
What are Apache Hive style partitions and Apache Parquet file format?
  • Apache Hive style partitions are compatible with most popular query engines, and should make data easily queryable and transportable.
  • Apache Parquet file format is an open source, column-oriented data file format that offers efficient data compression and data integrity.