Skip to main content

Setup the Zuora Connector for Azure Blob Storage

Zuora

Setup the Zuora Connector for Azure Blob Storage

Provides detailed instructions for setting up the setup of Zuora Connector for Azure Blob Storage.

Configuring your Azure Blob Storage destination within the Zuora Connector covers setting up Azure Storage Account credentials, managing permissions, configuring container access, and other necessary configurations to ensure a seamless data transfer process from Zuora to Azure Blob Storage.

  1. In the Azure portal, navigate to Storage accounts service and click + Create.
  2. In the Basics tab of the Create a storage account form, fill in the required details.
  3. In the Advanced settings, under Security, make sure Enable storage account key access is turned on. You may turn off (deselect) Allow enabling public access on containers. Under Data Lake Storage Gen2, select Enable hierarchical namespace.

    1-azure-settings-toggles.png
     
  4. In the Networking settings, you may limit Network access to either Enable public access from all networks or Enable public access from selected virtual networks and IP addresses. If the latter is selected, be sure to add the service's static IP to the address range of the chosen virtual network. All other settings can use the default selections.
  5. In the Data protection settings, you must turn off Enable soft delete for blobs, Enable soft delete for containers, and Enable soft delete for file shares.

    2-azure-turn-off-settings.png
     
  6. Once the remaining options have been configured to your preference, click Create.

Step 2: Create container and access token

  1. In the Azure portal, navigate to the Storage accounts service and click on the account that was created in the previous step.
  2. In the navigation pane, under Data storage, click Containers. Click + Container, choose a name for the container, and click Create.
  3. In the navigation pane, under "Security + networking", click Shared access signature.
  4. Update the required accessible services and permissions:
    1. Under Allowed services: select Blob and File.
    2. Under Allowed resource types: select Container and Object.
    3. Under Allowed permissions: select Read, Write, Delete, List, Add, Create, and Permanently Delete.
  5. Select a Start and expiry date/time based on your security posture (e.g., set the expiration date 6 months into the future), and click Generate SAS and connection string.
  6. Make a note of the SAS token that is generated.

    3-azure-setup-details.png

Step 4: Add Your Destination

  1. After completing the initial setup, share your container name and access ID with a Zuora representative who will create a connection link for you. 
  2. Using this link, you can securely input your Azure Blob Storage details, including the folder name, storage account name, and storage account SAS token
  3. Once you fill in these details and test the connection, saving the destination will initiate the onboarding process and begin transferring data from Zuora to Azure Blob Storage.

Verification and Data Transfer 

Once data synchronization is configured with the Azure Blob Storage connector, the data is automatically transferred to the specified Azure Blob Storage environment. This enables direct access and querying of the data within your Azure Blob Storage environment or through compatible data analysis tools. You can seamlessly manage and analyze this data within your Azure Blob Storage ecosystem.

Format of Transferred Data

Data transferred to the Azure Blob Storage destination will be loaded as Apache Parquet files in Apache Hive style partitions. The data organization structure is detailed below:

<container_name>/<folder_name>/<model_name>/dt=<transfer_date>/<file_part>_<transfer_timestamp>.parquet

where;

  • <container_name> and <folder_name> are provided during destination configuration.

  • <model_name> is the name of the data model being transferred (equivalent to a table name in relational data destinations).

  • <transfer_date> and <transfer_timestamp> are generated at transfer time based on the transfer's start time. <transfer_date> is of the form 2006-01-01, while <transfer_timestamp> is of the form 20060102150405.

  • <file_part> is a monotonically increasing integer for a given timestamp, without special meaning.

This structure allows for efficient storage and querying of data within Azure, providing seamless access for analytics and data processing tasks.

What are Apache Hive style partitions and Apache Parquet file format?

 

  • Apache Hive style partitions are compatible with most popular query engines, and should make data easily queryable and transportable.
  • Apache Parquet file format is an open source, column-oriented data file format that offers efficient data compression and data integrity.