Skip to main content

How to set up a BigQuery Data Source

This guide explains how to connect your BigQuery dataset to Propel. It covers how to create a GCP service account, set its required permissions, allow IPs of Propel to connect to BigQuery, and complete your setup in Propel.

The steps to follow are:

  1. Create a service account key
  2. Set service account permissions
  3. Complete Propel configuration

Requirements​

  • You have a Propel account.
  • You have a Google Cloud Platform account with BigQuery enabled.
  • You have permission to assign Job User, Data Viewer, and Admin roles at the dataset level.
  • You have the data you want to use with Propel in a BigQuery dataset.

1. Create a service account key​

Follow Google's Create a service account key instructions to set up service account key authentication.

Make a note of the entire service account key file. You will need it to configure the Data Source in Propel. It should have the following format:

{
  "type": "",
  "project_id": "",
  "private_key_id": "",
  "private_key": "",
  "client_email": "",
  "client_id": "",
  "auth_uri": "",
  "token_uri": "",
  "auth_provider_x509_cert_url": "",
  "client_x509_cert_url": ""
}

2. Set service account permissions​

Grant the service account the following permissions on the dataset:

ROLEREASON
BigQuery Job UserTo create queries
BigQuery Data ViewerTo view the data
BigQuery AdminTo create temporary tables (required to sync data)
tip

To learn how to set permissions in BigQuery, see Google's Introduction to IAM documentation.

3. Add IPs to allow-list​

To allow Propel to connect, configure your firewall and/or other access control mechanisms to allow incoming connections to your BigQuery data warehouse from the following Propel IP addresses:

3.239.194.48
3.239.194.49
3.239.194.50
3.239.194.51
3.239.194.52
3.239.194.53
3.239.194.54
3.239.194.55
54.196.30.169
52.72.201.213
18.213.226.96
3.224.126.197
3.217.26.199
3.17.239.162
3.15.73.135
18.219.73.236

4. Complete the Propel configuration​

The Propel BigQuery Data Source is currently in public preview and, as such, has the following limitations:

  • Propel must manually set it up.
  • In the Propel Console, the Data Source will appear as an S3 Data Source, not BigQuery.
  • Sync intervals can range from 5 minutes to 24 hours. Minute-level sync intervals are not available in the preview.

To complete the setup, complete the BigQuery setup form with the following information:

  • Account ID - Your Propel Account ID that you can find in the Console.
  • Project ID - The ID of your BigQuery project
  • Dataset name - The name of your BigQuery dataset
  • Tables or views - List the tables or views you want to use with Propel. You can use wildcards to select multiple tables or views. For example, my_dataset.my_table* will select all tables that start with my_table.
  • The service account JSON in a secure URL - Please send using a secure service like SendSecurely (make sure to give it a 24-hour expiry).