How to set up a BigQuery Data Source
This guide explains how to connect your BigQuery dataset to Propel. It covers how to create a GCP service account, set its required permissions, allow IPs of Propel to connect to BigQuery, and complete your setup in Propel.
The steps to follow are:
- You have a Propel account.
- You have a Google Cloud Platform account with BigQuery enabled.
- You have permission to assign Job User, Data Viewer, and Admin roles at the dataset level.
- You have the data you want to use with Propel in a BigQuery dataset.
1. Create a service account key
Follow Google's Create a service account key instructions to set up service account key authentication.
Make a note of the entire
service account key file. You will need it to configure the Data Source in Propel. It should have the following format:
2. Set service account permissions
Grant the service account the following permissions on the dataset:
|BigQuery Job User||To create queries|
|BigQuery Data Viewer||To view the data|
|BigQuery Admin||To create temporary tables (required to sync data)|
To learn how to set permissions in BigQuery, see Google's Introduction to IAM documentation.
3. Add IPs to allow-list
To allow Propel to connect, configure your firewall and/or other access control mechanisms to allow incoming connections to your BigQuery data warehouse from the following Propel IP addresses:
4. Complete the Propel configuration
The Propel BigQuery Data Source is currently in public preview and, as such, has the following limitations:
- Propel must manually set it up.
- In the Propel Console, the Data Pool will appear as an Amazon S3 Data Pool, not BigQuery.
- Sync intervals can range from 5 minutes to 24 hours. Minute-level sync intervals are not available in the preview.
To complete the setup, fill out the BigQuery setup form with the following information:
- Account ID - Your Propel Account ID that you can find in the Console.
- Project ID - The ID of your BigQuery project
- Dataset name - The name of your BigQuery dataset
- Tables or views - List the tables or views you want to use with Propel. You can use wildcards to select multiple tables or views. For example,
my_dataset.my_table*will select all tables that start with
- The service account JSON in a secure URL - Please send using a secure service like SendSecurely (make sure to give it a 24-hour expiry).