BigQuery
The Data Source enables you to connect BigQuery to Propel, providing an easy way to deploy a data API on top of your BigQuery data.
Consider using Propel on top of BigQuery when:
- You require sub-second query performance for dashboards or reports.
- You need to support high-concurrency and high-availability data workloads, such as customer-facing or mission-critical applications.
- You require fast data access through an API for web and mobile apps.
- You are building B2B SaaS or consumer applications that require multi-tenant access controls.
Set up guide
Follow our step-by-step BigQuery setup guide to connect your BigQuery data warehouse with Propel.
Architecture Overview
The BigQuery Data Source lets you use any table to power your Data Pools. Propel connects to the specified GCP project ID and BigQuery dataset using the provided service account. When you create a Data Pool from a BigQuery table or view, Propel automatically synchronizes the data from your BigQuery table into your Data Pool.
Features
BigQuery Data Pools support the following features:
| Feature name | Supported | Notes |
|---|---|---|
| Syncs inserts | ✅ | Propel incrementally syncs new records from the BigQuery table and appends them to Propel. Ideal for immutable event data. |
| Syncs updates and deletes | ❌ | Updates and deletes are not supported during preview. |
| Re-sync | ❌ | ReSyncs are not supported duting preview. |
| Configurable sync interval | ✅ | You can configure syncs to occur at intervals ranging from every minute to every 24 hours. |
| Sync Pausing / Resuming | ✅ | |
| Delete Job API | ✅ | See Delete Job API. |
| API configurable | ⚠️ | Only BigQuery Data Pools, not BigQuery Data Sources. See API reference docs. |
| Terraform configurable | ⚠️ | Only BigQuery Data Pools, not BigQuery Data Sources. See Propel Terraform docs. |
Supported data structures
Below are the BigQuery data structures that Propel supports.
| Name | Description |
|---|---|
| Tables | Includes regular, temporary, or object tables. |
| View | Includes regular and materialized views. |
Note: The BigQuery Data Source does not support the following data structures:
- Tables exported from Google Analytics
- Tables stored in a cloud storage service or Google Drive
Frequently Asked Questions
Does Propel read through to BigQuery on every query?
No, Propel only queries BigQuery to sync records, not to serve requests via the API.