Skip to main content

Data Pools

Data Pools provide the data that power Metrics. They are cached tables optimized for ultra-fast queries. Their data is synced from a table in your data warehouse or sent via an HTTP endpoint or an S3 bucket.

When new data comes into a Data Pool, Propel creates a Sync to keep track of the new records. Once the Sync succeeds, the records are available for Metric Queries.

A screen capture of the Data Pool Syncs.

Data Pool properties

Data Pools need additional information on the table columns to organize and optimize the data. These properties point to columns that determine how the Data Pool works.

Timestamp

The timestamp property identifies the table's primary timestamp column.

The Data Pool's timestamp will be the time dimension for any Metric powered by the Data Pool. With event data, the primary timestamp is typically when the event happened. Propel will optimize how the data is organized based on this timestamp. You cannot change this timestamp after you create the Data Pool. If you need Metrics with a different time dimension, you need to create a new Data Pool with a new timestamp.

Tenant ID

The tenant is the customer that signs up in your system.

The tenant ID property identifies the table column with your end customer's unique ID. Propel uses it to provide tenant isolation when querying Metrics data.

The tenant ID is an optional property.

How it works

Upon creating a Data Pool, you can optionally specify a tenant ID. This is a pointer to a specific column. This column will be used to control access. For example, a CUSTOMER_ID column can be used as the tenant ID. Once specified, Metrics can have tenant-level access policies.

When querying a Metric, Propel will only return values for a specific tenant. You specify the tenant ID when you mint the access token. That way, it is signed into the token and it cannot be modified from client-side code.

Setting up a Data Pool with a Tenant ID

⚠️ Note: Tenant ID can only be added when creating a new Data Pool.

  1. Create a Data Pool with an appropriate Data Source.
  2. Include the appropriate columns including a timestamp column and the desired identifier to use as tenant ID.
  3. On step 3 of Data Pool creation, specify both the timestamp column and the tenant ID.

Setting up a Metric with a Tenant ID aware access policy

  1. An Application is required in order to setup a Tenant ID aware access policy.
  2. With a Metric created on top of the tenant-aware Data Pool, select the metric and choose the "Access Control" tab.
  3. In the left hand application drop down menu, choose the application you wish to enable access control on.
  4. In the right hand policy drop down menu, select TENANT_ACCESS.

Access tokens created against this Application now require a tenant variable to be passed. If the tenant passed matches the tenant ID specified on the Data Pool, data will be returned. See the Authentication guide for more details on access tokens.

Excluded columns

With the excluded columns property, you can tell Propel not to sync specific columns of your table. They will not be part of the Data Pool's schema, and Propel will never sync their data. That means you cannot use excluded columns when defining Metrics.

Metrics are for aggregate data analysis, so some data in individual records may be irrelevant. Consider excluding columns when:

  • Your table has columns that you will not need for your Metrics, especially if they hold a large amount of data, like a giant JSON blob.
  • Your table has columns with personal identifiable information (PII) or other sensitive information that you do not need for your Metrics.

Unsupported columns

Certain column types are not supported by Propel. In the case where a column type is not supported, it will not be possible to include it when setting up a Data Pool.

Sync behaviors

The Sync behavior determines how the data is synced from the Data Source into the Data Pool.

NameDescription
Append onlySyncs new records. Updates and deletes are ignored.