Skip to main content

Data Pools

Data Pools provide the data that power Metrics. They are cached tables optimized for ultra-fast queries. Their data is synced from a table in your data warehouse or sent via an HTTP endpoint or an S3 bucket.

When new data comes into a Data Pool, Propel creates a Sync to keep track of the new records. Once the Sync succeeds, the records are available for Metric Queries.

A screen capture of the Data Pool Syncs.

Data Pool properties

Data Pools need additional information on the table columns to organize and optimize the data. These properties point to columns that determine how the Data Pool works.

Timestamp

The timestamp property identifies the table's primary timestamp column.

The Data Pool's timestamp will be the time dimension for any Metric powered by the Data Pool. With event data, the primary timestamp is typically when the event happened. Propel will optimize how the data is organized based on this timestamp. You cannot change this timestamp after you create the Data Pool. If you need Metrics with a different time dimension, you need to create a new Data Pool with a new timestamp.

Tenant ID

The tenant is the customer that signs up in your system.

The tenant ID property identifies the table column with your end customer's unique ID. Propel uses it to provide tenant isolation when querying Metrics data.

The tenant ID is an optional property. When a tenant ID is defined:

  • You can set TENANT_ACCESS access policies for a Metric to scope Metric Queries to a particular tenant.

Excluded columns

With the excluded columns property, you can tell Propel not to sync specific columns of your table. They will not be part of the Data Pool's schema, and Propel will never sync their data. That means you cannot use excluded columns when defining Metrics.

Metrics are for aggregate data analysis, so some data in individual records may be irrelevant. Consider excluding columns when:

  • Your table has columns that you will not need for your Metrics, especially if they hold a large amount of data, like a giant JSON blob.
  • Your table has columns with personal identifiable information (PII) or other sensitive information that you do not need for your Metrics.

Sync behaviors

The Sync behavior determines how the data is synced from the Data Source into the Data Pool.

NameDescription
Append onlySyncs new records. Updates and deletes are ignored.