Data Pools provide the data that power Metrics. They are cached tables optimized for ultra-fast queries. Their data is synced from a table in your data warehouse or sent via an HTTP endpoint or an S3 bucket.
When new data comes into a Data Pool, Propel creates a Sync to keep track of the new records. Once the Sync succeeds, the records are available for Metric Queries.
Data Pool properties
Data Pools need additional information on the table columns to organize and optimize the data. These properties point to columns that determine how the Data Pool works.
The timestamp property identifies the table's primary timestamp column.
The Data Pool's timestamp will be the time dimension for any Metric powered by the Data Pool. With event data, the primary timestamp is typically when the event happened. Propel will optimize how the data is organized based on this timestamp. You cannot change this timestamp after you create the Data Pool. If you need Metrics with a different time dimension, you need to create a new Data Pool with a new timestamp.
The tenant is the customer that signs up in your system.
The tenant ID property identifies the table column with your end customer's unique ID. Propel uses it to provide tenant isolation when querying Metrics data.
The tenant ID is an optional property.
How it works
Upon creating a Data Pool, you can optionally specify a tenant ID. This is a pointer to a specific column. This column will be used to control access. For example, a
CUSTOMER_ID column can be used as the tenant ID. Once specified, Metrics can have tenant-level access policies.
When querying a Metric, Propel will only return values for a specific tenant. You specify the tenant ID when you mint the access token. That way, it is signed into the token and it cannot be modified from client-side code.
Setting up a Data Pool with a Tenant ID
⚠️ Note: Tenant ID can only be added when creating a new Data Pool.
- Create a Data Pool with an appropriate Data Source.
- Include the appropriate columns including a timestamp column and the desired identifier to use as tenant ID.
- On step 3 of Data Pool creation, specify both the timestamp column and the tenant ID.
Setting up a Metric with a Tenant ID aware access policy
- An Application is required in order to setup a Tenant ID aware access policy.
- With a Metric created on top of the tenant-aware Data Pool, select the metric and choose the "Access Control" tab.
- In the left hand application drop down menu, choose the application you wish to enable access control on.
- In the right hand policy drop down menu, select
Access tokens created against this Application now require a
tenant variable to be passed. If the
tenant passed matches the tenant ID specified on the Data Pool, data will be returned. See the Authentication guide for more details on access tokens.
With the excluded columns property, you can tell Propel not to sync specific columns of your table. They will not be part of the Data Pool's schema, and Propel will never sync their data. That means you cannot use excluded columns when defining Metrics.
Metrics are for aggregate data analysis, so some data in individual records may be irrelevant. Consider excluding columns when:
- Your table has columns that you will not need for your Metrics, especially if they hold a large amount of data, like a giant JSON blob.
- Your table has columns with personal identifiable information (PII) or other sensitive information that you do not need for your Metrics.
Certain column types are not supported by Propel. In the case where a column type is not supported, it will not be possible to include it when setting up a Data Pool.
The Sync behavior determines how the data is synced from the Data Source into the Data Pool.
|Append only||Syncs new records. Updates and deletes are ignored.|