Skip to main content

Overview

With Propel, you can transform your data in real-time using SQL with our powerful Materialized Views. Data transformation is crucial for data aggregation and enrichment, setting the stage for dependable analytics and data applications.

Propel provides an extensive set of SQL functions, facilitating complex operations with ease. Furthermore, Propel ensures the integrity of your data throughout the transformation process, upholding the precision of your analytics.

Real-time transformations in SQL

Transformations in Propel are defined using SQL and calculated in real time. Propel accepts both PostgreSQL and Clickhouse SQL dialects, giving you the flexibility to work with the SQL dialect that you're most comfortable with. This ensures that you can maximize the power of SQL to define your data transformations precisely, optimizing the quality of your analytics.

High-level overview

In its simplest form, a Materialized View processes data as it is inserted into one Data Pool, transforms it, and writes it to another.

Propel's Materielized Views

Materialized Views support complex transformations, such as creating multiple Materialized Views from a single Data Pool or Materialized Views that perform joins across Data Pools to enrich data in real time.

How do Materialized Views work?

Propel's Materialized Views work similarly to those in Clickhouse, as Propel's implementation is based on it.

Propel Materialized Views work based on insert triggers. When new data is inserted into the source Data Pool, the Materialized View is triggered. This trigger executes the SQL statement defined in the Materialized View, transforming and storing the new data in the destination Data Pool. This means that the destination Data Pool is always up-to-date with the latest data from the source Data Pool without needing to run the SQL transformations on a schedule.

For every destination Data Pool, Propel will create one additional column: _propel_written_at.

Getting started with Materialized Views

The ClickHouse Data Pool is in preview and requires an onboarding session. Please schedule an onboarding session with one of our engineers to get started.

Things to know when working with Materialized Views

  • The destination Data Pool will not be automatically deleted when deleting a Materialized View.
  • Materialized Views cannot be modified. If you need to modify the fields, create a new one.
  • The destination Data Pools consume storage just like any other Data Pool.
  • Materialized Views incur data write costs just like any other Data Pool.
  • Data deleted or updated with the Batch update or delete API will not trigger the Materialized View and will not be propagated to the destination Data Pool.