Ingest Parquet files from an Amazon S3 bucket to Propel.

​

Architecture

Amazon S3 Data Pools connect to a specified Amazon S3 bucket and automatically synchronize Parquet files from the bucket into your Data Pool.

​

Features

Amazon S3 Parquet Data Pools support the following features:

​

How Propel syncs Parquet files in Amazon S3

The Amazon S3-based Data Pool syncs Parquet files from your S3 bucket. You specify:

1. Bucket name
2. File path
3. The schema of the data
4. Sync interval (1 minute to 24 hours)

During each sync, Propel retrieves and synchronizes all new files in the specified S3 bucket path

1. To sync all Parquet files, use the following path:

   Copy

   **/*.parquet
   

2. To sync files in a specific directory (e.g., “sales”):

   Copy

   sales/**/*.parquet
   

​

New records and updates

How records are ingested depends on the table engine you select when you create your Data Pool.

• MergeTree Data Pools (append-only data): Syncs inserts and ignores updates.
• ReplacingMergeTree Data Pools (mutable records): Syncs inserts and updates records with the same sorting key.

Read our guide on Selecting table engine and sorting key for details.

​

Schema changes

Propel enables the addition of new columns to Amazon S3 Data Pools through the AddColumnToDataPool job.

For breaking changes like column deletions or type modifications, recreate the Data Pool.

See our Changing Schemas section for more details.

​

Data Types

The table below shows default Parquet to Propel data type mappings. When creating an Amazon S3 Parquet Data Pool, you can customize these mappings.