Ingest data from S3.
Feature name | Supported | Notes |
---|---|---|
Syncs new records | ✅ | |
Configurable sync interval | ✅ | See the How Propel syncs section below. It can be configured to occur at intervals ranging from every minute to every 24 hours. |
Sync Pausing / Resuming | ✅ | |
Real-time updates | ✅ | See the Real-time updates section. |
Real-time deletes | ❌ | See the Real-time deletes section. |
Batch Delete API | ✅ | See Batch Delete API. |
Batch Update API | ✅ | See Batch Update API. |
API configurable | ✅ | See API docs. |
Terraform configurable | ✅ | See Terraform docs. |
*.parquet
pattern to sync only Parquet files, excluding other file types.AddColumnToDataPool
job.
For breaking changes like column deletions or type modifications, recreate the Data Pool.
See our Changing Schemas section for more details.
Parquet Type | Propel Type | Notes |
---|---|---|
BOOLEAN | BOOLEAN | |
INT8 | INT8 | |
UINT8 | INT16 | |
INT16 | INT16 | |
UINT16 | INT32 | |
INT32 | INT32 | |
UINT32 | INT64 | |
INT64 | INT64 | |
UINT64 | INT64 | |
FLOAT | FLOAT | |
DOUBLE | DOUBLE | |
DECIMAL(p ≤ 9, s=0) | INT32 | |
DECIMAL(p ≤ 9, s>0) | FLOAT | |
DECIMAL(p ≤ 18, s=0) | INT64 | |
DECIMAL(p ≤ 18, s>0) | DOUBLE | |
DECIMAL(p ≤ 76, s) | DOUBLE | |
DATE | DATE | |
TIME (ms) | INT32 | |
TIME (µs, ns) | INT64 | |
TIMESTAMP | TIMESTAMP | |
INT96 | TIMESTAMP | |
BINARY | STRING | |
STRING | STRING | |
ENUM | STRING | |
FIXED_LENGTH_BYTE_ARRAY | STRING | |
MAP | JSON | |
LIST | JSON |