Skip to main content

AWS S3

This Data Source enables you to use Parquet files stored in your AWS S3 bucket with Propel.

To setup an S3 Data Source, you provide AWS credentials for connecting to an S3 bucket where you store Parquet files of table data for use with Propel.

Each AWS S3 Data Source requires the name of the table and a path variable. The path variable allows you to specify where in your bucket the files referencing the table reside.

An S3 Data Source can contain one or many tables. Propel determines which Parquet files belong to which tables using the tables' S3 paths. For example, your S3 bucket might contain Parquet files representing sales data and signup data under two different paths:

s3://your-bucket
├── sales
│   ├── sales_1.parquet
│   ├── sales_2.parquet
│   └── sales_3.parquet
└── signups
    ├── signups_1.parquet
    ├── signups_2.parquet
    └── signups_3.parquet

A single S3 Data Source can contain tables for each by specifying two different S3 paths:

sales/**/*.parquet
signups/**/*.parquet

Setup guide​

Follow our step-by-step AWS S3 setup guide to connect your AWS S3 bucket to Propel.

Supported Sync behaviors​

  • Approximately every minute, Propel checks the AWS S3 bucket for new files at the tables' specified paths.
  • Append-only: Syncs data from new files only. Updated or deleted Parquet files are ignored. Append is suitable for immutable event data.

Usage​

Once you create your AWS S3 Data Source, you can then create Data Pools from your table:

  1. Click on "Data Pools" on the left-hand side menu.
  2. Click on the "New Data Pool" button.
  3. Select a name and description for your Data Pool.
  4. Select your AWS S3 Data Source.
  5. Click the "Create" button.

An animated screen capture of of how to use an AWS S3 Data Source to create a Data Pool.

After you create your Data Pool, you can define Metrics with your Data Pool's data.

Propel's guide to AWS S3 as a Data Source​

API reference documentation​

Below is the relevant API documentation for the S3 Data Source.

Objects​

Enums​

Inputs​

Queries​

Mutations​

Limits​

No limits at this point.