Webhooks

The Webhook Data Pool ingests JSON events via an HTTP URL from any application or SaaS service. Once the events are in the Propel Data Pool, you can use them to power your analytic dashboards, reports, and workflows.

Consider using the Webhook Data Pool when:

You need to power real-time analytics with streaming ingestion.
You need to send events from your application or a third-party SaaS service.
You are working with semi-structured JSON data.
You are capturing change events from a NoSQL database.
You are using Segment, Rudderstack, or Snowplow to collect events.
You are collecting events from OpenAI API calls.

Get started: Ingesting JSON events into Propel

Set up guide

Follow our step-by-step guide on How to ingest JSON events into Propel.

Architecture overview

Webhook Data Pools provides an HTTP URL to which you can send events from your applications, streaming infrastructure, or SaaS applications.

The architectural overview when sending Webhooks to Propel.

Features

Webhook Data Pools supports the following features:

Feature name	Supported	Notes
Event collection	✅	Collects events in JSON format.
Real-time updates	✅	See the Real-time updates section.
Real-time deletes	✅	See the Real-time deletes section.
Batch deletes	✅	See the Batch deletes section.
Bulk insert	✅	Up to 500 events per HTTP request.
API configurable	✅	See API reference docs.
Terraform configurable	✅	See Propel Terraform docs.

How does the Webhook Data Pool work?

When you create a Webhook Data Pool, you will get an HTTP URL to which you can post JSON events. These events will be collected in the Data Pool and made available via the APIs to power your analytics.

A screenshot of the Webhook Data Pool URL in the Propel Console

By default, the Webhook Data Pool has two columns: _propel_received_at and propel_payload.

Column	Type	Description
_propel_received_at	TIMESTAMP	The timestamp when the event was collected in UTC.
_propel_payload	JSON	The JSON Payload of the event.

During the setup of a Webhook Data Pool, you can optionally unpack top-level or nested keys from the incoming JSON event into specific columns. This provides convenience for commonly used keys, the ability to use them as Metric dimensions, and is particularly useful for setting an appropriate timestamp and tenant ID included in the payload.

See our step-by-step guide on How to ingest JSON events into Propel.

Authentication

Optionally, you can add basic authentication to your HTTP endpoint to secure your URL. If these parameters are not provided, anyone with the URL to your webhook will be able to upload data. While it's okay to test without HTTP Basic authentication, we recommend enabling it.

A screenshot of the Webhook Data Pool authentication setup in the Propel Console

Sending individual JSON events

You can send individual JSON events to the Webhook Data Pool by making an HTTP POST request to the Data Pool URL with the JSON event in the request body.

curl https://webhooks.us-east-2.propeldata.com/v1/WHK... \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "customer_id": 5,
    "order_id": 34,
    "store_id": 4445,
    "order_details": {
      "taco_count": 7,
      "total_price": 25.90,
      "checkout_time": "2023-07-31T15:20:10Z"
    },
    "created_at": "2023-07-31T14:50:35Z"
  }'

Sending a batch of JSON events

You can send a batch of JSON events to the Webhook Data Pool by making an HTTP POST request to the Data Pool URL with a JSON array of events in the request body. Each request can have a maximum of 500 events in the array.

curl https://webhooks.us-east-2.propeldata.com/v1/WHK... \
-X POST \
-H "Content-Type: application/json" \
-d '[
  {
    "customer_id": 5,
    "order_id": 34,
    "store_id": 4445,
    "order_details": {
      "taco_count": 7,
      "total_price": 25.90,
      "checkout_time": "2023-07-31T15:20:10Z"
    },
    "created_at": "2023-07-31T14:50:35Z"
  },
  {
    "customer_id": 8,
    "order_id": 22,
    "store_id": 1199,
    "order_details": {
      "taco_count": 3,
      "total_price": 15.75,
      "checkout_time": "2023-07-31T12:40:21Z"
    },
    "created_at": "2023-07-31T12:30:55Z"
  }
]'

If there are more than 500 events in a single request, Propel will return an HTTP 400 BAD REQUEST error.

Real-time updates

The Webhook Data Pool supports real-time updates. It uses each record’s primary key, consisting of the primary timestamp and a unique ID, to determine whether it should be inserted or updated within the Data Pool. This also means that if you send the same record twice, it will not be duplicated.

Updates are useful in the following scenarios:

When retrying requests: You can safely retry requests without worrying about creating duplicates.
When updating the properties of an object: For example, if you need to update the "taco_count" property of the "Order" object from "7" to "8.”

To update a record, make an HTTP POST request to the Webhook Data Pool URL with the primary timestamp and unique ID of the record you want to update.

The following example demonstrates how to update the taco_count and total_price fields in the record below. The record's primary timestamp is created_at and its unique ID is order_id.

{
  "customer_id": 5,
  "order_id": 34,
  "store_id": 4445,
  "order_details": {
    "taco_count": 7,
    "total_price": 25.9,
    "checkout_time": "2023-07-31T15:20:10Z"
  },
  "created_at": "2023-07-31T14:50:35Z"
}

To update the taco_count to "8" and update the total_price to "28.50", you need to make the following HTTP POST request. Note that the created_at (primary timestamp) and its order_id (unique ID) fields should remain the same as the original record to update it.

curl https://webhooks.us-east-2.propeldata.com/v1/WHK... \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "customer_id": 5,
    "order_id": 34,
    "store_id": 4445,
    "order_details": {
      "taco_count": 8,
      "total_price": 28.50,
      "checkout_time": "2023-07-31T15:20:10Z"
    },
    "created_at": "2023-07-31T14:50:35Z"
  }'

Real-time deletes

The Webhook Data Pool supports real-time deletes. It uses each record’s primary key, consisting of the primary timestamp and a unique ID, to determine whether it should be inserted or deleted within the Data Pool.

To delete a record, make an HTTP DELETE request to the Webhook Data Pool URL with the primary timestamp and unique ID of the record you want to update.

The following example shows how to delete a record with a timestamp of "2023-07-31T14:50:35Z" and a unique ID of "34".

curl https://webhooks.us-east-2.propeldata.com/v1/WHK... \
  -X DELETE \
  -H "Content-Type: application/json" \
  -d '{
          "order_id": 34,
          "created_at": "2023-07-31T14:50:35Z"
  }'

Schema changes

The Webhook Data Pool, at its core, handles semi-structured schema-less JSON data. That means you can add new properties to your payload whenever necessary. The entire payload will always be available in the _propel_payload column.

Propel will enforce the schema for required fields. Thus, if you stop providing data for a field that was previously unpacked into its own column and marked as required when creating the Data Pool, Propel will return an error.

Adding Columns

At any point, you can add columns to you Webhook Data Pool to evolve your schema with non-breaking changes.

A screen capture of where to add a column in the Data Pool schema page.

Once you click the “Add Column” button, you will be able to define the new column. The fields includes the JSON property to extract from the event, the name of the new column, and the type.

A screen capture of the fields to add a column.

Clicking “Add column” starts an asynchronous operation to add the column to the Data Pool. You can monitor the progress of the operation in the “Operations” tab.

A screen capture of the add column job in the Data Pool operations page.

Note that when you add a column, Propel will not backfill. To backfill existing rows, you can run a batch update operation.

Column deletions, column modifications, and data type changes are not supported because they are breaking changes to the schema.

Data types

The table below describes default data type mappings from JSON to Propel types. When creating a Webhook Data Pool, you can modify these default mappings. For instance, if you know that a column originally typed as a NUMBER contains a UNIX timestamp, you can convert it to a TIMESTAMP by changing the default mapping.

JSON type	Propel type
String	STRING
Number	DOUBLE
Object	JSON
Array	JSON
Boolean	BOOLEAN
Null	JSON

Key guides

Frequently Asked Questions

How long does it take for an event to be available via the API?

When an event is collected, the data will be available in Propel and served via the API in 1-3 minutes.

API reference documentation

Below is the relevant API documentation for the Webhook Data Pool.

Queries

Mutations

Limits

Up to 500 events (as a JSON array) can be sent for each POST.
The payload can be up to 1,048,320 bytes.

Best practices

Send as many events per request as you can, up to 500 or 1,048,320 bytes. This helps ingest them faster.
Retry on 429 and 500 errors with exponential backoff. This ensures no events are dropped.
Alert or page on any 413 Content Too Large error. This means the event count or content length was exceeded. It needs to be looked at, as retrying will not help.
Create the Webhook Data Pool with all the required fields. Pasting a sample event and entering the Webhook Data Pool creation flow and clicking "Extract top-level fields" will automatically do this.
Customize the primary timestamp to the proper event timestamp. Otherwise, the _propel_received_at will be used.
We recommend marking all the fields except the primary timestamp and unique ID as not required to minimize 400 errors (see next point). Even if there is a data fill issue, you want Propel to accept the request, and then you can run an update to backfill any missing data instead of dropping the event.
Alert or page on any 400 error. This means that the schema is incorrect and that Propel is rejecting the request. Why it's happening needs to be investigated, retrying will not help.
Double-check all the types. For objects, arrays, and dictionaries, make sure they are JSON in Propel.
Enable basic auth on the Webhook Data Pool to secure the endpoint.

Webhooks

Get started: Ingesting JSON events into Propel​

Architecture overview​

Features​

How does the Webhook Data Pool work?​

Authentication​

Sending individual JSON events​

Sending a batch of JSON events​

Real-time updates​

Real-time deletes​

Schema changes​

Adding Columns​

Data types​

Key guides​

Frequently Asked Questions​

API reference documentation​

Queries​

Mutations​

Limits​

Best practices​