Webhooks

The Webhook Data Pool ingests JSON events via an HTTP URL from any application or SaaS service. Once the events are in the Propel Data Pool, you can use them to power your analytic dashboards, reports, and workflows.

Consider using the Webhook Data Pool when:

• You need to power real-time analytics with streaming ingestion.
• You need to send events from your application or a third-party SaaS service.
• You are working with semi-structured JSON data.
• You are capturing change events from a NoSQL database.
• You are using Segment, Rudderstack, or Snowplow to collect events.
• You are collecting events from OpenAI API calls.

Get started: Ingesting JSON events into Propel

Set up guide

Follow our step-by-step guide on How to ingest JSON events into Propel.

Architecture overview

Webhook Data Pools provides an HTTP URL to which you can send events from your applications, streaming infrastructure, or SaaS applications.

Features

Webhook Data Pools supports the following features:

Feature name	Supported	Notes
Event collection	✅	Collects events in JSON format.
Real-time updates	✅	See the Real-time updates section.
Real-time deletes	✅	See the Real-time deletes section.
Batch Delete API	✅	See Batch Delete API.
Batch Update API	✅	See Batch Update API.
Bulk insert	✅	Up to 500 events per HTTP request.
API configurable	✅	See Management API docs.
Terraform configurable	✅	See Propel Terraform docs.

How does the Webhook Data Pool work?

When you create a Webhook Data Pool, you will get an HTTP URL to which you can post JSON events. These events will be collected in the Data Pool and made available via the APIs to power your analytics.

By default, the Webhook Data Pool has two columns: _propel_received_at and propel_payload.

Column	Type	Description
_propel_received_at	TIMESTAMP	The timestamp when the event was collected in UTC.
_propel_payload	JSON	The JSON Payload of the event.

During the setup of a Webhook Data Pool, you can optionally unpack top-level or nested keys from the incoming JSON event into specific columns. This provides convenience for commonly used keys, the ability to use them as Metric dimensions, and is particularly useful for setting an appropriate timestamp and tenant ID included in the payload.

See our step-by-step guide on How to ingest JSON events into Propel.

Authentication

Optionally, you can add basic authentication to your HTTP endpoint to secure your URL. If these parameters are not provided, anyone with the URL to your webhook will be able to upload data. While it's okay to test without HTTP Basic authentication, we recommend enabling it.

Sending individual JSON events

You can send individual JSON events to the Webhook Data Pool by making an HTTP POST request to the Data Pool URL with the JSON event in the request body.

curl https://webhooks.us-east-2.propeldata.com/v1/WHK... \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "customer_id": 5,
    "order_id": 34,
    "store_id": 4445,
    "order_details": {
      "taco_count": 7,
      "total_price": 25.90,
      "checkout_time": "2023-07-31T15:20:10Z"
    },
    "created_at": "2023-07-31T14:50:35Z"
  }'

Sending a batch of JSON events

You can send a batch of JSON events to the Webhook Data Pool by making an HTTP POST request to the Data Pool URL with a JSON array of events in the request body. Each request can have a maximum of 500 events in the array.

curl https://webhooks.us-east-2.propeldata.com/v1/WHK... \
-X POST \
-H "Content-Type: application/json" \
-d '[
  {
    "customer_id": 5,
    "order_id": 34,
    "store_id": 4445,
    "order_details": {
      "taco_count": 7,
      "total_price": 25.90,
      "checkout_time": "2023-07-31T15:20:10Z"
    },
    "created_at": "2023-07-31T14:50:35Z"
  },
  {
    "customer_id": 8,
    "order_id": 22,
    "store_id": 1199,
    "order_details": {
      "taco_count": 3,
      "total_price": 15.75,
      "checkout_time": "2023-07-31T12:40:21Z"
    },
    "created_at": "2023-07-31T12:30:55Z"
  }
]'

Errors

• HTTP 429 Too Many Requests - will be returned when your application is being rate limited by Propel. You need to retry with exponential backoff.
• HTTP 413 Content Too Large - will be returned if there are more than 500 events in a single request or the payload exceeds 1,048,320 bytes. You need to fix the request, retrying won't help.
• HTTP 400 Bad Request - will be returned when the schema is incorrect and that Propel is rejecting the request. You need to fix the request, retrying won't help.
• HTTP 500 Internal Server Error - will be returned if something went wrong in Propel. You need to retry with exponential backoff.

By setting the disable_partial_success=true query parameter, you can ensure that, if any individual event in a batch of events fails validation, the entire request will fail.

For example: https://webhooks.us-east-2.propeldata.com/v1/WHK00000000000000000000000000?disable_partial_success=true

Schema changes

The Webhook Data Pool, at its core, handles semi-structured schema-less JSON data. That means you can add new properties to your payload whenever necessary. The entire payload will always be available in the _propel_payload column.

Propel will enforce the schema for required fields. Thus, if you stop providing data for a field that was previously unpacked into its own column and marked as required when creating the Data Pool, Propel will return an error.

Adding Columns

At any point, you can add columns to you Webhook Data Pool to evolve your schema with non-breaking changes.

Once you click the “Add Column” button, you will be able to define the new column. The fields includes the JSON property to extract from the event, the name of the new column, and the type.

Clicking “Add column” starts an asynchronous operation to add the column to the Data Pool. You can monitor the progress of the operation in the “Operations” tab.

Note that when you add a column, Propel will not backfill. To backfill existing rows, you can run a batch update operation.

Column deletions, column modifications, and data type changes are not supported because they are breaking changes to the schema.

Data types

The table below describes default data type mappings from JSON to Propel types. When creating a Webhook Data Pool, you can modify these default mappings. For instance, if you know that a column originally typed as a NUMBER contains a UNIX timestamp, you can convert it to a TIMESTAMP by changing the default mapping.

JSON type	Propel type
String	STRING
Number	DOUBLE
Object	JSON
Array	JSON
Boolean	BOOLEAN
Null	JSON

Key guides

• Querying JSON data
• Selecting the right primary timestamp column for your Data Pool
• Understanding and configuring Tenant ID for your data pool
• Defining Metrics with JSON Fields

Frequently Asked Questions

How long does it take for an event to be available via the API?

When an event is collected, the data will be available in Propel and served via the API in 1-3 minutes.

Management API

Below is the relevant API documentation for the Webhook Data Pool.

Queries

• Get Data Pool by unique name
• Get Data Pool by ID
• Get Data Source by unique name
• Get Data Source by ID

Mutations

• Create Data Pool
• Modify Data Pool
• Delete Data Pool by ID
• Delete Data Pool by unique name
• Create Webhook Data Source
• Modify Webhook Data Source
• Delete Data Source by ID
• Delete Data Source by unique name

Limits

• Up to 500 events (as a JSON array) can be sent for each POST.
• The payload can be up to 1,048,320 bytes.

Best practices

• Send as many events per request as you can, up to 500 or 1,048,320 bytes. This helps ingest them faster.
• Retry on 429 and 500 errors with exponential backoff. This ensures no events are dropped.
• Alert or page on any 413 Content Too Large error. This means the event count or content length was exceeded. It needs to be looked at, as retrying will not help.
• Create the Webhook Data Pool with all the required fields. Pasting a sample event and entering the Webhook Data Pool creation flow and clicking "Extract top-level fields" will automatically do this.
• Customize the primary timestamp to the proper event timestamp. Otherwise, the _propel_received_at will be used.
• We recommend marking all the fields except the primary timestamp and unique ID as not required to minimize 400 errors (see next point). Even if there is a data fill issue, you want Propel to accept the request, and then you can run an update to backfill any missing data instead of dropping the event.
• Alert or page on any 400 error. This means that the schema is incorrect and that Propel is rejecting the request. Why it's happening needs to be investigated, retrying will not help.
• Double-check all the types. For objects, arrays, and dictionaries, make sure they are JSON in Propel.
• Enable basic auth on the Webhook Data Pool to secure the endpoint.