Skip to main content

Webhooks

The Webhook Data Pool ingests JSON events via an HTTP URL from any application or SaaS service. Once the events are in the Propel Data Pool, you can use them to power your analytic dashboards, reports, and workflows.

Consider using the Webhook Data Pool when:

  • You need to power real-time analytics with streaming ingestion.
  • You need to send events from your application or a third-party SaaS service.
  • You are working with semi-structured JSON data.
  • You are capturing change events from a NoSQL database.
  • You are using Segment, Rudderstack, or Snowplow to collect events.
  • You are collecting events from OpenAI API calls.

Get started: Ingesting JSON events into Propel​

Set up guide

Follow our step-by-step guide on How to ingest JSON events into Propel.

Architecture Overview​

Webhook Data Pools provides an HTTP URL to which you can send events from your applications, streaming infrastructure, or SaaS applications.

The architectural overview when sending Webhooks to Propel.

Features​

Webhook Data Pools supports the following features:

Feature nameSupportedNotes
Event collection✅Collects events in JSON format.
Real-time updates✅See the Real-time updates section.
Real-time deletes❌Real-time deletes are not supported.
Batch deletes✅See the Batch deletes section.
Bulk insert✅Up to 500 events per HTTP request.
Delete Job API✅See Delete Job API.
API configurable✅See API reference docs.
Terraform configurable✅See Propel Terraform docs.

How does the Webhook Data Pool work?​

When you create a Webhook Data Pool, you will get an HTTP URL to which you can post JSON events. These events will be collected in the Data Pool and made available via the APIs to power your analytics.

A screenshot of the Webhook Data Pool URL in the Propel Console

By default, the Webhook Data Pool has two columns: _propel_received_at and propel_payload.

ColumnTypeDescription
_propel_received_atTIMESTAMPThe timestamp when the event was collected in UTC.
_propel_payloadJSONThe JSON Payload of the event.

During the setup of a Webhook Data Pool, you can optionally unpack top-level or nested keys from the incoming JSON event into specific columns. This provides convenience for commonly used keys, the ability to use them as Metric dimensions, and is particularly useful for setting an appropriate timestamp and tenant ID included in the payload.

See our step-by-step guide on How to ingest JSON events into Propel.

Authentication​

Optionally, you can add basic authentication to your HTTP endpoint to secure your URL. If these parameters are not provided, anyone with the URL to your webhook will be able to upload data. While it's okay to test without HTTP Basic authentication, we recommend enabling it.

A screenshot of the Webhook Data Pool authentication setup in the Propel Console

Sending individual JSON events​

You can send individual JSON events to the Webhook Data Pool by making an HTTP POST request to the Data Pool URL with the JSON event in the request body.

curl https://webhooks.us-east-2.propeldata.com/v1/WHK... \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "customer_id": 5,
    "order_id": 34,
    "store_id": 4445,
    "order_details": {
      "taco_count": 7,
      "total_price": 25.90,
      "checkout_time": "2023-07-31T15:20:10Z"
    },
    "created_at": "2023-07-31T14:50:35Z"
  }'

Sending a batch of JSON events​

You can send a batch of JSON events to the Webhook Data Pool by making an HTTP POST request to the Data Pool URL with a JSON array of events in the request body. Each request can have a maximum of 500 events in the array.

curl https://webhooks.us-east-2.propeldata.com/v1/WHK... \
-X POST \
-H "Content-Type: application/json" \
-d '[
  {
    "customer_id": 5,
    "order_id": 34,
    "store_id": 4445,
    "order_details": {
      "taco_count": 7,
      "total_price": 25.90,
      "checkout_time": "2023-07-31T15:20:10Z"
    },
    "created_at": "2023-07-31T14:50:35Z"
  },
  {
    "customer_id": 8,
    "order_id": 22,
    "store_id": 1199,
    "order_details": {
      "taco_count": 3,
      "total_price": 15.75,
      "checkout_time": "2023-07-31T12:40:21Z"
    },
    "created_at": "2023-07-31T12:30:55Z"
  }
]'

If there are more than 500 events in a single request, Propel will return an HTTP 400 BAD REQUEST error.

Real-time updates​

The Webhook Data Pool supports real-time updates. It uses each record’s primary key, consisting of the primary timestamp and a unique ID, to determine whether it should be inserted or updated within the Data Pool. This also means that if you send the same record twice, it will not be duplicated.

Updates are useful in the following scenarios:

  • When retrying requests: You can safely retry requests without worrying about creating duplicates.
  • When updating the properties of an object: For example, if you need to update the "taco_count" property of the "Order" object from "7" to "8.”

To update a record, make an HTTP POST request to the Webhook Data Pool URL with the primary timestamp and unique ID of the record you want to update.

The following example demonstrates how to update the taco_count and total_price fields in the record below. The record's primary timestamp is created_at and its unique ID is order_id.

{
  "customer_id": 5,
  "order_id": 34,
  "store_id": 4445,
  "order_details": {
    "taco_count": 7,
    "total_price": 25.9,
    "checkout_time": "2023-07-31T15:20:10Z"
  },
  "created_at": "2023-07-31T14:50:35Z"
}

To update the taco_count to "8" and update the total_price to "28.50", you need to make the following HTTP POST request. Note that the created_at (primary timestamp) and its order_id (unique ID) fields should remain the same as the original record to update it.

curl https://webhooks.us-east-2.propeldata.com/v1/WHK... \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "customer_id": 5,
    "order_id": 34,
    "store_id": 4445,
    "order_details": {
      "taco_count": 8,
      "total_price": 28.50,
      "checkout_time": "2023-07-31T15:20:10Z"
    },
    "created_at": "2023-07-31T14:50:35Z"
  }'

Schema evolution​

The Webhook Data Pool, at its core, handles semi-structured schema-less JSON data. That means you can add new properties to your payload whenever necessary. The entire payload will always be available in the _propel_payload column. There are a couple of considerations to take into account:

  • If you stop sending a field that you unpacked into its own column and marked it as required when you created the Data Pool, Propel will return an error.
  • Unpacking of fields into individual columns can only be done at Data Pool creation time.

Data types​

The table below describes default data type mappings from JSON to Propel types. When creating a Webhook Data Pool, you can modify these default mappings. For instance, if you know that a column originally typed as a NUMBER contains a UNIX timestamp, you can convert it to a TIMESTAMP by changing the default mapping.

JSON typePropel type
StringSTRING
NumberDOUBLE
ObjectJSON
ArrayJSON
BooleanBOOLEAN
NullJSON

Key guides​

Frequently Asked Questions​

How long does it take for an event to be available via the API?

When an event is collected, the data will be available in Propel and served via the API in 1-3 minutes.

API reference documentation​

Below is the relevant API documentation for the Webhook Data Pool.

Queries​

Mutations​

Limits​

  • Up to 500 events (as a JSON array) can be sent for each POST.