How to ingest JSON events into Propel
This guide explains how to send JSON events to Propel over HTTPS from any application or SaaS service.
It covers how to set up a Webhook Data Pool, send events to it, and see the events arrive in the Data Pool.
- You have a Propel account.
Step 1: Create a Webhook Data Pool
As a first step, you need to create a Webhook Data Pool. In the Console, click on “Data Pools” in the left-hand menu. Click on “Create Data Pool” and select “Webhook”.
Define the schema
Next, you need to define the schema for your new Data Pool. You’ll notice that the default schema contains two columns:
|_propel_received_at||TIMESTAMP||The timestamp when the event was collected in UTC.|
|_propel_payload||JSON||The JSON Payload of the event.|
If you do no further editing, you can create your Webhook Data Pool as is, and any JSON sent to this webhook will be stored and available for querying in
However, take note of the Payload section on the right-hand side. Here, you can type or paste JSON for a sample event you intend to send to Propel. This lets you unpack top-level or nested keys from the incoming JSON event into specific columns. This feature provides convenience for commonly used keys, the ability to use them as Metric dimensions, and is particularly useful for setting an appropriate timestamp and tenant ID that are included as part of the payload.
For this guide, we will use the same TacoSoft sample data from the Quickstart. The example JSON event we will be collecting looks like this:
So, if we add our TacoSoft event JSON payload, we should see the following:
Note that Propel has confirmed the validity of the JSON. Propel provides the convenience of extracting all top-level or nested properties into the schema with just one click. In this example, we will click on "Extract nested properties". This action will create columns representing our sample event's nested JSON keys.
After clicking that, you should see this:
All nested keys are now part of our schema definition. The system has also auto-detected and set the appropriate data type, though the user can override this.
Before clicking Next, set
created_at as your primary Timestamp. We’ll want the time when the event was created, not when it was received, as the primary timestamp. If you need to control access to this data based on customer ID for multi-tenant SaaS applications, you can optionally specify the
customer_id column as your Tenant ID. However, in this example, we will not select a Tenant ID. Click Next.
Make sure your sample event and the schema you create are correct. A schema cannot be modified after a data pool is created.
Optionally, you can specify a username and password to enable HTTP basic authentication on the webhook URL that events will be sent to. For testing purposes, this can be left blank and edited later. Click on Next.
Set a name and description
Lastly, enter a unique name and description for your new Data Pool and click “Next”. For this example, we’ll name our new Data Pool “TacoHook”, and the description would be “Data Pool for Taco Orders received.”
Once your Webhook Data Pool is created, you will see the HTTP URL to which you can post JSON data on the overview screen. In our example, the Data Pool would be named "TacoHook".
Step 2: Send events to the Webhook Data Pool
You can send a batch of JSON events to the Webhook Data Pool by making an HTTP POST request to the Data Pool's URL. One way to easily test this is by using cURL from a terminal window. For example:
curl https://webhooks.us-east-2.propeldata.com/v1/WHK... \
-X POST \
-H "Content-Type: application/json" \
It is valid to send a single JSON object as an event or an array of JSON objects like the above example.
You should expect a 200 OK with multiple “Event processed successfully” messages in the body of the response, one for each event.
Step 3: See your events in the Data Pool
Lastly, we'll want to see the events we just sent in our new Data Pool. Click on “Data Pools”, and then select our “TacoHook” Data Pool.
Head over to the “Preview Data” tab, and you should see the events data as POSTed, reflected in the schema we defined.
It may take 1-3 minutes for events to process and show in the “Preview Data” page.
That's it! To recap, we created a Webhook Data Pool in Propel, sent data into Propel using it, and verified that it arrived in the Data Pool successfully.
You can learn more about using the GraphQL API you set up and check the examples.