Today, we’re launching a new way to power your customer-facing analytics: Webhooks. In addition to syncing data from your database and data warehouse, you can now send JSON events from your application or SaaS platform directly into Propel via an HTTP POST request.
What are Webhooks?
Webhooks are HTTP requests triggered by specific events in applications. When an event occurs, the application sends the event, typically in JSON format, to the specified URL as an HTTP request, i.e., the webhook. They are a simple yet powerful way to push data from one application to another, making them ideal for real-time analytics.
How Lumeo powers computer vision analytics with Webhooks
Lumeo enables enterprises to run computer vision pipelines on their installed commercial camera systems. When processing the video streams, Lumeo’s platform publishes the results of its computer vision models to Propel in real time via webhooks to power their customer-facing dashboards.
Bring data together from multiple sources
Webhooks provide the flexibility to ingest data from various data sources in real-time, making your analytics more comprehensive.
Webhooks can easily integrate with your existing applications, SaaS platforms, or streaming infrastructure, requiring minimal changes. This ensures you can get the data you need to power your analytics without expensive data engineering projects.
With Webhooks, you can process data the moment it’s generated, enabling you to provide real-time insights. This is particularly useful for applications dealing with fast-changing data or requiring immediate response based on the incoming data.
Webhooks simplify the data pipeline by reducing the steps and tools needed to ingest data. By directly pushing data from your application to Propel, you eliminate the need for intermediary data storage or transformation tools, simplifying your data architecture and reducing potential points of failure.
Webhooks are designed with scalability in mind. As your data grows, Webhooks can easily handle increased volumes, ensuring your analytics continue to run smoothly and efficiently.
Flexible schema definition and automatic JSON flattening
With Propel's Webhooks, you can customize the schema according to your JSON payload. It allows you to unpack and flatten both top-level and nested fields. Each unpacked field will then become its own column in the Data Pool. As events are received in the Webhook Data Pool the values are flattened into their corresponding columns.
Individual or batch event collection
You can send individual events or batches of JSON events to the Webhook Data Pool by making an HTTP POST request to the Data Pool's URL.
The examples below demonstrate how to send individual events and batches of events for the TacoSoft dataset from the Quickstart.
Sending individual events
Sending batches of events
The Webhook Data supports real-time updates. It uses each record’s primary key, consisting of the primary timestamp and a unique ID, to determine whether it should be inserted or updated within the Data Pool. This also means that if you send the same record twice, it will not be duplicated.
Updates are useful in the following scenarios:
- When retrying requests: You can safely retry requests without worrying about creating duplicates.
- When updating the properties of an object: For example, if you need to update the "taco_count" property of the "order_details" object from "7" to "8.”
You can optionally secure the Webhook Data Pool URL with HTTP basic authentication. This prevents third-party actors from posting events to your endpoint without your consent.
It’s available right now to all customers!
Now that you understand the power and flexibility of Webhooks in real-time data ingestion, it's time to start building! Whether you're a developer aiming to build analytics, or a data engineer seeking to serve your data through a low-latency API, Webhooks will accelerate your customer-facing analytics efforts.