Skip to main content

How to ingest data from Kafka into Propel

This guide explains how to connect Kafka with Propel to create customer-facing analytics dashboards.

It covers how to:

  1. Create a user in your Kafka cluster
  2. Make sure your Kafka cluster is accessible from Propel IPs
  3. Create a Kafka Data Pool

Requirements​

  • You have a Propel account.
  • A Kafka cluster with the topics to ingest.
  • Access to create users and grant permissions in your Kafka cluster.

1. Create a user in your Kafka cluster​

In this section, you'll set up a user for Propel. This user will connect to your Kafka cluster and have the necessary access to the topics.


1.1 Create the user "propel"​

First, you'll need to create the user. Kafka doesn't manage users directly; it relies on the underlying authentication system. So if you're using SASL/PLAIN for authentication, you would add the user to the JAAS configuration file.

  1. Open the JAAS configuration file (e.g., kafka_server_jaas.conf) in a text editor.
  2. Add the following entry to create the user "propel" with the password:
KafkaServer {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  username="propel"
  password="YOUR_SUPER_SECURE_PASSWORD"
  user_propel="YOUR_SUPER_SECURE_PASSWORD";
};

Replace YOUR_SUPER_SECURE_PASSWORD with a secure password.

  1. Save the file.

1.2: Configure the Kafka Server to Use the JAAS Config​

  1. Set the KAFKA_OPTS environment variable to point to your JAAS config file:
export KAFKA_OPTS="-Djava.security.auth.login.config=/path/to/your/kafka_server_jaas.conf"
  1. Restart the Kafka server for the changes to take effect.

1.3: Grant Describe and Read Access to the "propel" User​

Now, you'll use Kafka's Access Control Lists (ACLs) to grant permissions to the "propel" user.

Use the kafka-acls CLI to add ACLs for the "propel" user so that it can operate on the "propel-*" consumer groups.

Make sure to replace localhost:2181 with your Zookeeper server.

bin/kafka-acls.sh \
  --authorizer-properties zookeeper.connect=localhost:2181 \
  --add \
  --allow-principal 'User:propel' \
  --operation Describe \
  --operation Read \
  --operation Delete \
  --group 'propel-' --resource-pattern-type prefixed

For each topic you need to ingest to Propel, run the following command:

bin/kafka-acls.sh \
    --authorizer-properties zookeeper.connect=localhost:2181 \
    --add \
    --allow-principal 'User:propel' \
    --operation Describe \
    --operation Read \
    --topic 'YOUR_TOPIC'

These commands grant Describe and Read access to the topics for the user "propel".

Step 1.4: Verify the ACLs​

Verify that the ACLs have been correctly set by listing the ACLs for the topics you authorized.

bin/kafka-acls.sh \
    --authorizer-properties zookeeper.connect=localhost:2181 \
    --list \
    --topic YOUR_TOPIC \

You should see the ACLs you added for the user "propel".

2. Make sure your Kafka cluster is accessible from Propel IPs.​

To ensure that Propel can connect to your Kafka cluster, you need to authorize access from the following IP addresses:

3.17.239.162
3.15.73.135
18.219.73.236

3. Create a Kafka Data Pool​

To create a Kafka Data Pool, go to the Data Pools section in the Console, click Create Data Pool and click on the Kafka tile.

Select Kafka Data Pool

If you create a Kafka Data Pool for the first time, you must create your Kafka credentials for Propel to connect to your Kafka servers.

Kafka Data Pool create new credentials

3.1 Create your Kafka credentials​

To create your Kafka credentials, you will need the following details:

  • Bootstrap servers: The list of addresses for your Kafka cluster's brokers.
  • Authentication type: The authentication protocol used by your Kafka cluster: SCRAM-SHA-256, SCRAM-SHA-512, PLAIN, or NONE.
  • TLS: Whether your Kafka cluster uses TLS for secure communication.
  • Username: The username for the user you created in your Kafka cluster.
  • Password: The password for the user you created in your Kafka cluster.

Create Kafka credentials

3.2 Test your credentials​

After entering your Kafka credentials, click Create and test credentials to ensure Propel can successfully connect to your Kafka cluster. If the connection is successful, you will see a confirmation message. If not, check your entered credentials and try again.

Kafka credentials test

3.3 Introspect your Kafka topics​

Here, you will see a list of topics available to ingest. If you don't see the topic you want to ingest, make sure your user has the right permissions to access the topic.

Kafka topic introspection

3.4 Select the topic to ingest and timestamp​

Here, you will see a list of topics available to ingest. Select the topic you want to ingest into this Data Pool. You will see the schema of the Data Pool.

Create Kafka Data Pool select topic

Next, you need to select the timestamp column. This is the column that will be used to order the data in the Data Pool. By default, Propel selects the _timestamp generated by Kafka.

Create Kafka Data Pool select timestamp

3.5 Name your Data Pool and start ingesting​

After you've selected the topic, provide a name for your Data Pool. This name will be used to identify the Data Pool in Propel. Once you've named your Data Pool, click Create Data Pool. Propel will then start ingesting data from the selected Kafka topics into your Data Pool.

Create Kafka Data Pool select name

3.6 Look at the data in your Data Pool​

Once you've started ingesting data, you can view the data in your Data Pool. Go to the Data Pools section in the Console, click on your Kafka Data Pool, and click on the Preview Data tab. Here, you can see the data that has been ingested from your Kafka topic.

Data Pool preview data

What's next?​

Now that you've ingested data from Kafka into Propel, you can begin using the APIs with the data in your Data Pool. Here are some suggestions for your next steps: