This guide covers how to:
  1. Create a user in your Kafka cluster
  2. Make sure your Kafka cluster is accessible from Propel IPs
  3. Create a Kafka Data Pool in Propel

​
Requirements

  • A Propel account.
  • A Kafka cluster with the topics to ingest.
  • Access to create users and grant permissions in your Kafka cluster.

​
1. Create a user in your Kafka cluster

First, you’ll need to create a user with the necessary permissions for Propel to connect to your Kafka cluster.
1

Create the user

Kafka doesn’t manage users directly; it relies on the underlying authentication system. So if you’re using SASL/PLAIN for authentication, you would add the user to the JAAS configuration file.
  1. Open the JAAS configuration file (e.g., kafka_server_jaas.conf) in a text editor.
  2. Add the following entry to create the user “propel” with the password:
KafkaServer {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  username="propel"
  password="YOUR_SUPER_SECURE_PASSWORD"
  user_propel="YOUR_SUPER_SECURE_PASSWORD";
};
Replace YOUR_SUPER_SECURE_PASSWORD with a secure password.
  1. Save the file.
2

Set environment variable

Set the KAFKA_OPTS environment variable to point to your JAAS config file:
export KAFKA_OPTS="-Djava.security.auth.login.config=/path/to/your/kafka_server_jaas.conf"
Restart the Kafka server for the changes to take effect.
3

Grant permissions

Now, you’ll use Kafka’s Access Control Lists (ACLs) to grant permissions to the “propel” user.Use the kafka-acls CLI to add ACLs for the “propel” user so that it can operate on the propel-* consumer groups.
bin/kafka-acls.sh \
  --authorizer-properties zookeeper.connect=localhost:2181 \
  --add \
  --allow-principal 'User:propel' \
  --operation Describe \
  --operation Read \
  --operation Delete \
  --group 'propel-' --resource-pattern-type prefixed
For each topic you need to ingest to Propel, run the following command:
bin/kafka-acls.sh \
  --authorizer-properties zookeeper.connect=localhost:2181 \
  --add \
  --allow-principal 'User:propel' \
  --operation Describe \
  --operation Read \
  --topic 'YOUR_TOPIC'
Make sure to replace localhost:2181 with your Zookeeper server.These commands grant Describe and Read access to the topics for the user “propel”.
4

Verify the ACLs

Verify that the ACLs have been correctly set by listing the ACLs for the topics you authorized.
bin/kafka-acls.sh \
  --authorizer-properties zookeeper.connect=localhost:2181 \
  --list \
  --topic YOUR_TOPIC
You should see the ACLs you added for the user “propel”.

​
2. Make sure your Kafka cluster is accessible from Propel IPs.

To ensure that Propel can connect to your Kafka cluster, you need to authorize access from the following IP addresses: 
18.219.73.236
3.15.73.135
3.17.239.162

​
3. Create a Kafka Data Pool

1

Create a Kafka Data Pool

Go to the “Data Pools” section in the Console, click “Create Data Pool” and click on the “Kafka” tile.
Select Kafka Data Pool
If you create a Kafka Data Pool for the first time, you must create your Kafka credentials for Propel to connect to your Kafka servers.
Kafka Data Pool create new credentials
2

Create your Kafka credentials

To create your Kafka credentials, you will need the following details:
  • Bootstrap servers: The list of addresses for your Kafka cluster’s brokers.
  • Authentication type: The authentication protocol used by your Kafka cluster: SASL/SCRAM-SHA-256, SASL/SCRAM-SHA-512, SASL/PLAIN, or NONE.
  • TLS: Whether your Kafka cluster uses TLS for secure communication.
  • Username: The username for the user you created in your Kafka cluster.
  • Password: The password for the user you created in your Kafka cluster.
Create Kafka credentials
3

Test your credentials

After entering your Kafka credentials, click “Create and test credentials” to ensure Propel can successfully connect to your Kafka cluster. If the connection is successful, you will see a confirmation message. If not, check your entered credentials and try again.
Kafka credentials test
4

Introspect your Kafka topics

Here, you will see a list of topics available to ingest. If you don’t see the topic you want to ingest, make sure your user has the right permissions to access the topic.
Kafka topic introspection
5

Select the topic to ingest and timestamp

Here, you will see a list of topics available to ingest. Select the topic you want to ingest into this Data Pool. You will see the schema of the Data Pool.
Create Kafka Data Pool select topic
Next, you need to select the timestamp column. This is the column that will be used to order the data in the Data Pool. By default, Propel selects the _timestamp generated by Kafka.
Create Kafka Data Pool select timestamp
6

Name your Data Pool and start ingesting

After you’ve selected the topic, provide a name for your Data Pool. This name will be used to identify the Data Pool in Propel. Once you’ve named your Data Pool, click “Create Data Pool”. Propel will then start ingesting data from the selected Kafka topics into your Data Pool.
Create Kafka Data Pool select name
7

Look at the data in your Data Pool

Once you’ve started ingesting data, you can view the data in your Data Pool. Go to the “Data Pools” section in the Console, click on your Kafka Data Pool, and click on the “Preview Data” tab. Here, you can see the data that has been ingested from your Kafka topic.
Data Pool preview data