Kafka setup guide

Kafka doesn’t manage users directly; it relies on the underlying authentication system. So if you’re using SASL/PLAIN for authentication, you would add the user to the JAAS configuration file.

1. Open the JAAS configuration file (e.g., kafka_server_jaas.conf) in a text editor.
2. Add the following entry to create the user “propel” with the password:

KafkaServer {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  username="propel"
  password="YOUR_SUPER_SECURE_PASSWORD"
  user_propel="YOUR_SUPER_SECURE_PASSWORD";
};

Replace YOUR_SUPER_SECURE_PASSWORD with a secure password.

3. Save the file.

Set the KAFKA_OPTS environment variable to point to your JAAS config file:

export KAFKA_OPTS="-Djava.security.auth.login.config=/path/to/your/kafka_server_jaas.conf"

Restart the Kafka server for the changes to take effect.

Now, you’ll use Kafka’s Access Control Lists (ACLs) to grant permissions to the “propel” user.

Use the kafka-acls CLI to add ACLs for the “propel” user so that it can operate on the propel-* consumer groups.

bin/kafka-acls.sh \
  --authorizer-properties zookeeper.connect=localhost:2181 \
  --add \
  --allow-principal 'User:propel' \
  --operation Describe \
  --operation Read \
  --operation Delete \
  --group 'propel-' --resource-pattern-type prefixed

For each topic you need to ingest to Propel, run the following command:

bin/kafka-acls.sh \
  --authorizer-properties zookeeper.connect=localhost:2181 \
  --add \
  --allow-principal 'User:propel' \
  --operation Describe \
  --operation Read \
  --topic 'YOUR_TOPIC'

Make sure to replace localhost:2181 with your Zookeeper server.

These commands grant Describe and Read access to the topics for the user “propel”.

Verify that the ACLs have been correctly set by listing the ACLs for the topics you authorized.

bin/kafka-acls.sh \
  --authorizer-properties zookeeper.connect=localhost:2181 \
  --list \
  --topic YOUR_TOPIC

You should see the ACLs you added for the user “propel”.

In Confluent Cloud, you generally use API keys for authentication rather than user/password combinations.

1. Log into your Confluent Cloud account.
2. Go to the “Environments” section in the sidebar and click on your environment.
3. Click on the “Clusters” tab and select your cluster.
4. Click on “API Keys” and then on “Create key”.
5. Choose the “Service account”, then “Create new one” and name it “Propel”.
6. In the “Add ACLs to service account” section, assign:
   • DESCRIBE and READ operations to the topic you need to ingest to Propel.
   • DESCRIBE, READ and DELETE operations to the consumer group propel-*.

7. Click “Next” and get your Key and Secret that you will use as a user and password to connect to your Kafka cluster.

After setting the permissions, you can verify them by clicking on the API key in the “API Keys” section and reviewing the roles and resources it has access to.

1. Sign in to the AWS IAM Management Console.

2. In the navigation pane, choose “Users” and then choose “Create User”.

3. For “Username”, enter “propel” and click “Next”.

4. Select “Attach policies directly”, search and select the AmazonMSKReadOnlyAccess policy, and click “Next”.

5. Then, create a custom policy that grants describe, read, and delete permissions on MSK consumer groups prefixed with “propel-”:

   1. Click “Create policy”, then choose the JSON tab.
   2. Paste the following JSON policy into the editor:
   Copy

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "kafka-cluster:DescribeGroup",
           "kafka-cluster:DeleteGroup",
           "kafka-cluster:ReadGroup"
         ],
         "Resource": "arn:aws:kafka:<region>:<account-id>:group/<cluster-name>/propel-*"
       }
     ]
   }
   

   3. Replace <region>, <account-id>, and <cluster-name> with your actual AWS region, account ID, and MSK cluster name.
   4. Click “Review policy”, give your policy a name (e.g., “PropelMSKPolicy”), and click “Create policy”.

1. Click on the user “propel” you just created.
2. Click on the “Security credentials” tab and click on “Create access key”.
3. Select “Other” and click “Next”.
4. Enter any tags and click “Create access key” (optional: add metadata to the user by attaching tags as key-value pairs).
5. You now have the access key and secret you can use as user and password to connect to your Kafka cluster. Save these credentials securely, as you will not have access to the secret access key again after this step.

Follow these steps to enable SASL/SCRAM for authentication with Redpanda.

1. Edit the Redpanda configuration file (usually located at /etc/redpanda/redpanda.yaml).

2. Enable SASL/SCRAM by adding or updating the following configuration:

redpanda:
  kafka_api:
    - address: 0.0.0.0
      port: 9092
      name: sasl_listen

See Redpanda docs for enabling SASL/SCRAM.

3. Enable TLS encryption.

SASL provides authentication, but not encryption. To enable SASL authentication with TLS encryption for the Kafka API, in redpanda.yaml, enter:

redpanda:
  kafka_api:
    - address: 0.0.0.0
      port: 9092
      name: sasl_tls_listener
      authentication_method: sasl
  kafka_api_tls:
    - name: sasl_tls_listener
      key_file: broker.key
      cert_file: broker.crt
      truststore_file: ca.crt
      enabled: true
      require_client_auth: false

See Redpanda docs for enabling TLS encryption.

4. Confirm the SCRAM mechanism is enabled

To check if SASL/SCRAM is enabled, run the following command:

rpk cluster config get sasl_mechanisms

You should see SCRAM in the output.

See Redpanda docs for checking SASL/SCRAM.

5. Restart Redpanda to apply the changes:

Restart your Redpanda server to apply the changes.

Redpanda uses rpk, a command-line tool, to manage users and ACLs.

Create the user “propel”:

rpk acl user create propel --password '<YOUR_SUPER_SECURE_PASSWORD>' --mechanism SCRAM-SHA-256

Replace <YOUR_SUPER_SECURE_PASSWORD> with a secure password.

Grant DESCRIBE, READ, and DELETE permissions to the “propel” user for the topics you need to ingest. Use the rpk acl command to add ACLs for the “propel” user so that it can operate on the propel-* consumer groups.

rpk acl create --allow-principal 'User:propel' --operation describe --operation read --operation delete --resource-pattern-type prefixed --group 'propel-'

For each topic you need to ingest to Propel, run the following command:

rpk acl create --allow-principal 'User:propel' --operation describe --operation read --topic YOUR_TOPIC

These commands grant DESCRIBE and READ access to the topic “YOUR_TOPIC” for the user “propel”.

Verify that the ACLs have been correctly set by listing the ACLs for the topic “YOUR_TOPIC”:

rpk acl list --topic YOUR_TOPIC

You should see the ACLs you added for the user “propel”.

Go to the “Data Pools” section in the Console, click “Create Data Pool” and click on the “Kafka” tile.

If you create a Kafka Data Pool for the first time, you must create your Kafka credentials for Propel to connect to your Kafka servers.

To create your Kafka credentials, you will need the following details:

• Bootstrap servers: The list of addresses for your Kafka cluster’s brokers.
• Authentication type: The authentication protocol used by your Kafka cluster: SASL/SCRAM-SHA-256, SASL/SCRAM-SHA-512, SASL/PLAIN, or NONE.
• TLS: Whether your Kafka cluster uses TLS for secure communication.
• Username: The username for the user you created in your Kafka cluster.
• Password: The password for the user you created in your Kafka cluster.

After entering your Kafka credentials, click “Create and test credentials” to ensure Propel can successfully connect to your Kafka cluster. If the connection is successful, you will see a confirmation message. If not, check your entered credentials and try again.

Here, you will see a list of topics available to ingest. If you don’t see the topic you want to ingest, make sure your user has the right permissions to access the topic.

Here, you will see a list of topics available to ingest. Select the topic you want to ingest into this Data Pool. You will see the schema of the Data Pool.

Next, you need to select the timestamp column. This is the column that will be used to order the data in the Data Pool. By default, Propel selects the _timestamp generated by Kafka.

After you’ve selected the topic, provide a name for your Data Pool. This name will be used to identify the Data Pool in Propel. Once you’ve named your Data Pool, click “Create Data Pool”. Propel will then start ingesting data from the selected Kafka topics into your Data Pool.

Once you’ve started ingesting data, you can view the data in your Data Pool. Go to the “Data Pools” section in the Console, click on your Kafka Data Pool, and click on the “Preview Data” tab. Here, you can see the data that has been ingested from your Kafka topic.