Home>Articles>Kafka SSL Client Authentication in Multi-Tenancy Architecture
cloud_hand_users
Articles

Kafka SSL Client Authentication in Multi-Tenancy Architecture

Apache Kafka is the key product for not only messaging transformations but also real-time data processing, in addition to many other use cases. Architectures hosted inside the cloud claim to be secure in terms of communication and providing general security. But when it comes to the multiple client/consumer communication from a server/producer, Kafka provides in-built support for SSL as well as user-based authentication. In the below article, we will set up such an authentication mechanism step-by-step.

kafka Server client Authentication graphic fastnforward

 

The step by step solution is divided into three parts:

  1. SSL support for one or more brokers: Generate the key and the certificate for each machine in the cluster. You can use Java’s KeyTool utility to accomplish this task. We will generate the key into a temporary KeyStore initially so that we can export and sign it later with CA.
  2. Kafka Configurations (We used Kafka 2.11-2.3.0).
  3. Running the whole set up.

Instruction to install this Use Case

  1. SSL support for one or more brokers

We will use Java’s key tool utility to accomplish this task. We will generate the key into a temporary KeyStore initially so that we can export and sign it later with CA.

We are going to use one Kafka server and two clients (consumers). Also, here we are using self-signed certificates else we need to have TrustStore and KeyStore JKSs for each server.

Points to note:

  1. Please create a folder for creating and keeping all cert files.
  2. Please provide identical details and passwords for all. In my case I have used,

issuer = C = de, ST = RF, L = Mainz, O = Technaura, OU = consulting, CN = swarnava.c, emailAddress = swarnava.c @technaura.com

Only the CN for client2 I have given a different user for testing purpose. Please generate your certificate carefully, else there will be a problem in the next part

keytool - keystore kafka.server.keystore.jks - alias localhost - validity 365 - genkey - keyalg RSA

openssl req – new – x509 – keyout ca – key – out ca – cert – days 365
keytool - keystore kafka.server.truststore.jks - alias CARoot -
import -file ca - cert
keytool - keystore kafka.client1.truststore.jks - alias CARoot -
import -file ca - cert
keytool - keystore kafka.client2.truststore.jks - alias CARoot -
import -file ca - cert
keytool - keystore kafka.server.keystore.jks - alias localhost - certreq - file cert - file
openssl x509 - req - CA ca - cert - CAkey ca - key - in cert - file - out cert - signed - days 365 - CAcreateserial
keytool - keystore kafka.server.keystore.jks - alias CARoot -
import -file ca - cert
keytool - keystore kafka.server.keystore.jks - alias localhost -
import -file cert - signed
keytool - keystore kafka.client1.keystore.jks - alias localhost - validity 365 - genkey - keyalg RSAkeytool - keystore kafka.client1.keystore.jks - alias localhost - certreq - file cert - file
openssl x509 - req - CA ca - cert - CAkey ca - key - in cert - file - out cert - signed - days 365 - CAcreateserial
keytool - keystore kafka.client1.keystore.jks - alias CARoot -
import -file ca - cert
keytool - keystore kafka.client1.keystore.jks - alias localhost -
import -file cert - signed
keytool - keystore kafka.client2.keystore.jks - alias localhost - validity 365 - genkey - keyalg RSA
keytool - keystore kafka.client2.keystore.jks - alias localhost - certreq - file cert - file
keytool - keystore kafka.client2.keystore.jks - alias localhost - certreq - file cert - file
openssl x509 - req - CA ca - cert - CAkey ca - key - in cert - file - out cert - signed - days 365 - CAcreateserial
keytool - keystore kafka.client2.keystore.jks - alias CARoot -
import -file ca - cert

keytool - keystore kafka.client2.keystore.jks - alias localhost -
import -file cert - signed

Once everything is generated you can see the generated files,

λ ls

ca-cert ca-key cert-signed kafka.client1.truststore.jks kafka.client2.truststore.jks kafka.server.truststore.jks ca-cert.srl cert-file kafka.client1.keystore.jks kafka.client2.keystore.jks kafka.server.keystore.jks

B. Kafka Configuration

B.1 Changing the server.properties file with below lines,

listeners=PLAINTEXT://localhost:9092,SSL://localhost:9093

Also add:

ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.endpoint.identification.algorithm=

ssl.keymanager.algorithm=SunX509


ssl.keystore.location=<path-to-file>/kafka.server.keystore.jks


ssl.keystore.password=<password>


ssl.keystore.type=JKS


ssl.protocol=TLS


ssl.trustmanager.algorithm=PKIX


ssl.truststore.location=<path-to-file>/kafka.server.truststore.jks


ssl.truststore.password=<password>


ssl.truststore.type=JKS


authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer


allow.everyone.if.no.acl.found=true


ssl.client.auth=required

B.2 Creating new files client-ssl.properties, client-ssl1.properties and client-ssl2.properties inside kafka_2.11-2.3.0\config respectively

security.protocol = SSL
ssl.truststore.location = < path - to - file > /kafka.client2.truststore.jks
ssl.truststore.password = < password >
ssl.enabled.protocols = TLSv1 .2, TLSv1 .1, TLSv1
ssl.truststore.type = JKS
ssl.endpoint.identification.algorithm =
ssl.keystore.location = < path - to - file > /kafka.client2.keystore.jks
ssl.keystore.password = < password >
ssl.key.password = < password >

 

security.protocol = SSL
ssl.truststore.location = C: /Work/Software / NewKafkaZoo / ssl - bothways / kafka.client1.truststore.jks
ssl.truststore.password = < password >
ssl.enabled.protocols = TLSv1 .2, TLSv1 .1, TLSv1
ssl.truststore.type = JKS
ssl.endpoint.identification.algorithm =
ssl.keystore.location = C: /Work/Software / NewKafkaZoo / ssl - bothways / kafka.client1.keystore.jks
ssl.keystore.password = < password >
ssl.key.password = < password >

 

security.protocol = SSL
ssl.truststore.location = < path - to - file > /kafka.client2.truststore.jks
ssl.truststore.password = < password >
ssl.enabled.protocols = TLSv1 .2, TLSv1 .1, TLSv1
ssl.truststore.type = JKS
ssl.endpoint.identification.algorithm =
ssl.keystore.location = < path - to - file > /kafka.client2.keystore.jks
ssl.keystore.password = < password >
ssl.key.password = < password >

B.3 Running Kafka and Zookeeper first,

Zkserver

.\bin\windows\kafka-server-start.bat .\config\server.properties

B.4 Opening a new terminal and creating a new topic and checking the same,

.\bin\ windows\ kafka - topics.bat--create--zookeeper localhost: 2181--topic test1--partitions 1--replication - factor 1

.\bin\ windows\ kafka – topics.bat–list–zookeeper localhost: 2181

B.5 Checking the created certificate

openssl s_client - debug - connect localhost: 9093 - tls1

It will return you the below details at the end. It means your certificate is generated properly.

Wtmz24ChQdgNcygKXLq1AHgDetoHz57hrx5f75 / gh31nDdgHpv4xKyO40TSIH + 8 v

PqgvvrogH0lgLCwsJfqwPEJbWZjL6pvLsBfPB8NMICMXpL50ZA ==
-- -- - END CERTIFICATE-- -- -

subject = C = de, ST = RF, L = Mainz, O = Technaura, OU = consulting, CN = swarnava.c
issuer = C = de, ST = RF, L = Mainz, O = Technaura, OU = consulting, CN = swarnava.c, emailAddress = swarnava.c @technaura.com
-- -

Acceptable client certificate CA names
C = de, ST = RF, L = Mainz, O = Technaura, OU = consulting, CN = swarnava.c, emailAddress = swarnava.c @technaura.com

Client Certificate Types: ECDSA sign, RSA sign, DSA sign

Peer signing digest: MD5 - SHA1
Peer signature type: RSA

Server Temp Key: ECDH, P - 256, 256 bits
-- -

New, TLSv1 .0, Cipher is ECDHE - RSA - AES256 - SHA
Server public key is 2048 bit

Secure Renegotiation IS supported

We will run the set up for three different scenarios, i.e. without authentication, only server-side authentication, server and client-side authentication.

C. Running the whole set up

Command for producing using console producer

.\bin\ windows\ kafka - console - producer.bat--broker - list < broker host: port > --topic < topic - name > --producer.config config\ < config file >

Command for consuming using console consumer

.\bin\ windows\ kafka - console - consumer.bat--bootstrap - server < server host: port > --topic < topic - name > --consumer.config config\ < config file >

C.1 Without Authentication-

PRODUCER

Snap shot of SSL Authentication set up Producer

CONSUMER

snapshot of consumer SSL authentication setup

C.2 Only Server-side Authentication

Created another topic ‘test2’

PRODUCER

Snap shot of producer server-side Authentication

 

CONSUMER 1

Snapshot of server side authentication Consumer side

CONSUMER 2

Snapshot of server side authentic Consumer2

Note: Please check the config files used.

C.3 Server and Client-side Authentication-

For authorization of topic – ‘test2’ only for the user – swarnava.c use below two commands.

.\bin\ windows\ kafka - acls.bat--authorizer kafka.security.auth.SimpleAclAuthorizer--authorizer - properties zookeeper.connect = localhost: 2181--add--allow - principal User: "CN=swarnava.c,OU=consulting,O=Technaura,L=Mainz,ST=RF,C=de"--cluster--producer--topic test2

 

.\bin\ windows\ kafka - acls.bat--authorizer kafka.security.auth.SimpleAclAuthorizer--authorizer - properties zookeeper.connect = localhost: 2181--add--allow - principal User: "CN=swarnava.c,OU=consulting,O=Technaura,L=Mainz,ST=RF,C=de"--group = * --consumer--topic test2

Snapshot of server and client side authentication

PRODUCER

snapshot of testing server client side authentication

CONSUMER 1 (With User – swarnava.c)

Snapshot of server client side authentication test

CONSUMER 2 (Without User – swarnava.c)

Snapshot of client server side authentication test

Congrats. You are done. This is the complete implementation of the SSL in Kafka.

 

Thumbs up green emoji

 

Written by Swarnava Chakraborty. Swarnava is a Technical Lead (consultancy and delivery) at Technaura Systems GmbH.

For similar articles, please read further:

What is SSL/TSL Client Authentication?
SSL Client Authentication: It’s a matter of trust
Multi-Tenant vs Single Tenant Architecture
Failure detection and alert from real-time data flow using Kafka and KSQL