Updated: Aug 22, 2020
Apache Kafka is middleware solution for enterprise application. It was initiated by LinkedIn lead by Neha Narkhede and Jun Rao. Initially it was designed for monitoring and tracking system, later on it became part of one of the leading project of Apache.
Why Use Kafka?
Disk based persistence
Kafka Use Cases
1. Enterprise messaging system
Kafka has topic based implementation for message system. One or more consumers can consume the message and commit as per application need.
Suitable for both online and offline messaging consumer system.
2. Message Store with playback capability
Kafka provides the message retention on the topic. Retention of the message can be configured for the specified duration.
Each message is backed up with distributed file system.
Supports the storage size for 50K to 50 TB.
3. Stream processing
Kafka is capable enough to process the message in real time in batch mode or in message wise. it provides the aggregation of message processing for specified time window.
Download and Install Kafka
Kafka requires below JRE and Zookeeper. Download and Install the below components.
ZooKeeper : http://zookeeper.apache.org/releases.html
Installation (on Windows)
1. JDK Setup
Set the JAVA_HOME under system environment variables from the path Control Panel -> System -> Advanced system settings -> Environment Variables.
Search for a PATH variable in the “System Variable” section in “Environment Variables” dialogue box you just opened.
Edit the PATH variable and append “;%JAVA_HOME%\bin”
To confirm the Java installation just open cmd and type “java –version”, you should be able to see version of the java you just installed
2. Zookeeper Installation:
Goto your Zookeeper config directory. It would be zookeeper home directory (i.e: c:\zookeeper-3.4.10\conf)
Rename file "zoo_sample.cfg" to "zoo.cfg".
Open zoo.cfg in any text editor and Find & edit dataDir=/tmp/zookeeper to :\zookeeper-3.4.10\data.
Add entry in System Environment Variables as we did for Java.
Add in System Variables ZOOKEEPER_HOME = C:\zookeeper-3.4.10
Edit System Variable named "PATH" and append ;%ZOOKEEPER_HOME%\bin;
You can change the default Zookeeper port in zoo.cfg file (Default port 2181).
Run Zookeeper by opening a new cmd and type zkserver.
3. Kafka Setup:
Go to your Kafka config directory. For me its C:\kafka_2.10-0.10.2.0\config.
Edit file "server.properties" and Find & edit line "log.dirs=/tmp/kafka-logs" to "log.dir= C:\kafka_2.10-0.10.2.0\kafka-logs".
If your Zookeeper is running on some other machine or cluster you can edit " zookeeper.connect=localhost:2181" to your custom IP and port.
Goto kafka installation folder and type below command from a command line. \bin\windows\kafka-server-start.bat .\config\server.properties.
Your Kafka will run on default port 9092 & connect to zookeeper’s default port which is 2181.
Now create a topic with name “test.topic” with replication factor 1, in case one Kafka server is running(standalone setup).
If you have a cluster with more than 1 Kafka server running, you can increase the replication-factor accordingly which will increase the data availability and act like a fault-tolerant system.
Open a new command prompt in the location C:\kafka_2.11-0.9.0.0\bin\windows and type following command and hit enter.
kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test.topic
Creating a Producer
· Open a new command prompt in the location C:\kafka_2.11-0.9.0.0\bin\windows.