P-1 Setting Up a Kafka Cluster with Zookeeper for High Availability
Installation Guide : / installation-guide-for-kafka-cluster-with-...
Introduction:
Apache Kafka is a powerful distributed streaming platform known for its real-time data streaming capabilities. To harness its full potential, setting up a Kafka cluster with high availability is essential. This article will guide you through the process of creating a Kafka cluster with three Zookeeper nodes and three broker nodes, ensuring a robust and fault-tolerant environment for your data streaming needs.
Requirements:
Before diving into the setup, make sure you have the following prerequisites in place:
Install Java:
Kafka relies on Java to run, so ensure that all machines in your cluster have Java installed.
Kafka Binary:
Download the Kafka binary and copy it to all three machines where you plan to set up Kafka.
Setting Up Zookeeper Cluster with High Availability:
Zookeeper plays a crucial role in maintaining Kafka cluster metadata. Here's how to set up Zookeeper with high availability:
Configure DNS Entries:
Ensure DNS entries are properly configured on all servers. You can edit the /etc/hosts file to map server IP addresses to hostnames.
bash
Copy code
vim /etc/hosts
Download and Extract Zookeeper:
Download the Zookeeper distribution tarball on all nodes and extract it.
bash
Copy code
wget https://downloads.apache.org/kafka/3....
tar -zxvf kafka_2.12–3.4.0.tgz
Create a Common Directory:
Create a directory named /app on all nodes and move the Kafka binary to that directory.
bash
Copy code
mkdir /app
mv kafka_2.12–3.4.0 /app
Configure Zookeeper Properties:
Create and edit the zookeeper.properties file in the Kafka configuration directory (/app/kafka_2.12–3.4.0/config/zookeeper.properties) on all nodes. Add the following properties:
properties
Copy code
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/app/data/zookeeper
clientPort=2181
maxClientCnxns=0
admin.enableServer=false
server.1=192.168.213.133:2888:3888
server.2=192.168.213.134:2888:3888
server.3=192.168.213.135:2888:3888
Set Zookeeper IDs:
On each Zookeeper node, create a unique ID file in the /app/data/zookeeper directory:
On Zookeeper Node 1:
bash
Copy code
vi /app/data/zookeeper/myid
Add the value 1 and save the file.
Repeat this process on Zookeeper Node 2 and Node 3 with their respective IDs.
Start Zookeeper:
Start Zookeeper on all nodes:
bash
Copy code
./bin/zookeeper-server-start.sh -daemon ./config/zookeeper.properties
After a short while, Zookeeper services should be up and running.
Setting Up Kafka Cluster with Multiple Brokers:
Now that your Zookeeper cluster is running smoothly, it's time to set up the Kafka cluster with multiple brokers:
Create Kafka Data Directory:
Create a directory for Kafka data on all nodes:
bash
Copy code
mkdir /app/data/kafka-logs
Configure Server Properties:
Edit the server.properties file in the Kafka configuration directory on each node. Customize the following properties, ensuring that each broker has a unique broker.id:
On Centos-1:
bash
Copy code
vim /app/kafka_2.12–3.4.0/config/server.properties
properties
Copy code
broker.id=0 # unique value
listeners=PLAINTEXT://localhost:9092
delete.topic.enable=true
auto.create.topics.enable=false
advertised.listeners=PLAINTEXT://localhost:9092
log.dirs=/app/data/kafka-logs
num.partitions=3
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=2
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=2
zookeeper.connect=192.168.213.133:2181,192.168.213.134:2181,192.168.213.135:2181
Repeat this process on Centos-2 and Centos-3, incrementing the broker.id accordingly.
Start Kafka Brokers:
Start the Kafka brokers on all nodes:
bash
Copy code
./bin/kafka-server-start.sh -daemon ./config/server.properties
Check Broker Status:
Verify that the Kafka brokers are running and connected to Zookeeper:
bash
Copy code
ss -tulpn | grep java
Conclusion:
You've successfully set up a Kafka cluster with three Zookeeper nodes and three broker nodes, ensuring high availability for your data streaming needs. This robust and fault-tolerant architecture will help you process real-time data efficiently and reliably. Enjoy harnessing the power of Kafka for your applications!
Watch video P-1 Setting Up a Kafka Cluster with Zookeeper for High Availability online without registration, duration hours minute second in high quality. This video was added by user TechwithAayushPawar 01 January 1970, don't forget to share it with your friends and acquaintances, it has been viewed on our site 214 once and liked it 6 people.