Apache Kafka is a distributed stream processing platform, which is widely used in log collection, monitoring data aggregation, streaming data processing and other scenarios. In development and testing environments, in order to save resources, we often need to build multiple Kafka Broker instances on a physical machine to simulate clusters in production environments. This article will introduce in detail how to set up multiple Kafka Broker instances on a single machine.
Environmental preparation
- Operating system: Ubuntu 20.04
- Java version: 1.8+
- Kafka version: 2.8.0
Install Java
Make sure that Java 1.8 or higher is installed on your system. You can check the Java version by:
java -version
If Java is not installed, you can use the following command to install it:
sudo apt update sudo apt install openjdk-8-jdk
Download Kafka
VisitKafka official websiteDownload the latest version of Kafka compression package and unzip it to the right place:
wget /kafka/2.8.0/kafka_2.13-2.8. tar -xzf kafka_2.13-2.8. -C /opt cd /opt/kafka_2.13-2.8.0
Configure multiple Broker instances
Suppose we need to run three Kafka Broker instances on the same machine, we will create a configuration file for each instance separately.
Copy the configuration file
First copy the default file and create a separate configuration file for each Broker instance:
cp config/ config/ cp config/ config/
Modify the configuration file
Edit each newly created configuration file to ensure that the configuration of each Broker instance is unique, especially parameters such as
=1 listeners=PLAINTEXT://:9092 =/tmp/kafka-logs-1 =2 listeners=PLAINTEXT://:9093 =/tmp/kafka-logs-2
Start Zookeeper
Kafka relies on Zookeeper for coordination, so first you need to start Zookeeper:
bin/ config/
Start Kafka Broker instance
Start each Kafka Broker instance separately:
bin/ config/ & bin/ config/ &
Test cluster
To verify that the cluster is working properly, we can create a topic and publish and subscribe to messages from different Broker instances.
Create a topic
bin/ --create --topic test-topic --bootstrap-server localhost:9092 --replication-factor 2 --partitions 3
Post a message
bin/ --broker-list localhost:9092 --topic test-topic
Enter some test messages, such as:
Hello, Kafka!
This is a test message.
Subscribe to Message
Open another terminal window and subscribe to the topic to receive messages:
bin/ --bootstrap-server localhost:9092 --topic test-topic --from-beginning
If everything works fine, you should be able to see all the messages posted earlier.
This configuration is very useful for development and testing because it allows us to simulate a real production environment with limited resources. Hope this article helps you!
Method extension
Apache Kafka is a distributed stream processing platform that is widely used to build real-time data pipelines and streaming applications. In some cases, for testing or development purposes, you may need to run multiple Kafka broker instances on a stand-alone machine to simulate a cluster environment. Here is a simple example showing how to configure and start multiple Kafka broker instances on a stand-alone machine.
Environmental preparation
- Install Java: Make sure that Java 8 or higher is installed on your machine.
- Download Kafka: Download the latest version of Kafka from the official Apache Kafka website and unzip it.
Configuration file modification
Kafka's configuration file is located in the config directory. You need to create a separate configuration file for each broker instance and make the necessary modifications to avoid port conflicts and other resource conflicts.
1. Copy and modify
Suppose you want to run two broker instances, you can do this:
- Copy config/ to config/ and config/.
- Modify :
=0 listeners=PLAINTEXT://:9092 =/tmp/kafka-logs-0
Modify :
=1 listeners=PLAINTEXT://:9093 =/tmp/kafka-logs-1
Start ZooKeeper
Kafka uses ZooKeeper to manage cluster metadata. Start ZooKeeper first:
bin/ config/
Start Kafka Brokers
Start two Kafka broker instances in different terminal windows:
Start the first broker
bin/ config/
Start the second broker
bin/ config/
Create a topic
You can use the command line tools provided by Kafka to create topics and specify multiple brokers:
bin/ --create --topic my-topic --bootstrap-server localhost:9092,localhost:9093 --replication-factor 2 --partitions 3
Send and receive messages
Send a message
bin/ --topic my-topic --bootstrap-server localhost:9092
Enter a message in the open terminal, for example:
Hello, Kafka!
Receive message
Run the consumer in another terminal window:
bin/ --topic my-topic --from-beginning --bootstrap-server localhost:9092,localhost:9093
You should be able to see the message you sent earlier.
This is very useful for testing and development scenarios and can help you better understand and debug Kafka's cluster behavior. In Apache Kafka, you can simulate a small cluster environment by configuring multiple Broker instances even in a stand-alone environment. This setup is very useful for testing and learning how Kafka works.
The following will describe in detail how to configure multiple Kafka Broker instances on a single machine.
Preparation
Install Java: Make sure that Java (JDK 8 or higher) is installed on your system.
Download Kafka: Download the latest version of Kafka from Kafka's official website and unzip it into a directory, such as /opt/kafka_2.13-3.0.0.
Create multiple configuration files: In order to run multiple Broker instances on the same machine, each instance needs to have its own configuration file.
Configuration File
Suppose we want to run three Broker instances (Broker 0, Broker 1, and Broker 2) on the same machine. We need to create a separate configuration file for each Broker. Kafka's default configuration file is located in config/.
Copy the configuration file:
Copy the default file three times, named, and respectively.
- Modify configuration files: Make necessary modifications to each copied configuration file, mainly changing the following items:
- : Each Broker must have a unique ID.
- port: Each Broker needs to listen for a different port.
- : Each Broker needs to use a different log directory.
The example configuration is as follows:
=0 port=9092 =/tmp/kafka-logs-0
=1 port=9093 =/tmp/kafka-logs-1
=2 port=9094 =/tmp/kafka-logs-2
Start Zookeeper
kafka relies on Zookeeper to manage cluster metadata. Start Zookeeper first:
bin/ config/
Start multiple Broker instances
Start each Broker instance in turn:
# Start Broker 0bin/ config/ & # Start Broker 1bin/ config/ & # Start Broker 2bin/ config/ &
Create a topic
You can use the command line tool provided by Kafka to create a topic and specify multiple Brokers as replicas:
bin/ --create --topic my-topic --bootstrap-server localhost:9092,localhost:9093,localhost:9094 --replication-factor 3 --partitions 3
test
Production news:
bin/ --broker-list localhost:9092 --topic my-topic
Enter some messages in this terminal.
Consumption message: Open another terminal and run the consumer command:
bin/ --bootstrap-server localhost:9092 --topic my-topic --from-beginning
Close Broker and Zookeeper
Close all Broker instances and Zookeeper:
# Close Broker 0bin/ # Close Broker 1bin/ # Close Broker 2bin/ # Close Zookeeperbin/
Summarize
You can successfully configure and run multiple Kafka Broker instances on one machine to form a small Kafka cluster. This is very helpful for learning and testing the distributed features of Kafka.
The above is the detailed explanation of the Kafka single-machine multi-broker instance cluster construction tutorial. For more information about Kafka multi-broker cluster construction, please pay attention to my other related articles!