mirror of
https://github.com/openimsdk/open-im-server.git
synced 2025-10-29 23:38:11 +08:00
4f40022105
* feat: add standardizer optimize makefile * feat: add standardizer optimize makefile * feat: add openim test docs * feat: add openim kafka docs * feat: add openim kafka docs * feat: add openim kafka docs
162 lines
5.3 KiB
Markdown
162 lines
5.3 KiB
Markdown
# OpenIM Kafka Guide
|
|
|
|
This document aims to provide a set of concise guidelines to help you quickly install and use Kafka through Docker Compose.
|
|
|
|
## Installing Kafka
|
|
|
|
With the Docker Compose script provided by OpenIM, you can easily install Kafka. Use the following command to start Kafka:
|
|
|
|
```bash
|
|
docker compose up -d
|
|
```
|
|
|
|
After executing this command, Kafka will be installed and started. You can confirm the Kafka container is running with the following command:
|
|
|
|
```bash
|
|
docker ps | grep kafka
|
|
```
|
|
|
|
The output of this command, as shown below, displays the status information of the Kafka container:
|
|
|
|
```
|
|
be416b5a0851 bitnami/kafka:3.5.1 "/opt/bitnami/script…" 3 days ago Up 2 days 9092/tcp, 0.0.0.0:19094->9094/tcp, :::19094->9094/tcp kafka
|
|
```
|
|
|
|
### References
|
|
|
|
- Official Docker installation documentation: [Click here](http://events.jianshu.io/p/b60afa35303a)
|
|
- Detailed installation guide: [Tutorial on Towards Data Science](https://towardsdatascience.com/how-to-install-apache-kafka-using-docker-the-easy-way-4ceb00817d8b)
|
|
|
|
## Using Kafka
|
|
|
|
### Entering the Kafka Container
|
|
|
|
To execute Kafka commands, you first need to enter the Kafka container. Use the following command:
|
|
|
|
```bash
|
|
docker exec -it kafka bash
|
|
```
|
|
|
|
### Kafka Command Tools
|
|
|
|
Inside the Kafka container, you can use various command-line tools to manage Kafka. These tools include but are not limited to:
|
|
|
|
- `kafka-topics.sh`: For creating, deleting, listing, or altering topics.
|
|
- `kafka-console-producer.sh`: Allows sending messages to a specified topic from the command line.
|
|
- `kafka-console-consumer.sh`: Allows reading messages from the command line, with the ability to specify topics.
|
|
- `kafka-consumer-groups.sh`: For managing consumer group information.
|
|
|
|
### Kafka Client Tool Installation
|
|
|
|
For easier Kafka management, you can install Kafka client tools. If you installed Kafka through OpenIM's Docker Compose, you can install the Kafka client tools with the following command:
|
|
|
|
```bash
|
|
make install.kafkactl
|
|
```
|
|
|
|
### Automatic Topic Creation
|
|
|
|
When installing Kafka through OpenIM's Docker Compose method, OpenIM automatically creates the following topics:
|
|
|
|
- `latestMsgToRedis`
|
|
- `msgToPush`
|
|
- `offlineMsgToMongoMysql`
|
|
|
|
These topics are created using the `scripts/create-topic.sh` script. The script waits for Kafka to be ready before executing the commands to create topics:
|
|
|
|
```bash
|
|
# Wait for Kafka to be ready
|
|
until /opt/bitnami/kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9092; do
|
|
echo "Waiting for Kafka to be ready..."
|
|
sleep 2
|
|
done
|
|
|
|
# Create topics
|
|
/opt/bitnami/kafka/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 8 --topic latestMsgToRedis
|
|
/opt/bitnami/kafka/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 8 --topic msgToPush
|
|
/opt/bitnami/kafka/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 8 --topic offlineMsgToMongoMysql
|
|
|
|
echo "Topics created."
|
|
```
|
|
|
|
The optimized and expanded documentation further details some basic commands for operations inside the Kafka container, as well as basic commands for managing Kafka using `kafkactl`. Here is a more detailed guide.
|
|
|
|
|
|
## Basic Commands in the Kafka Container
|
|
|
|
### Listing Topics
|
|
|
|
To list all existing topics, you can use the following command:
|
|
|
|
```bash
|
|
kafka-topics.sh --list --bootstrap-server localhost:9092
|
|
```
|
|
|
|
### Creating a New Topic
|
|
|
|
When creating a new topic, you can specify the number of partitions and the replication factor. Here is the command to create a new topic:
|
|
|
|
```bash
|
|
kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic your_topic_name
|
|
```
|
|
|
|
### Producing Messages
|
|
|
|
To send messages to a specific topic, you can use the producer command. The following command prompts you to enter messages, which are sent to the specified topic with each press of the Enter key:
|
|
|
|
```bash
|
|
kafka-console-producer.sh --broker-list localhost:9092 --topic your_topic_name
|
|
```
|
|
|
|
### Consuming Messages
|
|
|
|
To read messages from a specific topic, you can use the consumer command. The following command reads new messages from the specified topic and outputs them on the console:
|
|
|
|
```bash
|
|
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic your_topic_name --from-beginning
|
|
```
|
|
|
|
The `
|
|
|
|
--from-beginning` parameter reads messages from the beginning of the topic. If this parameter is omitted, only new messages will be read.
|
|
|
|
|
|
## Basic Commands Using `kafkactl`
|
|
|
|
`kafkactl` is a command-line tool for managing and operating Kafka clusters. It offers a more modern way to interact with Kafka.
|
|
|
|
### Listing Topics
|
|
|
|
To list all topics, you can use:
|
|
|
|
```bash
|
|
kafkactl get topics
|
|
```
|
|
|
|
### Creating a New Topic
|
|
|
|
To create a new topic with `kafkactl`, use:
|
|
|
|
```bash
|
|
kafkactl create topic your_topic_name --partitions 1 --replication-factor 1
|
|
```
|
|
|
|
### Producing Messages
|
|
|
|
To send messages to a topic, you can use:
|
|
|
|
```bash
|
|
kafkactl produce your_topic_name --value "your message"
|
|
```
|
|
|
|
Here, `"your message"` is the content of the message you want to send.
|
|
|
|
### Consuming Messages
|
|
|
|
To consume messages from a topic, use:
|
|
|
|
```bash
|
|
kafkactl consume your_topic_name --from-beginning
|
|
```
|
|
|
|
Again, the `--from-beginning` parameter will start consuming messages from the beginning of the topic. If you do not wish to start from the beginning, you can omit this parameter. |