Message cooperation started with Spring Boot Apache Kafka edition

table of contents

1. Synopsis up to the last time
2. Purpose of the article
3. Prerequisite environment
4. Build a messaging platform (Kafka edition)
5. Create Sink, Source, Processor (Kafka edition)
6. Operation check
7. Summary

Synopsis up to the last time

We have taken the first step in designing and implementing a message-driven system. Message linkage started with Spring Boot

Purpose of the article

Let's take the second step.

While you can easily benefit from message driving RabbitMQ cannot be configured as a cluster It seems that the upper limit of scale and ensuring node level fault tolerance are inferior to Apache Kafka.

Let's touch Apache Kafka as well.

The completed code is here. Although not explained in the article, it comes with a test class.

Prerequisite environment

• Spring Tool Suite must be installed.
• JRE 8 or above must be installed.

I will not use Docker this time. Others are the same as last time, so version information is omitted.

Build a messaging platform (Kafka edition)

As mentioned above, Apache Kafka is adopted.

Since quickstart is posted on the official Apache Kafka website. Follow this procedure to install.

First, get the HTTP link from Apache Kafka distribution page.

A page structure in which the suggested URL changes each time you access it. Make a note of the URL displayed in "We suggest ~" unless you have a specific reason. Let's cooperate with load distribution.

Please refer to the following. Screenshot when meisei-u.ac.jp is suggested.

Next, unzip the downloaded file and move it to a lower folder.

# wget [kafka.URL of tgz]
# cd [Folder after decompression]

Enter the start command. First of all, Zoo Keeper.

# bin/zookeeper-server-start.sh config/zookeeper.properties

Next is Kafka itself.

# bin/kafka-server-start.sh config/server.properties

Let's check the operation a little. Create a topic (message destination) named test.

# bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test

Display the topic list. Confirm that the topic has been created.

# bin/kafka-topics.sh --list --bootstrap-server localhost:9092
test

With the steps so far, Kafka is ready to process your message.

Create Sink, Source, Processor (Kafka edition)

Using Spring Cloud Stream as before I will make Sink, Source and Processor, but ... No change in the implementation code. Unit tests are also from the same source.

The previous code is here. The code for this time is here.

Only pom.xml is modified. Removed RabbitMQ related and added Apache Kafka related. If you compare the completed pom.xml with the previous one, you can see it, so I will omit the details.

The most important changes are: The guide is here.

(Omission)
		<dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-stream-binder-kafka</artifactId>
		</dependency>
(Omission)

Since no property value is set in the app The point is that the connection destination information (localhost: 9092) to kafka and various parameters are set based on the default.

Operation check

Run maven clean package for each project. Start each Jar generated under target as follows.

$ java -jar target/hello-source-kafka-0.0.1-SNAPSHOT.jar --server.port=8080
$ java -jar target/hello-sink-kafka-0.0.1-SNAPSHOT.jar  --server.port=8082
$ java -jar target/hello-processor-kafka-0.0.1-SNAPSHOT.jar --server.port=8090

In the startup log, it is displayed that Kafka is being used. The following is an example of Sink log output.

2019-12-07 12:45:28.595  INFO 4986 --- [           main] o.s.c.s.b.k.p.KafkaTopicProvisioner      : Using kafka topic for outbound: hello-sink
2019-12-07 12:45:28.600  INFO 4986 --- [           main] o.a.k.clients.admin.AdminClientConfig    : AdminClientConfig values: 
	bootstrap.servers = [localhost:9092]
(Omission)
	ssl.truststore.type = JKS
2019-12-07 12:45:28.707  INFO 4986 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka version: 2.3.1
2019-12-07 12:45:28.709  INFO 4986 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka commitId: 18a913733fb71c01
2019-12-07 12:45:28.710  INFO 4986 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka startTimeMs: 1575722728705

Send a POST request to Source with curl.

$ curl -v localhost:8080 -d '{"tweet":"Hello"}' -H 'Content-Type: application/json'

A message is output to the console that started Sink's Jar.

Received Hello processing!

Summary

Using Spring Boot, Apache Kafka, and Spring Cloud Stream I learned how to design / implement microservice message linkage.

By using Spring Cloud Stream Dependence on messaging infrastructure can be removed from app design and implementation.

Please play with each messaging platform by exchanging it.

References

Spring Cloud Stream Kafka Binder Reference Guide

Apache Kafka QuickStart

Try Apache Kafka