Message cooperation started with Spring Boot Apache Kafka edition

table of contents

  1. Synopsis up to the last time
  2. Purpose of the article
  3. Prerequisite environment
  4. Build a messaging platform (Kafka edition)
  5. Create Sink, Source, Processor (Kafka edition)
  6. Operation check
  7. Summary

Synopsis up to the last time

We have taken the first step in designing and implementing a message-driven system. Message linkage started with Spring Boot

Purpose of the article

Let's take the second step.

While you can easily benefit from message driving RabbitMQ cannot be configured as a cluster It seems that the upper limit of scale and ensuring node level fault tolerance are inferior to Apache Kafka.

Let's touch Apache Kafka as well.

The completed code is here. Although not explained in the article, it comes with a test class.

Prerequisite environment

I will not use Docker this time. Others are the same as last time, so version information is omitted.

Build a messaging platform (Kafka edition)

As mentioned above, Apache Kafka is adopted.

Since quickstart is posted on the official Apache Kafka website. Follow this procedure to install.

First, get the HTTP link from Apache Kafka distribution page.

A page structure in which the suggested URL changes each time you access it. Make a note of the URL displayed in "We suggest ~" unless you have a specific reason. Let's cooperate with load distribution.

Please refer to the following. Screenshot when meisei-u.ac.jp is suggested. スクリーンショット 2019-12-07 22.34.10.png

Next, unzip the downloaded file and move it to a lower folder.

# wget [kafka.URL of tgz]
# cd [Folder after decompression]

Enter the start command. First of all, Zoo Keeper.

# bin/zookeeper-server-start.sh config/zookeeper.properties

Next is Kafka itself.

# bin/kafka-server-start.sh config/server.properties

Let's check the operation a little. Create a topic (message destination) named test.

# bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test

Display the topic list. Confirm that the topic has been created.

# bin/kafka-topics.sh --list --bootstrap-server localhost:9092
test

With the steps so far, Kafka is ready to process your message.

Create Sink, Source, Processor (Kafka edition)

Using Spring Cloud Stream as before I will make Sink, Source and Processor, but ... No change in the implementation code. Unit tests are also from the same source.

The previous code is here. The code for this time is here.

Only pom.xml is modified. Removed RabbitMQ related and added Apache Kafka related. If you compare the completed pom.xml with the previous one, you can see it, so I will omit the details.

The most important changes are: The guide is here.

(Omission)
		<dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-stream-binder-kafka</artifactId>
		</dependency>
(Omission)

Since no property value is set in the app The point is that the connection destination information (localhost: 9092) to kafka and various parameters are set based on the default.

Operation check

Run maven clean package for each project. Start each Jar generated under target as follows.

$ java -jar target/hello-source-kafka-0.0.1-SNAPSHOT.jar --server.port=8080
$ java -jar target/hello-sink-kafka-0.0.1-SNAPSHOT.jar  --server.port=8082
$ java -jar target/hello-processor-kafka-0.0.1-SNAPSHOT.jar --server.port=8090

In the startup log, it is displayed that Kafka is being used. The following is an example of Sink log output.

2019-12-07 12:45:28.595  INFO 4986 --- [           main] o.s.c.s.b.k.p.KafkaTopicProvisioner      : Using kafka topic for outbound: hello-sink
2019-12-07 12:45:28.600  INFO 4986 --- [           main] o.a.k.clients.admin.AdminClientConfig    : AdminClientConfig values: 
	bootstrap.servers = [localhost:9092]
(Omission)
	ssl.truststore.type = JKS
2019-12-07 12:45:28.707  INFO 4986 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka version: 2.3.1
2019-12-07 12:45:28.709  INFO 4986 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka commitId: 18a913733fb71c01
2019-12-07 12:45:28.710  INFO 4986 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka startTimeMs: 1575722728705

Send a POST request to Source with curl.

$ curl -v localhost:8080 -d '{"tweet":"Hello"}' -H 'Content-Type: application/json'

A message is output to the console that started Sink's Jar.

Received Hello processing!

Summary

Using Spring Boot, Apache Kafka, and Spring Cloud Stream I learned how to design / implement microservice message linkage.

By using Spring Cloud Stream Dependence on messaging infrastructure can be removed from app design and implementation.

Please play with each messaging platform by exchanging it.

References

Spring Cloud Stream Kafka Binder Reference Guide

Apache Kafka QuickStart

Try Apache Kafka

Recommended Posts

Message cooperation started with Spring Boot Apache Kafka edition
Message cooperation started with Spring Boot
Get started with Spring boot
Getting Started with Spring Boot
Download with Spring Boot
Hello World (REST API) with Apache Camel + Spring Boot 2
Spring Security usage memo: Cooperation with Spring MVC and Boot
Hello World (console app) with Apache Camel + Spring Boot 2
Generate barcode with Spring Boot
Hello World with Spring Boot
Implement GraphQL with Spring Boot
Hello World with Spring Boot!
Run LIFF with Spring Boot
SNS login with Spring Boot
Access Apache Kafka with Micronaut
File upload with Spring Boot
Spring Boot starting with copy
Spring Boot starting with Docker
Hello World with Spring Boot
Set cookies with Spring Boot
Use Spring JDBC with Spring Boot
Add module with Spring Boot
Create microservices with Spring Boot
Send email with spring boot
Spring Boot validation message changes
I tried to get started with Swagger using Spring Boot
gRPC on Spring Boot with grpc-spring-boot-starter
Create an app with Spring Boot 2
Hot deploy with Spring Boot development
Database linkage with doma2 (Spring boot)
Spring Boot programming with VS Code
Until "Hello World" with Spring Boot
Inquiry application creation with Spring Boot
Going out of message (Spring boot)
Get validation results with Spring Boot
(Intellij) Hello World with Spring Boot
Create an app with Spring Boot
Easy Pub/Sub messaging with Apache Kafka
Google Cloud Platform with Spring Boot 2.0.0
Check date correlation with Spring Boot
I tried GraphQL with Spring Boot
[Java] LINE integration with Spring Boot
Beginning with Spring Boot 0. Use Spring CLI
I tried Flyway with Spring Boot
Spring Boot gradle build with Docker
Processing at application startup with Spring Boot
Hello World with Eclipse + Spring Boot + Maven
Send regular notifications with LineNotify + Spring Boot
Perform transaction confirmation test with Spring Boot
HTTPS with Spring Boot and Let's Encrypt
Try using Spring Boot with VS Code
Start web application development with Spring Boot
Launch Nginx + Spring Boot application with docker-compose
I tried Lazy Initialization with Spring Boot 2.2.0
Implement CRUD with Spring Boot + Thymeleaf + MySQL
Asynchronous processing with Spring Boot using @Async
Implement paging function with Spring Boot + Thymeleaf
(IntelliJ + gradle) Hello World with Spring Boot
How to split Spring Boot message file
Use cache with EhCashe 2.x with Spring Boot
Form class validation test with Spring Boot