Circuit Breaker Pattern with Apache Camel

Circuit Breaker Pattern

The Circuit Breaker Pattern is one of the design patterns. When an application accesses a remote service, a temporary failure may occur due to a high load or DB failover. This is a temporary failure, and if you wait for it to recover, the process will succeed without error. The circuit breaker pattern is used in such cases, and if a failure is detected, access is not performed until recovery. This has the advantages of not causing a large number of errors, not continuing processing because it is known that an error will occur, and not imposing an unnecessary load on the remote side. It is called the circuit breaker pattern because it trips the breaker when the load is high.

The circuit breaker has the following three states.

• CLOSED
  Normal condition. The number of the latest errors is counted, and when the number exceeds the threshold value, it shifts to OPEN.
• OPEN
  Requests are restricted due to an error. After a certain period of time, it will shift to HALF-OPEN.
• HALF-OPEN
  Check if the remote system has recovered from the failure. If a small number of requests are executed and the processing is successful, it will move to CLOSED (normal).

The circuit breaker pattern was published (?) In 2007 in the book Release It! (Although I only glance at it). The Japanese version of Relase It! Has also been released, and the following Second Edition was also released in 2018 (no Japanese version).

• Release It! Second Edition

The circuit breaker pattern can also be found on Martin Fowler's blog.

• CircuitBreaker by Martin Fowler

There is also part 2 of this article. Circuit Breaker Pattern with Apache Camel Part 2 (Try Hystrix Dashboard)

Use the circuit breaker pattern in Apache Camel

The following two circuit breakers are available in Apache Camel.

• Circuit Breaker Load Balancer
• Hystrix EIP

Circuit Breaker Load Balancer is a simple implementation that realizes the circuit breaker pattern. It has fewer features than the Hystrix EIP, but is easy to use. However, since it is Deprecated in v2.23.0, the following Hystrix EIP will be used. Hystrix EIP is a component for using Camel with a library for fault tolerance called Hystrix developed by NetFlix. Hystrix also has many functions as a circuit breaker.

For Hystrix EIP, I will create a sample program and run it later.

The official websites of Circuit Breaker Load Balancer and Hystrix EIP are as follows.

-Load Balancer (Official Site) -Hystrix EIP (Official Site)

• As far as the following sources are confirmed, it seems that the circuit breaker of Load Balancer is Deprecated from 2.19.0. https://github.com/apache/camel/blob/camel-2.19.x/camel-core/src/main/java/org/apache/camel/model/loadbalancer/CircuitBreakerLoadBalancerDefinition.java

Hystrix EIP

Now let's try Hystrix EIP. First, add the following libraries. x.x.x specifies the version of Camel you are using.

pom.xml

<dependency>
    <groupId>org.apache.camel</groupId>
    <artifactId>camel-hystrix</artifactId>
    <version>x.x.x</version>
</dependency>

Hystrix has many options and can be set as follows:

--Set the timeout to 5 seconds (executionTimeoutInMilliseconds). --The circuit breaker waits 5 seconds before requesting again when the state becomes open. No request is sent while waiting, but consume continues. If the request still fails after 5 seconds, wait 5 seconds again (circuitBreakerSleepWindowInMilliseconds). --Set the time window to detect errors to 5 seconds (circuitBreakerSleepWindowInMilliseconds). --Set the error threshold when opening the state to 20% (circuitBreakerErrorThresholdPercentage). --Set the minimum number of requests to consider migrating to Open in the window to 2 (circuitBreakerRequestVolumeThreshold).

						.hystrixConfiguration()
							.circuitBreakerErrorThresholdPercentage(20)
							.executionTimeoutInMilliseconds(5000)
							.circuitBreakerSleepWindowInMilliseconds(5000)
							.circuitBreakerRequestVolumeThreshold(2)

In the case of XML DSL, the description is as follows.

		<route id="main_route">
			<from uri="timer:trigger?period=300" />
				<log message="Client request" />
				<hystrix>
					<hystrixConfiguration circuitBreakerErrorThresholdPercentage="20" executionTimeoutInMilliseconds="5000" circuitBreakerSleepWindowInMilliseconds="5000" circuitBreakerRequestVolumeThreshold="2" />
					<to uri="http4://localhost:8888/hello"/>
					<onFallback>
						<transform>
							<constant>Fallback message</constant>
						</transform>
					</onFallback>
				</hystrix>
			    <log message="response: ${body}" />
		</route>

Create the following as a route to try the circuit breaker setting earlier. This route is processed by the Timer component every 300ms. "to (" http4: // localhost: 8888 / hello ")" is now used to make a request to the remote system.

			public void configure() throws Exception {
				from("timer:trigger?period=300")
					.log("Client request")
					.hystrix()
						.hystrixConfiguration()
							.circuitBreakerErrorThresholdPercentage(20)
							.executionTimeoutInMilliseconds(5000)
							.circuitBreakerSleepWindowInMilliseconds(5000)
							.circuitBreakerRequestVolumeThreshold(2)
						.end()
						.to("http4://localhost:8888/hello")
					.end()
					.log("response: ${body}");
			}

Fallback

Hystrix also supports fallback. Fallback is a degenerate operation (Kana?) In Japanese, which means to limit the functions. Hystrix calls the onFallback method when the error exceeds the threshold. In the above source, the Fallback message is stored in the BODY of the message. In the Closed state, the thread that executes the command and the thread that falls back are separate so that they do not affect the execution of the command.

Also, for fallback, there are onFallback and onFallbackViaNetwork methods. Use the onFallback method to handle fallback within the same app, and the onFallbackViaNetwork method to call a remote service. The onFallbackViaNetwork method has its own thread pool for complete thread isolation. This completely separates threads even in the OPEN state.

If you output the log with the onFallbackViaNetwork method, the thread name will be "hystrix-CamelHystrix-10" as shown below, and you can see that it is a separate thread.

[2019-02-06 20:32:38.985], [INFO ], route1, hystrix-CamelHystrix-10, route1, Fallback message

Property

The table below lists some of the properties that are often used.

For other properties, refer to the official website below.

-Hystrix EIP (Official Site)

Check the status with JMX

You can get the values of various properties of Hystrix with JMX. The following is an example of displaying JMX information using hawtio.

The following three are especially important.

• Circuit breaker open
  Indicates whether the circuit breaker is OPEN.
• Hystrix error count
  Shows the number of errors in the time window.
• Hystrix error percentage
  Shows the percentage of errors in the time window.

Below is the entire source including the main method.

	public static void main(String[] args) {
		try {
			CamelContext context = new DefaultCamelContext();
			context.addRoutes(createRouteBuilder());

			context.start();
		} catch (Exception e) {
			e.printStackTrace();
		}
	}

	static RouteBuilder createRouteBuilder() {
        return new RouteBuilder() {
			public void configure() throws Exception {
				from("timer:trigger?period=300")
					.log("Client request")
					.hystrix()
						.hystrixConfiguration()
							.circuitBreakerErrorThresholdPercentage(20)
							.executionTimeoutInMilliseconds(5000)
							.circuitBreakerSleepWindowInMilliseconds(5000)
							.circuitBreakerRequestVolumeThreshold(2)
						.end()
						.to("http4://localhost:8888/hello")
					.end()
					.log("response: ${body}");
			}
        };
    }

reference

-Hystrix EIP (Official Site)

-Load Balancer (Official Site)

• Create Resilient Camel Applications With Hystrix

• CircuitBreaker by Martin Fowler

-Hystrix Component (official site)

• Netflix/Hystrix wiki(github)

• Circuit Breaker Pattern in Apache Camel

• Release It! Second Edition