2018/04/08 postscript: Added a description about Java 10 to the summary.
2017/06/02 postscript: The contents of the survey have been summarized and the description has been significantly updated.
2017/06/01 postscript: Added "Efforts for this issue after JDK 8/9" at the end of the article. It seems that it has been fixed in 8u121.b34 and 8u131.b06 of OpenJDK8 (I will check separately if it is really fixed). Both workarounds have been released since 2017, so it's a good idea to check if you don't update your JDK / JRE frequently.
Java10 officially now supports Docker containers. The Java program on the Docker container can now grasp the resource settings such as CPU and memory set in the Docker container, so the menicoa problem described in this article will be solved.
Regarding Docker support for Java10, the following Qiita article summarizes in detail including operation check.
https://qiita.com/c9katayama/items/b427a008623f0e228c50
If you limit the number of CPUs to a small number with Docker, taskset, etc. in the Menicoa environment, the performance of Java programs may drop significantly. There are two ways to deal with it.
If you only use the option to specify the CPU core ID, such as the --cpuset-cpus of the --docker command, you can handle it with a JDK update. (Java 8 8u121.b34 or later or 8u131.b06 or later) If you use an option that does not specify a CPU core ID, such as --cpus in the --docker command, you need to tune the appropriate JVM parameter settings when you start the Java program.
In a menicore environment (a machine with a large number of CPU cores), performance will drop significantly when a Java program is executed on Docker with resource restrictions such as the number of CPU cores. The same applies when a Java program is executed using a container technology or command (ex. Tasket command) that uses resource restrictions by cgroups other than Docker.
In Java, the following JVM parameters are set based on the hardware resources of the machine.
--Number of GC (garbage collect) threads --Number of JIT compilation threads --Heap size
This is fine in itself. However, in Java, even if hardware resources are restricted by cgroups such as Docker, the JVM parameters are determined based on the hardware resources of the host machine, not the restricted hardware resources.
Therefore, for example, if you execute a Java program on a "Docker container with 20 CPU cores" on a host machine with 20 CPU cores, Java will determine the JVM parameters based on the 20 cores. .. However, the fact that only one CPU core can be used in the Docker container remains the same, so Java programs suffer from the overhead of a large number of GC threads and JIT compilation threads in addition to application processing, resulting in a significant decrease in performance. ..
Be careful of those who meet the following conditions.
--The host machine is a menicoa environment. (Equipped with dozens of CPU cores) --Java runs on the Docker container. --The above Docker container limits CPU resources. (With docker command --cpuset-cpus, --cpus, etc. Specify option)
There are two main approaches to dealing with this.
As described in "(Reference) Addressing this issue after JDK8 / 9" in this article, cgroup-related issues have been addressed to some extent in the new version of JDK8 / 9. In the case of JDK8, it will be solved by updating to the following version or later.
--8u121.b34 or later --8u131.b06 or later
The confirmation results in the Docker container with the JDK before and after the action are listed below. I have confirmed it in an environment with 8 logical CPU cores, but in the JDK after dealing with it, the resource limit is properly reflected in the JVM parameter by the docker command --cpuset-cpus.
#--------------------------------------------------------------
#In the JDK before action--cpuset-The setting of the cpus option is not reflected in the JVM parameter
#--------------------------------------------------------------
$ docker run --rm --cpuset-cpus 0-1 openjdk:8u77-b03 java -XX:+PrintFlagsFinal -version | egrep "CICompilerCount |ParallelGCThreads"
intx CICompilerCount := 4 {product}
uintx ParallelGCThreads = 8 {product}
openjdk version "1.8.0_03-Ubuntu"
OpenJDK Runtime Environment (build 1.8.0_03-Ubuntu-8u77-b03-3ubuntu3-b03)
OpenJDK 64-Bit Server VM (build 25.03-b03, mixed mode)
#--------------------------------------------------------------
#After dealing with it, in JDK--cpuset-The setting of the cpus option is reflected in the JVM parameter
#--------------------------------------------------------------
$ docker run --rm --cpuset-cpus 0-1 openjdk:8u131-b11 java -XX:+PrintFlagsFinal -version | egrep "CICompilerCount |ParallelGCThreads"
intx CICompilerCount := 2 {product}
uintx ParallelGCThreads = 2 {product}
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-0ubuntu1.16.04.2-b11)
OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)
However, in the case of an option that does not specify a specific CPU core ID like --cpus of the docker command, it seems that the problem cannot be avoided even with Java after handling. I haven't tried it, but it seems likely that this problem cannot be avoided for the --cpu-quota and --cpu-period options as well. In this case, it is necessary to tune the JVM parameter settings, which is another workaround.
#--------------------------------------------------------------
#After dealing with it, even in JDK--No effect with cpus option settings
#--------------------------------------------------------------
docker run --rm --cpus 2.0 openjdk:8u131-b11 java -XX:+PrintFlagsFinal -version | egrep "CICompilerCount |ParallelGCThreads"
intx CICompilerCount := 4 {product}
uintx ParallelGCThreads = 8 {product}
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-0ubuntu1.16.04.2-b11)
OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)
It is solved by tuning by setting the JVM parameter when executing the Java program on the Docker container. It's more time-consuming than a JDK update, but it's the most reliable.
You may want to set the GC to serial GC if the following conditions are true:
--Limiting CPU resources to one or less CPU cores --The main processing content is batch processing (don't worry about Stop the World)
#Change GC to serial GC
java -XX:+UseSerialGC ...
However, it is said that serial GC will be abolished in the future, so be careful if you plan to update Java.
If you want to use parallel GC, which is the default GC of Java8, you can handle it by reducing the number of threads of Parallel GC. The appropriate number of threads depends on the CPU resource limit specified in the docker command, so tuning is required.
#Set the number of Parallel GC threads to 1
java XX:ParallelGCThreads=1 ...
Since I have never set CSM, G1, etc., I will not describe it, but basically you can set the number of threads according to the CPU resource.
JIT compilation also runs on multiple threads, and a reasonable number of threads will be generated in the Menikoa environment. For this reason, tuning is also required according to the CPU resource limit.
#Set the number of JIT compilation threads to 1
java -XX:CICompilerCount=1 ...
Determine the JVM parameters based on the Java program on the virtual machine and the hardware resources of the virtual machine. Therefore, even if the host machine is in a menicoa environment, the JVM parameters are set based on the hardware resources assigned to the virtual machine.
This problem has been recognized since JDK9, and it seems that Issue is backported to JDK8 as far as the support status of Issue is seen. And it has been dealt with in JDK8 "8u121.b34" and "8u131.b06".
JDK-8140793 - getAvailableProcessors may incorrectly report the number of cpus in Docker container --Issue about CPU issues covered in this article. --This problem has been resolved by the countermeasure of another case JDK-6515172. --Looking at the related issues, "8u121.b34" and "8u131.b06" have already been dealt with.
JDK-8172318 - [linux] Experimental support for cgroup memory limits in container (ie Docker) environments --Since the same problem as this article occurs in memory, Issue to deal with so that the memory size of the Docker container controlled by cgroups on the JVM side can be grasped. --Introduced "experimentally" in JDK9. --A backport has been requested for JDK8. --Similar to the problem with JDK-6515172 / JDK-8140793, looking at the related issues, "8u121.b34" and "8u131.b06" have been addressed.
Recommended Posts