Parsing the target code is required for native code generation and optimization processing performed by the HotSpot compiler. The native heap (C heap) is used as the memory area used for parsing, and when parsing large methods or complex syntax, the C heap is also used accordingly. This post is about seeing how the HotSpot compiler increases the C heap.
By the way, Java programs are optimized like any other language, but not when compiled with javac. When it is optimized, the Java program is executed, and when the code is compiled by the HotSpot compiler, it is optimized by the HotSpot compiler.
To check the memory usage, let the compiler perform parsing until the compiler gives up, and then check the change in memory usage before and after that. The HotSpot compiler couldn't find a good way to give up on Out Of Nodes, resulting in endlessly long code.
See https://github.com/takimai39/sample/blob/master/OutOfNodes.java for the entire program.
OutOfNodes.java
import java.io.*;
public class OutOfNodes {
public static void main(String[] args) {
for ( int n = 0 ; n < 2 ; n++ ) {
// CompileThreshold=Since it is 10000, the HotSpot compiler is not used for the first loop.
System.out.println("Hit return to continue"); keyin();
for ( int x = 0 ; x < 9999; x++) {
challenge();
}
}
System.out.println("Hit return to exit"); keyin();
}
private static void keyin() {
BufferedReader userInputStreamReader = new BufferedReader(new InputStreamReader(System.in));
try {
String inputString = userInputStreamReader.readLine();
}
catch (Exception e) {
e.printStackTrace();
}
}
private static String getX() {
String str = "ABCDEFGHIJKLMNOPQRSTUVWXYZ";
return(str);
}
private static String challenge() {
String a[] = new String[300];
String b[] = new String[300];
a[0] = getX();
a[1] = getX();
// ...Omitted, endlessly repeat this pattern 230 times...
b[0] = 0 + a[0];
b[1] = 1 + a[1];
// ...Repeat 230 times in the same way...
return(b[0]);
}
}
Try running it with the PrintCompilation option to see what methods the HotSpot compiler is compiling. The display format is from the left Timestamp compilation_number Attribute character Class name :: Method name (code size)
$ java -XX:+PrintCompilation OutOfNodes
Hit return to continue
3106 1 n java.lang.System::arraycopy (native) (static)
3108 2 java.lang.Integer::stringSize (21 bytes)
3110 3 java.lang.Integer::getChars (131 bytes)
3118 4 java.lang.Object::<init> (1 bytes)
3118 5 java.lang.Math::min (11 bytes)
3118 6 java.lang.AbstractStringBuilder::ensureCapacityInternal (16 bytes)
3121 7 java.lang.String::length (6 bytes)
3121 9 java.lang.AbstractStringBuilder::append (48 bytes)
3124 10 java.lang.String::<init> (67 bytes)
3125 11 java.util.Arrays::copyOfRange (63 bytes)
3127 12 java.lang.StringBuilder::append (8 bytes)
3129 8 java.lang.String::getChars (62 bytes)
3130 13 java.lang.AbstractStringBuilder::<init> (12 bytes)
3130 14 java.lang.StringBuilder::<init> (7 bytes)
3130 15 java.lang.StringBuilder::toString (17 bytes)
3130 16 OutOfNodes::getX (5 bytes)
3130 17 java.lang.StringBuilder::append (8 bytes)
3132 18 java.lang.AbstractStringBuilder::append (62 bytes)
Hit return to continue
841500 19 OutOfNodes::challenge (7992 bytes)
841658 20 % OutOfNodes::main @ 20 (55 bytes)
Hit return to exit
846756 19 OutOfNodes::challenge (7992 bytes) COMPILE SKIPPED: out of nodes during split (not retryable)
On another terminal screen, try to get the core file while observing the process size. After confirming COMPILE SKIPPED :, when checking the size again, it increases by about 64M bytes.
$ pmap `jps | grep OutOfNodes | awk '{print $1}' ` | grep total
total 2127304K
$ pmap `jps | grep OutOfNodes | awk '{print $1}' ` | grep total
total 2192840K
For later confirmation, get the core file for each situation with the gcore command.
You can output commands such as pmap, but there is CORE ANALYZER as a tool to check the memory area allocated by malloc () etc. It's a great tool, very much appreciated by anyone checking the core file. Commands such as heap / v can be used with gdb plus included in this core analyzer. In addition, you can check the breakdown of each arena with heap / c, so you can use it to check how many bytes of area are malloced from the output result.
$ /work/core_analyzer-master/bin/Linux/ptmalloc/gdb-7.11.1/gdb /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.141-2.6.10.5.el7.x86_64/bin/java ./core.24437.before
(gdb) heap /v Before the increase in memory usage
Warning: 17 mmap memory blocks were found while mp_ reports 16
Python Exception <type 'exceptions.NameError'> Installation error: gdb.execute_unwinders function is missing:
Tuning params & stats:
mmap_threshold=131072
pagesize=4096
n_mmaps=16
n_mmaps_max=65536
total mmap regions created=16
mmapped_mem=11202560
sbrk_base=0x15b8000
Main arena (0x7fb6ab64a760) owns regions:
[0x15d900f - 0x15d9000] Total 17179869184.0GBFailed to get the first chunk at 0x15d8fff
Dynamic arena (0x7fb670000020) owns regions:
[0x7fb6700008c0 - 0x7fb670021000] Total 129KB in-use 1(312) free 3(129KB)
Dynamic arena (0x7fb66c000020) owns regions:
[0x7fb66c0008c0 - 0x7fb66c021000] Total 129KB in-use 1(312) free 2(129KB)
Dynamic arena (0x7fb678000020) owns regions:
[0x7fb6780008c0 - 0x7fb678021000] Total 129KB in-use 1(312) free 2(129KB)
Dynamic arena (0x7fb674000020) owns regions:
[0x7fb6740008c0 - 0x7fb674021000] Total 129KB in-use 11(3KB) free 1(126KB)
Dynamic arena (0x7fb67c000020) owns regions:
[0x7fb67c0008c0 - 0x7fb67c021000] Total 129KB in-use 1(312) free 2(129KB)
Dynamic arena (0x7fb68c000020) owns regions:
[0x7fb68c0008c0 - 0x7fb68c021000] Total 129KB in-use 10(1KB) free 3(128KB)
Dynamic arena (0x7fb688000020) owns regions:
[0x7fb6880008c0 - 0x7fb688021000] Total 129KB in-use 3(22KB) free 2(107KB)
Dynamic arena (0x7fb694000020) owns regions:
[0x7fb6940008c0 - 0x7fb694021000] Total 129KB in-use 1(312) free 2(129KB)
Dynamic arena (0x7fb690000020) owns regions:
[0x7fb6900008c0 - 0x7fb690021000] Total 129KB in-use 0(0) free 2(129KB)
Dynamic arena (0x7fb698000020) owns regions:
[0x7fb6980008c0 - 0x7fb698021000] Total 129KB in-use 0(0) free 2(129KB)
Dynamic arena (0x7fb6a4000020) owns regions:
[0x7fb6a40008c0 - 0x7fb6a40fe000] Total 1013KB in-use 1622(964KB) free 4(36KB)
mmap-ed large memory blocks:
...
Below is the COMPILE SKIPPED HotSpot compiler: out of nodes...It is the memory usage after becoming.
(gdb) heap /v After increased memory usage
Warning: 18 mmap memory blocks were found while mp_ reports 16
Python Exception <type 'exceptions.NameError'> Installation error: gdb.execute_unwinders function is missing:
Tuning params & stats:
mmap_threshold=8527872
pagesize=4096
n_mmaps=16
n_mmaps_max=65536
total mmap regions created=23
mmapped_mem=11202560
sbrk_base=0x15b8000
Main arena (0x7fb6ab64a760) owns regions:
[0x15d900f - 0x15d9000] Total 17179869184.0GBFailed to get the first chunk at 0x15d8fff
Dynamic arena (0x7fb670000020) owns regions:
[0x7fb6700008c0 - 0x7fb670021000] Total 129KB in-use 1(312) free 3(129KB)
Dynamic arena (0x7fb66c000020) owns regions:
[0x7fb66c0008c0 - 0x7fb66c021000] Total 129KB in-use 1(312) free 2(129KB)
Dynamic arena (0x7fb678000020) owns regions:
[0x7fb6780008c0 - 0x7fb67807e000] Total 501KB in-use 14(6KB) free 4(494KB)
+ Dynamic arena (0x7fb674000020) owns regions: <--This arena is especially increasing!
+ [0x7fb664000030 - 0x7fb667b15000] Total 59MB in-use 4(127KB) free 4(58MB)
+ [0x7fb6740008c0 - 0x7fb678000000] Total 63MB in-use 49(51KB) free 12(63MB)
Dynamic arena (0x7fb67c000020) owns regions:
[0x7fb67c0008c0 - 0x7fb67c021000] Total 129KB in-use 1(312) free 2(129KB)
Dynamic arena (0x7fb68c000020) owns regions:
[0x7fb68c0008c0 - 0x7fb68c021000] Total 129KB in-use 10(1KB) free 3(128KB)
Dynamic arena (0x7fb688000020) owns regions:
[0x7fb6880008c0 - 0x7fb688021000] Total 129KB in-use 3(22KB) free 2(107KB)
Dynamic arena (0x7fb694000020) owns regions:
[0x7fb6940008c0 - 0x7fb694021000] Total 129KB in-use 4(560) free 1(129KB)
Dynamic arena (0x7fb690000020) owns regions:
[0x7fb6900008c0 - 0x7fb690021000] Total 129KB in-use 0(0) free 1(129KB)
Dynamic arena (0x7fb698000020) owns regions:
[0x7fb6980008c0 - 0x7fb698021000] Total 129KB in-use 10(6KB) free 2(122KB)
Dynamic arena (0x7fb6a4000020) owns regions:
[0x7fb6a40008c0 - 0x7fb6a40fe000] Total 1013KB in-use 1643(839KB) free 7(161KB)
mmap-ed large memory blocks:
...
You can see a further breakdown of arena above with the heap / c command. The increased area is 56380088 bytes free, which is almost free and unused. In other words, the area that was once used by malloc () but is no longer needed and is freed is counted.
(gdb) heap /c 0x7fb664000030
Dynamic arena (0x7fb674000020): [0x7fb664000020 - 0x7fb667b15000]
[0x7fb664000030 - 0x7fb6675c4ae8] 56380088 bytes free
[0x7fb6675c4af0 - 0x7fb6675ccad8] 32744 bytes inuse
[0x7fb6675ccae0 - 0x7fb6677ac8b8] 1965528 bytes free
[0x7fb6677ac8c0 - 0x7fb6677b48a8] 32744 bytes inuse
[0x7fb6677b48b0 - 0x7fb667994868] 1966008 bytes free
[0x7fb667994870 - 0x7fb66799c858] 32744 bytes inuse
[0x7fb66799c860 - 0x7fb6679a4848] 32744 bytes inuse
[0x7fb6679a4850 - 0x7fb667b15000] 1509296 bytes free
Total inuse 4 blocks 130976 bytes
Total free 4 blocks 61820920 bytes
(gdb)
Given that there is only a core file for information, it is very difficult to determine what the free () area was previously used for. However, if you still think about something, the empty_block address information may be available as a hint in this case. The HotSpot compiler allocates a memory area to be used in units called Chunk, and sets the address of empty_block when it finishes using it.
Excerpt from OpenJDK source code indexSet.cpp
...
184 //---------------------------- IndexSet::free_block() -------------------------
185 // Add a BitBlock to the free list.
186
187 void IndexSet::free_block(uint i) {
188 debug_only(check_watch("free block", i));
189 assert(i < _max_blocks, "block index too large");
190 BitBlock *block = _blocks[i];
191 assert(block != &_empty_block, "cannot free the empty block");
192 block->set_next((IndexSet::BitBlock*)Compile::current()->indexSet_free_block_list());
193 Compile::current()->set_indexSet_free_block_list(block);
194 set_block(i,&_empty_block);
195 }
Check the address of empty_block with gdb and see how much you can find in the core file. You can see that there are many empty_block addresses in the core after it becomes COMPILE SKIPPED.
(gdb) info var empty_block
All variables matching regular expression "empty_block":
File /usr/src/debug/java-1.7.0-openjdk-1.7.0.141-2.6.10.5.el7.x86_64/openjdk/hotspot/src/share/vm/opto/indexSet.cpp:
IndexSet::BitBlock IndexSet::_empty_block;
Non-debugging symbols:
0x00007fb6ab053ac0 IndexSet::_empty_block
(gdb)
$ od -An -x -w8 core.24437.before | grep "7fb6 0000 3ac0 ab05" | wc -l
1
$ od -An -x -w8 core.24437.after | grep "7fb6 0000 3ac0 ab05" | wc -l
54655
$
When I dump the area around that area, it's ugly.
$ gdb /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.141-2.6.10.5.el7.x86_64/bin/java ./core.24437.after
...
(gdb) x/20gx 0x7fb665322f30
0x7fb665322f30: 0x00007fb6ab053ac0 0x00007fb6ab053ac0
0x7fb665322f40: 0x00007fb6ab053ac0 0x00007fb6ab053ac0
0x7fb665322f50: 0x00007fb6ab053ac0 0x00007fb6ab053ac0
0x7fb665322f60: 0x00007fb6ab053ac0 0x00007fb6ab053ac0
0x7fb665322f70: 0x00007fb6ab053ac0 0x00007fb6ab053ac0
0x7fb665322f80: 0x00007fb6ab053ac0 0x00007fb6ab053ac0
0x7fb665322f90: 0x00007fb6ab053ac0 0x00007fb6ab053ac0
0x7fb665322fa0: 0x00007fb6756461c0 0x00007fb6ab053ac0
0x7fb665322fb0: 0x00007fb6ab053ac0 0x00007fb6ab053ac0
0x7fb665322fc0: 0x00007fb6ab053ac0 0x00007fb6ab053ac0
(gdb)
0x7fb665322fd0: 0x00007fb6ab053ac0 0x00007fb6ab053ac0
0x7fb665322fe0: 0x00007fb6ab053ac0 0x00007fb6ab053ac0
0x7fb665322ff0: 0x00007fb6ab053ac0 0x00007fb6ab053ac0
0x7fb665323000: 0x00007fb6ab053ac0 0x00007fb6ab053ac0
0x7fb665323010: 0x00007fb6ab053ac0 0x00007fb6ab053ac0
0x7fb665323020: 0x00007fb6ab053ac0 0x00007fb6ab053ac0
0x7fb665323030: 0x00007fb6ab053ac0 0x00007fb6ab053ac0
0x7fb665323040: 0x00007fb6ab053ac0 0x00007fb6ab053ac0
0x7fb665323050: 0x00007fb6ab053ac0 0x00007fb6ab053ac0
0x7fb665323060: 0x00007fb6ab053ac0 0x00007fb6ab053ac0
(gdb)
** What is a node? ** ** The node in the message COMPILE SKIPPED: out of nodes during split ... is one data after being decomposed into a Tree structure by parsing at compile time.
I think it is rare that the memory usage of 64Mbyte increases and the compile time of 5 seconds leads to actual harm. Since it always fails to compile, specify exclude in the .hotspot_compiler file to prevent it from being targeted by the HotSpot compiler. Or, conversely, instead of excluding it, it is possible to increase MaxNodeLimit and compile it. If you can change the Java program you're running, you could simplify the code in question.
I think it will be a decision after thorough testing, but I personally thought that the default settings could be used if there was no actual harm.
Recommended Posts