1. Analyze memory usage
1.1 Enter the docker container
# View container listdocker ps # Enter the container command line terminal according to the container name or iddocker exec -it <container_name> bash
1.2 View the current java process list through jps
Three different levels of process list (mainly obtaining the lvmid of the java process)
root@21e4300860c8:/# jps 1 jar 79 Jps root@21e4300860c8:/# jps -l 1 / 127 root@21e4300860c8:/# jps -lv 1 / -Xmx512m -Xms512m -=dev,jiewli -= -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=18000 139 -=/usr/lib/jvm/java-8-openjdk-amd64 -Xms8m
1.3 Statistics the memory pool capacity of the java process through jstat -gccapacity
root@21e4300860c8:/# jstat -gccapacity -h 20 1 250 NGCMN NGCMX NGC S0C S1C EC OGCMN OGCMX OGC OC MCMN MCMX MC CCSMN CCSMX CCSC YGC FGC 174592.0 174592.0 174592.0 7680.0 8704.0 157184.0 349696.0 349696.0 349696.0 349696.0 0.0 1136640.0 98280.0 0.0 1048576.0 11904.0 46 4 174592.0 174592.0 174592.0 7680.0 8704.0 157184.0 349696.0 349696.0 349696.0 349696.0 0.0 1136640.0 98280.0 0.0 1048576.0 11904.0 46 4 174592.0 174592.0 174592.0 7680.0 8704.0 157184.0 349696.0 349696.0 349696.0 349696.0 0.0 1136640.0 98280.0 0.0 1048576.0 11904.0 46 4 174592.0 174592.0 174592.0 7680.0 8704.0 157184.0 349696.0 349696.0 349696.0 349696.0 0.0 1136640.0 98280.0 0.0 1048576.0 11904.0 46 4 174592.0 174592.0 174592.0 7680.0 8704.0 157184.0 349696.0 349696.0 349696.0 349696.0 0.0 1136640.0 98280.0 0.0 1048576.0 11904.0 46 4 174592.0 174592.0 174592.0 7680.0 8704.0 157184.0 349696.0 349696.0 349696.0 349696.0 0.0 1136640.0 98280.0 0.0 1048576.0 11904.0 46 4 174592.0 174592.0 174592.0 7680.0 8704.0 157184.0 349696.0 349696.0 349696.0 349696.0 0.0 1136640.0 98280.0 0.0 1048576.0 11904.0 46 4
illustrate:
- The new generation space is
Survivor0、Survivor1、Eden(Survivor Space 0, Survivor Space 1, Garden of Eden) consists of three parts -
OGC = sum(all OC), At present, there is only one space in the elderly, soOGCandOCequal. You can explore the differences when there are multiple spaces in the elderly. - JVM parameters
-Xmx512mThe maximum heap memory is equal toNGCMX + OGCMX - JVM parameters
-Xms512mThe minimum heap memory is equal toNGCMN + OGCMN - Minimum value of the heap
-XmsParameters and maximum values-XmxSetting the parameters to the same can prevent the heap from automatically expanding. Because the initial allocation of the JVM is-XmsSpecifies that the default is physical memory1/64; The maximum memory allocated by the JVM is-XmxSpecifies that the default is physical memory1/4. When the default free heap memory is less than 40%, the JVM will increase the heap until-XmxThe maximum limit of the spare heap memory is greater than 70%, the JVM will reduce the heap until-XmsMinimum limit for . Therefore, the server is generally set up-Xms、-XmxEqual to avoid resizing the heap after each GC. The heap memory of an object is recycled by an automatic memory management system called a garbage collector. - Full GC trigger condition:
- Call
hour - Insufficient space for the elderly
- Insufficient space in the method area
- The average size of the elderly after passing through Minor GC (Queen Generation GC) is greater than the available memory in the elderly
- When copying from the Eden area and From Space area to the To Space area, the object size is larger than the available memory in To Space, the object will be transferred to the old age, and the available memory in the old age is smaller than the size of the object.
JVM parameters-XX:MetaspaceSizeThe memory size of the metaspace is initialized, the default is 20.79MB. After exceeding 20.79MB, Full GC will be triggered every dimensional space amplification, so a larger value can reduce the number of Full GCs.
JVM parameters-XX:MaxMetaspaceSizeThe maximum memory size of the metaspace is unlimited by default (exceeding the host memory is considered consistent with the host memory). Setting this value can avoid occupying a large amount of host memory resources in the event of a failure and affecting other services.
| Column code | List name | Capacity (kB) | Capacity (MB) |
|---|---|---|---|
| NGCMN | Initial New Generation Space Capacity | 174592.0 | 170.5 |
| NGCMX | Maximum space capacity of the new generation | 174592.0 | 170.5 |
| NGC | Current new generation capacity | 174592.0 | 170.5 |
| S0C | Current survivor space 0 capacity | 7680.0 | 7.5 |
| S1C | Current Survivor Space 1 Capacity | 8704.0 | 8.5 |
| EC | Current space capacity of Eden area | 157184.0 | 153.5 |
| OGCMN | Initial space capacity for old age | 349696.0 | 341.5 |
| OGCMX | Maximum space capacity for older generations | 349696.0 | 341.5 |
| OGC | Current space capacity for the elderly | 349696.0 | 341.5 |
| MCMN | Initial metaspace capacity | 0.0 | 0 |
| MCMX | Maximum metaspace capacity | 1136640.0 | 1,110 |
| MC | Current metaspace capacity | 98280.0 | 95.9765625 |
| CCSMN | Compressed initial capacity of class space | 0.0 | 0 |
| CCSMX | Maximum compressed class space capacity | 1048576.0 | 1,024 |
| CCSC | The currently compressed class space capacity | 11904.0 | 11.625 |
| YGC | Number of GC events in the new generation | 46 | |
| FGC | Number of full GC events | 4 |
1.4 You can also use jstat -gc to count
root@7f9fbd4518a4:/# jstat -gc -h 20 1 250 S0C S1C S0U S1U EC EU OC OU MC MU CCSC CCSU YGC YGCT FGC FGCT GCT 10240.0 10240.0 2416.0 0.0 154112.0 123601.6 349696.0 60088.4 81112.0 75681.7 9432.0 8462.8 38 1.640 3 0.431 2.071 10240.0 10240.0 2416.0 0.0 154112.0 123601.6 349696.0 60088.4 81112.0 75681.7 9432.0 8462.8 38 1.640 3 0.431 2.071 10240.0 10240.0 2416.0 0.0 154112.0 123601.6 349696.0 60088.4 81112.0 75681.7 9432.0 8462.8 38 1.640 3 0.431 2.071 10240.0 10240.0 2416.0 0.0 154112.0 123893.9 349696.0 60088.4 81112.0 75681.7 9432.0 8462.8 38 1.640 3 0.431 2.071 10240.0 10240.0 2416.0 0.0 154112.0 123893.9 349696.0 60088.4 81112.0 75681.7 9432.0 8462.8 38 1.640 3 0.431 2.071 10240.0 10240.0 2416.0 0.0 154112.0 123893.9 349696.0 60088.4 81112.0 75681.7 9432.0 8462.8 38 1.640 3 0.431 2.071 10240.0 10240.0 2416.0 0.0 154112.0 123893.9 349696.0 60088.4 81112.0 75681.7 9432.0 8462.8 38 1.640 3 0.431 2.071 10240.0 10240.0 2416.0 0.0 154112.0 123893.9 349696.0 60088.4 81112.0 75681.7 9432.0 8462.8 38 1.640 3 0.431 2.071 10240.0 10240.0 2416.0 0.0 154112.0 123893.9 349696.0 60088.4 81112.0 75681.7 9432.0 8462.8 38 1.640 3 0.431 2.071 10240.0 10240.0 2416.0 0.0 154112.0 123893.9 349696.0 60088.4 81112.0 75681.7 9432.0 8462.8 38 1.640 3 0.431 2.071
illustrate:
| Column code | List name | KB | MB | Remark |
|---|---|---|---|---|
| S0C | Survivor space 0 capacity | 10240.0 | 10 | |
| S1C | Survivor Space 1 Capacity | 10240.0 | 10 | |
| S0U | Survivor Space 1 Utilization Rate | 2416.0 | 2.35 | |
| S1U | Survivor Space 1 Utilization Rate | 0.0 | 0.0 | |
| EC | Eden space capacity | 154112.0 | 150.5 | |
| EU | Eden Space Utilization | 123601.0 | 120.70 | |
| OC | Space capacity for the elderly | 349696.0 | 341.5 | |
| OU | Space utilization rate in the elderly | 60088.0 | 58.67 | |
| MC | Metaspace capacity | 81112.0 | 79.21 | |
| MU | Metaspace utilization | 75681.7 | 73.90 | |
| CCSC | Compressed space capacity | 9432.0 | 9.21 | Compressed class space capacity, corresponding parameter -XX:CompressedClassSpaceSize |
| CCSU | Compressed space utilization | 8462.0 | 8.26 | |
| YGC | Number of GC events in the new generation | 38 | ||
| YGCT | Time of the new generation of GC events | 1.640 | ||
| FGC | Full GC events | 3 | ||
| FGCT | Full GC event time | 0.431 | ||
| GCT | Total GC time | 2.071 |
1.5 Statistics of garbage collection statistics of java processes through jstat -gcutil
No limiting the number of acquisitions, try to simulate the system call situation under normal circumstances during the acquisition process until the gc event is triggered.
Trigger a new generation of GC eventMinor GC(observeYGCQuantity +1)
root@21e4300860c8:/# jstat -gcutil -h 20 1 250 S0 S1 E O M CCS YGC YGCT FGC FGCT GCT 0.00 8.68 98.93 16.94 92.54 89.70 49 1.761 4 0.908 2.669 0.00 8.68 98.96 16.94 92.54 89.70 49 1.761 4 0.908 2.669 0.00 8.68 98.96 16.94 92.54 89.70 49 1.761 4 0.908 2.669 0.00 8.68 98.96 16.94 92.54 89.70 49 1.761 4 0.908 2.669 0.00 8.68 98.96 16.94 92.54 89.70 49 1.761 4 0.908 2.669 6.80 0.00 0.00 16.95 92.52 89.72 50 1.766 4 0.908 2.673 6.80 0.00 0.74 16.95 92.52 89.72 50 1.766 4 0.908 2.673 6.80 0.00 1.27 16.95 92.52 89.72 50 1.766 4 0.908 2.673 6.80 0.00 2.67 16.95 92.52 89.72 50 1.766 4 0.908 2.673 6.80 0.00 2.67 16.95 92.52 89.72 50 1.766 4 0.908 2.673 6.80 0.00 2.67 16.95 92.52 89.72 50 1.766 4 0.908 2.673 6.80 0.00 2.67 16.95 92.52 89.72 50 1.766 4 0.908 2.673
Trigger a full GC eventFull GC, If the space is too large, it cannot be triggered manually in general, and it is necessary to wait through time (the project start time divided byFGCIt can be seen how long it takes to trigger Full GC), which is usually triggered once every 1 hour.
Maybe it can alsosurvivorSpace utilization is close to 100% triggering Full GC.
root@21e4300860c8:/# jstat -gcutil -h 20 1 250 S0 S1 E O M CCS YGC YGCT FGC FGCT GCT 0.00 99.67 53.34 19.88 93.38 90.50 45 1.680 3 0.627 2.308 0.00 99.67 55.03 19.88 93.38 90.50 45 1.680 3 0.627 2.308 0.00 99.67 55.03 19.88 93.38 90.50 45 1.680 3 0.627 2.308 0.00 99.67 55.03 19.88 93.38 90.50 45 1.680 3 0.627 2.308 0.00 99.67 56.72 19.88 93.38 90.50 45 1.680 3 0.627 2.308 0.00 99.67 56.72 19.88 93.38 90.50 45 1.680 3 0.627 2.308 28.13 0.00 0.00 21.17 93.02 90.13 46 1.729 4 0.627 2.356 0.00 0.00 0.00 16.94 92.41 89.38 46 1.729 4 0.908 2.637 0.00 0.00 0.84 16.94 92.41 89.38 46 1.729 4 0.908 2.637 0.00 0.00 0.84 16.94 92.41 89.38 46 1.729 4 0.908 2.637 0.00 0.00 0.84 16.94 92.41 89.38 46 1.729 4 0.908 2.637 0.00 0.00 0.90 16.94 92.41 89.38 46 1.729 4 0.908 2.637 0.00 0.00 0.92 16.94 92.41 89.38 46 1.729 4 0.908 2.637 0.00 0.00 0.92 16.94 92.41 89.38 46 1.729 4 0.908 2.637
1.6 Check the metaspace situation
# View a parameter of a java instance# Check the value of MetaspaceSize, the default is -XX:MetaspaceSize=21807104 (20.79 MB)jinfo -flag MetaspaceSize <jvmid> # Check the value of MaxMetaspaceSize, the default is -XX:MaxMetaspaceSize=18446744073709547520 (the value exceeds the maximum memory of the host, and is considered infinite or consistent with the host)jinfo -flag MaxMetaspaceSize <jvmid>
1.7 Check the status of the docker container
#On the docker container (host) terminaldocker stats # Check the status of a docker containerdocker stats <container_name | id>
The command result is as follows:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 38b70395877a knx-organization-service 0.49% 426.4MiB / 3.682GiB 11.31% 4.88MB / 4.35MB 47.6MB / 0B 57 7f9fbd4518a4 user-account-service 0.38% 479.4MiB / 3.682GiB 12.71% 1.84MB / 2.67MB 72.1MB / 0B 48
illustrate:
| List name | describe |
|---|---|
| CONTAINER ID and Name | The ID and name of the container |
| CPU % and MEM % | Percentage of host CPU and memory that the container is using |
| MEM USAGE / LIMIT | The total memory that the container is using and the total amount of memory allowed to be used |
| NET I/O | The amount of data a container sends and receives through its network interface |
| BLOCK I/O | The amount of data the container has read and written from the block device on the host |
| PIDs | The number of processes or threads created by the container |
I/O devices are roughly divided into two categories: block devices and character devices. The block device stores information in fixed-sized blocks, each block has its own address.
The size of the data block is usually between 512 bytes and 32768 bytes. The basic feature of a block device is that each block can be read and written independently of other blocks. Disks are the most common block devices
Based on the first result, we can know:
- The maximum memory limit for docker containers is
3.682GiB, in fact, is the entire memory of the host, which is equivalent to unlimited. - The memory size used by the docker container is
426.4MiB,and11.31% * 3.682GiBConsistent, it is actually the actual memory usage size of the java process in the container (very close), which means that the docker container does not consume much extra memory except for the service. - The number of processes or threads in the docker container is 57, and the process has only one java service, so it can be regarded as the number of threads for the java service.
1.8 Calculate performance parameters
It is known that the current space capacity of the elderly is341.5 MB, and then based on the results collected from the two event processes, namely, the Cenozoic GC and Full GC, it can be calculated:
- Space utilization rate in the elderly: 16.94% ~ 21.17%
- Space utilization interval for the elderly: 57.8501 MB ~ 72.29555 MB
- By default, the old age accounts for 2/3 of the heap memory, and the minimum heap size that can be calculated can be calculated: 72.29555/0.66=109.53871 MB
- Push the minimum heap size up a closest 1024 divisor: 1024/8=128 MB
- Non-heap memory will not be collected by garbage, and can be regarded as a permanent generation. The amount of current use is basically only needed, according to
CCSCCalculate the closest 1024 divisor upwards: 1024/64=16 MB - The current number of threads is 57. Considering that the current instance is a development environment, the concurrency rate is very low, and the maximum number of threads is 100.
2. Modify JVM parameters -Xmx, -Xms, -XX:MetaspaceSize, -XX:MaxMetaspaceSize
Java8 has already removed the permanent generation space (PermGen or permanent generation)-XX:PermSize、-XX:MaxPermSizeBoth parameters are invalid.
Instead, metaspace:
-
-XX:MetaspaceSize: Minimum metaspace size: It is not the initial metaspace size. The metaspace is 0 at the beginning and continues to expand. Until MetaspaceSize, Full GC will not be triggered. Once the amplification exceeds MetaspaceSize, Full GC will be triggered every amplification. -
-XX:MaxMetaspaceSize: Maximum metaspace size: the upper limit of amplification, the default is unlimited. Set a value to avoid a service from occupying the entire server's memory due to errors loading classes, thereby affecting the operation of other services.
Modify the project imageDockerfilemiddlejavaJVM parameters of the command
java \ -Xmx128m \ -Xms128m \ -XX:MetaspaceSize=128m \ -XX:MaxMetaspaceSize=256m \ ......
And rebuild the image and run it.
3. Estimate the memory limit of Docker containers
Max memory = [-Xmx] + [-XX:MaxMetaspaceSize] + number_of_threads * [-Xss]
Summarize
The above is personal experience. I hope you can give you a reference and I hope you can support me more.