What I personally fix well
Socket is also included in the FD, but it is overlooked. This kind of error occurs almost first. Nginx Too many open files error etc. Check with ʻulimit -n`, but note that it is executed by the user who encountered the error. It may be different for root.
[anyuser]$ ulimit -n
1024
In this case, only 1024 FDs can be used.
Modify limits.conf. Note that depending on the case, only root is often set to FD, and if nothing is set, the default will be 1024. Change both the soft limit and the hard limit. The value is determined as appropriate. The following is when changing the FD of anyuser.
/etc/security/limits.conf
* soft core unlimited
* hard core unlimited
root soft nofile 65536
root hard nofile 65536
anyuser soft nofile 65536 ← Add here
anyuser hard nofile 65536 ← Add here
Even if it is described in /etc/security/limits.conf, it may not be reflected. Ssh to the server, check the process ID of the process, and check with cat / proc / <process_ID> / limits.
If throughput does not come out even though it is not a resource bottleneck, any port may be exhausted.
In most cases, if you check the server with netstat, a large amount of TIME_WAIT may occur and any port may be used up.
It is quite common. https://qiita.com/kuni-nakaji/items/c07004c7d9e5bb683bc2 This can be confirmed with the following command.
[root@]# netstat -antp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:111 0.0.0.0: LISTEN 1/systemd
tcp 0 0 127.0.0.1:25 0.0.0.0: LISTEN 1211/master
tcp 0 0 0.0.0.0:12127 0.0.0.0: LISTEN 8689/sshd
tcp 0 0 10.140.180.223:12127 10.140.50.217:61738 ESTABLISHED 14604/sshd: appladm
tcp6 0 0 :::40042 ::: LISTEN 12232/java
tcp6 0 0 :::111 ::: LISTEN 1/systemd
tcp6 0 0 :::8080 ::: LISTEN 12232/java
tcp6 0 0 :::34710 ::: LISTEN 12232/java
tcp6 0 0 :::12120 ::: LISTEN 12232/java
tcp6 0 0 ::1:25 ::: LISTEN 1211/master
tcp6 0 0 :::12127 ::: LISTEN 8689/sshd
tcp6 0 0 10.140.180.223:42881 10.140.197.150:3306 ESTABLISHED 12232/java
Modify the following ip_local_port_range and modify tcp_tw_reuse to 1. (Add if there is no entry for tcp_tw_reuse.) It is not recommended to modify other parameters.
/etc/sysctl.conf
net.ipv4.ip_local_port_range = 30000 65500
net.ipv4.tcp_tw_reuse = 0
Reflect with sysctl -p when completed.