[LINUX] Socket communication possible SYN flooding on port 443. Sending cookies. Appears in the log
Introduction
Since possible SYN flooding on port 443. Sending cookies. was output to / var / log / messages, I made a note about Linux socket communication.
environment
- CentOS6
- Apache
What are SYN_cookies?
https://ja.wikipedia.org/wiki/SYN_cookies
tcp sequence number
https://tomslifestylelab.com/tcp-sequencing/
reference
https://qastack.jp/server/294209/possible-syn-flooding-in-log-despite-low-number-of-syn-recv-connections https://github.com/hiboma/hiboma/blob/master/kernel/net/net-backlog.md https://qiita.com/nk_yohn3301/items/515733ed244ee3f35522 https://blog.ssrf.in/posts/linux-backlog-memo/ https://christina04.hatenablog.com/entry/2016/12/31/124314 https://kazuhira-r.hatenablog.com/entry/2019/07/10/015733 http://nishitki.hatenablog.com/entry/2018/11/18/230425 https://qiita.com/smallpalace/items/14ea25ae07178f5ea6bd https://qiita.com/sato4557/items/f1205bef8dbfe8022832 https://blog.cloudflare.com/jp/syn-packet-handling-in-the-wild-jp/ http://d.hatena.ne.jp/nyant/20111216/1324043063 http://triplesky.blogspot.jp/2014/07/centos6dmesgpossible-syn-flooding-on.html
Conclusion
How to create a queue to receive syn by the kernel
https://blog.ssrf.in/posts/linux-backlog-memo/
The value of the backlog argument of the listen system call becomes the size of the queue The backlog is specified by the ListenBacklog directive in httpd.conf. It has been omitted.
https://thinkit.co.jp/article/118/2?page=0%2C1
Also, even if you set it in ListenBacklog, the restriction by OS has priority, and the restriction differs depending on the OS, so you have to be careful.
Relationship with OS https://software.fujitsu.com/jp/manual/manualfiles/M050000/B1WN4991/03/ihsaa/ihs00102.htm
It is the value obtained by adding the setting value of this directive to the maximum value (tcp_max_syn_backlog) of the connection establishing queue set in the Linux system.
Cause
If there is more syn communication than tcp_max_syn_backlog (the number of possible queues for connections that have not received an ACK), the OS will be recognized as a SYN Flood attack and will reject SYN packets, / var / log / message. Possible SYN flooding on port 443. Sending packets. was output to. 128 by default for tcp_max_syn_backlog
counter-measure
If there is no DDOS and ** possible SYN flooding on port 443. Sending cookies ** is logged, the following three kernel parameters need to be adjusted.
https://note.com/ujisakura/n/n443807235887
net.ipv4.tcp_max_syn_backlog Number of queues waiting for ACK after receiving SYN (affects SYN flood occurrence)
- Unit: Number of queues
- The number of queues that can be held for connections that have received SYN per port (= socket) on the Listen port and have not received ACK. Anything above this value will be recognized by the OS as a SYN Flood attack and will reject SYN packets.
Example net.ipv4.tcp_max_syn_backlog = 128→8192
net.core.somaxconn Socket limit for TCP connections accepted at the same time Number of TCP session queues that receive ACKs from SYN on the Listne port and accept the state before ESTAB at a time
- Unit: Number of queues `* It is displayed in Send-Q of netstat. `` ** * It is recommended to match the numbers of syn_backlog and somaxconn. ** ** Example net.core.somaxconn = 128→8192
** The backlog queue length is set to the value of the backlog argument specified during listen (). However, if the specified value is larger than the sysctl net.core.somaxconn, it will be truncated to the net.core.somaxconn value. ** **
If you are using httpd
When MaxClients is full, syn is detained at the value of sommax. If you are causing synflood, you may also need to increase the value of MaxClients
Is it possible to specify the backlog at the time of listening with ListenBackLog?
https://software.fujitsu.com/jp/manual/manualfiles/M050000/B1WN4991/03/ihsaa/ihs00102.htm
→ ListenBackLog is added to the kernel tcp_max_syn_backlog value. → The default of tcp_max_syn_backlog is 2048
ListenBackLog directive Set the number of additions for the connection waiting queue. The maximum number of connection waiting queues is the value obtained by adding the setting value of this directive to the maximum value (tcp_max_syn_backlog) of the connection establishing queue set in the Linux system. The number of additions can be from 1 to 2147483647.
https://milestone-of-se.nesuke.com/sv-basic/linux-basic/ss-netstat/
root@web2:~
# ss -tln | grep 443
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 :::443 :::*
Send-Q of listen is the value of backlog. If it is default, it is 128 of net.core.somaxconn = 128
For example, in nginx, the default is 511, and 511 is the backlog value as it is. It depends on the middle.
net.core.netdev_max_backlog Buffering area passed from NIC to OS before receiving LISTEN port
- Unit: Number of packets Example net.core.netdev_max_backlog = 1000→5000
Apache ListenBacklog value
Even if you set it in ListenBacklog, the limitation by OS has priority, and the limitation differs depending on the OS, so you have to be careful. Since the limit differs depending on the OS, it seems better to increase it here as well
Below the above reasons etc.
TCP session state transition
netstat status status list
Reference: [[Revised new edition] Linux system construction and operation technology for professionals](https://www.amazon.co.jp/%EF%BC%BB%E6%94%B9%E8%A8%82% E6% 96% B0% E7% 89% 88% EF% BC% BD% E3% 83% 97% E3% 83% AD% E3% 81% AE% E3% 81% 9F% E3% 82% 81% E3% 81% AELinux% E3% 82% B7% E3% 82% B9% E3% 83% 86% E3% 83% A0% E6% A7% 8B% E7% AF% 89% E3% 83% BB% E9% 81% 8B% E7% 94% A8% E6% 8A% 80% E8% A1% 93-% E4% B8% AD% E4% BA% 95% E6% 82% A6% E5% 8F% B8-ebook / dp / B01LZURAPK / ref = pd_rhf_gw_p_img_3? _encoding = UTF8 & psc = 1 & refRID = H7NNY7NRDFMMF1V27RMH)
Reference: https://www.atmarkit.co.jp/ait/articles/0207/20/news003.html
TIME_WAIT The state of receiving ACK in "CLOSING". Waiting for a timeout after active closing. Wait for a while to avoid reusing the same sequence number, port number, etc. (Wait for packets that were delayed on the network to arrive within this time, so do not conflict with them), then "CLOSED" Transition to and exit. Waiting for the end of the connection. After waiting for a while, transition to CLOSED and exit (disappears from netstat display)
Linux kernel TCP / IP protocol stack
https://blog.cloudflare.com/jp/syn-packet-handling-in-the-wild-jp/
-
There are ** two ** file descriptors and queues for sockets
-
backlog --The Accept queue contains fully established connections and your application is always available. ** When the process calls accept (), the socket is dequeued and passed to the application. ** This means that the connected socket will be available to the user process via the file descriptor.
Reference: http://u-kipedia.hateblo.jp/entry/2015/01/01/001135
Diagram containing file descriptors
See also: https://wiki.bit-hive.com/linuxkernelmemo/pg/listen%20backlog%20%E3%80%903.6%E3%80%91 Figure with two cues
** So when the user process accept (), it will be associated with the file descriptor from the backlog in this figure **
https://blog.ssrf.in/posts/linux-backlog-memo/
- When receiving SYN, store it in the listen queue
- Returns SYN / ACK and moves to accept queue at the same time
- TCP sockets have a queue called backlog
- There is a Listen Queue to store when a SYN is received and an Accept Queue to store when an ACK is received.
- Queue implementation is https://elixir.bootlin.com/linux/v4.15/source/include/net/request_sock.h#L50
- SYN_RECV, ESTABLISHED state is queued
- Removed from queue when accepted (2)
Kernel parameters
https://qiita.com/sion_cojp/items/c02b5b5586b48eaaa469
It is the following parameters to change the queue size in the above figure.
| parameter | Contents | default value |
|---|---|---|
| net.core.somaxconn = 65535 | The number of TCP session queues that can be accepted at one time. The number of TCP sessions is managed by the backlog, and those exceeding that are stored in the queue. Set the number of this queue here |
128 |
| net.core.netdev_max_backlog = 65536 | Maximum number of packets that can be queued when receiving a packet | 1000 |
| net.ipv4.tcp_max_syn_backlog = 65536 | The number of connections that can hold a connection that accepts SYN per socket and does not receive an ACK. If this value is exceeded, the OS will reject SYN packets. You can check the overflowed packets with netstat. However, this value is net.core.If greater than somaxconn, net.core.Give priority to the value of somaxconn. |
1024 |
Other
| parameter | Contents | default value |
|---|---|---|
| net.ipv4.tcp_syncookies = 1 | SYN flood attack countermeasures | 1 |
| net.core.rmem_max = 16777216 | Default and maximum size of TCP and UDP receive buffers. | 124928 |
| net.core.wmem_max = 16777216 | Default and maximum size of TCP and UDP send buffers. | 124928 |
| net.nf_conntrack_max = 1053616 | Maximum number of ESTABLISHED lists(Number of management on the OS side) | variable |
| net.ipv4.conf.default.accept_source_route = 0 | Deny packets with source routes. Originally the purpose is to test the route, so turn it off | 1 |
| net.ipv4.tcp_rmem = 4096 349520 16777216 | Data receive buffer size | variable |
| net.ipv4.tcp_timestamps = 0 | Measures to disconnect from the WAN side server when multiple hosts are clients in a NAT environment | 1 |
| net.ipv4.tcp_fin_timeout = 5 | FIN timeout time | 60 |
| net.ipv4.tcp_syn_retries = 3 | The number of retries to send tcp SYN. If it doesn't work even if you send SYN this number of times, give up |
5 |
| net.ipv4.tcp_synack_retries = 3 | When opening a connection as an acceptor, the kernel packs an ACK into the SYN and sends it to see the SYN it received earlier. | 5 |
| net.ipv4.ip_local_port_range = 10000 65535 | TCP/Range of ports used when sending IP 1024 if possible-65535 is good, but iptables and AWS have Network ACL Similarly, if you do not open it as needed, the packet will stop in the middle and you will be addicted to it, so be careful. $ cat /proc/sys/net/ipv4/ip_local_port_Can be confirmed in range |
32768 61000 |
| net.ipv4.tcp_tw_reuse = 1 | Reuse the connection from yourself. tcp_tw_recycle reuses the connection from the other party. | 0 |
| net.ipv4.tcp_tw_recycle = 0 | Reuse the connection from the other party. | 0 |
| net.ipv4.tcp_rfc1337 = 1 | Comply with RFC1337. TIME_If RST is received in WAIT state, TIME_Close the socket without waiting for the end of the WAIT period | 0 |
| net.ipv4.tcp_fin_timeout = 5 | The timeout should be as short as possible | 60 |
| net.ipv4.tcp_max_tw_buckets = 65536 | Time held by the system at the same time-Maximum number of wait sockets. To prevent DoS attacks, it is better not to lower it. |
dynamic |
| net.ipv4.tcp_orphan_retries = 0 | The number of resends before terminating the closed TCP connection from here. A web server can consume resources, so it's better to keep it low. |
0 |
| net.ipv4.tcp_keepalive_probes = 5 | The number of times TCP sends keepalive probes. When this number is reached, the connection is considered broken. |
9 |
| net.ipv4.tcp_slow_start_after_idle = 0 | If there is no communication for a while even after keepalive, reset the gesture window size. Off is good in an environment where keepalive is used and there is little congestion. |
1 |
| net.ipv4.conf.eth180.arp_ignore = 1 | Do not let the hanging AP servers respond to arp (do not remember the MAC address) If the IP and mac address are not related, you will not be able to communicate with the device you want to send. |
0 |
| net.ipv4.conf.eth180.arp_announce = 2 | Do not let the hanging AP servers respond to arp (do not remember the MAC address) If the IP and mac address are not related, you will not be able to communicate with the device you want to send. |
0 |
| vm.swappiness = 0 | Do not use swap with real memory | 0 |
| vm.overcommit_memory = 2 | Do not allocate more memory to processes than real memory | 0 |
| vm.overcommit_ratio = 99 | If it kills itself with OOM killer, it detects out of memory and handles the error. vm.overcommit_memory =Set when 2 |
50 |
| kernel.shmmax =Number of memory bytes | Maximum size of shared memory. Server on-board memory(1GB)Because it changes according to | dynamic? |
| kernel.shmall = 68719476736 | Maximum number of shared memory pages for the entire system | 68719476736 |
| kernel.msgmnb = 65536 | Maximum number of messages that can be held in one message queue | 65536 |
| kernel.msgmax = 65536 | Maximum message queue ID | 65536 |
| kernel.sysrq = 0 | SysRq is a specific key (Alt) if the kernel did yesterday when the server became unable to operate the keyboard.+ PrintScreen)Can be interrupted by. 0 because it is not used | 0 |
| kernel.core_uses_pid = 1 | At the time of core dump".PID"Do not add to the core file name | 0 |
Apache backlog
https://thinkit.co.jp/article/118/2?page=0%2C1
ListenBackLog directive
Connections that connect when the MaxClients limit is exceeded are Instead of being rejected, up to the number set by the "ListenBacklog" directive (default value is 511) is stored in the queue as waiting for processing and put on hold. The connections stored in the queue have a mechanism to respond to requests in sequence as soon as the established connection is released (Fig. 2).
** Even if you set it in ListenBacklog, the restriction by OS has priority, and the restriction differs depending on the OS, so you have to be careful. ** **
Supplement: Socket communication confirmation command
netstat command
netstat -npt
https://network-beginners-handbook.com/netstat/
"Protocol" is the transport layer protocol name such as "TCP" or "UDP".
- If you run "netstat" without options, UDP will not be displayed.
- UDP is not displayed because UDP does not establish a connection. --netstat -a also shows udp
- "State" represents the TCP state transition.
Checking the syn queue size
SYN_RECV is the state before becoming "ESTABLISHED") So when the syn flood is up If you grep with SYN_RECV and wc, you can see the current number of queues
Confirm drop
$ netstat -s
TcpExt:
...
1380 times the listen queue of a socket overflowed
1380 SYNs to LISTEN sockets dropped
...
https://blog.ssrf.in/posts/linux-backlog-memo/
LINUX_MIB_LISTENOVERFLOWS and LINUX_MIB_LISTENDROPS increase when the queue is full
# nstat -az TcpExtListenDrops
#kernel
TcpExtListenDrops 119 0.0
root@web2:~
# nstat -az TcpExtListenOverflows
#kernel
TcpExtListenOverflows 119 0.0
https://qastack.jp/superuser/661188/what-is-in-the-local-address-of-netstat-output