Symptoms

There are many records for connection tracking in the file "/proc/self/net/nf_conntrack" on the node for a container's IP address "xx.xx.xx.73":

ipv4     2 tcp      6 363312 ESTABLISHED src=xx.xx.xx.73 dst=yy.yy.yy.254 sport=80 dport=60440 [UNREPLIED] src=yy.yy.yy.254 dst=xx.xx.xx.73 sport=60440 dport=80 mark=0 secmark=0 use=2
ipv4     2 tcp      6 363090 ESTABLISHED src=xx.xx.xx.73 dst=46.202.252.61 sport=80 dport=63423 [UNREPLIED] src=yy.yy.yy.61 dst=xx.xx.xx.73 sport=63423 dport=80 mark=0 secmark=0 use=2
ipv4     2 tcp      6 362908 ESTABLISHED src=xx.xx.xx.73 dst=yy.yy.yy.6 sport=80 dport=1853 [UNREPLIED] src=yy.yy.yy.6 dst=xx.xx.xx.73 sport=1853 dport=80 mark=0 secmark=0 use=2

Cause

Possible causes for this to happen are:

  1. slow network connection between this container and the clients who use the service, Apache port 80) in this case.
  2. DDoS attack (SYN flood);
  3. traffic shaping enabled for a container.

The situation occurs in the following way:

  1. the client sends SYN packet to the server;
  2. the server replies with ACK packet and then sends data;
  3. while this backward packets is on the way to the client, the latter sends RST or FIN to terminate the connection;
  4. the connection gets closed on the client's side and on the server's side, but since ACK was seen in one direction only, the connection tracking is not initiated properly.

As the result, the table can be overflown, those records will be kept for a long time, 5 days in the default configuration.

Resolution

There are several publicly suggested solutions:

  1. decrease the value of net.netfilter.nf_conntrack_tcp_timeout_established from 432000 seconds (5 days) to some reasonable low value, e.g. 7200 (2 hours);

    [root@hwnode ~]# sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=7200 >> /etc/sysctl.conf
  2. disable connection tracking for affected services/ports in the container (-j NOTRACK);

  3. set net.netfilter.nf_conntrack_tcp_loose to 0.

    [root@hwnode ~]# sysctl -w net.netfilter.nf_conntrack_tcp_loose=0 >> /etc/sysctl.conf

Internal content