Packet drops on this system for connections using ip_conntrack
or nf_conntrack
. Following messages seen in /var/log/kern
on the centos nodes when one of the instances drops packets:
$ tail -f /var/log/kern
Jul 4 03:47:16 centos kernel: : nf_conntrack: table full, dropping packet
Jul 4 03:47:16 centos kernel: : nf_conntrack: table full, dropping packet
This can happen when you are being attacked, or is also very likely to happen on a busy server even if there is no malicious activity.
NOTE: By default, CentOS will set this maximum to 65,536 connections. This is enough for lightly loaded servers, but can easily be exhausted on heavy traffic servers.
View the current maximum configured connections
$ cat /proc/sys/net/netfilter/nf_conntrack_max
To see the current used connections
$ cat /proc/sys/net/netfilter/nf_conntrack_count
Increase maximum configured connections limit
# Temporarily Solution
echo 524288 > /proc/sys/net/netfilter/nf_conntrack_max
# Permanent Solution
# Add following line on /etc/rc.d/rc.local
$ vim /etc/rc.d/rc.local
echo 524288 > /proc/sys/net/netfilter/nf_conntrack_max
$ chmod a+x /etc/rc.d/rc.local