Theo Julienne

Ethernet MTU and TCP MSS: Why connections stall

Fri, 21 Aug 2020 00:00:00 +0000

MTU and MSS are two terms that are easily mistaken and their misconfiguration is often the cause of networking problems. Spending enough time working on production systems that interface with large networks of computers or the Internet almost guarantees coming across situations where an interface was configured with the wrong MTU, or a firewall was filtering ICMP. This results in a client being unable to transfer large amounts of data when smaller transfers work fine. This post will walk through MTU, MSS and packet size negotiation for TCP connections, and the common situations where it breaks down. This post was inspired by multiple discussions during the course of investigating errors on production systems as part of my role at GitHub.

If you want to take away a simple snippet from this post, the summary is:

The MTU of the interfaces on either side of a physical
or logical link must be equal. Don't block ICMP.

The examples mentioned in this blog post will be reproducible in the lab from theojulienne/blog-lab-mtu - clone this repository and bring up the lab, then poke around at these examples in a real system:

$ git clone https://github.com/theojulienne/blog-lab-mtu.git
$ cd blog-lab-mtu
$ vagrant up
$ vagrant ssh

Ethernet MTU: Maximum Transmission Unit

The MTU (Maximum Transmission Unit) on an Ethernet network specifies the maximum payload size of the data to be transmitted along with an Ethernet header on a network. Typically this payload will be an IP packet, in which case the MTU specifies the maximum combined size of the IP header and IP data.

The MTU is specified at the interface level as it is a link-level setting, and is typically propagated down to the underlying network card driver. The expectation is that packets that are larger than this configured size that appear to be transmitted over the wire are invalid or corrupt and should be dropped. In a valid configuration, hosts connected together via a link will have the same MTU specified:

If interfaces on either side of a link have mismatching MTU configurations, then the smaller side will treat packets larger than the local MTU as invalid and drop the packets before any software has the chance to see them.

Streams of data that are larger than the MTU will be broken up into packets that completely fill an Ethernet frame, up to the MTU in each. If the remote end has a smaller MTU configured for the same link, those larger packets will be dropped. MTU should be configured the same on both interfaces on either side of a link, and so the MTU should be considered a bidirectional maximum.

MTU in the lab

The lab in this blog post can be used to observe this in an example system. In one terminal, bring up the lab hosts inside the Vagrant machine:

$ vagrant ssh -- /vagrant/bin/run-lab

In another terminal, vagrant ssh then enable the first scenario from above with matching MTU of 1500 on client and server:

vagrant@blog-lab-mtu:/vagrant$ /vagrant/bin/scenario direct_1500

Log in to the client and server hosts and observe that we can send a packet with 1400 bytes of payload as expected, since both hosts have an MTU of 1500. The -s 1400 argument to ping sets the payload size, and the -M do argument instructs ping to set the DF (Don’t Fragment) bit, ensuring that the whole IP packet must arrive in one piece or not at all.

vagrant@blog-lab-mtu:/vagrant$ /vagrant/bin/shell client
root@client:/# ping -c 2 -M do -s 1400 server-direct
PING server-direct (172.28.0.40) 1400(1428) bytes of data.
1408 bytes from server-direct (172.28.0.40): icmp_seq=1 ttl=64 time=0.096 ms
1408 bytes from server-direct (172.28.0.40): icmp_seq=2 ttl=64 time=0.080 ms

--- server-direct ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 2ms
rtt min/avg/max/mdev = 0.080/0.088/0.096/0.008 ms
root@client:/# exit
vagrant@blog-lab-mtu:/vagrant$ /vagrant/bin/shell server
root@server:/# ping -c 2 -M do -s 1400 client-direct
PING client-direct (172.28.0.10) 1400(1428) bytes of data.
1408 bytes from client-direct (172.28.0.10): icmp_seq=1 ttl=64 time=0.079 ms
1408 bytes from client-direct (172.28.0.10): icmp_seq=2 ttl=64 time=0.071 ms

--- client-direct ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 32ms
rtt min/avg/max/mdev = 0.071/0.075/0.079/0.004 ms
root@server:/# 

Now switch to the second scenario with mismatching MTU, and observe that 1400 byte payloads no longer succeed:

vagrant@blog-lab-mtu:/vagrant$ /vagrant/bin/scenario direct_mismatch

vagrant@blog-lab-mtu:/vagrant$ /vagrant/bin/shell client
root@client:/# ping -c 2 -M do -s 1400 server-direct
PING server-direct (172.28.0.40) 1400(1428) bytes of data.
ping: local error: Message too long, mtu=1200
ping: local error: Message too long, mtu=1200

--- server-direct ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 10ms

root@client:/# exit
vagrant@blog-lab-mtu:/vagrant$ /vagrant/bin/shell server
root@server:/# ping -c 2 -M do -s 1400 client-direct
PING client-direct (172.28.0.10) 1400(1428) bytes of data.

--- client-direct ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 19ms

root@server:/# 

Notice that the client host is immediately able to observe that it cannot send a packet this large since the MTU on the interface is 1200. However, the server host believes the MTU of the link is 1500, so sends the packet, however the client is unable to receive it. This occurs at such a low level that neither host is aware of the failure - the packet just disappears.

TCP MSS: Maximum Segment Size

The TCP MSS (Maximum Segment Size) sounds very similar to MTU, and since it relates to the maximum size of network packets, they are easy to conflate even though they are quite different. A TCP segment is the TCP header and TCP data that forms part of a single packet. The MSS specifies the expected maximum size of the data component of this segment that a host expects it would be able to receive without the IP packet being fragmented. IP fragmentation is typically disabled for TCP packets on modern networking stacks due to the added complexity and overhead, so the MSS represents the maximum size that the host expects to be able to receive in any given packet.

Rather than being an interface-level configuration like MTU, the MSS advertisement forms a part of the typical TCP handshake and is calculated based on the underlying MTU of the interface that a local host will use to communicate with a remote host. MSS can be thought of as a TCP hint around how much data can be included in a single TCP packet, given the current MTU. Each host calculates the MSS it will advertise by taking the local MTU and subtracting the size of the IP and TCP headers, then includes that MSS in the TCP options of the SYN or SYN-ACK packet as part of the TCP three-way handshake.

This is not a negotiation of a single MSS, but rather each host is giving the remote host an indication of the maximum size of a single packet it expects will be possible to send back. This number must be less than the MTU minus IP/TCP headers, since there’s no way any larger packet could arrive given the local MTU. Each host will use the remote host’s advertised MSS as a hint for what size individual outgoing packets should be. Since this is a configurable hint, it is also only unidirectional, and although a host may advertise a lower MSS than it can otherwise handle, that doesn’t in any way restrict it from sending packets larger than the MSS it advertised (providing the remote host allowed for it).

The simple MSS exchange happens to work around small misconfigurations of MTU, such as the trivial example described above:

In this case, the client would advertise an MSS of 1200 (MTU) - 20 (IP hdr) - 20 (TCP hdr) = 1160, which would cause the server to refrain from sending packets that contained more than 1160 bytes in the TCP payload, which also ensures it would be able to arrive within the bounds of the MTU of 1200 once those headers are added on.

However, the above network is still misconfigured, since even though TCP happens to work around it, other protocols will fail since they don’t exchange MSS values. MSS is actually intended to allow hosts to work around valid configurations where their own local networks have different MTU, such as the following:

In this example, if the server with a valid MTU of 9000 attempted to send an Ethernet frame containing more than 1500 bytes without fragmentation being allowed, that packet would not be able to make it to the client. The intermediary router, being the first host that is aware of this problem as it is aware of the MTU of both links, would send an ICMPv4 “Fragmentation required, but DF set” message or an ICMPv6 “Packet Too Big” message back to the sender to inform it that forwarding the packet without breaking it up is not possible (and that the IP header had the DF, or Don’t Fragment, bit set).

However, TCP will succeed in unrestricted communications between these hosts due to the MSS advertisements. The server in this configuration will receive an MSS from the client that will ensure no Ethernet frames with a payload larger than 1500 bytes are generated, so they will be received successfully.

MSS in the lab

Select the scenario from above with the client on a network with 1500 MTU and the server on a network with 9000 MTU:

vagrant@blog-lab-mtu:/vagrant$ /vagrant/bin/scenario client_net_smaller

Running a ping from the side with the larger MTU, we can observe that packets larger than the client’s MTU cause the intermediary router to return an ICMP message since it is unable to forward the packet:

vagrant@blog-lab-mtu:/vagrant$ /vagrant/bin/shell server
root@server:/# ping -c 2 -M do -s 3000 client
PING client (172.29.0.10) 3000(3028) bytes of data.
From 172.30.0.20 icmp_seq=1 Frag needed and DF set (mtu = 1500)
ping: local error: Message too long, mtu=1500

--- client ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 18ms
pipe 2
root@server:/# 

In a slightly more complex example, open up a few terminals and spin up a simple HTTP server that sends a large payload and observe in a tcpdump that the MSS advertisements allow the connection to succeed despite the differing MTU:

# reset everything so Linux doesn't remember that ICMP frag message from above
vagrant@blog-lab-mtu:/vagrant$ /vagrant/bin/flush-all-route-caches

vagrant@blog-lab-mtu:/vagrant$ /vagrant/bin/shell server
root@server:/# sample-http-server 

vagrant@blog-lab-mtu:/vagrant$ /vagrant/bin/shell server
root@server:/# tcpdump -i any icmp or port 80

vagrant@blog-lab-mtu:~$ /vagrant/bin/shell client
root@client:/# curl http://server/

The tcpdump should return something like the following - note the MSS advertised by each side in the first 2 SYN packets is the MTU minus the IP and TCP header size of 40 bytes - mss 1460 and mss 8960. The packets with an HTTP payload are broken into smaller packets with a TCP segment of just 1448 bytes - small enough to fit inside an MTU of 1500 with an IP and TCP header with 12 additional bytes for TCP options (you can observe those options where it says [nop,nop,TS val 3808245569 ecr 3455879517]).

IP client.51424 > server.80: Flags [S], seq 4195639166, win 64240, options [mss 1460,sackOK,TS val 3456553938 ecr 0,nop,wscale 6], length 0
IP server.80 > client.51424: Flags [S.], seq 3403777541, ack 4195639167, win 62636, options [mss 8960,sackOK,TS val 3808919991 ecr 3456553938,nop,wscale 6], length 0
IP client.51424 > server.80: Flags [.], ack 1, win 1004, options [nop,nop,TS val 3456553939 ecr 3808919991], length 0
IP client.51424 > server.80: Flags [P.], seq 1:71, ack 1, win 1004, options [nop,nop,TS val 3456553939 ecr 3808919991], length 70: HTTP: GET / HTTP/1.1
IP server.80 > client.51424: Flags [.], ack 71, win 978, options [nop,nop,TS val 3808919991 ecr 3456553939], length 0
IP server.80 > client.51424: Flags [P.], seq 1:114, ack 71, win 978, options [nop,nop,TS val 3808920010 ecr 3456553939], length 113: HTTP: HTTP/1.1 200 OK
IP client.51424 > server.80: Flags [.], ack 114, win 1003, options [nop,nop,TS val 3456553958 ecr 3808920010], length 0
IP server.80 > client.51424: Flags [.], seq 114:1562, ack 71, win 978, options [nop,nop,TS val 3808920011 ecr 3456553958], length 1448: HTTP
IP server.80 > client.51424: Flags [.], seq 1562:3010, ack 71, win 978, options [nop,nop,TS val 3808920011 ecr 3456553958], length 1448: HTTP
IP server.80 > client.51424: Flags [.], seq 3010:4458, ack 71, win 978, options [nop,nop,TS val 3808920011 ecr 3456553958], length 1448: HTTP
IP server.80 > client.51424: Flags [P.], seq 4458:4915, ack 71, win 978, options [nop,nop,TS val 3808920011 ecr 3456553958], length 457: HTTP
IP client.51424 > server.80: Flags [.], ack 1562, win 1002, options [nop,nop,TS val 3456553959 ecr 3808920011], length 0
IP client.51424 > server.80: Flags [.], ack 3010, win 995, options [nop,nop,TS val 3456553959 ecr 3808920011], length 0
IP client.51424 > server.80: Flags [.], ack 4458, win 984, options [nop,nop,TS val 3456553959 ecr 3808920011], length 0
IP client.51424 > server.80: Flags [.], ack 4915, win 980, options [nop,nop,TS val 3456553959 ecr 3808920011], length 0
IP client.51424 > server.80: Flags [F.], seq 71, ack 4915, win 1002, options [nop,nop,TS val 3456553960 ecr 3808920011], length 0
IP server.80 > client.51424: Flags [F.], seq 4915, ack 72, win 978, options [nop,nop,TS val 3808920013 ecr 3456553960], length 0
IP client.51424 > server.80: Flags [.], ack 4916, win 1002, options [nop,nop,TS val 3456553961 ecr 3808920013], length 0

One interesting note is that on many modern network devices, running a packet capture may result in tcpdump and similar tools observing packets that appear larger than the configured MTU due to Large Receive Offload and Large Send Offload and other technologies which coalesce multiple packets that are part of the same flow into a single pseudo-packet. On receive, the network card will coalesce subsequent packets from a stream together before passing them to the kernel as a single packet for faster processing. On send, the kernel will provide one larger packet that the network card will split appropriately as it sends over the wire based on the configured MSS.

This packet coelescing has been intentionally disabled in the lab to make it simpler to observe when packets are being split up on the (virtual) wire, however if the same example was run on real server the HTTP payload would likely appear to tcpdump as a single larger packet, though it would still be broken up the same way on the wire.

Path MTU: Hidden bottlenecks

Although in the above example, TCP MSS was able to work around a simple configuration where hosts had valid but differing MTUs on their links, this is still not a complete solution as there may be additional intermediary links involved with an MTU that is lower than either the client or server link.

In this example, all Ethernet payloads larger than 1200 bytes from either side will not be able to be forwarded past the first hop (if IP fragmentation is disabled). However, both client and server will advertise an MSS that will allow for Ethernet payloads larger than 1200 bytes to be sent.

With full visibility of the network, using a diagram like we have here, we can see that packets can only make it between client and server if they are no more than 1200 bytes including headers. This is the Path MTU, or the minimum MTU of all links on the path between communicating hosts. In practice, where hosts are communicating arbitrarily over the Internet and where multiple paths could be available between those hosts, we don’t have visibility into the full system and therefore we are unable to put a specific number on the Path MTU up front. Instead, it must be possible for hosts to discover this Path MTU as needed during existing communications, as the need for it arises.

Path MTU Discovery

Path MTU Discovery is the process of hosts working from the local MTU and the remote initial MSS advertisement as hints, and arriving at the actual Path MTU of the (current) full path between those hosts in each direction.

The process starts by assuming that the advertised MSS is correct for the full path, after reducing it if the local link’s MTU minus IP/TCP header size is smaller (since we couldn’t send a larger packet regardless of the MSS). When a packet is sent that is larger than the smallest link along the path, it will at least make it one hop to the first router, since we know the local link MTU is large enough to fit it.

When a router receives a packet on one interface and needs to forward it to another interface that the packet cannot fit on, the router sends back an ICMPv4 “Fragmentation required” message or an ICMPv6 “Packet Too Big” message. The router includes the MTU of the next (smaller) hop in that message, since it knows it. Upon receipt of that message, the originating host is able to reduce the calculated Path MTU for communications with that remote host, and resend the data as multiple smaller packets. From then on, packet size is correctly limited by the size of the MTU of the smallest link in the path observed so far.

A full example is below, though note that in practice there may not be complete symmetry in the path in each direction, multiple hops may progressively have smaller MTU values along the way, and the path may even change throughout the lifetime of a single connection:

This example shows how critical it is for TCP that ICMP messages of this type are forwarded correctly. This exchange is where most problems around MTU occur in production systems, when firewalls along the path block or throttle ICMP traffic in a way that inhibits Path MTU Discovery. Don’t block ICMP, it will break Path MTU Discovery and also TCP connections with large data transfers where the initial MSS advertisement is not enough to limit the Path MTU. At the very least, don’t block ICMPv4 “Fragmentation required” or ICMPv6 “Packet Too Big”, even if you block other ICMP messages.

The common traceroute utility observes hops between hosts using the TTL to observe each hop via TTL Exceeded messages, and this can be extended to show Path MTU (as well as the hops along the way), which is functionality that the tracepath utility provides. tracepath sends large packets, starting at the maximum sendable on the local link, to a remote host and shows any ICMP messages and the adjusted Path MTU along the way as it gradually increases TTL and decreases packet size. tracepath is a good first place to start when diagnosing issues observed between 2 hosts where MTU misconfiguration or ICMP filtering is suspected.

Path MTU Discovery in the lab

Select the scenario from above with the client on a network with 1500 MTU, the server on a network with 9000 MTU, and an additional intermediary network with 1200 MTU that packets must traverse:

vagrant@blog-lab-mtu:/vagrant$ /vagrant/bin/scenario hidden_smaller

Observe that neither side can immediately ascertain the correct Path MTU and must see an ICMP message from the intermediary router before they become aware of the smaller link:

vagrant@blog-lab-mtu:/vagrant$ /vagrant/bin/shell client
root@client:/# ping -c 2 -M do -s 1400 server
PING server (172.31.0.40) 1400(1428) bytes of data.
From vagrant_router-a_1.vagrant_client_router_a (172.29.0.20) icmp_seq=1 Frag needed and DF set (mtu = 1200)
ping: local error: Message too long, mtu=1200

--- server ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 2ms

root@client:/# exit
vagrant@blog-lab-mtu:/vagrant$ /vagrant/bin/shell server
root@server:/# ping -c 2 -M do -s 1400 client
PING client (172.29.0.10) 1400(1428) bytes of data.
From vagrant_router-b_1.vagrant_router_b_server (172.31.0.30) icmp_seq=1 Frag needed and DF set (mtu = 1200)
ping: local error: Message too long, mtu=1200

--- client ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 3ms

root@server:/# 

Bringing up the example HTTP server from earlier, we can also observe the full process off Path MTU Discovery. In this case, note that we observe the tcpdump from the router-b on its interface towards server since it has a better vantage point for observing retransmits.

# reset everything so Linux doesn't remember that ICMP frag message from above
vagrant@blog-lab-mtu:/vagrant$ /vagrant/bin/flush-all-route-caches

vagrant@blog-lab-mtu:/vagrant$ /vagrant/bin/shell server
root@server:/# sample-http-server 

vagrant@blog-lab-mtu:/vagrant$ /vagrant/bin/shell router-b # better vantage point
root@router-b:/# tcpdump -i eth1 icmp or port 80

vagrant@blog-lab-mtu:~$ /vagrant/bin/shell client
root@client:/# curl http://server/

The tcpdump will return something like the following:

IP client.51428 > server.80: Flags [S], seq 644598568, win 64240, options [mss 1460,sackOK,TS val 3457600577 ecr 0,nop,wscale 6], length 0
IP server.80 > client.51428: Flags [S.], seq 1840446146, ack 644598569, win 62636, options [mss 8960,sackOK,TS val 3809966630 ecr 3457600577,nop,wscale 6], length 0
IP client.51428 > server.80: Flags [.], ack 1, win 1004, options [nop,nop,TS val 3457600578 ecr 3809966630], length 0
IP client.51428 > server.80: Flags [P.], seq 1:71, ack 1, win 1004, options [nop,nop,TS val 3457600578 ecr 3809966630], length 70: HTTP: GET / HTTP/1.1
IP server.80 > client.51428: Flags [.], ack 71, win 978, options [nop,nop,TS val 3809966630 ecr 3457600578], length 0
IP server.80 > client.51428: Flags [P.], seq 1:114, ack 71, win 978, options [nop,nop,TS val 3809966650 ecr 3457600578], length 113: HTTP: HTTP/1.1 200 OK
IP client.51428 > server.80: Flags [.], ack 114, win 1003, options [nop,nop,TS val 3457600598 ecr 3809966650], length 0
IP server.80 > client.51428: Flags [.], seq 114:1562, ack 71, win 978, options [nop,nop,TS val 3809966650 ecr 3457600598], length 1448: HTTP
IP router-b > server: ICMP client unreachable - need to frag (mtu 1200), length 556
IP server.80 > client.51428: Flags [.], seq 1562:3010, ack 71, win 978, options [nop,nop,TS val 3809966650 ecr 3457600598], length 1448: HTTP
IP router-b > server: ICMP client unreachable - need to frag (mtu 1200), length 556
IP server.80 > client.51428: Flags [.], seq 3010:4458, ack 71, win 978, options [nop,nop,TS val 3809966650 ecr 3457600598], length 1448: HTTP
IP router-b > server: ICMP client unreachable - need to frag (mtu 1200), length 556
IP server.80 > client.51428: Flags [P.], seq 4458:4915, ack 71, win 978, options [nop,nop,TS val 3809966650 ecr 3457600598], length 457: HTTP
IP client.51428 > server.80: Flags [.], ack 114, win 1003, options [nop,nop,TS val 3457600598 ecr 3809966650,nop,nop,sack 1 {4458:4915}], length 0
IP server.80 > client.51428: Flags [.], seq 114:1262, ack 71, win 978, options [nop,nop,TS val 3809966650 ecr 3457600598], length 1148: HTTP
IP server.80 > client.51428: Flags [.], seq 1262:2410, ack 71, win 978, options [nop,nop,TS val 3809966650 ecr 3457600598], length 1148: HTTP
IP server.80 > client.51428: Flags [.], seq 2410:3558, ack 71, win 978, options [nop,nop,TS val 3809966650 ecr 3457600598], length 1148: HTTP
IP server.80 > client.51428: Flags [.], seq 3558:4458, ack 71, win 978, options [nop,nop,TS val 3809966650 ecr 3457600598], length 900: HTTP
IP client.51428 > server.80: Flags [.], ack 1262, win 993, options [nop,nop,TS val 3457600599 ecr 3809966650,nop,nop,sack 1 {4458:4915}], length 0
IP client.51428 > server.80: Flags [.], ack 2410, win 976, options [nop,nop,TS val 3457600599 ecr 3809966650,nop,nop,sack 1 {4458:4915}], length 0
IP client.51428 > server.80: Flags [.], ack 3558, win 967, options [nop,nop,TS val 3457600599 ecr 3809966650,nop,nop,sack 1 {4458:4915}], length 0
IP client.51428 > server.80: Flags [.], ack 4915, win 970, options [nop,nop,TS val 3457600599 ecr 3809966650], length 0
IP server.80 > client.51428: Flags [F.], seq 4915, ack 71, win 978, options [nop,nop,TS val 3809966651 ecr 3457600599], length 0
IP client.51428 > server.80: Flags [F.], seq 71, ack 4916, win 1002, options [nop,nop,TS val 3457600600 ecr 3809966651], length 0
IP server.80 > client.51428: Flags [.], ack 72, win 978, options [nop,nop,TS val 3809966652 ecr 3457600600], length 0

Note that the advertised MSS values are the same as the earlier example, which does not reflect the Path MTU since it is not yet known. Each of the initial large packets sent from the server to the client cause an ICMP fragmentation message:

IP server.80 > client.51428: Flags [.], seq 114:1562, ack 71, win 978, options [nop,nop,TS val 3809966650 ecr 3457600598], length 1448: HTTP
IP router-b > server: ICMP client unreachable - need to frag (mtu 1200), length 556
IP server.80 > client.51428: Flags [.], seq 1562:3010, ack 71, win 978, options [nop,nop,TS val 3809966650 ecr 3457600598], length 1448: HTTP
IP router-b > server: ICMP client unreachable - need to frag (mtu 1200), length 556
IP server.80 > client.51428: Flags [.], seq 3010:4458, ack 71, win 978, options [nop,nop,TS val 3809966650 ecr 3457600598], length 1448: HTTP
IP router-b > server: ICMP client unreachable - need to frag (mtu 1200), length 556

The server then resends the failing packets, this time respecting the newly calculated Path MTU of 1200:

IP server.80 > client.51428: Flags [.], seq 114:1262, ack 71, win 978, options [nop,nop,TS val 3809966650 ecr 3457600598], length 1148: HTTP
IP server.80 > client.51428: Flags [.], seq 1262:2410, ack 71, win 978, options [nop,nop,TS val 3809966650 ecr 3457600598], length 1148: HTTP
IP server.80 > client.51428: Flags [.], seq 2410:3558, ack 71, win 978, options [nop,nop,TS val 3809966650 ecr 3457600598], length 1148: HTTP
IP server.80 > client.51428: Flags [.], seq 3558:4458, ack 71, win 978, options [nop,nop,TS val 3809966650 ecr 3457600598], length 900: HTTP

We can also use tracepath to perform Path MTU Discovery and observe which routers are responding - the tool starts with the local network’s MTU then discovers the reduced MTU link as it progresses:

# reset everything so Linux doesn't remember that ICMP frag message from above
vagrant@blog-lab-mtu:~$ /vagrant/bin/flush-all-route-caches

vagrant@blog-lab-mtu:~$ /vagrant/bin/shell client
root@client:/# tracepath -n server
 1?: [LOCALHOST]                      pmtu 1500
 1:  172.29.0.20                                           0.088ms 
 1:  172.29.0.20                                           0.032ms 
 2:  172.29.0.20                                           0.030ms pmtu 1200
 2:  172.30.0.30                                           0.046ms 
 3:  172.31.0.40                                           0.058ms reached
     Resume: pmtu 1200 hops 3 back 3 
root@client:/# exit
vagrant@blog-lab-mtu:~$ /vagrant/bin/shell server
root@server:/# tracepath -n client
 1?: [LOCALHOST]                      pmtu 9000
 1:  172.31.0.30                                           0.159ms 
 1:  172.31.0.30                                           0.034ms 
 2:  172.31.0.30                                           0.027ms pmtu 1200
 2:  172.30.0.20                                           0.092ms 
 3:  172.29.0.10                                           0.060ms reached
     Resume: pmtu 1200 hops 3 back 3 
root@server:/# 

The tracepath tool is extremely useful in determining whether a connection stalling failure is indeed a Path MTU blackhole due to a router or firewall blocking ICMP packets.

Path MTU Discovery and Anycast

One final complexity occurs when routers have multiple equal-cost paths (ECMP) to multiple hosts that share the same IP address, a common situation with deployments of Anycast. In this case, routers hash packets across the different available paths and attempt to be consistent so that packets from the same connection arrive on the same remote host (and/or travel via the same path).

However, the input to the hash may not (and typically does not) understand that an ICMP fragmentation or packet too big message is related to the TCP connection that triggered it since the IP source is different to a normal returning packet, and instead is a router along the way, not the expected remote host. This leads to a situation where one host receives the TCP packets for a connection, and another unrelated host receives the ICMP packet relating to that connection, which gets disregarded. This introduces a Path MTU blackhole, as if ICMP were being filtered.

In practice, there are ways to work around this issue. One way is to broadcast those ICMP messages to all hosts. An alternative approach is used by GLB Director which allows the routers to perform the ICMP-unaware ECMP hashing, but then re-hashes it correctly at the first software load balancer layer. GLB inspects inside ICMP messages, since they contain part of the triggering packet, and hashes those packets the same way they would be hashed if they were the original TCP packet that triggered them, ensuring ICMP messages land on the same host as the related TCP connection. In general, it’s important that any system involving hashing or otherwise manipulating TCP packets ensures that ICMP messages relating to the stream are sent to the appropriate host, as they are a crucial part of the way that TCP operates.

Wrapping up

It is often possible to ignore the details of MTU, MSS advertisement and Path MTU Discovery and have things continue to work to a certain extent. However, when these systems fail, connections will stall entirely and in a very blocking way for users. This is often seen only on large transfers, as smaller data transfers don’t trigger the issue, since the packets remain small. It’s also often only intermittent in cases where only one path between hosts has a reduced Path MTU, or just one path has a router blocking ICMP packets.

Thankfully, the rule for keeping networks functioning correctly with regards to MTU can be summarised simply as:

The MTU of the interfaces on either side of a physical
or logical link must be equal. Don't block ICMP.

Asking if this rule holds true both internally and externally in any trouble ticket that has the pattern of “Why is my connection stalling when (action that transfers large data) but not when (action that transfers small data)?” will almost always yield an MTU misconfiguration or ICMP filtering and a root cause.

Scaling Linux Services: Before accepting connections

Fri, 03 Jul 2020 00:00:00 +0000

When writing services that accept TCP connections, we tend to think of our work as starting from the point where our service accepts a new client connection and finishing when we complete the request and close the socket. For services at scale, operations can happen at such a high rate that some of the default resource limits of the Linux kernel can break this abstraction and start causing impact to incoming connections outside of that connection lifecycle. This post focuses on some standard resource limitations that exist before the client socket is handed to the application - all of which came up during the course of investigating errors on production systems as part of my role at GitHub (in some cases, multiple times across different applications).

In its most basic form (ignoring non-blocking variants), listening for TCP connections requires a call to listen() to actually start allowing incoming connections, followed by repeated calls to accept() to take the next pending connection and return a file descriptor that is for that particular client. In C this pattern looks something like:

int server_fd = socket(AF_INET, SOCK_STREAM, 0);
bind(server_fd, /* ... */);

// start listening for connections
listen(server_fd, 512);

while (running) {
    // block until a connection arrives and then accept it
    int client_fd = accept(server_fd, NULL, NULL);

    // ... handle client_fd ...
}

close(server_fd);

This is often hidden behind further layers of abstraction, and we tend to hide away all the implementation details of accepting connections and view it as a stream of new connections that we pick up and then process in parallel. However, when building a system that runs at scale, this abstraction tends to break down because there are resource limitations introduced in the period where connections are being established but are not yet returned by accept(). Before that point, those client connections are considered as being part of the server/listen socket and not as independent resources exposed to the application.

The examples mentioned in this blog post will be reproducible in the lab from theojulienne/blog-lab-scaling-accept - clone this repository and bring up the lab, then poke around at these examples in a real system:

$ git clone https://github.com/theojulienne/blog-lab-scaling-accept.git
$ cd blog-lab-scaling-accept
$ vagrant up
$ vagrant ssh

From SYN to accept()

The Linux kernel maintains 2 queues of connections that maintain the backlog of connections that are not yet accept()ed by the application:

When a SYN packet is received to initiate a new connection to a listen socket, a SYN-ACK is sent to the client and the half-completed connection state is stored in the “SYN backlog” or “Request Socket Queue”. This represents connections that have not yet been fully validated as having two-way communication between the hosts, where the server hasn’t yet validated that the remote end has successfully received a packet from the server (the SYN could have come from another host spoofing the source IP).

Once the client responds to the server’s SYN-ACK with an ACK, the connection has then completed the full TCP 3-way handshake, and the server knows that two-way communication has been established. At this point, the client connection is ready to be provided to the application at a future call to accept(), and is added to the appropriately named “accept queue”.

SYN backlog

Connections in the SYN backlog remain there for a period of time relative to the Round Trip Time between the server and the client. If there are N slots in the backlog then you can have at most N connections in this backlog per average RTT, after which the backlog overflows.

This won’t actually cause the connections to fail by default on Linux, but instead it will cause SYN cookies to be sent. This is due to the server being unable to validate that the client is who they say they at the point where only a SYN has been received. Before receiving an ACK that contains the sequence number that the server sent to the client in the SYN-ACK, the client could be spoofing packets as coming from a different IP. This is a common Denial of Service attack called a SYN Flood which is so common that the Linux kernel has built in mitigation by sending a SYN cookie when there is no room in the SYN backlog for the new connection.

This means that if the SYN backlog overflows under normal circumstances with no DoS attack, the kernel allows the connection to move further along in the handshake, and will not store any resources for the connection until an ACK completes the handshake – sending SYN cookies during normal circumstances does however indicate that the rate of connections is probably too high for the default limits as SYN cookies are really only meant for mitigating SYN floods.

The circumstances in which the kernel will send SYN cookies is configured via the sysctl net.ipv4.tcp_syncookies, by default this is set to 1 which indicates that SYN cookies should be sent when the SYN backlog overflows, but can also be set to 0 to disable entirely or 2 to force SYN cookies to be sent 100% of the time.

When SYN cookies are enabled as needed (the default), they are triggered when the number of pending connections in the SYN backlog is more than the configured accept queue backlog size for the socket - the logic for this is here. When SYN cookies are fully disabled, net.ipv4.tcp_max_syn_backlog configures the number of connections allowed in the SYN backlog separately - the logic for this case is here.

In the default configuration where the backlog overflows and SYN cookies are sent, the kernel increments the TCPReqQFullDoCookies counter and logs this line to the kernel log, which is often confused for an indicator of a real SYN flood even when it’s just due to legitimate connections coming in too fast:

TCP: request_sock_TCP: Possible SYN flooding on port 8080. Sending cookies.  Check SNMP counters.

If SYN cookies are explicitly disabled, the kernel increments the TCPReqQFullDrop counter and the following would be logged instead:

TCP: request_sock_TCP: Possible SYN flooding on port 8080. Dropping request.  Check SNMP counters.

If you see this message for internal service-to-service connections, chances are you’re dealing with a scaling problem or a thundering herd problem, and not an actual SYN flood or intentional bad actor.

Those “SNMP Counters” that the kernel mentions are available in nstat in the network namespace:

$ nstat -a | grep TCPReqQFull
TcpExtTCPReqQFullDoCookies      706                0.0

SYN backlog in the lab

To see how the SYN backlog behaves in the lab, we can simulate latency on local connections with the following:

vagrant@blog-lab-scaling-accept:~$ sudo /vagrant/reset-lab.sh
vagrant@blog-lab-scaling-accept:~$ sudo tc qdisc add dev lo root netem delay 200ms
vagrant@blog-lab-scaling-accept:~$ ping 127.0.0.1
PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=401 ms
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=400 ms

This means we can now simulate overflowing the SYN backlog. To make it a bit easier to reproduce, reduce the size of the SYN backlog from the default of 128 by reducing net.core.somaxconn (because SYN cookies are enabled):

vagrant@blog-lab-scaling-accept:~$ sudo sysctl net.core.somaxconn
net.core.somaxconn = 128
vagrant@blog-lab-scaling-accept:~$ sudo sysctl -w net.core.somaxconn=10
net.core.somaxconn = 10
vagrant@blog-lab-scaling-accept:~$ sudo systemctl restart nginx
vagrant@blog-lab-scaling-accept:~$ 

At this point, if more than 10 connections arrive within 400ms (before the SYN-ACK and ACK handshake complete), then 10 connections will be in the SYN backlog, which is the maximum configured. This will trigger SYN cookies to be sent, triggering the message and counters above. Let’s test out if this works, run a simulation that opens N concurrent connections:

vagrant@blog-lab-scaling-accept:~$ python /vagrant/test_send_concurrent_connections.py 127.0.0.1:80 20
Waiting, Ctrl+C to exit.

And verify that SYN cookies were sent as expected:

vagrant@blog-lab-scaling-accept:~$ sudo dmesg -c
[ 2571.784749] TCP: request_sock_TCP: Possible SYN flooding on port 80. Sending cookies.  Check SNMP counters.
vagrant@blog-lab-scaling-accept:~$ sudo nstat | grep ReqQ
TcpExtTCPReqQFullDoCookies      10                 0.0
vagrant@blog-lab-scaling-accept:~$

Try disabling the simulated latency with sudo tc qdisc del dev lo root and re-run the same test, now connections move through fast enough that SYN cookies are not sent.

This also demonstrates the way the Round Trip Time effects the amount of connections that can burst into a LISTEN socket - test_send_concurrent_connections.py does actually attempt to initiate complete connections and nginx on the other end is readily calling accept() as fast as it can, but because 20 connections are initiated at once, 20 SYN packets arrive before any handshake can continue, and the SYN backlog overflows. You can imagine that in the real world, with some of these values often defaulting to 128, and users often being far away from servers (on the other side of the world), it’s pretty easy to accidentally trigger this scenario without a true SYN flood.

Accept queue

Once an ACK packet is received and validated, a new client connection is ready for the application to process. This connection is moved into the accept queue, waiting for the application to call accept() and receive it.

Unlike the SYN backlog, the accept queue has no backup plan for when it overflows. Back in the original call to listen(), a backlog argument was provided, which indicates how many connections can have completed the 3-way handshake and be waiting in the kernel for the application to accept.

This is the first common issue: If the backlog number provided to listen() is not large enough to contain any number of connections that could reasonably complete their handshake between 2 calls to accept(), then a connection will be dropped on the floor, and the application generally won’t even notice - the next call to accept() will succeed without any indication of a dropped connection! This is particularly likely to happen when some reasonable amount of work can happen between calls to accept() or when the incoming connections tend to arrive at the same time (such as jobs running on many servers with cron, or a thundering herd of reconnection attempts).

However, even when you specify a high enough backlog value to listen(), there’s another location that silently limits this value. The net.core.somaxconn sysctl specifies a network-system-wide maximum for the backlog of any socket. When a larger backlog value is provided, the kernel silently caps it at the value of net.core.somaxconn. This is the next most common issue: Both net.core.somaxconn and the backlog argument to listen() need to be adjusted appropriately so that the backlog is actually adjusted as expected.

As an added complexity, the net.core.somaxconn sysctl is not system global, but is global to a Linux network namespace. For new network namespaces, such as used by most Docker containers launched, the values from the default network namespace are not inherited, but instead set to the built-in kernel default:

vagrant@blog-lab-scaling-accept:~$ sudo /vagrant/reset-lab.sh
vagrant@blog-lab-scaling-accept:~$ sudo sysctl net.core.somaxconn
net.core.somaxconn = 128
vagrant@blog-lab-scaling-accept:~$ sudo docker run -it ubuntu sysctl net.core.somaxconn
net.core.somaxconn = 128
vagrant@blog-lab-scaling-accept:~$ sudo sysctl -w net.core.somaxconn=1024
net.core.somaxconn = 1024
vagrant@blog-lab-scaling-accept:~$ sudo docker run -it ubuntu sysctl net.core.somaxconn
net.core.somaxconn = 128
vagrant@blog-lab-scaling-accept:~$ 

Therein lies the third common issue: Running in containers that have their own network namespace (which is most of them launched by Kubernetes/Docker), even if the system has net.core.somaxconn correctly tweaked, that value will be ignored, and so the container must also have net.core.somaxconn tweaked to match the application running inside of it.

If any of these situations occurs and causes impact, they are visible in the ListenOverflows counter:

$ nstat -a | grep ListenOverflows
TcpExtListenOverflows           6811               0.0

Accept queue in the lab

To see how the accept queue behaves in the lab, we can run a server that is bad at accepting connections (it sleeps a lot) with a backlog of 10:

vagrant@blog-lab-scaling-accept:~$ sudo /vagrant/reset-lab.sh
vagrant@blog-lab-scaling-accept:~$ sudo sysctl -w net.core.somaxconn=1024
net.core.somaxconn = 1024
vagrant@blog-lab-scaling-accept:~$ python /vagrant/laggy_server.py 10
Listening with backlog 10

In another window, send a bunch of connections to the laggy server, then exit after a few seconds:

vagrant@blog-lab-scaling-accept:~$ python /vagrant/test_send_concurrent_connections.py 127.0.0.1:8080 20
Waiting, Ctrl+C to exit.
(wait a few seconds)
^C
vagrant@blog-lab-scaling-accept:~$ sudo nstat | grep ListenOverflows
TcpExtListenOverflows           44                 0.0
vagrant@blog-lab-scaling-accept:~$ 

At this point, we can see how important the backlog argument to listen() is. Next, let’s review how somaxconn caps this by setting up a server with a longer backlog, that we expect to be capped:

vagrant@blog-lab-scaling-accept:~$ sudo /vagrant/reset-lab.sh
vagrant@blog-lab-scaling-accept:~$ sudo sysctl -w net.core.somaxconn=10
net.core.somaxconn = 10
vagrant@blog-lab-scaling-accept:~$ python /vagrant/laggy_server.py 1024
Listening with backlog 1024

All looks good - notice how we effectively called listen(1024) and nothing went wrong. Running the same commands as we did when we provided the small backlog value, we can observe the same problem when the backlog is silently truncated by the kernel:

vagrant@blog-lab-scaling-accept:~$ python /vagrant/test_send_concurrent_connections.py 127.0.0.1:8080 20
Waiting, Ctrl+C to exit.
(wait a few seconds)
^C
vagrant@blog-lab-scaling-accept:~$ sudo nstat | grep ListenO
TcpExtListenOverflows           77                 0.0
vagrant@blog-lab-scaling-accept:~$ 

Monitoring counters

A few counters were mentioned above that can be read with nstat:

TcpExtTCPReqQFullDoCookies - detecting where SYN cookies were used to mitigate a lack of SYN backlog space
TcpExtTCPReqQFullDrop - detecting where SYNs were dropped because SYN cookies were disabled and the SYN backlog was full
TcpExtListenOverflows - detecting when a TCP connection completed the 3-way handshake but the accept queue was full or when a SYN was received while the accept queue was full

Because these counters are all part of the network namespace and are global within a namespace, no information about which socket or application caused the issue will be visible, and if the process is in a container, it will not be visible in the base system counters (in the default network namespace). Because of this, every network namespace will need to be inspected/monitored for these counters, and then additional work needs to be done to trace this back to the application, potentially using the kernel log lines as a hint in the case of the SYN backlog variant.

Detecting backlog overflows with tracing

The metrics available from the counters above provide a minimal picture of backlog overflows and dropped connections, but ideally we would be able to inspect this situation system-wide, seeing through any network namespaces, and be able to link it back to an application and even socket/port.

This is possible using kprobes and eBPF tracing using bcc, by hooking kernel functions that handle these failure cases and inspecting the context at that point in time. This allows us to extract realtime data from systems in production, unlike the counters which are at best vague indicators of an underlying issue existing somewhere on the system.

We know that the ListenOverflows counter is incremented any time the accept backlog overflows - we can start there and work back to building a tracing program.

Searching the Linux source code for ListenOverflows shows what this counter is called internally - LINUX_MIB_LISTENOVERFLOWS. Searching the kernel source tree shows all the places in the kernel that increment that counter - in this case, the best candidates are tcp_conn_request in net/ipv4/tcp_input.c and tcp_v4_syn_recv_sock in net/ipv4/tcp_ipv4.c. They handle slightly different points in the connection lifecycle:

tcp_conn_request handles a SYN packet that initiates a new connection against a LISTEN socket and places it in the SYN backlog
tcp_v4_syn_recv_sock handles an ACK packet that completes a connection and adds it to the accept queue

These map to the earlier diagram:

tcp_conn_request drops connections if the accept queue is full as a safety mechanism, even though it wouldn’t be adding to it yet. tcp_v4_syn_recv_sock also drops connections if the accept queue is full, and rightly so, since it would be adding to it. If a SYN packet has already been accepted and added to the SYN backlog while the accept queue had available space, but was full by the point the ACK arrived, drops will occur in tcp_v4_syn_recv_sock when the ACK is received. If a new SYN arrives while the accept queue is full, then tcp_conn_request will drop instead. For local testing in a VM, tcp_conn_request is the easier one to test since it doesn’t require careful timing between SYN and ACK to reproduce.

The code path that increments the ListenOverflows counter in each of those functions looks like the following:

int tcp_conn_request(struct request_sock_ops *rsk_ops,
             const struct tcp_request_sock_ops *af_ops,
             struct sock *sk, struct sk_buff *skb)
{
    /* ... */

    if (sk_acceptq_is_full(sk)) {
        NET_INC_STATS(sock_net(sk), LINUX_MIB_LISTENOVERFLOWS);
        goto drop;
    }

    /* ... */

drop:
    tcp_listendrop(sk);
    return 0;
}

struct sock *tcp_v4_syn_recv_sock(const struct sock *sk, struct sk_buff *skb,
                  struct request_sock *req,
                  struct dst_entry *dst,
                  struct request_sock *req_unhash,
                  bool *own_req)
{
    /* ... */

    if (sk_acceptq_is_full(sk))
        goto exit_overflow;
    
    /* ... */

exit_overflow:
    NET_INC_STATS(sock_net(sk), LINUX_MIB_LISTENOVERFLOWS);
    /* ... */
}

The function called to check for overflow is a simple inlined check:

static inline bool sk_acceptq_is_full(const struct sock *sk)
{
    return READ_ONCE(sk->sk_ack_backlog) > READ_ONCE(sk->sk_max_ack_backlog);
}

When looking to trace functions that are inlined like this, the easiest way to hook the condition is to attach to the start of the calling function (in this case tcp_conn_request and tcp_v4_syn_recv_sock) which is not inlined, and perform the same conditional check in our eBPF code. We can do this with kprobes:

/* generic handler */
static inline void handle_sk_potential_overflow(struct pt_regs *ctx, int syn_recv, const struct sock *sk)
{
    /* we need to read these using bpf_probe_read to ensure the read is safe */
    u32 sk_ack_backlog = 0;
    u32 sk_max_ack_backlog = 0;
    bpf_probe_read(&sk_ack_backlog, sizeof(sk_ack_backlog), (void *)&sk->sk_ack_backlog);
    bpf_probe_read(&sk_max_ack_backlog, sizeof(sk_max_ack_backlog), (void *)&sk->sk_max_ack_backlog);

    if (sk_ack_backlog > sk_max_ack_backlog) {
        /* handle the condition */
    }
}

/* when a SYN arrives */
struct tcp_request_sock_ops;
void kprobe__tcp_conn_request(struct pt_regs *ctx, struct request_sock_ops *rsk_ops,
             const struct tcp_request_sock_ops *af_ops,
             const struct sock *sk, struct sk_buff *skb)
{
    handle_sk_potential_overflow(ctx, 0, sk);
}

/* when an ACK arrives for a SYN_RECV socket */
void kprobe__tcp_v4_syn_recv_sock(struct pt_regs *ctx, const struct sock *sk) /* no need for the remaining unused args */
{
    handle_sk_potential_overflow(ctx, 1, sk);
}

At this point, we have a hook on the same condition that would trigger incrementing the ListenOverflows counter in TCP over IPv4. This can be fleshed out to read detailed information about the socket and return it to the userspace tracing program.

Tracing backlog overflows in the lab

One such program is included in the lab, let’s start up our laggy server again:

vagrant@blog-lab-scaling-accept:~$ sudo /vagrant/reset-lab.sh
vagrant@blog-lab-scaling-accept:~$ python /vagrant/laggy_server.py 10
Listening with backlog 10

In another session, start up the tracing program:

vagrant@blog-lab-scaling-accept:~$ sudo python /vagrant/trace_backlog_overflow.py 
CONTAINER/HOST          PID PROCESS              NETNSID    BIND                 PKT     BL    MAX

And finally, run our client:

vagrant@blog-lab-scaling-accept:~$ python /vagrant/test_send_concurrent_connections.py 127.0.0.1:8080 20
Waiting, Ctrl+C to exit.

The result will be traces showing where the backlog overflowed the configured maximum, along with details of the process and whether it occured during SYN or ACK:

vagrant@blog-lab-scaling-accept:~$ sudo python /vagrant/trace_backlog_overflow.py
CONTAINER/HOST          PID PROCESS              NETNSID    BIND                 PKT     BL    MAX
blog-lab-scaling-acc  14742 python /vagrant/lagg 4026531992 127.0.0.1:8080       SYN     11     10
blog-lab-scaling-acc  14742 python /vagrant/lagg 4026531992 127.0.0.1:8080       SYN     11     10
blog-lab-scaling-acc  14742 python /vagrant/lagg 4026531992 127.0.0.1:8080       SYN     11     10
blog-lab-scaling-acc  14742 python /vagrant/lagg 4026531992 127.0.0.1:8080       SYN     11     10
blog-lab-scaling-acc  14742 python /vagrant/lagg 4026531992 127.0.0.1:8080       SYN     11     10
blog-lab-scaling-acc  14742 python /vagrant/lagg 4026531992 127.0.0.1:8080       SYN     11     10
blog-lab-scaling-acc  14742 python /vagrant/lagg 4026531992 127.0.0.1:8080       SYN     11     10
blog-lab-scaling-acc  14742 python /vagrant/lagg 4026531992 127.0.0.1:8080       SYN     11     10
blog-lab-scaling-acc  14742 python /vagrant/lagg 4026531992 127.0.0.1:8080       SYN     11     10
blog-lab-scaling-acc  14742 python /vagrant/lagg 4026531992 127.0.0.1:8080       SYN     11     10
blog-lab-scaling-acc  14742 python /vagrant/lagg 4026531992 127.0.0.1:8080       SYN     11     10
blog-lab-scaling-acc  14742 python /vagrant/lagg 4026531992 127.0.0.1:8080       SYN     11     10
blog-lab-scaling-acc  14742 python /vagrant/lagg 4026531992 127.0.0.1:8080       SYN     11     10
blog-lab-scaling-acc  14742 python /vagrant/lagg 4026531992 127.0.0.1:8080       SYN     11     10
blog-lab-scaling-acc  14742 python /vagrant/lagg 4026531992 127.0.0.1:8080       SYN     11     10
blog-lab-scaling-acc  14742 python /vagrant/lagg 4026531992 127.0.0.1:8080       SYN     11     10

As discussed earlier, if many SYN packets are queued while there is room in the accept queue, but then the queue overflows anyway, the ACK packets trigger it instead (run the client again in another window to see this output):

vagrant@blog-lab-scaling-accept:~$ sudo tc qdisc add dev lo root netem delay 200ms
vagrant@blog-lab-scaling-accept:~$ sudo python /vagrant/trace_backlog_overflow.py
CONTAINER/HOST          PID PROCESS              NETNSID    BIND                 PKT     BL    MAX
blog-lab-scaling-acc  14742 python /vagrant/lagg 4026531992 127.0.0.1:8080       ACK     11     10
blog-lab-scaling-acc  14742 python /vagrant/lagg 4026531992 127.0.0.1:8080       ACK     11     10
blog-lab-scaling-acc  14742 python /vagrant/lagg 4026531992 127.0.0.1:8080       ACK     11     10
blog-lab-scaling-acc  14742 python /vagrant/lagg 4026531992 127.0.0.1:8080       ACK     11     10
blog-lab-scaling-acc  14742 python /vagrant/lagg 4026531992 127.0.0.1:8080       ACK     11     10
blog-lab-scaling-acc  14742 python /vagrant/lagg 4026531992 127.0.0.1:8080       ACK     11     10
blog-lab-scaling-acc  14742 python /vagrant/lagg 4026531992 127.0.0.1:8080       ACK     11     10
blog-lab-scaling-acc  14742 python /vagrant/lagg 4026531992 127.0.0.1:8080       ACK     11     10
blog-lab-scaling-acc  14742 python /vagrant/lagg 4026531992 127.0.0.1:8080       ACK     11     10

Detecting misconfigured listen sockets with tracing

We can also trace calls to listen() to observe when the system places caps on the backlog size, to see a leading indicator for applications launching on the machine that have misconfigured values for the backlog or somaxconn. This will only work at the time that applications start up, since we’ll be hooking listen() as it is actually called, which is typically just when the application first launches.

The easiest way to find the kernel source responsible for this is to start from the listen() syscall, which is __sys_listen() in the kernel:

int __sys_listen(int fd, int backlog)
{
    struct socket *sock;
    int err, fput_needed;
    int somaxconn;

    sock = sockfd_lookup_light(fd, &err, &fput_needed);
    if (sock) {
        somaxconn = sock_net(sock->sk)->core.sysctl_somaxconn;
        if ((unsigned int)backlog > somaxconn)
            backlog = somaxconn;

        err = security_socket_listen(sock, backlog);
        if (!err)
            err = sock->ops->listen(sock, backlog);

        fput_light(sock->file, fput_needed);
    }
    return err;
}

From this snippet alone, it’s easy to observe the way that somaxconn (sysctl_somaxconn) from the current network namespace (sock_net(sock->sk)) silently limits the backlog. By tracing the listen syscall itself, we’ll get the originally requested backlog, but to see what the limit was eventually set to we need to hook a point in time after that limit (we could also read the sysctl ourselves and repeat the logic).

Conveniently, sock->ops->listen(...) calls the underlying listen operation in the address family, in this case inet_listen:

int inet_listen(struct socket *sock, int backlog)
{
    struct sock *sk = sock->sk;

    /* ... */
}

At this point, inet_listen is called with the capped backlog. By combining a trace on the listen() syscall itself with a trace on inet_listen, we can map the original requested value to the actual value used:

BPF_HASH(requested_backlogs, u32, u64);

TRACEPOINT_PROBE(syscalls, sys_enter_listen) {
    u32 pid = bpf_get_current_pid_tgid();

    u64 backlog = args->backlog;
    requested_backlogs.update(&pid, &backlog);

    return 0;
}

void kprobe__inet_listen(struct pt_regs *ctx, struct socket *sock, int backlog)
{
    struct sock *sk = sock->sk;
    u32 pid = bpf_get_current_pid_tgid();

    u64 *requested_backlog;
    requested_backlog = requested_backlogs.lookup(&pid);
    if (requested_backlog == NULL) {
        return;
    }

    requested_backlogs.delete(&pid);

    /*
        `*requested_backlogs` contains the argument to listen()
        `backlog` contains the capped backlog
    */
}

With some extra work we can also peek inside the network namespace to see the value of somaxconn at the time of the call. From above, we can use sock_net(sock->sk)->core.sysctl_somaxconn if we have a struct socket *sock - or since we now have the internal struct sock *sk, we’ll want sock_net(sk)->core.sysctl_somaxconn. The sock_net function is pretty simple:

static inline
struct net *sock_net(const struct sock *sk)
{
    return read_pnet(&sk->sk_net);
}

/* ... elsewhere */
static inline struct net *read_pnet(const possible_net_t *pnet)
{
#ifdef CONFIG_NET_NS
    return pnet->net;
#else
    return &init_net;
#endif
}

We’ll need to simulate that function chain with a call to bpf_probe_read to ensure it’s safe to read that member, otherwise the eBPF validator will complain because the dereferencing is a bit too complex for bcc to automatically safeguard with a bpf_probe_read call. That gives us the following:

static inline
struct net *safe_sock_net(const struct sock *sk)
{
#ifdef CONFIG_NET_NS
    struct net *ret;
    // was: read_pnet(&sk->sk_net)->net
    bpf_probe_read(&ret, sizeof(ret), (void *)&(sk->sk_net.net));
    return ret;
#else
    return &init_net;
#endif
}

void kprobe__inet_listen(struct pt_regs *ctx, struct socket *sock, int backlog)
{
    struct sock *sk = sock->sk;
    /* ... */

    u32 netns_somaxconn = 0;
    bpf_probe_read(&netns_somaxconn, sizeof(netns_somaxconn), (void *)&(safe_sock_net(sk)->core.sysctl_somaxconn));

    /* ... */
}

At this point we have the requested backlog (the argument to listen()), the actual backlog calculated (after any somaxconn capping) and the value for somaxconn as well. We can fill this out with further information about the pid and namespace and provide it to the userspace side of the tracing app.

Tracing accept queue overflows in the lab

A fleshed out tracing app for the above is also included in the lab, this time we need to start the tracing app before our listen()ing server:

vagrant@blog-lab-scaling-accept:~$ sudo /vagrant/reset-lab.sh
vagrant@blog-lab-scaling-accept:~$ sudo python /vagrant/trace_listen_backlog_capped.py 
CONTAINER/HOST          PID PROCESS              NETNSID    BIND                    REQ    MAX ACTUAL

Now we can start up our server with a small backlog, then with a larger backlog, and then with a backlog limited by somaxconn:

vagrant@blog-lab-scaling-accept:~$ python /vagrant/laggy_server.py 10
Listening with backlog 10
^C
vagrant@blog-lab-scaling-accept:~$ python /vagrant/laggy_server.py 128
Listening with backlog 128
^C
vagrant@blog-lab-scaling-accept:~$ python /vagrant/laggy_server.py 1024
Listening with backlog 1024
^C

The tracing app will show all these cases, along with where it was silently limited:

vagrant@blog-lab-scaling-accept:~$ sudo python /vagrant/trace_listen_backlog_capped.py 
CONTAINER/HOST          PID PROCESS              NETNSID    BIND                    REQ    MAX ACTUAL
blog-lab-scaling-acc  14849 python /vagrant/lagg 4026531992 127.0.0.1:8080           10    128     10 
blog-lab-scaling-acc  14851 python /vagrant/lagg 4026531992 127.0.0.1:8080          128    128    128 
blog-lab-scaling-acc  14853 python /vagrant/lagg 4026531992 127.0.0.1:8080         1024    128    128 LIMITED

This also works through container network namespaces:

vagrant@blog-lab-scaling-accept:~$ sudo docker run -v /vagrant:/vagrant -it python:2.7 python /vagrant/laggy_server.py 1024
Listening with backlog 1024
^C
vagrant@blog-lab-scaling-accept:~$ sudo sysctl -w net.core.somaxconn=1024
net.core.somaxconn = 1024
vagrant@blog-lab-scaling-accept:~$ sudo docker run -v /vagrant:/vagrant -it python:2.7 python /vagrant/laggy_server.py 1024
Listening with backlog 1024
^C
vagrant@blog-lab-scaling-accept:~$ sudo docker run --sysctl net.core.somaxconn=1024 -v /vagrant:/vagrant -it python:2.7 python /vagrant/laggy_server.py 1024
Listening with backlog 1024
^C
vagrant@blog-lab-scaling-accept:~$ 

Note that setting the somaxconn value on the base system had no effect, but setting the value in the container with docker run --sysctl did change it:

vagrant@blog-lab-scaling-accept:~$ sudo python /vagrant/trace_listen_backlog_capped.py 
CONTAINER/HOST          PID PROCESS              NETNSID    BIND                    REQ    MAX ACTUAL
380944436fe3          15274 python /vagrant/lagg 4026532171 127.0.0.1:8080         1024    128    128 LIMITED
9e82132ad3d6          15387 python /vagrant/lagg 4026532171 127.0.0.1:8080         1024    128    128 LIMITED
af3b584488ea          15495 python /vagrant/lagg 4026532171 127.0.0.1:8080         1024   1024   1024 

Wrapping up

We’ve seen how scaling something as simple as a TCP service can run into unexpected resource limitations, even before the program accepts a connection. Key takeaways are:

The backlog argument to listen should always be set high enough so that any valid burst of connections that could arrive between calls to accept() will be queued up
net.core.somaxconn must be set high enough that any backlog value is not restricted - if not, it will be silently capped
net.core.somaxconn is namespaced, so must be set in the namespace/container that the socket is created in
SYN flood warnings are often logged due to the SYN backlog being too small, and it should be treated as a potential scaling issue unless a real SYN flood is suspected
The number of connections in the SYN backlog are relative to the RTT between the client and server, not just the rate of connections

The tracing tools provided here are fairly lightweight in kernel space and are mostly gated to failure cases and hook relatively low rate system calls, making them reasonably safe to use on production systems, unless CPU resource usage is an active bottleneck at the time. These tracing tools may help track down which application or socket the ListenOverflow or TCPReqQFull* counters are being triggered by, as well as monitoring applications launching for any misconfigurations without needing to audit the configuration of each one individually.

Debugging network stalls on Kubernetes

Sat, 06 Jul 2019 00:00:00 +0000

We’ve talked about Kubernetes before, and over the last couple of years it’s become the standard deployment pattern at GitHub. We now run a large portion of both internal and public-facing services on Kubernetes. As our Kubernetes clusters have grown, and our targets on the latency of our services have become more stringent, we began to notice that certain services running on Kubernetes in our environment were experiencing sporadic latency that couldn’t be attributed to the performance characteristics of the application itself.

Essentially, applications running on our Kubernetes clusters would observe seemingly random latency of up to and over 100ms on connections, which would cause downstream timeouts or retries. Services were expected to be able to respond to requests in well under 100ms, which wasn’t feasible when the connection itself was taking so long. Separately, we also observed very fast MySQL queries, which we expected to take a matter of milliseconds and that MySQL observed taking only milliseconds, were being observed taking 100ms or more from the perspective of the querying application.

The problem was initially narrowed down to communications that involved a Kubernetes node, even if the other side of a connection was outside Kubernetes. The most simple reproduction we had was a Vegeta benchmark that could be run from any internal host, targeting a Kubernetes service running on a node port, and would observe the sporadically high latency. In this post, we’ll walk through how we tracked down the underlying issue.

Removing complexity to find the path at fault

Using an example reproduction, we wanted to narrow down the problem and remove layers of complexity. Initially, there were too many moving parts in the flow between Vegeta and pods running on Kubernetes to determine if this was a deeper network problem, so we needed to rule some out.

The client, Vegeta, creates a TCP connection to any kube-node in the cluster. Kubernetes runs in our data centers as an overlay network (a network that runs on top of our existing datacenter network) that uses IPIP (which encapsulates the overlay network’s IP packet inside the datacenter’s IP packet). When a connection is made to that first kube-node, it performs stateful Network Address Translation (NAT) to convert the kube-node’s IP and port to an IP and port on the overlay network (specifically, of the pod running the application). On return, it undoes each of these steps. This is a complex system with a lot of state, and a lot of moving parts that are constantly updating and changing as services deploy and move around.

As part of running a tcpdump on the original Vegeta benchmark, we observed the latency during a TCP handshake (between SYN and SYN-ACK). To simplify some of the complexity of HTTP and Vegeta, we can use hping3 to just “ping” with a SYN packet and see if we observe the latency in the response packet—then throw away the connection. We can filter it to only include packets over 100ms and get a simpler reproduction case than a full Layer 7 Vegeta benchmark or attack against the service. The following “pings” a kube-node using TCP SYN/SYN-ACK on the “node port” for the service (30927) with an interval of 10ms, filtered for slow responses:

theojulienne@shell ~ $ sudo hping3 172.16.47.27 -S -p 30927 -i u10000 | egrep --line-buffered 'rtt=[0-9]{3}\.'
len=46 ip=172.16.47.27 ttl=59 DF id=0 sport=30927 flags=SA seq=1485 win=29200 rtt=127.1 ms
len=46 ip=172.16.47.27 ttl=59 DF id=0 sport=30927 flags=SA seq=1486 win=29200 rtt=117.0 ms
len=46 ip=172.16.47.27 ttl=59 DF id=0 sport=30927 flags=SA seq=1487 win=29200 rtt=106.2 ms
len=46 ip=172.16.47.27 ttl=59 DF id=0 sport=30927 flags=SA seq=1488 win=29200 rtt=104.1 ms
len=46 ip=172.16.47.27 ttl=59 DF id=0 sport=30927 flags=SA seq=5024 win=29200 rtt=109.2 ms
len=46 ip=172.16.47.27 ttl=59 DF id=0 sport=30927 flags=SA seq=5231 win=29200 rtt=109.2 ms

Our first new observation from the sequence numbers and timings is that this isn’t a one-off, but is often grouped, like a backlog that eventually gets processed.

Next up, we want to narrow down which component(s) were potentially at fault. Is it the kube-proxy iptables NAT rules that are hundreds of rules long? Is it the IPIP tunnel and something on the network handling them poorly? One way to validate this is to test each step of the system. What happens if we remove the NAT and firewall logic and only use the IPIP part:

Linux thankfully lets you just talk directly to an overlay IP when you’re on a machine that’s part of the same network, so that’s pretty easy to do:

theojulienne@kube-node-client ~ $ sudo hping3 10.125.20.64 -S -i u10000 | egrep --line-buffered 'rtt=[0-9]{3}\.'
len=40 ip=10.125.20.64 ttl=64 DF id=0 sport=0 flags=RA seq=7346 win=0 rtt=127.3 ms
len=40 ip=10.125.20.64 ttl=64 DF id=0 sport=0 flags=RA seq=7347 win=0 rtt=117.3 ms
len=40 ip=10.125.20.64 ttl=64 DF id=0 sport=0 flags=RA seq=7348 win=0 rtt=107.2 ms

Based on our results, the problem still remains! That rules out iptables and NAT. Is it TCP that’s the problem? Let’s see what happens when we perform a normal ICMP ping:

theojulienne@kube-node-client ~ $ sudo hping3 10.125.20.64 --icmp -i u10000 | egrep --line-buffered 'rtt=[0-9]{3}\.'
len=28 ip=10.125.20.64 ttl=64 id=42594 icmp_seq=104 rtt=110.0 ms
len=28 ip=10.125.20.64 ttl=64 id=49448 icmp_seq=4022 rtt=141.3 ms
len=28 ip=10.125.20.64 ttl=64 id=49449 icmp_seq=4023 rtt=131.3 ms
len=28 ip=10.125.20.64 ttl=64 id=49450 icmp_seq=4024 rtt=121.2 ms
len=28 ip=10.125.20.64 ttl=64 id=49451 icmp_seq=4025 rtt=111.2 ms
len=28 ip=10.125.20.64 ttl=64 id=49452 icmp_seq=4026 rtt=101.1 ms
len=28 ip=10.125.20.64 ttl=64 id=50023 icmp_seq=4343 rtt=126.8 ms
len=28 ip=10.125.20.64 ttl=64 id=50024 icmp_seq=4344 rtt=116.8 ms
len=28 ip=10.125.20.64 ttl=64 id=50025 icmp_seq=4345 rtt=106.8 ms
len=28 ip=10.125.20.64 ttl=64 id=59727 icmp_seq=9836 rtt=106.1 ms

Our results show that the problem still exists. Is it the IPIP tunnel that’s causing the problem? Let’s simplify things further:

Is it possible that it’s every packet between these two hosts?

theojulienne@kube-node-client ~ $ sudo hping3 172.16.47.27 --icmp -i u10000 | egrep --line-buffered 'rtt=[0-9]{3}\.'
len=46 ip=172.16.47.27 ttl=61 id=41127 icmp_seq=12564 rtt=140.9 ms
len=46 ip=172.16.47.27 ttl=61 id=41128 icmp_seq=12565 rtt=130.9 ms
len=46 ip=172.16.47.27 ttl=61 id=41129 icmp_seq=12566 rtt=120.8 ms
len=46 ip=172.16.47.27 ttl=61 id=41130 icmp_seq=12567 rtt=110.8 ms
len=46 ip=172.16.47.27 ttl=61 id=41131 icmp_seq=12568 rtt=100.7 ms
len=46 ip=172.16.47.27 ttl=61 id=9062 icmp_seq=31443 rtt=134.2 ms
len=46 ip=172.16.47.27 ttl=61 id=9063 icmp_seq=31444 rtt=124.2 ms
len=46 ip=172.16.47.27 ttl=61 id=9064 icmp_seq=31445 rtt=114.2 ms
len=46 ip=172.16.47.27 ttl=61 id=9065 icmp_seq=31446 rtt=104.2 ms

Behind the complexity, it’s as simple as two kube-node hosts sending any packet, even ICMP pings, to each other. They’ll still see the latency, if the target host is a “bad” one (some are worse than others).

Now there’s one last thing to question: we clearly don’t observe this everywhere, so why is it just on kube-node servers? And does it occur when the kube-node is the sender or the receiver? Luckily, this is also pretty easy to narrow down by using a host outside Kubernetes as a sender, but with the same “known bad” target host (from a staff shell host to the same kube-node). We can observe this is still an issue in that direction:

theojulienne@shell ~ $ sudo hping3 172.16.47.27 -p 9876 -S -i u10000 | egrep --line-buffered 'rtt=[0-9]{3}\.'
len=46 ip=172.16.47.27 ttl=61 DF id=0 sport=9876 flags=RA seq=312 win=0 rtt=108.5 ms
len=46 ip=172.16.47.27 ttl=61 DF id=0 sport=9876 flags=RA seq=5903 win=0 rtt=119.4 ms
len=46 ip=172.16.47.27 ttl=61 DF id=0 sport=9876 flags=RA seq=6227 win=0 rtt=139.9 ms
len=46 ip=172.16.47.27 ttl=61 DF id=0 sport=9876 flags=RA seq=7929 win=0 rtt=131.2 ms

And then perform the same from the previous source kube-node to a staff shell host (which rules out the source host, since a ping has both an RX and TX component):

theojulienne@kube-node-client ~ $ sudo hping3 172.16.33.44 -p 9876 -S -i u10000 | egrep --line-buffered 'rtt=[0-9]{3}\.'
^C
--- 172.16.33.44 hping statistic ---
22352 packets transmitted, 22350 packets received, 1% packet loss
round-trip min/avg/max = 0.2/7.6/1010.6 ms

Looking into the packet captures of the latency we observed, we get some more information. Specifically, that the “sender” host (bottom) observes this timeout while the “receiver” host (top) does not—see the Delta column (in seconds):

Additionally, by looking at the difference between the ordering of the packets (based on the sequence numbers) on the receiver side of the TCP and ICMP results above, we can observe that ICMP packets always arrive in the same sequence they were sent, but with uneven timing, while TCP packets are sometimes interleaved, but a subset of them stall. Notably, we observe that if you count the ports of the SYN packets, the ports are not in order on the receiver side, while they’re in order on the sender side.

There is a subtle difference between how modern server NICs – like we have in our data centers—handle packets containing TCP vs ICMP. When a packet arrives, the NIC hashes the packet “per connection” and tries to divvy up the connections across receive queues, each (approximately) delegated to a given CPU core. For TCP, this hash includes both source and destination IP and port. In other words, each connection is hashed (potentially) differently. For ICMP, just the IP source and destination are hashed, since there are no ports.

Another new observation is that we can tell that ICMP observes stalls on all communications between the two hosts during this period from the sequence numbers in ICMP vs TCP, while TCP does not. This tells us that the RX queue hashing is likely in play, almost certainly indicating the stall is in processing RX packets, not in sending responses.

This rules out kube-node transmits, so we now know that it’s a stall in processing packets, and that it’s on the receive side on some kube-node servers.

Deep dive into Linux kernel packet processing

To understand why the problem could be on the receiving side on some kube-node servers, let’s take a look at how the Linux kernel processes packets.

Going back to the simplest traditional implementation, the network card receives a packet and sends an interrupt to the Linux kernel stating that there’s a packet that should be handled. The kernel stops other work, switches context to the interrupt handler, processes the packet, then switches back to what it was doing.

This context switching is slow, which may have been fine on a 10Mbit NIC in the 90s, but on modern servers where the NIC is 10G and at maximal line rate can bring in around 15 million packets per second, on a smaller server with eight cores that could mean the kernel is interrupted millions of times per second per core.

Instead of constantly handling interrupts, many years ago Linux added NAPI, the networking API that modern drivers use for improved performance at high packet rates. At low rates, the kernel still accepts interrupts from the NIC in the method we mentioned. Once enough packets arrive and cross a threshold, it disables interrupts and instead begins polling the NIC and pulling off packets in batches. This processing is done in a “softirq”, or software interrupt context. This happens at the end of syscalls and hardware interrupts, which are times that the kernel (as opposed to userspace) is already running.

This is much faster, but brings up another problem. What happens if we have so many packets to process that we spend all our time processing packets from the NIC, but we never have time to let the userspace processes actually drain those queues (read from TCP connections, etc.)? Eventually the queues would fill up, and we’d start dropping packets. To try and make this fair, the kernel limits the amount of packets processed in a given softirq context to a certain budget. Once this budget is exceeded, it wakes up a separate thread called ksoftirqd (you’ll see one of these in ps for each core) which processes these softirqs outside of the normal syscall/interrupt path. This thread is scheduled using the standard process scheduler, which already tries to be fair.

With an overview of the way the kernel is processing packets, we can see there is definitely opportunity for this processing to become stalled. If the time between softirq processing calls grows, packets could sit in the NIC RX queue for a while before being processed. This could be something deadlocking the CPU core, or it could be something slow preventing the kernel from running softirqs.

Narrow down processing to a core/method

At this point, it makes sense that this could happen, and we know we’re observing something that looks a lot like it. The next step is to confirm this theory, and if we do, understand what’s causing it.

Let’s revisit the slow round trip packets we saw before:

len=46 ip=172.16.53.32 ttl=61 id=29573 icmp_seq=1953 rtt=99.3 ms
len=46 ip=172.16.53.32 ttl=61 id=29574 icmp_seq=1954 rtt=89.3 ms
len=46 ip=172.16.53.32 ttl=61 id=29575 icmp_seq=1955 rtt=79.2 ms
len=46 ip=172.16.53.32 ttl=61 id=29576 icmp_seq=1956 rtt=69.1 ms
len=46 ip=172.16.53.32 ttl=61 id=29577 icmp_seq=1957 rtt=59.1 ms

len=46 ip=172.16.53.32 ttl=61 id=29790 icmp_seq=2070 rtt=75.7 ms
len=46 ip=172.16.53.32 ttl=61 id=29791 icmp_seq=2071 rtt=65.6 ms
len=46 ip=172.16.53.32 ttl=61 id=29792 icmp_seq=2072 rtt=55.5 ms

As discussed previously, these ICMP packets are hashed to a single NIC RX queue and processed by a single CPU core. If we want to understand what the kernel is doing, it’s helpful to know where (cpu core) and how (softirq, ksoftirqd) it’s processing these packets so we can catch them in action.

Now it’s time to use the tools that allow live tracing of a running Linux kernel - bcc is what was used here. This allows you to write small C programs that hook arbitrary functions in the kernel, and buffer events back to a userspace Python program which can summarize and return them to you. The “hook arbitrary functions in the kernel” is the difficult part, but it actually goes out of its way to be as safe as possible to use, because it’s designed for tracing exactly this type of production issue that you can’t simply reproduce in a testing or dev environment.

The plan here is simple: we know the kernel is processing those ICMP ping packets, so let’s hook the kernel function icmp_echo which takes an incoming ICMP “echo request” packet and initiates sending the ICMP “echo response” reply. We can identify the packet using the incrementing icmp_seq shown by hping3 above.

The code for this bcc script looks complex, but breaking it down it’s not as scary as it sounds. The icmp_echo function is passed a struct sk_buff *skb, which is the packet containing the ICMP echo request. We can delve into this live and pull out the echo.sequence (which maps to the icmp_seq shown by hping3 above), and send that back to userspace. Conveniently, we can also grab the current process name/id as well. This gives us results like the following, live as the kernel processes these packets:

TGID    PID     PROCESS NAME    ICMP_SEQ
     0       swapper/11      770
     0       swapper/11      771
     0       swapper/11      772
     0       swapper/11      773
     0       swapper/11      774
 20086   prometheus      775
     0       swapper/11      776
     0       swapper/11      777
     0       swapper/11      778
  4542    spokes-report-s 779

One thing to note about this process name is that in a post-syscall softirq context, you see the process that made the syscall show as the “process”, even though really it’s the kernel processing it safely within the kernel context.

With that running, we can now correlate back from the stalled packets observed with hping3 to the process that’s handling it. A simple grep on that capture for the icmp_seq values with some context shows what happened before these packets were processed. The packets that line up with the above hping3 icmp_seq values have been marked along with the rtt’s we observed above (and what we’d have expected if <50ms rtt’s weren’t filtered out):

TGID    PID     PROCESS NAME    ICMP_SEQ ** RTT
--
 10436   cadvisor        1951
 10436   cadvisor        1952
    76      ksoftirqd/11    1953 ** 99ms
    76      ksoftirqd/11    1954 ** 89ms
    76      ksoftirqd/11    1955 ** 79ms
    76      ksoftirqd/11    1956 ** 69ms
    76      ksoftirqd/11    1957 ** 59ms
    76      ksoftirqd/11    1958 ** (49ms)
    76      ksoftirqd/11    1959 ** (39ms)
    76      ksoftirqd/11    1960 ** (29ms)
    76      ksoftirqd/11    1961 ** (19ms)
    76      ksoftirqd/11    1962 ** (9ms)
--
 10436   cadvisor        2068
 10436   cadvisor        2069
    76      ksoftirqd/11    2070 ** 75ms
    76      ksoftirqd/11    2071 ** 65ms
    76      ksoftirqd/11    2072 ** 55ms
    76      ksoftirqd/11    2073 ** (45ms)
    76      ksoftirqd/11    2074 ** (35ms)
    76      ksoftirqd/11    2075 ** (25ms)
    76      ksoftirqd/11    2076 ** (15ms)
    76      ksoftirqd/11    2077 ** (5ms)

The results tells us a few things. First, these packets are being processed by ksoftirqd/11 which conveniently tells us this particular pair of machines have their ICMP packets hashed to core 11 on the receiving side. We can also see that every time we see a stall, we always see some packets processed in cadvisor’s syscall softirq context, followed by ksoftirqd taking over and processing the backlog, exactly the number we’d expect to work through the backlog.

The fact that cadvisor is always running just prior to this immediately also implicates it in the problem. Ironically, cadvisor “analyzes resource usage and performance characteristics of running containers”, yet it’s triggering this performance problem. As with many things related to containers, it’s all relatively bleeding-edge tooling which can result in some somewhat expected corner cases of bad performance.

What is cadvisor doing to stall things?

With the understanding of how the stall can happen, the process causing it, and the CPU core it’s happening on, we now have a pretty good idea of what this looks like. For the kernel to hard block and not schedule ksoftirqd earlier, and given we see packets processed under cadvisor’s softirq context, it’s likely that cadvisor is running a slow syscall which ends with the rest of the packets being processed:

That’s a theory but how do we validate this is actually happening? One thing we can do is trace what’s running on the CPU core throughout this process, catch the point where the packets are overflowing budget and processed by ksoftirqd, then look back a bit to see what was running on the CPU core. Think of it like taking an x-ray of the CPU every few milliseconds. It would look something like this:

Conveniently, this is something that’s already mostly supported. The perf record tool samples a given CPU core at a certain frequency and can generate a call graph of the live system, including both userspace and the kernel. Taking that recording and manipulating it using a quick fork of a tool from Brendan Gregg’s FlameGraph that retained stack trace ordering, we can get a one-line stack trace for each 1ms sample, then get a sample of the 100ms before ksoftirqd is in the trace:

# record 999 times a second, or every 1ms with some offset so not to align exactly with timers
sudo perf record -C 11 -g -F 999
# take that recording and make a simpler stack trace.
sudo perf script 2>/dev/null | ./FlameGraph/stackcollapse-perf-ordered.pl | grep ksoftir -B 100

This results in the following:

(hundreds of traces that look similar)
cadvisor;[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];entry_SYSCALL_64_after_swapgs;do_syscall_64;sys_read;vfs_read;seq_read;memcg_stat_show;mem_cgroup_nr_lru_pages;mem_cgroup_node_nr_lru_pages
cadvisor;[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];entry_SYSCALL_64_after_swapgs;do_syscall_64;sys_read;vfs_read;seq_read;memcg_stat_show;mem_cgroup_nr_lru_pages;mem_cgroup_node_nr_lru_pages
cadvisor;[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];entry_SYSCALL_64_after_swapgs;do_syscall_64;sys_read;vfs_read;seq_read;memcg_stat_show;mem_cgroup_iter
cadvisor;[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];entry_SYSCALL_64_after_swapgs;do_syscall_64;sys_read;vfs_read;seq_read;memcg_stat_show;mem_cgroup_nr_lru_pages;mem_cgroup_node_nr_lru_pages
cadvisor;[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];[cadvisor];entry_SYSCALL_64_after_swapgs;do_syscall_64;sys_read;vfs_read;seq_read;memcg_stat_show;mem_cgroup_nr_lru_pages;mem_cgroup_node_nr_lru_pages
ksoftirqd/11;ret_from_fork;kthread;kthread;smpboot_thread_fn;smpboot_thread_fn;run_ksoftirqd;__do_softirq;net_rx_action;ixgbe_poll;ixgbe_clean_rx_irq;napi_gro_receive;netif_receive_skb_internal;inet_gro_receive;bond_handle_frame;__netif_receive_skb_core;ip_rcv_finish;ip_rcv;ip_forward_finish;ip_forward;ip_finish_output;nf_iterate;ip_output;ip_finish_output2;__dev_queue_xmit;dev_hard_start_xmit;ipip_tunnel_xmit;ip_tunnel_xmit;iptunnel_xmit;ip_local_out;dst_output;__ip_local_out;nf_hook_slow;nf_iterate;nf_conntrack_in;generic_packet;ipt_do_table;set_match_v4;ip_set_test;hash_net4_kadt;ixgbe_xmit_frame_ring;swiotlb_dma_mapping_error;hash_net4_test
ksoftirqd/11;ret_from_fork;kthread;kthread;smpboot_thread_fn;smpboot_thread_fn;run_ksoftirqd;__do_softirq;net_rx_action;gro_cell_poll;napi_gro_receive;netif_receive_skb_internal;inet_gro_receive;__netif_receive_skb_core;ip_rcv_finish;ip_rcv;ip_forward_finish;ip_forward;ip_finish_output;nf_iterate;ip_output;ip_finish_output2;__dev_queue_xmit;dev_hard_start_xmit;dev_queue_xmit_nit;packet_rcv;tpacket_rcv;sch_direct_xmit;validate_xmit_skb_list;validate_xmit_skb;netif_skb_features;ixgbe_xmit_frame_ring;swiotlb_dma_mapping_error;__dev_queue_xmit;dev_hard_start_xmit;__bpf_prog_run;__bpf_prog_run

There’s a lot there, but looking through it you can see it’s the cadvisor-then-ksoftirqd pattern we saw from the ICMP tracer above. What does it mean?

Each line is a trace of the CPU at a point in time. Each call down the stack is separated by ; on that line. Looking at the middle of the lines we can see the syscall being called is read(): .... ;do_syscall_64;sys_read; ... So cadvisor is spending a lot of time in a read() syscall relating to mem_cgroup_* functions (the top of the call stack / end of line).

The call stack trace isn’t convenient to see what’s being read, so let’s use strace to see what cadvisor is doing and find 100ms-or-slower syscalls:

theojulienne@kube-node-bad ~ $ sudo strace -p 10137 -T -ff 2>&1 | egrep '<0\.[1-9]'
[pid 10436] <... futex resumed> )       = 0 <0.156784>
[pid 10432] <... futex resumed> )       = 0 <0.258285>
[pid 10137] <... futex resumed> )       = 0 <0.678382>
[pid 10384] <... futex resumed> )       = 0 <0.762328>
[pid 10436] <... read resumed> "cache 154234880\nrss 507904\nrss_h"..., 4096) = 658 <0.179438>
[pid 10384] <... futex resumed> )       = 0 <0.104614>
[pid 10436] <... futex resumed> )       = 0 <0.175936>
[pid 10436] <... read resumed> "cache 0\nrss 0\nrss_huge 0\nmapped_"..., 4096) = 577 <0.228091>
[pid 10427] <... read resumed> "cache 0\nrss 0\nrss_huge 0\nmapped_"..., 4096) = 577 <0.207334>
[pid 10411] <... epoll_ctl resumed> )   = 0 <0.118113>
[pid 10382] <... pselect6 resumed> )    = 0 (Timeout) <0.117717>
[pid 10436] <... read resumed> "cache 154234880\nrss 507904\nrss_h"..., 4096) = 660 <0.159891>
[pid 10417] <... futex resumed> )       = 0 <0.917495>
[pid 10436] <... futex resumed> )       = 0 <0.208172>
[pid 10417] <... futex resumed> )       = 0 <0.190763>
[pid 10417] <... read resumed> "cache 0\nrss 0\nrss_huge 0\nmapped_"..., 4096) = 576 <0.154442>

Sure enough, we see the slowread() calls. From the content being read and mem_cgroup context above, these read() calls are to a memory.stat file which shows the memory usage and limits of a cgroup (the resource isolation technology used by Docker). cadvisor is polling this file to get resource utilization details for the containers. Let’s see if it’s the kernel or cadvisor that’s doing something unexpected by attempting the read ourselves:

theojulienne@kube-node-bad ~ $ time cat /sys/fs/cgroup/memory/memory.stat >/dev/null

real    0m0.153s
user    0m0.000s
sys    0m0.152s
theojulienne@kube-node-bad ~ $ 

Since we can reproduce it, this indicates that it’s the kernel hitting a pathologically bad case.

What causes this read to be so slow

At this point it’s much more simple to find similar issues reported by others. As it turns out, this has been reported to cadvisor as an excessive CPU usage problem, it just hadn’t been observed that latency was also being introduced to the network stack randomly as well. In fact, some folks internally had noticed cadvisor was consuming more CPU than expected, but it didn’t seem to be causing an issue since our servers had plenty of CPU capacity, and so the CPU usage hadn’t yet been investigated.

The overview of the issue is that the memory cgroup is accounting for memory usage inside a namespace (container). When all processes in that cgroup exit, the memory cgroup is released by Docker. However, “memory” isn’t just process memory, and although processes memory usage itself is gone, it turns out the kernel also assigns cached content like dentries and inodes (directory and file metadata) that are cached to the memory cgroup. From that issue:

“zombie” cgroups: cgroups that have no processes and have been deleted but still have memory charged to them (in my case, from the dentry cache, but it could also be from page cache or tmpfs).

Rather than the kernel iterating over every page in the cache at cgroup release time, which could be very slow, they choose to wait for those pages to be reclaimed and then finally clean up the cgroup once all are reclaimed when memory is needed, lazily. In the meantime, the cgroup still needs to be counted during stats collection.

From a performance perspective, they are trading off time on a slow process by amortizing it over the reclamation of each page, opting to make the initial cleanup fast in return for leaving some cached memory around. That’s fine, when the kernel reclaims the last of the cached memory, the cgroup eventually gets cleaned up, so it’s not really a “leak”. Unfortunately the search that memory.stat performs, the way it’s implemented on the kernel version (4.9) we’re running on some servers, combined with the huge amount of memory on our servers, means it can take a significantly long time for the last of the cached data to be reclaimed and for the zombie cgroup to be cleaned up.

It turns out we had nodes that had such a large number of zombie cgroups that some had reads/stalls of over a second.

The workaround on that cadvisor issue, to immediately free the dentries/inodes cache systemwide, immediately stopped the read latency, and also the network latency stalls on the host, since the dropping of the cache included the cached pages in the “zombie” cgroups and so they were also freed. This isn’t a solution, but it does validate the cause of the issue.

As it turns out newer kernel releases (4.19+) have improved the performance of the memory.stat call and so this is no longer a problem after moving to that kernel. In the interim, we had existing tooling that was able to detect problems with nodes in our Kubernetes clusters and gracefully drain and reboot them, which we used to detect the cases of high enough latency that would cause issues, and treat them with a graceful reboot. This gave us breathing room while OS and kernel upgrades were rolled out to the remainder of the fleet.

Wrapping up

Since this problem manifested as NIC RX queues not being processed for hundreds of milliseconds, it was responsible for both high latency on short connections and latency observed mid-connection such as between MySQL query and response packets. Understanding and maintaining performance of our most foundational systems like Kubernetes is critical to the reliability and speed of all services that build on top of them. As we invest in and improve on this performance, every system we run benefits from those improvements.

GLB: GitHub's open source load balancer

Wed, 08 Aug 2018 00:00:00 +0000

At GitHub, we serve tens of thousands of requests every second out of our network edge, operating on GitHub’s metal cloud. We’ve previously introduced GLB, our scalable load balancing solution for bare metal datacenters, which powers the majority of GitHub’s public web and git traffic, as well as fronting some of our most critical internal systems such as highly available MySQL clusters. Today we’re excited to share more details about our load balancer’s design, as well as release the GLB Director as open source.

GLB Director is a Layer 4 load balancer which scales a single IP address across a large number of physical machines while attempting to minimise connection disruption during any change in servers. GLB Director does not replace services like haproxy and nginx, but rather is a layer in front of these services (or any TCP service) that allows them to scale across multiple physical machines without requiring each machine to have unique IP addresses.

Scaling an IP using ECMP

The basic property of a Layer 4 load balancer is the ability to take a single IP address and spread inbound connections across multiple servers. To scale a single IP address to handle more traffic than any single machine can process, we need to not only split amongst backend servers, but also need to be able to scale up the servers that handle the load balancing themselves. This is essentially another layer of load balancing.

Typically we think of an IP address as referencing a single physical machine, and routers as moving a packet to the next closest router to that machine. In the simplest case where there’s always a single best next hop, routers pick that hop and forward all packets there until the destination is reached.

In reality, most networks are far more complicated. There is often more than a single path available between two machines, for example where multiple ISPs are available or even when two routers are joined together with more than one physical cable to increase capacity and provide redundancy. This is where Equal-Cost Multi-Path (ECMP) routing comes in to play - rather than routers picking a single best next hop, where they have multiple hops with the same cost (usually defined as the number of ASes to the destination), they instead hash traffic so that connections are balanced across all available paths of equal cost.

ECMP is implemented by hashing each packet to determine a relatively consistent selection of one of the available paths. The hash function used here varies by device, but typically it’s a consistent hash based on the source and destination IP address as well as the source and destination port for TCP traffic. This means that multiple packets for the same ongoing TCP connection will typically traverse the same path, meaning that packets will arrive in the same order even when paths have different latencies. Notably in this case, the paths can change without any disruption to connections because they will always end up at the same destination server, and at that point the path it took is mostly irrelevant.

An alternative use of ECMP can come in to play when we want to shard traffic across multiple servers rather than to the same server over multiple paths. Each server can announce the same IP address with BGP or another similar network protocol, causing connections to be sharded across those servers, with the routers blissfully unaware that the connections are being handled in different places, not all ending on the same machine as would traditionally be the case.

While this shards traffic as we had hoped, it has one huge drawback: when the set of servers that are announcing the same IP change (or any path or router along the way changes), connections must rebalance to maintain an equal balance of connections on each server. Routers are typically stateless devices, simply making the best decision for each packet without consideration to the connection it is a part of, which means some connections will break in this scenario.

In the above example on the left, we can imagine that each colour represents an active connection. A new proxy server is added to announce the same IP. The router diligently adjusts the consistent hash to move 1/3 connections to the new server while keeping 2/3 connections where they were. Unfortunately for those 1/3 connections that were already in progress, the packets are now arriving on a server that doesn’t know about the connection, and so they fail.

Split director/proxy load balancer design

The issue with the previous ECMP-only solution is that it isn’t aware of the full context for a given packet, nor is it able to store data for each packet/connection. As it turns out, there are commonly used patterns to help out with this situation by implementing some stateful tracking in software, typically using a tool like Linux Virtual Server (LVS). We create a new tier of “director” servers that take packets from the router via ECMP, but rather than relying on the router’s ECMP hashing to choose the backend proxy server, we instead control the hashing and store state (which backend was chosen) for all in-progress connections. When we change the set of proxy tier servers, the director tier hopefully hasn’t changed, and our connection will continue.

Although this works well in many cases, it does have some drawbacks. In the above example, we add both a LVS director and backend proxy server at the same time. The new director receives some set of packets, but doesn’t have any state yet (or has delayed state), so hashes it as a new connection and may get it wrong (and cause the connection to fail). A typical workaround with LVS is to use multicast connection syncing to keep the connection state shared amongst all LVS director servers. This still requires connection state to propagate, and also still requires duplicate state - not only does each proxy need state for each connection in the Linux kernel network stack, but every LVS director also needs to store a mapping of connection to backend proxy server.

Removing all state from the director tier

When we were designing GLB, we decided we wanted to improve on this situation and not duplicate state at all. GLB takes a different approach to that described above, by using the flow state already stored in the proxy servers as part of maintaining established Linux TCP connections from clients.

For each incoming connection, we pick a primary and secondary server that could handle that connection. When a packet arrives on the primary server and isn’t valid, it is forwarded to the secondary server. The hashing to choose the primary/secondary server is done once, up front, and is stored in a lookup table, and so doesn’t need to be recalculated on a per-flow or per-packet basis. When a new proxy server is added, for 1/N connections it becomes the new primary, and the old primary becomes the secondary. This allows existing flows to complete, because the proxy server can make the decisions with its local state, the single source of truth. Essentially this gives packets a “second chance” at arriving at the expected server that holds their state.

Even though the director will still send connections to the wrong server, that server will then know how to forward on the packet to the correct server. The GLB director tier is completely stateless in terms of TCP flows: director servers can come and go at any time, and will always pick the same primary/secondary server providing their forwarding tables match (but they rarely change). To change proxies, some care needs to be taken, which we describe below.

Maintaining invariants: rendezvous hashing

The core of the GLB Director design comes down to picking that primary and secondary server consistently, and to allow the proxy tier servers to drain and fill as needed. We consider each proxy server to have a state, and carefully adjust the state as a way of adding and removing servers.

We create a static binary forwarding table, which is generated identically on each director server, to map incoming flows to a given primary and secondary server. Rather than having complex logic to pick from all available servers at packet processing time, we instead use some indirection by creating a table (65k rows), with each row containing a primary and secondary server IP address. This is stored in memory as flat array of binary data, taking about 512kb per table. When a packet arrives, we consistently hash it (based on packet data alone) to the same row in that table (using the hash as an index into the array), which provides a consistent primary and secondary server pair.

We want each server to appear approximately equally in both the primary and secondary fields, and to never appear in both in the same row. When we add a new server, we desire some rows to have their primary become secondary, and the new server become primary. Similarly, we desire the new server to become secondary in some rows. When we remove a server, in any rows where it was primary, we want the secondary to become primary, and another server to pick up secondary.

This sounds complex, but can be summarised succinctly with a couple of invariants:

As we change the set of servers, the relative order of existing servers should be maintained.
The order of servers should be computable without any state other than the list of servers (and maybe some predefined seeds).
Each server should appear at most once in each row.
Each server should appear approximately an equal number of times in each column.

Reading the problem that way, Rendezvous hashing is an ideal choice, since it can trivially satisfy these invariants. Each server (in our case, the IP) is hashed along with the row number, the servers are sorted by that hash (which is just a number), and we get a unique order for servers for that given row. We take the first two as the primary and secondary respectively.

Relative order will be maintained because the hash for each server will be the same regardless of which other servers are included. The only information required to generate the table is the IPs of the servers. Since we’re just sorting a set of servers, the servers only appear once. Finally, if we use a good hash function that is pseudo-random, the ordering will be pseudo-random, and so the distribution will be even as we expect.

Draining, filling, adding and removing proxies

Adding or removing proxy servers require some care in our design. This is because a forwarding table entry only defines a primary/secondary proxy, so the draining/failover only works with at most a single proxy host in draining. We define the following valid states and state transitions for a proxy server:

When a proxy server is active, draining or filling, it is included in the forwarding table entries. In a stable state, all proxy servers are active, and the rendezvous hashing described above will have an approximately even and random distribution of each proxy server in both the primary and secondary columns.

As a proxy server transitions to draining, we adjust the entries in the forwarding table by swapping the primary and secondary entries we would have otherwise included:

This has the effect of sending packets to the server that was previously secondary first. Since it receives the packets first, it will accept SYN packets and therefore take any new connections. For any packet it doesn’t understand as relating to a local flow, it forwards it to the other server (the previous primary), which allows existing connections to complete.

This has the effect of draining the desired server of connections gracefully, after which point it can be removed completely, and proxies can shuffle in to fill the empty secondary slots:

A node in filling looks just like active, since the table inherently allows a second chance:

This implementation requires that no more than one proxy server at a time is in any state other than active, which in practise has worked well at GitHub. The state changes to proxy servers can happen as quickly as the longest connection duration that needs to be maintained. We’re working on extensions to the design that support more than just a primary and secondary, and some components (like the header listed below) already include initial support for arbitrary server lists.

Encapsulation within the datacenter

We now have an algorithm to consistently pick backend proxy servers and operate on them, but how do we actually move packets around the datacenter? How do we encode the secondary server inside the packet so the primary can forward a packet it doesn’t understand?

Traditionally in the LVS setup, an IP over IP (IPIP) tunnel is used. The client IP packet is encapsulated inside an internal datacenter IP packet and forwarded on to the proxy server, which decapsulates it. We found that it was difficult to encode the additional server metadata inside IPIP packets, as the only standard space available was the IP Options, and our datacenter routers passed packets with unknown IP options to software for processing (which they called “Layer 2 slow path”), taking speeds from millions to thousands of packets per second.

To avoid this, we needed to hide the data inside a different packet format that the router wouldn’t try to understand. We initially adopted raw Foo-over-UDP (FOU) with a custom Generic Route Encapsulation (GRE) payload, essentially encapsulating everything inside a UDP packet. We recently transitioned to Generic UDP Encapsulation (GUE), which is a layer on top FOU which provides a standard for encapsulating IP protocols inside a UDP packet. We place our secondary server’s IP inside the private data of the GUE header. From a router’s perspective, these packets are all internal datacenter UDP packets between two normal servers.

 0                   1                   2                   3  
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+\
|          Source port          |        Destination port       | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ UDP
|             Length            |            Checksum           | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+/
| 0 |C|   Hlen  |  Proto/ctype  |             Flags             | GUE
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|     Private data type (0)     |  Next hop idx |   Hop count   |\
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
|                             Hop 0                             | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ GLB
|                              ...                              | private
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ data
|                             Hop N                             | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+/

Another benefit to using UDP is that the source port can be filled in with a per-connection hash so that they are flow within the datacenter over different paths (where ECMP is used within the datacenter), and received on different RX queues on the proxy server’s NIC (which similarly use a hash of TCP/IP header fields). This is not possible with IPIP because most commodity datacenter NICs are only able to understand plain IP, TCP/IP and UDP/IP (and a few others). Notably, the NICs we use cannot look inside IP/IP packets.

When the proxy server wants to send a packet back to the client, it doesn’t need to be encapsulated or travel back through our director tier, it can be sent directly to the client (often called “Direct Server Return”). This is typical of this sort of load balancer design and is especially useful for content providers where the majority of traffic flows outbound with a relatively small amount of traffic inbound.

This leaves us with a packet flow that looks like the following:

DPDK for 10G+ line rate packet processing

Since we first publicly discussed our initial design, we’ve completely rewritten glb-director to use DPDK, an open source project that allows very fast packet processing from userland by bypassing the Linux kernel. This has allowed us to achieve NIC line rate processing on commodity NICs with commodity CPUs, and allows us to trivially scale our director tier to handle as much inbound traffic as our public connectivity requires. This is particularly important during DDoS attacks, where we do not want our load balancer to be a bottleneck.

One of our initial goals with GLB was that our load balancer could run on commodity datacenter hardware without any server-specific physical configuration. Both GLB director and proxy servers are provisioned like normal servers in our datacenter. Each server has a bonded pair of network interfaces, and those interfaces are shared between DPDK and Linux on GLB director servers.

Modern NICs support SR-IOV, a technology that enables a single NIC to act like multiple NICs from the perspective of the operating system. This is typically used by virtual machine hypervisors to ask the real NIC (“Physical Function”) to create multiple pretend NICs for each VM (“Virtual Functions”). To enable DPDK and the Linux kernel to share NICs, we use flow bifurcation, which sends specific traffic (destined to GLB-run IP addresses) to our DPDK process on a Virtual Function while leaving the rest of the packets with the Linux kernel’s networking stack on the Physical Function.

We’ve found that the packet processing rates of DPDK on a Virtual Function are acceptable for our requirements. GLB Director uses a DPDK Packet Distributor pattern to spread the work of encapsulating packets across any number of CPU cores on the machine, and since it is stateless this can be highly parallelised.

GLB Director supports matching and forwarding inbound IPv4 and IPv6 packets containing TCP payloads, as well as inbound ICMP Fragmentation Required messages used as part of Path MTU Discovery, by peeking into the inner layers of the packet during matching.

Bringing test suites to DPDK with Scapy

One problem that typically arises in creating (or using) technologies that operate at high speeds due to using low-level primitives (like communicating with the NIC directly) is that they become significantly more difficult to test. As part of creating the GLB Director, we also created a test environment that supports simple end-to-end packet flow testing of our DPDK application, by leveraging the way DPDK provides an Environment Abstraction Layer (EAL) that allows a physical NIC and a libpcap-based local interface to appear the same from the view of the application.

This allowed us to write tests in Scapy, a wonderfully simple Python library for reading, manipulating and writing packet data. By creating a Linux Virtual Ethernet Device, with Scapy on one side and DPDK on the other, we were able to pass in custom crafted packets and validate what our software would provide on the other side, being a fully GUE-encapsulated packet directed to the expected backend proxy server.

This allows us to test more complex behaviours such as traversing layers of ICMPv4/ICMPv6 headers to retrieve the original IPs and TCP ports for correct forwarding of ICMP messages from external routers.

Healthchecking of proxies for auto-failover

Part of the design of GLB is to handle server failure gracefully. The current design of having a designated primary/secondary for a given forwarding table entry / client means that we can work around single-server failure by running health checks from the perspective of each director. We run a service called glb-healthcheck which continually validates each backend server’s GUE tunnel and arbitrary HTTP port.

When a server fails, we swap the primary/secondary entries anywhere that server is primary. This performs a “soft drain” of the server, which provides the best chance for connections to gracefully fail over. If the healthcheck failure is a false positive, connections won’t be disrupted, they will just traverse a slightly different path.

Second chance on proxies with iptables

The final component that makes up GLB is a Netfilter module and iptables target that runs on every proxy server and allows the “second chance” design to function.

This module provides a simple task deciding whether the inner TCP/IP packet inside every GUE packet is valid locally according to the Linux kernel TCP stack, and if it isn’t, forwards it to the next proxy server (the secondary) rather than decapsulating it locally.

In the case where a packet is a SYN (new connection) or is valid locally for an established connection, it simply accepts it locally. We then use the Linux kernel 4.x GUE support provided as part of the fou module to receive the GUE packet and process it locally.

Available today as open source

When we started down the path of writing a better datacenter load balancer, we decided that we wanted to release it open source so that others could benefit from and share in our work. We’re excited to be releasing all the components discussed here as open source at github/glb-director. We hope this will allow others to reuse our work and contribute to a common standard software load balancing solution that runs on commodity hardware in physical datacenter environments.

Also, we’re hiring!

GLB and the GLB Director has been an ongoing project designed, authored, reviewed and supported by various members of GitHub’s Production Engineering organisation, including @joewilliams, @nautalice, @ross, @theojulienne and many others. If you’re interested in joining us in building great infrastructure projects like GLB, our Data Center team is hiring production engineers specialising in Traffic Systems, Network and Facilities.

GLB part 2: HAProxy zero-downtime, zero-delay reloads with multibinder

Thu, 01 Dec 2016 00:00:00 +0000

Recently we introduced GLB, the GitHub Load Balancer that powers GitHub.com. The GLB proxy tier, which handles TCP connection and TLS termination is powered by HAProxy, a reliable and high performance TCP and HTTP proxy daemon. As part of the design of GLB, we set out to solve a few of the common issues found when using HAProxy at scale.

Prior to GLB, each host ran a single monolithic instance of HAProxy for all our public services, with frontends for each external IP set, and backends for each backing service. With the number of services we run, this became unwieldy, our configuration was over one thousand lines long with many interdependent ACLs and no modularization. Migrating to GLB we decided to split the configuration per-service and support running multiple isolated load balancer instances on a single machine. Additionally, we wanted to be able to update a single HAProxy configuration easily without any downtime, additional latency on connections or disrupting any other HAProxy instance on the host. Today we are releasing our solution to this problem, multibinder.

HAProxy almost-safe reloads

HAProxy uses the SO_REUSEPORT socket option, which allows multiple processes to create LISTEN sockets on the same IP/port combination. The Linux kernel then balances new connections between all available LISTEN sockets. In this diagram, we see the initial stage of an HAProxy reload starting with a single process (left) and then causing a second process to start (right) which binds to the same IP and port, but with a different socket:

This works great so far, until the original process terminates. HAProxy sends a signal to the original process stating that the new process is now accept()ing and handling connections (left), which causes it to stop accepting new connections and close its own socket before eventually exiting once all connections complete (right):

Unfortunately there’s a small period between when this process last calls accept() and when it calls close() where the kernel will still route some new connections to the original socket. The code then blindly continues to close the socket, and all connections that were queued up in that LISTEN socket get discarded (because accept() is never called for them):

For small scale sites, the chance of a new connection arriving in the few microseconds between these calls is very low. Unfortunately at the scale we run HAProxy, a customer impacting number of connections would hit this issue each and every time we reload HAProxy. Previously we used the official solution offered by HAProxy, dropping SYN packets during this small window, causing the client to retry the SYN packet shortly afterwards. Other potential solutions to the same problem include using tc qdisc to stall the SYN packets as they come in, and then un-stall the queue once the reload is complete. During development of GLB, we weren’t satisfied with either solution and sought out one without any queue delays and sharing of the same LISTEN socket.

Supporting zero-downtime, zero-delay reloads

The way other services typically support zero-downtime reloads is to share a LISTEN socket, usually by having a parent process that holds the socket open and fork()s the service when it needs to reload, leaving the socket open for the new process to consume. This creates a slightly different situation, where the kernel has a single LISTEN socket and clients are queued for accept() by either process. The file descriptors in each process may be different, but they will point to the same in-kernel socket structure.

In this scenario, a new process would be started that inherits the same LISTEN socket (left), and when the original pid stops calling accept(), connections remain queued for the new process to process because the kernel LISTEN socket and queue are shared (right):

Unfortunately, HAProxy doesn’t support this method directly. We considered patching HAProxy to add built-in support but found that the architecture of HAProxy favours process isolation and non-dynamic configuration, making it a non-trivial architectural change. Instead, we created multibinder to solve this problem generically for any daemon that needs zero-downtime reload capabilities, and integrated it with HAProxy by using a few tricks with existing HAProxy configuration directives to get the same result.

Multibinder is similar to other file-descriptor sharing services such as einhorn, except that it runs as an isolated service and process tree on the system, managed by your usual process manager. The actual service, in this case HAProxy, runs separately as another service, rather than as a child process. When HAProxy is started, a small wrapper script calls out to multibinder and requests the existing LISTEN socket to be sent using Ancillary Data over an UNIX Domain Socket. The flow looks something like the following:

Once the socket is provided to the HAProxy wrapper, it leaves the LISTEN socket in the file descriptor table and writes out the HAProxy configuration file from an ERB template, injecting the file descriptors using file descriptor binds like fd@N (where N is the file descriptor received from multibinder), then calls exec() to launch HAProxy which uses the provided file descriptor rather than creating a new socket, thus inheriting the same LISTEN socket. From here, we get the ideal setup where the original HAProxy process can stop calling accept() and connections simply queue up for the new process to handle.

Example & multiple instances

Along with the release of multibinder, we’re also providing examples of running multiple HAProxy instances with multibinder leveraging systemd service templates. Following these instructions you can launch a set of HAProxy servers using separate configuration files, each using the same system-wide multibinder instance to request their binds and having true zero-downtime, zero-delay reloads.

Introducing the GitHub Load Balancer

Thu, 22 Sep 2016 00:00:00 +0000

At GitHub we serve billions of HTTP, Git and SSH connections each day. To get the best performance we run on bare metal hardware. Historically one of the more complex components has been our load balancing tier. Traditionally we scaled this vertically, running a small set of very large machines running haproxy, and using a very specific hardware configuration allowing dedicated 10G link failover. Eventually we needed a solution that was scalable and we set out to create a load balancer solution that would run on commodity hardware in our typical data center configuration.

Over the last year we’ve developed our new load balancer, called GLB (GitHub Load Balancer). Today, and over the next few weeks, we will be sharing the design and releasing its components as open source software.

Out with the old, in with the new

GitHub is growing and our monolithic, vertically scaled load balancer tier had met its match and a new approach was required. Our original design was based around a small number of large machines each with dedicated links to our network spine. This design tied networking gear, the load balancing hosts and load balancer configuration together in such a way that scaling horizontally was deemed too difficult. We set out to find a better way.

We first identified the goals of the new system, design pitfalls of the existing system and prior art that we could draw experience and inspiration from. After some time we determined that the following would produce a successful load balancing tier that we could maintain into the future:

Runs on commodity hardware
Scales horizontally
Supports high availability, avoids breaking TCP connections during normal operation and failover
Supports connection draining
Per service load balancing, with support for multiple services per load balancer host
Can be iterated on and deployed like normal software
Testable at each layer, not just integration tests
Built for multiple POPs and data centers
Resilient to typical DDoS attacks, and tools to help mitigate new attacks

Design

To achieve these goals we needed to rethink the relationship between IP addresses and hosts, the constituent layers of our load balancing tier and how connections are routed, controlled and terminated.

Stretching an IP

In a typical setup, you assign a single public facing IP address to a single physical machine. DNS can then be used to split traffic over multiple IPs, letting you shard traffic across multiple servers. Unfortunately, DNS entries are cached fairly aggressively (often ignoring the TTL), and some of our users may specifically whitelist or hardcode IP addresses. Additionally, we offer a certain set of IPs for our Pages service which customers can use directly for their apex domain. Rather than relying on adding additional IPs to increase capacity, and having an IP address fail when the single server failed, we wanted a solution that would allow a single IP address to be served by multiple physical machines.

Routers have a feature called Equal-Cost Multi-Path (ECMP) routing, which is designed to split traffic destined for a single IP across multiple links of equal cost. ECMP works by hashing certain components of an incoming packet such as the source and destination IP addresses and ports. By using a consistent hash for this, subsequent packets that are part of the same TCP flow will hash to the same path, avoiding out of order packets and maintaining session affinity.

This works great for routing packets across multiple paths to the same physical destination server. Where it gets interesting is when you use ECMP to split traffic destined for a single IP across multiple physical servers, each of which terminate TCP connections but share no state, like in a load balancer. When one of these servers fails or is taken out of rotation and is removed from the ECMP server set a rehash event occurs. 1/N connections will get reassigned to the remaining servers. Since these servers don’t share connection state these connections get terminated. Unfortunately, these connections may not be the same 1/N connections that were mapped to the failing server. Additionally, there is no way to gracefully remove a server for maintenance without also disrupting 1/N active connections.

L4/L7 split design

A pattern that has been used by other projects is to split the load balancers into a L4 and L7 tier. At the L4 tier, the routers use ECMP to shard traffic using consistent hashing to a set of L4 load balancers - typically using software like ipvs/LVS. LVS keeps connection state, and optionally syncs connection state with multicast to other L4 nodes, and forwards traffic to the L7 tier which runs software such as haproxy. We call the L4 tier “director” hosts since they direct traffic flow, and the L7 tier “proxy” hosts, since they proxy connections to backend servers.

This L4/L7 split has an interesting benefit: the proxy tier nodes can now be removed from rotation by gracefully draining existing connections, since the connection state on the director nodes will keep existing connections mapped to their existing proxy server, even after they are removed from rotation for new connections. Additionally, the proxy tier tends to be the one that requires more upkeep due to frequent configuration changes, upgrades and scaling so this works to our advantage.

If the multicast connection syncing is used, then the L4 load balancer nodes handle failure slightly more gracefully, since once a connection has been synced to the other L4 nodes, the connection will no longer be disrupted. Without connection syncing, providing the director nodes hash connections the same way and have the same backend set, connections may successfully continue over a director node failure. In practise, most installations of this tiered design just accept connection disruption under node failure or node maintenance.

Unfortunately, using LVS for the director tier has some significant drawbacks. Firstly, multicast was not something we wanted to support, so we would be relying on the nodes having the same view of the world, and having consistent hashing to the backend nodes. Without connection syncing, certain events, including planned maintenance of nodes, could cause connection disruption. Connection disruption is something we wanted to avoid due to how git cannot retry or resume if the connection is severed mid-flight. Finally, the fact that the director tier requires connection state at all adds an extra complexity to DDoS mitigation such as synsanity - to avoid resource exhaustion, syncookies would now need to be generated on the director nodes, despite the fact that the connections themselves are terminated on the proxy nodes.

Designing a better director

We decided early on in the design of our load balancer that we wanted to improve on the common pattern for the director tier. We set out to design a new director tier that was stateless and allowed both director and proxy nodes to be gracefully removed from rotation without disruption to users wherever possible. Users live in countries with less than ideal internet connectivity, and it was important to us that long running clones of reasonably sized repositories would not fail during planned maintenance within a reasonable time limit.

The design we settled on, and now use in production, is a variant of Rendezvous hashing that supports constant time lookups. We start by storing each proxy host and assign a state. These states handle the connection draining aspect of our design goals and will be discussed further in a future post. We then generate a single, fixed-size forwarding table and fill each row with a set of proxy servers using the ordering component of Rendezvous hashing. This table, along with the proxy states, are sent to all director servers and kept in sync as proxies come and go. When a TCP packet arrives on the director, we hash the source IP to generate consistent index into the forwarding table. We then encapsulate the packet inside another IP packet (actually Foo-over-UDP) destined to the internal IP of the proxy server, and send it over the network. The proxy server receives the encapsulated packet, decapsulates it, and processes the original packet locally. Any outgoing packets use Direct Server Return, meaning packets destined to the client egress directly to the client, completely bypassing the director tier.

Stay tuned

Now that you have a taste of the system that processed and routed the request to this blog post we hope you stay tuned for future posts describing our director design in depth, improving haproxy hot configuration reloads and how we managed to migrate to the new system without anyone noticing.

SYN Flood Mitigation with synsanity

Tue, 12 Jul 2016 00:00:00 +0000

GitHub hosts a wide range of user content, and like all large websites this often causes us to become a target of denial of service attacks. Around a year ago, GitHub was on the receiving end of a large, unusual and very well publicised attack involving both application level and volumetric attacks against our infrastructure.

Our users rely on us to be highly available and we take this seriously. Although the attackers are doing the wrong thing, there’s no use blaming the attacker for their attacks being successful. Our commitment is to own our own availability, and that we have a responsibility to mitigate these sorts of attacks to the maximum extent technically possible.

In an effort to reduce the impact of these attacks, we began work on a series of additional mitigation strategies and systems to better prepare us for a future attack of a similar nature. Today we’re sharing our mitigation for one of the attacks we received: synsanity, a SYN flood DDoS mitigation module for Linux 3.x.

What is a SYN flood anyway?

SYN floods are one of the oldest and most common attacks, so common that the Linux kernel includes some built in support for mitigating them. When a client connects to a server using TCP, it uses the three-way handshake to synchronise:

A SYN packet is essentially the client telling the server “I’d like to connect”. During this handshake, both client and server generate random Initial Sequence Numbers (ISNs), which are used to synchronise the TCP connection between the two parties. These sequence numbers let TCP keep track of which messages have been sent and acknowledged by the other party.

A SYN flood abuses this handshake by only going part way through the handshake. Rather than progressing through the normal sequence, an attacker floods the target server with as many SYN packets as they can muster, from as many different hosts as they can, and spoofing the origin IP as much as they can.

The host receiving the SYN flood must respond to each and every packet with a SYN-ACK, but unfortunately the source IP was likely spoofed, so they go nowhere (or worse, come back as rejected). These packets are almost indistinguishable from real SYN packets from real clients, which means it’s hard or impossible to filter out the bad ones on the server. Even external DDoS scrubbing services can only guess whether a packet is legitimate or part of a flood, making it difficult to mitigate an attack without impacting legitimate traffic.

To make matters worse, when the server is handling normal connections and receives the ACK from a real client, it still needs to know that it came from a SYN packet it sent, so it must also keep a list of connections (in state SYN_RECV) for which a SYN has been received and an ACK has not yet been received.

During a SYN flood, this behaviour is undesirable. If the queue of connections in SYN_RECV has no size limit, memory will get exhausted pretty quickly. If it does have a size limit, as is the case in Linux, then there’s no more space to store state and the connections will simply fail as the packets are dropped.

SYN cookies

SYN cookies are a clever way of avoiding the storage of TCP connection state during the initial handshake, deferring that storage until a valid ACK has been received. It works by crafting the Initial Sequence Number (ISN) in the SYN-ACK packet sent by the server in such a way that it cryptographically hashes details about the initial SYN packet and its TCP options, so that when the ACK is received (with a sequence number 1 larger than the ISN), the server can validate that it generated the SYN-ACK packet for which an ACK is now being received. The server stores no state for the connection until the ACK (containing the validated SYN cookie) is received, and only at that point is state regenerated and stored.

Since this hash is calculated with a secret that only the server knows, it doesn’t significantly weaken the sequence number selection and it’s still difficult for someone to forge an ACK (or other packet) for a different connection without having seen the SYN-ACK from the real server.

SYN cookies have been around for a while, and they have fairly minimal impact on the reliability and spoof-protection of TCP. Rather than enabling them constantly, the Linux kernel by default automatically enables SYN cookies only when the SYN receive queue is full. This means that under normal circumstances when no SYN flood is occurring, you get no impact at all, but during a SYN flood, you accept the minimal impact of SYN cookies (in return for not dropping connections). The extra CPU cost of creating SYN cookies is offset by the fact that you no longer have a limited resource, and in practise this is an excellent trade-off.

In Linux 3.x, SYN cookies are generated inside a machine-wide lock on the LISTEN socket that the packet was destined for. This implementation causes all SYN cookies to be generated serially across all cores, defeating the benefits of a multi-processor system. To make matters worse, all cores spin waiting for the lock to become available. This was fine back in the days when an average attacker could only send a few MBits of SYN packets your way, mostly thanks to the networks being much slower. These days however, with servers attached to transit providers with multiple 10GB+ links the whole way down the line, it’s now possible to completely saturate CPU resources.

While Linux 4.x has a patch to send SYN cookies under a per-CPU-core socket lock, which does fix the problem, we wanted a solution that allowed us to use an existing, maintained kernel with upstream security patches. We didn’t want to roll and maintain an entire custom kernel and all related future security patches just to mitigate this form of attack. Patching Linux 3.x to backport the socket lock change was also a similar maintenance burden we wanted to avoid.

SYNPROXY

One solution to get the best of both worlds was the SYNPROXY iptables module. It sits in netfilter in the kernel, before the Linux TCP stack, and as the name suggests, proxies all connections while generating SYN cookies. When a SYN packet comes in, it responds with a SYN-ACK and throws away all state. On receipt of a valid ACK packet matching the SYN cookie, it then sends a SYN downstream and completes the usual TCP handshake. For every subsequent packet in each direction, it modifies the sequence numbers so that it is transparent to both sides.

This is quite an intrusive way of solving the problem since it touches every packet during the entire connection, but it does successfully mitigate SYN floods. Unfortunately we found that in practise under our load and with the amount of malformed packets we receive, it quickly broke down and caused a kernel panic. Additionally, it had to be enabled all the time, since there was no simple way to activate it only when under attack. This meant that we would have to accept the minimal impact of SYN cookies constantly, and at our scale this still would likely cause issues for some of our users.

We decided that it was more complicated than it needed to be for our use case, and we wanted a simpler solution that would only touch the packets that needed to be touched to mitigate a SYN flood. We also decided that a mitigation should only cause potential (even if minimal) impact during mitigation, and not under normal operation.

synsanity

Enter synsanity, our solution to mitigate SYN floods on Linux 3.x. synsanity is inspired by SYNPROXY, in that it is an iptables module that sits inside iptables between the Linux TCP stack and the network card. The major difference is that rather than touch all packets, synsanity simply generates a SYN cookie identically to the way the Linux kernel would generate one if the SYN queue was full, and once it validates the ACK packet, it allows it through to the standard Linux SYN cookie code, which creates and completes the connection. After this point, synsanity doesn’t touch any further packet in the TCP connection.

Similar to the way that Linux only enables SYN cookies when the SYN queue overflows, we only enable synsanity when the SYN queue overflows as well. We match the core Linux code exactly, except that we do it in an iptables module, outside the LISTEN lock. Since an iptables module can be compiled and maintained outside the Linux kernel source tree itself, we don’t need to use a custom Linux kernel, and can instead just maintain and deploy a single module to our servers.

synsanity has allowed us to mitigate multiple attacks that would have previously caused a partial or complete service outage, both long running attacks and large volume attacks.

synsanity sending SYN cookies during a 300kpps SYN flood

Open Source

We believe that if you need to hide your mitigation to keep it secure, it’s not designed well enough. The best and most secure tools are shared, open and subject to community scrutiny, so today we’re open sourcing synsanity so that everyone can benefit from this work.