MTU (maximum transmission unit)

The maximum transmission unit (MTU) is the largest packet, in bytes, that a network link can carry in one piece. Most Ethernet and Wi-Fi networks have an MTU of 1500 bytes. Anything bigger has to be split into fragments or is dropped.

MTU on an overlay network

Every packet a host sends to another host’s overlay address gets wrapped before it crosses the underlay network. Nebula adds its own header and an authentication tag, then sends the result inside a UDP packet with its own IP header, which adds about 60 bytes over IPv4. The overlay’s MTU has to be smaller than the underlay’s by at least that much, or the wrapped packets won’t fit.

Nebula’s default overlay MTU, the tun.mtu setting, is 1300 bytes. That leaves room for the wrapping on a standard 1500-byte link, plus extra headroom for paths with a smaller MTU, such as PPPoE connections, mobile networks, or other tunnels along the way.

Signs of an MTU problem

MTU problems can be confusing, because small packets get through and only large ones don’t. ping works and SSH logs in, but copying a file stalls. Web pages or API calls hang partway, or TLS connections never finish setting up.

To check whether full-size packets reach another host, ping its overlay IP with a 1272-byte payload and fragmentation turned off. With 28 bytes of ICMP and IP headers, that makes a 1300-byte packet:

# Linux
ping -M do -s 1272 100.100.0.9
# macOS
ping -D -s 1272 100.100.0.9
# Windows
ping -f -l 1272 100.100.0.9

Changing the MTU

Set the tun.mtu config override, 1200 or higher, on a network, on a tag to apply it to every host that carries the tag, or on a single host. Upcoming Nebula releases cap it at 8937, with a warning, if it’s set higher.

Lower it when hosts sit behind links that add their own overhead and large transfers stall. Raise it only when every path between the hosts supports larger packets, such as jumbo frames within a single cloud availability zone. In AWS, for example, the underlay MTU inside an availability zone is 9001, which allows an overlay MTU of 8600. Traffic that leaves the zone drops back to 1500, though, and the oversized packets get fragmented, which cloud providers often rate limit heavily.

Setting it on the network or on a tag, rather than host by host, keeps it consistent across hosts that talk to each other.