Fix Slow Network (NAT) after Debian Wheezy Kernel Update 3.2.0-4

NOTE: This issue was fixed with 3.2.60-1+deb7u3 update that came out in Debian’s security update stream.

Router with FirewallI noticed a few weeks ago that after a Debian kernel update on my Debian-based router, network performance degraded terribly. Linux clients behind this Debian firewall did not seem to be effected nearly as much as the Windows clients — Windows machines could not upload at all to the Internet once this Debian update was in place on the router.

At first I thought it was Comcast, before I realized that it was mostly the Windows machines that had slow network performance. Sometimes download performance was effected as well – some sites just stalling, and Pandora was practically unlistenable.

After searching around a bit, I found an old bug where the network address translation Linux kernel code had been patched for handling the defragmentation of packets that exceeded MTU values, if I’m remembering right. Apparently this “fix” caused a number of problems with the 3.2.0-4 Debian GNU/Linux kernel when it was implemented along with some security updates.

I started playing around with it on my own, and managed to find a Debian bug where a couple patches were available that patched it back. This is very, very good, because the network connection was pretty much unusable if you were using IP Masquerading or NAT as a firewall/router.

The bug is documented on the Debian bugsite, along with the kernel patches. But if you’d like a step-by-step, this is what I did to fix the problem on 2 different routers so far:

Prepare

You’ll need some disk space — probably around 10G free. Always back up — if following these steps results in an unbootable machine for you, don’t blame me. It very well could. Particularly if you don’t pay attention, or know things that I can’t even imagine you don’t know. Which is hard. You’ve been warned. It’s a kernel recompile! I’d say wait for Debian to release it in the channel, but it’s been weeks, and I’m sure some of you have been suffering as much as me.

Install Debian Packages

This is a kernel compile – we’ll be keeping all of Debian’s customizations, along with their current kernel, just with our 2 little extra patches applied. As such, you’ll need some source to compile, and the Debian scripts that automate the Debian Way. It’s a boatload of packages…

# apt-get install devscripts
# apt-get build-dep linux

I know, sweetie.

To The Kernel Source and Patch

I like to do my dirty work in /usr/src – and when doing it, I like to be root, not any of that sudo or fakeroot stuff. So if you’re playing it safe and wise, you’ll need to fakeroot these compiles. I leave it to you. But if you’re willing to be root, here’s the easy:

# cd /usr/src
# mkdir linux-deb
# cd linux-deb
# apt-get source linux

NOTE! You might want to specify “linux=3.2.60-1+deb7u1” instead of just the plain “linux” there. That way you’re sure to get the right version – the version with the problem, that matches with this fix.

As for the patches, I’ll link to the ones provided in the bug report that you can get with wget — I’ve also included them as full text below if you’d rather, in case the cut & paste for these long URI’s don’t work right for you.

If you can these two long lines pasted, you’ll get two files outputted to your working directory that are those patches. Saw this from Teodor Milkov in the bug – thanks Teo!

# wget
--no-check-certificate
"https://bugs.debian.org/cgi-bin/bugreport.cgi?msg=50;filename=revert-net-ip-ipv6-handle-gso-skbs-in-forwarding-pat.patch;att=1;bug=754294"
-O revert-net-ip-ipv6-handle-gso-skbs-in-forwarding-pat.patch


# wget
--no-check-certificate
"https://bugs.debian.org/cgi-bin/bugreport.cgi?msg=50;filename=revert-net-ipv4-ip_forward-fix-inverted-local_df-tes.patch;att=2;bug=754294"
-O revert-net-ipv4-ip_forward-fix-inverted-local_df-tes.patch

Compile Kernel with the Patches

Now you’ll just cd down into the top of your Debian kernel build tree, and apply these patches and compile. This command line is for the amd64 architecture. You maybe have a different one.. ?

And replace that -j 8 with the number of CPU cores you have (or less)

# cd linux-3.2.60
# debian/bin/test-patches -f amd64 -j 8 ../revert-net-ipv4-ip_forward-fix-inverted-local_df-tes.patch ../revert-net-ip-ipv6-handle-gso-skbs-in-forwarding-pat.patch

Now go make some dinner. Do some yoga! Dig in the earth, or paint a room. That will take some time. The first error up top at the very beginning is normal.

Install the new Debian Kernel Package

Now you should have a nice new linux-image-3.2.0-4 deb package file, along with another with debug headers, and just your regular headers. ūüėČ This new Debian package, version-wise, is the same as the one in the main stream, only with a ~test — so I believe we should get newer-versioned kernels automatically when they come out.

Install this deb with the normal

dpkg -i linux-image-3.2.0-4-amd64_3.2.60-1+deb7u1a~test_amd64.deb

It’ll do all your modules and initrd stuff for you, and call your grub menu rebuilder doohicky.

One of my routers failed the install, complaining that it couldn’t make a symlink to the initrd file from / to /boot — that’s because there was no initrd. I solved it by removing my current kernel-image package (ignore the scary warnings if you’re foolhearty) and then running the dpkg -i again on it, where the initrd was made just fine. The other router had no problem with it. Go figure.

Hope this helps some of you if you’re having those terrible network performance problems after that last Debian kernel update. I wish they could get these fixed sooner.

Anyway, here are those patches if you need to cut and paste your own, instead of wgetting from those obnoxiously long URI’s. Just put them in any named file, and then be sure to call them by those names from the test-patches step.

diff --git a/net/ipv4/ip_forward.c b/net/ipv4/ip_forward.c
index 7593f3a..e0d9f02 100644
--- a/net/ipv4/ip_forward.c
+++ b/net/ipv4/ip_forward.c
@@ -42,12 +42,12 @@
 static bool ip_may_fragment(const struct sk_buff *skb)
 {
     return unlikely((ip_hdr(skb)->frag_off & htons(IP_DF)) == 0) ||
-        skb->local_df;
+           !skb->local_df;
 }
 
 static bool ip_exceeds_mtu(const struct sk_buff *skb, unsigned int mtu)
 {
-    if (skb->len <= mtu)
+    if (skb->len <= mtu || skb->local_df)
         return false;
 
     if (skb_is_gso(skb) && skb_gso_network_seglen(skb) <= mtu)
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -2588,22 +2588,5 @@ static inline bool skb_is_recycleable(co
 
     return true;
 }
-
-/**
- * skb_gso_network_seglen - Return length of individual segments of a gso packet
- *
- * @skb: GSO skb
- *
- * skb_gso_network_seglen is used to determine the real size of the
- * individual segments, including Layer3 (IP, IPv6) and L4 headers (TCP/UDP).
- *
- * The MAC/L2 header is not accounted for.
- */
-static inline unsigned int skb_gso_network_seglen(const struct sk_buff *skb)
-{
-    unsigned int hdr_len = skb_transport_header(skb) -
-                   skb_network_header(skb);
-    return hdr_len + skb_gso_transport_seglen(skb);
-}
 #endif    /* __KERNEL__ */
 #endif    /* _LINUX_SKBUFF_H */
--- a/net/ipv4/ip_forward.c
+++ b/net/ipv4/ip_forward.c
@@ -39,68 +39,6 @@
 #include <net/route.h>
 #include <net/xfrm.h>
 
-static bool ip_may_fragment(const struct sk_buff *skb)
-{
-    return unlikely((ip_hdr(skb)->frag_off & htons(IP_DF)) == 0) ||
-           !skb->local_df;
-}
-
-static bool ip_exceeds_mtu(const struct sk_buff *skb, unsigned int mtu)
-{
-    if (skb->len <= mtu || skb->local_df)
-        return false;
-
-    if (skb_is_gso(skb) && skb_gso_network_seglen(skb) <= mtu)
-        return false;
-
-    return true;
-}
-
-static bool ip_gso_exceeds_dst_mtu(const struct sk_buff *skb)
-{
-    unsigned int mtu;
-
-    if (skb->local_df || !skb_is_gso(skb))
-        return false;
-
-    mtu = dst_mtu(skb_dst(skb));
-
-    /* if seglen > mtu, do software segmentation for IP fragmentation on
-     * output.  DF bit cannot be set since ip_forward would have sent
-     * icmp error.
-     */
-    return skb_gso_network_seglen(skb) > mtu;
-}
-
-/* called if GSO skb needs to be fragmented on forward */
-static int ip_forward_finish_gso(struct sk_buff *skb)
-{
-    struct sk_buff *segs;
-    int ret = 0;
-
-    segs = skb_gso_segment(skb, 0);
-    if (IS_ERR(segs)) {
-        kfree_skb(skb);
-        return -ENOMEM;
-    }
-
-    consume_skb(skb);
-
-    do {
-        struct sk_buff *nskb = segs->next;
-        int err;
-
-        segs->next = NULL;
-        err = dst_output(segs);
-
-        if (err && ret == 0)
-            ret = err;
-        segs = nskb;
-    } while (segs);
-
-    return ret;
-}
-
 static int ip_forward_finish(struct sk_buff *skb)
 {
     struct ip_options * opt    = &(IPCB(skb)->opt);
@@ -110,9 +48,6 @@ static int ip_forward_finish(struct sk_b
     if (unlikely(opt->optlen))
         ip_forward_options(skb);
 
-    if (ip_gso_exceeds_dst_mtu(skb))
-        return ip_forward_finish_gso(skb);
-
     return dst_output(skb);
 }
 
@@ -152,7 +87,8 @@ int ip_forward(struct sk_buff *skb)
     if (opt->is_strictroute && opt->nexthop != rt->rt_gateway)
         goto sr_failed;
 
-    if (!ip_may_fragment(skb) && ip_exceeds_mtu(skb, dst_mtu(&rt->dst))) {
+    if (unlikely(skb->len > dst_mtu(&rt->dst) && !skb_is_gso(skb) &&
+             (ip_hdr(skb)->frag_off & htons(IP_DF))) && !skb->local_df) {
         IP_INC_STATS(dev_net(rt->dst.dev), IPSTATS_MIB_FRAGFAILS);
         icmp_send(skb, ICMP_DEST_UNREACH, ICMP_FRAG_NEEDED,
               htonl(dst_mtu(&rt->dst)));
--- a/net/ipv6/ip6_output.c
+++ b/net/ipv6/ip6_output.c
@@ -381,17 +381,6 @@ static inline int ip6_forward_finish(str
     return dst_output(skb);
 }
 
-static bool ip6_pkt_too_big(const struct sk_buff *skb, unsigned int mtu)
-{
-    if (skb->len <= mtu || skb->local_df)
-        return false;
-
-    if (skb_is_gso(skb) && skb_gso_network_seglen(skb) <= mtu)
-        return false;
-
-    return true;
-}
-
 int ip6_forward(struct sk_buff *skb)
 {
     struct dst_entry *dst = skb_dst(skb);
@@ -515,7 +504,7 @@ int ip6_forward(struct sk_buff *skb)
     if (mtu < IPV6_MIN_MTU)
         mtu = IPV6_MIN_MTU;
 
-    if (ip6_pkt_too_big(skb, mtu)) {
+    if (skb->len > mtu && !skb_is_gso(skb)) {
         /* Again, force OUTPUT device used as source address */
         skb->dev = dst->dev;
         icmpv6_send(skb, ICMPV6_PKT_TOOBIG, 0, mtu);
  • Digo Garcia

    I have Linux clients that were affected…

  • Yeah, I think they all were – I noticed a small bit on Linux. But the Windows clients were practically useless. They wouldn’t even do an upload speed test. And downloads were almost impossible! I’m amazed this happened, really. I’ve never seen such an all-pervasive things happen.