Wifi latency using RALink RT3090 (rt2800pci)

After updating to Ubuntu 18.04 Bionic, I found my RALink RT3090 wireless card using the rt2800pci driver introduced latency. This was most noticeable as a delay while typing over SSH.

This is because power management is enabled on the card, which puts it into sleep mode and it takes some time to wake up again.

You can see the power management state of the card with:

sudo iwconfig

Amongst other things, this will show:

Power Management:on

You can test whether disabling power management solves the problem with:

sudo iwconfig DEV power off

Where DEV is the device name (eg: wlan0, wlp3s0)

If this solves the problem, then apply the setting on boot. Edit the file /etc/NetworkManager/conf.d/default-wifi-powersave-on.conf and change:

[connection]
wifi.powersave = 3

So it reads:

wifi.powersave = 2

realtek RTL8188CUS slow on Raspberry Pi

I recently had an adventure troubleshooting slow wifi on one of my Raspberry Pi systems. No matter what I did, I could not get more than 57 kilobytes per second transfer speed to it.

finding the netdev_priv struct

In addition to the net_device struct within the kernel, network drivers also have their own private or device-specific struct which stores stats unique to the individual hardware.

The name of the struct varies for each device type, however the location remains the same, right after net_device:

linux-2.6.32-358.14.1.el6.x86_64/include/linux/netdevice.h

/**
 *      netdev_priv - access network device private data
 *      @dev: network device
 *
 * Get network device private data
 */
static inline void *netdev_priv(const struct net_device *dev)
{
        return (char *)dev + ALIGN(sizeof(struct net_device), NETDEV_ALIGN);
}

A search in cscope for netdev_priv will show you many functions within drivers which update the priv structure via pointers, for example this one in e1000_main.c:

struct e1000_adapter *adapter = netdev_priv(netdev);

So, given that we know what the device-specific struct is called, and where the device-specific private struct is, how do you find it and read it?

We'll need the ability to read kernel memory using crash, either run on a live system, or in a vmcore captured with kdump:

crash /usr/lib/debug/lib/modules/2.6.32-358.14.1.el6.x86_64/vmlinux /var/crash/2013-08-26/vmcore

We'll also need the debugging symbols for the driver in question:

crash> mod -s e1000 /usr/lib/debug/lib/modules/2.6.32-358.14.1.el6.x86_64/kernel/drivers/net/e1000/e1000.ko.debug
     MODULE       NAME                 SIZE  OBJECT FILE
ffffffffa01550e0  e1000              170678  /usr/lib/debug/lib/modules/2.6.32-358.14.1.el6.x86_64/kernel/drivers/net/e1000/e1000.ko.debug 

The net command will show you the network devices in the system:

crash> net
   NET_DEVICE     NAME   IP ADDRESS(ES)
ffff88007e76b820  lo     127.0.0.1
ffff8800372c0020  eth0   192.168.1.126

We can then cast net_device against this to see the in-kernel device struct:

crash> net_device 0xffff8800372c0020
struct net_device {
  name = "eth0\000\000\060:03.0\000\000\000",
...

But we want to get after this struct and cast it against the struct in the driver.

We need to know how big net_device is:

crash> struct -o net_device
struct net_device {
...
}
SIZE: 1728

We then find the net address plus the size of net_device:

crash> px 0xffff8800372c0020+1728
$1 = 0xffff8800372c06e0

We can now cast the device-specific private struct against this new address:

crash> e1000_adapter 0xffff8800372c06e0
struct e1000_adapter {
  vlgrp = 0x0,
  mng_vlan_id = 65535,
  bd_number = 0,
  rx_buffer_len = 1522,
...

and we're done!

how to display tcp rto

Calculating the TCP Retransmission Timeout always makes my head explode, yet it can be useful when troubleshooting TCP algorithms like FRTO or TIME_WAIT timeout when TCP Timestamps are in use.

This excellent post by sgros gives a very thorough overview of the RTO algorithms in the kernel and how RTO is actually calculated:

• http://sgros.blogspot.com/2012/02/calculating-tcp-rto.html

However, I don't want to do that, and I certainly don't want to do it for every socket in realtime. Is there an easy way to display the RTO for every socket?

There sure is:

ss -i

The -i option of ss is the flag to "Show internal TCP information", check man ss for more options.

Here's an example of a socket on my system including its RTO:

State   Recv-Q   Send-Q    Local Address:Port      Peer Address:Port
ESTAB   0        0           10.99.0.123:51308   192.168.34.111:http
cubic wscale:6,7 rto:660 rtt:396/38.75 ato:40 cwnd:10 send 272.3Kbps rcv_space:14600
                 ^^RTO^^

ss is one of the "next generation" iproute2 commands to replace the older netstat. Much like ip addr has replaced ifconfig, ip route has replaced route, and ip neigh has replaced arp.

how to network jumbo frames to a kvm guest

Getting a jumbo frame to a KVM guest is not something which works out of the box, but can be configured to work quite easily.

One way get jumbo frames to a KVM guest is to directly pass the guest a physical NIC through PCI Passthrough, or a Virtual Function through SR-IOV.

Another way is to use the traditional libvirt bridge. Create a bridge on the host, and bridge the guest's NICs into this. The reason this doesn't work by default is all the interfaces are created with the default Ethernet MTU of 1500 bytes.

You can set MTU of the physical NIC, the bridge, and the guest interface with the network-scripts, but you cannot automatically set the MTU of the tap interface with the network-scripts. The tap interface is how the virtual network interface connects to the "real world". You'll see these tap devices with names like "vnet0" in your bridge.

So how do we do this?

First, some requirements:

• A physical interface which supports jumbo frames.
  The Linux bridge code will only allow the bridge to change to the smallest MTU of all its interfaces.
• A bridge with the larger required MTU
  Technically a jumbo frame covers MTU from 1501 to 9000, but almost everyone just wants to set 9000
  
• virtio-net or Intel e1000 network interface inside the guest
  The real-life Realtek 8139 does not support an MTU of 9000, therefore there is no driver support for large MTU in the emulated version either.

Our first step is to make all tap interfaces on the host have an MTU of 9000. We do this by adding a udev rule to the top of the network device creation as follows:

/etc/udev/rules.d/70-persistent-net.rules (on host)

SUBSYSTEM=="net", ACTION=="add", KERNEL=="vnet*", ATTR{mtu}="9000"

Then set your physical interface MTU to 9000:

/etc/sysconfig/network-scripts/ifcfg-eth0 (on host):

DEVICE=eth0
HWADDR=[physical MAC address]
TYPE=Ethernet
ONBOOT=yes
BRIDGE=br0
MTU=9000

And do the same inside the bridge:

/etc/sysconfig/network-scripts/ifcfg-br0 (on host)

DEVICE=br0
TYPE=Bridge
ONBOOT=yes
DELAY=0
MTU=9000

Now inside your guest, set your virtio or e1000 interface to also have the larger MTU:

/etc/sysconfig/network-scripts/ifcfg-eth0 (in guest)

DEVICE=eth0
HWADDR=[guest MAC address]
TYPE=Ethernet
ONBOOT=yes
MTU=9000

You'll need to perform a "service network restart" on the host, or at least bring the bridge and eth interfaces down and up. Start your guest and confirm all interfaces have been made with the correct MTU:

# ip link
2: eth0: mtu 9000 qdisc pfifo_fast state UP qlen 1000
    link/ether 01:23:45:67:89:0A brd ff:ff:ff:ff:ff:ff
3: br0: mtu 9000 qdisc noqueue state UNKNOWN
    link/ether 01:23:45:67:89:0A brd ff:ff:ff:ff:ff:ff
4: vnet0: mtu 9000 qdisc pfifo_fast state UNKNOWN qlen 500
    link/ether fe:54:00:01:23:45 brd ff:ff:ff:ff:ff:ff

This should be enough to send a "Do Not Fragment" 9000 byte frame over the network. Let's test with a ping:

# ping -c 4 -s 8972 -M do 172.16.0.2
PING 172.16.0.2 (172.16.0.2) 8972(9000) bytes of data.
8980 bytes from 172.16.0.2: icmp_seq=1 ttl=64 time=1.27 ms
8980 bytes from 172.16.0.2: icmp_seq=2 ttl=64 time=0.284 ms
8980 bytes from 172.16.0.2: icmp_seq=3 ttl=64 time=0.202 ms
8980 bytes from 172.16.0.2: icmp_seq=4 ttl=64 time=0.260 ms

Success!

how to persist ethtool settings across reboot

You can use the ethtool command to read and change information about your network interfaces.

For example, ethtool -g ethX reads the size of the ring buffer on the NIC, and ethtool -G ethx rx A tx B changes it. Use man ethtool to discover more settings.

But these options don't persist across reboot, so how do you make sure your settings are kept permanent?

You can enter the ethtool commands in /etc/rc.local (or your distribution's equivalent) where commands are run after the current runlevel completes, but this isn't ideal. Network services may have started during the runlevel and ethtool commands tend to interrupt network traffic. It would be more preferable to have the commands applied as the interface is brought up.

The network service in CentOS has the ability to do this. The script /etc/sysconfig/network-scripts/ifup-post checks for the existence of /sbin/ifup-local, and if it exists, runs it with the interface name as a parameter (eg: /sbin/ifup-local eth0)

We can create this file with touch /sbin/ifup-local make it executable with chmod +x /sbin/ifup-local set its SELinux context with chcon --reference /sbin/ifup /sbin/ifup-local and then open it in an editor.

A simple script to apply the same settings to all interfaces would be something like:

#!/bin/bash
if [ -n "$1" ]; then
    /sbin/ethtool -G $1 rx 4096 tx 4096
    /sbin/ethtool -K $1 tso on gso on
fi

Keep in mind this will attempt to apply settings to ALL interfaces, even the loopback.

If we have different interfaces we want to apply different settings to, or want to skip the loopback, we can make a case statement:

#!/bin/bash
case "$1" in
 eth0)
  /sbin/ethtool -G $1 rx 16384 tx 16384
  /sbin/ethtool -K $1 gso on gro on
  ;;
 eth1)
  /sbin/ethtool -G $1 rx 64 tx 64
  /sbin/ethtool -K $1 tso on gso on
  /sbin/ip link set $1 txqueuelen 0
  ;;
esac
exit 0

Now ethtool settings are applied to interfaces as they start, all potential interruptions to network communication are done as the interface is brought up, and your server can continue to boot with full network capabilities.