Pages - Menu

Sunday, August 5, 2012

how to persist ethtool settings across reboot

You can use the ethtool command to read and change information about your network interfaces.

For example, ethtool -g ethX reads the size of the ring buffer on the NIC, and ethtool -G ethx rx A tx B changes it. Use man ethtool to discover more settings.

But these options don't persist across reboot, so how do you make sure your settings are kept permanent?

You can enter the ethtool commands in /etc/rc.local (or your distribution's equivalent) where commands are run after the current runlevel completes, but this isn't ideal. Network services may have started during the runlevel and ethtool commands tend to interrupt network traffic. It would be more preferable to have the commands applied as the interface is brought up.

The network service in CentOS has the ability to do this. The script /etc/sysconfig/network-scripts/ifup-post checks for the existence of /sbin/ifup-local, and if it exists, runs it with the interface name as a parameter (eg: /sbin/ifup-local eth0)

We can create this file with touch /sbin/ifup-local make it executable with chmod +x /sbin/ifup-local set its SELinux context with chcon --reference /sbin/ifup /sbin/ifup-local and then open it in an editor.

A simple script to apply the same settings to all interfaces would be something like:
#!/bin/bash
if [ -n "$1" ]; then
    /sbin/ethtool -G $1 rx 4096 tx 4096
    /sbin/ethtool -K $1 tso on gso on
fi
Keep in mind this will attempt to apply settings to ALL interfaces, even the loopback.

If we have different interfaces we want to apply different settings to, or want to skip the loopback, we can make a case statement:
#!/bin/bash
case "$1" in
 eth0)
  /sbin/ethtool -G $1 rx 16384 tx 16384
  /sbin/ethtool -K $1 gso on gro on
  ;;
 eth1)
  /sbin/ethtool -G $1 rx 64 tx 64
  /sbin/ethtool -K $1 tso on gso on
  /sbin/ip link set $1 txqueuelen 0
  ;;
esac
exit 0
Now ethtool settings are applied to interfaces as they start, all potential interruptions to network communication are done as the interface is brought up, and your server can continue to boot with full network capabilities.

12 comments:

Anonymous said...

Nice! Just what I was looking for. I still can't comprehend why the defaults are lower than the maximum - there's no advantage.

Jamie said...

Actually large buffers can be detrimental, have a read about bufferbloat:

http://en.wikipedia.org/wiki/Bufferbloat
http://gettys.wordpress.com/bufferbloat-faq/

Ideally you want to set your buffers so they are as small as possible when the link is contended, without dropping packets.

Of course that depends on the system, traffic, load, interrupts, so many other things.

Anonymous said...

Hm, interesting read, I'll tinker with the settings some more and do some test to see what's best in my case.

Anonymous said...

Nice find. However your use of exec in this context is wrong. The first call to exec will replace the running script and therefore not run any subsequent ethtool commands within the matching case statement.

Jamie said...

Ah, thanks! I have corrected.

qk said...

this doesn't work for my F16 - script /sbin/ifup-local is not called

Jamie said...

Do some debugging on the script. Try "echo -e $(date)\n $1 >> /root/debug.txt" to see what's being called when the system starts up. You can redirect stderr and stdout to a file with "&> /root/file.txt" at the end of your ethool command to see if they run and if they produce any errors.

I've got a system where I have a virt bridge configured (br0) with the network interface (eth0) in it. For that system I need to specify "br0" as the case but "eth0" as the interface name to change in the ip/ethtool lines.

Joe Rice said...

You can do this on CentOS via the ETHTOOL_OPTS in the nic config files in
/etc/sysconfig/network-scripts (eg. ifcfg-eth0).

For example:

ETHTOOL_OPTS="-G rx 4096"

It isn't documented very well, but there is a RedHat bug report describing a fix to allow multiple ethtool changes. The fix was available in initscripts-9.03.24-1.el6.

https://bugzilla.redhat.com/show_bug.cgi?id=692410

Jamie said...

That's true, but anything earlier than that bug doesn't have the ability to use ETHTOOL_OPTS. It can also be problematic to do multiple things at once. If you want to do something besides just run ethool commands, like run a command which isn't ethtool, then you end up using ifup-local anyway.

mlan said...

Thank you to the author for this post and to the contributors for this thread. I have been scouring rhel/centos docs trying to find out why ETHTOOL_OPTS would not accept a -G flag or similar arguments. The bug report that was posted was very enlightening. I am also dealing with a bridged virtual interface on a vendor's el5 build. Although I was not able to get the script posted on this page to work for me (I'm not strong in bash), a simple if/then comparing the token to the bridge interface was enough to call ethtool and set the buffer sizes on the physical ethernet interfaces. Thanks again, you have saved me a big headache until I can get a new release from the vendor on el6.

Jonathon Reinhart said...

Thanks a lot for this post! I have an old RHEL 5 VM that I needed this for.

In my case, the RHEL 5 VM is running on ESXi. Both e1000 and vmxnet NICs/drivers are defaulting to TSO=on, but sending the frames out un-segmented! Adding a script to turn TSO off worked like a charm.

Anonymous said...

Apparently ETHTOOL_OPTS requires the nic to specified after the -G i.e. ETHTOOL_OPTS="-G eth0 rx 4096"