I have a system with an onboard Promise FastTrack SATA RAID controller. This isn't a "real" RAID controller, but one which relies on the OS to perform some of the RAID work. This is commonly called "FakeRAID" and is not desirable.
Despite setting the controller into AHCI Mode in the BIOS, I still get a disk called /dev/mapper/pdc_abcdef or similar and cannot use the disk /dev/sdb directly. I wanted the FakeRAID gone.
The FakeRAID is managed by Device-Mapper RAID called "dmraid".
An old CentOS 5 post
suggests removing the dmraid package might work, but this wanted to
remove anaconda and many other things, so didn't seem a good option.
There is a kernel boot parameter to disable Device-Mapper RAID so I applied this.
In /etc/default/grub add nodmraid to GRUB_CMDLINE_LINUX_DEFAULT:
GRUB_CMDLINE_LINUX_DEFAULT="quiet rhgb nodmraid"
Then re-create the GRUB2 configuration file:
grub2-mkconfig -o /boot/grub2/grub.cfg
Reboot, and no more FakeRAID drive. Success.
Showing posts with label rhel. Show all posts
Showing posts with label rhel. Show all posts
Monday, December 7, 2015
Sunday, June 1, 2014
kernel panic scsi_wait_scan on elrepo kernel-ml
Browsing around ELRepo, you can see they have the elrepo-kernel repository which contains the long-life kernel-lt (currently 3.10) and mainline kernel-ml (currently 3.14) packages.
I'd always been curious about these, so I spun up a test VM and did a CentOS 6 Basic Server install, then added ELRepo and installed them.
kernel-lt works fine first time.
However, kernel-ml will not boot, and simply repeats:
I'd always been curious about these, so I spun up a test VM and did a CentOS 6 Basic Server install, then added ELRepo and installed them.
kernel-lt works fine first time.
However, kernel-ml will not boot, and simply repeats:
FATAL: Module scsi_wait_scan not found.
Thursday, August 29, 2013
finding the netdev_priv struct
In addition to the net_device struct within the kernel, network drivers also have their own private or device-specific struct which stores stats unique to the individual hardware.
The name of the struct varies for each device type, however the location remains the same, right after net_device:
linux-2.6.32-358.14.1.el6.x86_64/include/linux/netdevice.h
We'll need the ability to read kernel memory using crash, either run on a live system, or in a vmcore captured with kdump:
We need to know how big net_device is:
The name of the struct varies for each device type, however the location remains the same, right after net_device:
linux-2.6.32-358.14.1.el6.x86_64/include/linux/netdevice.h
/**
* netdev_priv - access network device private data
* @dev: network device
*
* Get network device private data
*/
static inline void *netdev_priv(const struct net_device *dev)
{
return (char *)dev + ALIGN(sizeof(struct net_device), NETDEV_ALIGN);
}
A search in cscope for netdev_priv will show you many functions within drivers which update the priv structure via pointers, for example this one in e1000_main.c:struct e1000_adapter *adapter = netdev_priv(netdev);So, given that we know what the device-specific struct is called, and where the device-specific private struct is, how do you find it and read it?
We'll need the ability to read kernel memory using crash, either run on a live system, or in a vmcore captured with kdump:
crash /usr/lib/debug/lib/modules/2.6.32-358.14.1.el6.x86_64/vmlinux /var/crash/2013-08-26/vmcoreWe'll also need the debugging symbols for the driver in question:
crash> mod -s e1000 /usr/lib/debug/lib/modules/2.6.32-358.14.1.el6.x86_64/kernel/drivers/net/e1000/e1000.ko.debug
MODULE NAME SIZE OBJECT FILE
ffffffffa01550e0 e1000 170678 /usr/lib/debug/lib/modules/2.6.32-358.14.1.el6.x86_64/kernel/drivers/net/e1000/e1000.ko.debug
The net command will show you the network devices in the system:crash> net NET_DEVICE NAME IP ADDRESS(ES) ffff88007e76b820 lo 127.0.0.1 ffff8800372c0020 eth0 192.168.1.126We can then cast net_device against this to see the in-kernel device struct:
crash> net_device 0xffff8800372c0020
struct net_device {
name = "eth0\000\000\060:03.0\000\000\000",
...
But we want to get after this struct and cast it against the struct in the driver.We need to know how big net_device is:
crash> struct -o net_device
struct net_device {
...
}
SIZE: 1728
We then find the net address plus the size of net_device:crash> px 0xffff8800372c0020+1728 $1 = 0xffff8800372c06e0We can now cast the device-specific private struct against this new address:
crash> e1000_adapter 0xffff8800372c06e0
struct e1000_adapter {
vlgrp = 0x0,
mng_vlan_id = 65535,
bd_number = 0,
rx_buffer_len = 1522,
...
and we're done!
Wednesday, January 16, 2013
how to network jumbo frames to a kvm guest
Getting a jumbo frame to a KVM guest is not something which works out of the box, but can be configured to work quite easily.
One way get jumbo frames to a KVM guest is to directly pass the guest a physical NIC through PCI Passthrough, or a Virtual Function through SR-IOV.
Another way is to use the traditional libvirt bridge. Create a bridge on the host, and bridge the guest's NICs into this. The reason this doesn't work by default is all the interfaces are created with the default Ethernet MTU of 1500 bytes.
You can set MTU of the physical NIC, the bridge, and the guest interface with the network-scripts, but you cannot automatically set the MTU of the tap interface with the network-scripts. The tap interface is how the virtual network interface connects to the "real world". You'll see these tap devices with names like "vnet0" in your bridge.
So how do we do this?
First, some requirements:
/etc/udev/rules.d/70-persistent-net.rules (on host)
/etc/sysconfig/network-scripts/ifcfg-eth0 (on host):
/etc/sysconfig/network-scripts/ifcfg-br0 (on host)
/etc/sysconfig/network-scripts/ifcfg-eth0 (in guest)
One way get jumbo frames to a KVM guest is to directly pass the guest a physical NIC through PCI Passthrough, or a Virtual Function through SR-IOV.
Another way is to use the traditional libvirt bridge. Create a bridge on the host, and bridge the guest's NICs into this. The reason this doesn't work by default is all the interfaces are created with the default Ethernet MTU of 1500 bytes.
You can set MTU of the physical NIC, the bridge, and the guest interface with the network-scripts, but you cannot automatically set the MTU of the tap interface with the network-scripts. The tap interface is how the virtual network interface connects to the "real world". You'll see these tap devices with names like "vnet0" in your bridge.
So how do we do this?
First, some requirements:
- A physical interface which supports jumbo frames.
The Linux bridge code will only allow the bridge to change to the smallest MTU of all its interfaces. - A bridge with the larger required MTU
Technically a jumbo frame covers MTU from 1501 to 9000, but almost everyone just wants to set 9000 - virtio-net or Intel e1000 network interface inside the guest
The real-life Realtek 8139 does not support an MTU of 9000, therefore there is no driver support for large MTU in the emulated version either.
/etc/udev/rules.d/70-persistent-net.rules (on host)
SUBSYSTEM=="net", ACTION=="add", KERNEL=="vnet*", ATTR{mtu}="9000"
Then set your physical interface MTU to 9000:/etc/sysconfig/network-scripts/ifcfg-eth0 (on host):
DEVICE=eth0 HWADDR=[physical MAC address] TYPE=Ethernet ONBOOT=yes BRIDGE=br0 MTU=9000And do the same inside the bridge:
/etc/sysconfig/network-scripts/ifcfg-br0 (on host)
DEVICE=br0 TYPE=Bridge ONBOOT=yes DELAY=0 MTU=9000Now inside your guest, set your virtio or e1000 interface to also have the larger MTU:
/etc/sysconfig/network-scripts/ifcfg-eth0 (in guest)
DEVICE=eth0 HWADDR=[guest MAC address] TYPE=Ethernet ONBOOT=yes MTU=9000You'll need to perform a "service network restart" on the host, or at least bring the bridge and eth interfaces down and up. Start your guest and confirm all interfaces have been made with the correct MTU:
# ip link
2: eth0: mtu 9000 qdisc pfifo_fast state UP qlen 1000
link/ether 01:23:45:67:89:0A brd ff:ff:ff:ff:ff:ff
3: br0: mtu 9000 qdisc noqueue state UNKNOWN
link/ether 01:23:45:67:89:0A brd ff:ff:ff:ff:ff:ff
4: vnet0: mtu 9000 qdisc pfifo_fast state UNKNOWN qlen 500
link/ether fe:54:00:01:23:45 brd ff:ff:ff:ff:ff:ff
This should be enough to send a "Do Not Fragment" 9000 byte frame over the network. Let's test with a ping:# ping -c 4 -s 8972 -M do 172.16.0.2 PING 172.16.0.2 (172.16.0.2) 8972(9000) bytes of data. 8980 bytes from 172.16.0.2: icmp_seq=1 ttl=64 time=1.27 ms 8980 bytes from 172.16.0.2: icmp_seq=2 ttl=64 time=0.284 ms 8980 bytes from 172.16.0.2: icmp_seq=3 ttl=64 time=0.202 ms 8980 bytes from 172.16.0.2: icmp_seq=4 ttl=64 time=0.260 msSuccess!
Sunday, August 5, 2012
how to persist ethtool settings across reboot
You can use the ethtool command to read and change information about your network interfaces.
For example, ethtool -g ethX reads the size of the ring buffer on the NIC, and ethtool -G ethx rx A tx B changes it. Use man ethtool to discover more settings.
But these options don't persist across reboot, so how do you make sure your settings are kept permanent?
You can enter the ethtool commands in /etc/rc.local (or your distribution's equivalent) where commands are run after the current runlevel completes, but this isn't ideal. Network services may have started during the runlevel and ethtool commands tend to interrupt network traffic. It would be more preferable to have the commands applied as the interface is brought up.
The network service in CentOS has the ability to do this. The script /etc/sysconfig/network-scripts/ifup-post checks for the existence of /sbin/ifup-local, and if it exists, runs it with the interface name as a parameter (eg: /sbin/ifup-local eth0)
We can create this file with touch /sbin/ifup-local make it executable with chmod +x /sbin/ifup-local set its SELinux context with chcon --reference /sbin/ifup /sbin/ifup-local and then open it in an editor.
A simple script to apply the same settings to all interfaces would be something like:
If we have different interfaces we want to apply different settings to, or want to skip the loopback, we can make a case statement:
For example, ethtool -g ethX reads the size of the ring buffer on the NIC, and ethtool -G ethx rx A tx B changes it. Use man ethtool to discover more settings.
But these options don't persist across reboot, so how do you make sure your settings are kept permanent?
You can enter the ethtool commands in /etc/rc.local (or your distribution's equivalent) where commands are run after the current runlevel completes, but this isn't ideal. Network services may have started during the runlevel and ethtool commands tend to interrupt network traffic. It would be more preferable to have the commands applied as the interface is brought up.
The network service in CentOS has the ability to do this. The script /etc/sysconfig/network-scripts/ifup-post checks for the existence of /sbin/ifup-local, and if it exists, runs it with the interface name as a parameter (eg: /sbin/ifup-local eth0)
We can create this file with touch /sbin/ifup-local make it executable with chmod +x /sbin/ifup-local set its SELinux context with chcon --reference /sbin/ifup /sbin/ifup-local and then open it in an editor.
A simple script to apply the same settings to all interfaces would be something like:
#!/bin/bash
if [ -n "$1" ]; then
/sbin/ethtool -G $1 rx 4096 tx 4096
/sbin/ethtool -K $1 tso on gso on
fi
Keep in mind this will attempt to apply settings to ALL interfaces, even the loopback.If we have different interfaces we want to apply different settings to, or want to skip the loopback, we can make a case statement:
#!/bin/bash case "$1" in eth0) /sbin/ethtool -G $1 rx 16384 tx 16384 /sbin/ethtool -K $1 gso on gro on ;; eth1) /sbin/ethtool -G $1 rx 64 tx 64 /sbin/ethtool -K $1 tso on gso on /sbin/ip link set $1 txqueuelen 0 ;; esac exit 0Now ethtool settings are applied to interfaces as they start, all potential interruptions to network communication are done as the interface is brought up, and your server can continue to boot with full network capabilities.
Saturday, August 4, 2012
kvm virtual machine won't start
Recently I brought my KVM host down for an upgrade. I shut down the guests within their OSes, confirmed they were powered off with a virsh list --all, and everything went well with the upgrade.
However, upon returning the guests to service, one of them would not start up. Typing virsh start I got the helpful error message
Error restoring domain: cannot send monitor command
Connection reset by peer
Why was this one VM broken but the others were fine?
I played around with virsh autostart and virsh autostart --disable but that had no effect, nor did a reboot of the hypervisor.
After some searching around, it turns out libvirt has the capability to keep "managed save states" of guests, kinda like a sleep mode or snapshot, to save you fully powering a guest OS off.
For some reason, a managed save for this one guest had been created, perhaps it had not shut down properly, or perhaps there's an errorr in libvirt. I could view the saved state with
# virsh list --all --managed-save
Id Name State
----------------------------------
- guest1 shut off
- guest2 saved
Now a virsh managedsave-remove guest2 returned it to the "shut off" state, and I could start it properly with virsh start as per usual.
However, upon returning the guests to service, one of them would not start up. Typing virsh start I got the helpful error message
Error restoring domain: cannot send monitor command
Connection reset by peer
Why was this one VM broken but the others were fine?
I played around with virsh autostart and virsh autostart --disable but that had no effect, nor did a reboot of the hypervisor.
After some searching around, it turns out libvirt has the capability to keep "managed save states" of guests, kinda like a sleep mode or snapshot, to save you fully powering a guest OS off.
For some reason, a managed save for this one guest had been created, perhaps it had not shut down properly, or perhaps there's an errorr in libvirt. I could view the saved state with
# virsh list --all --managed-save
Id Name State
----------------------------------
- guest1 shut off
- guest2 saved
Now a virsh managedsave-remove guest2 returned it to the "shut off" state, and I could start it properly with virsh start as per usual.
Saturday, April 21, 2012
starting rtorrent on system startup
rTorrent is a curses-based bittorrent client for Linux.
It's very light on system resources but has great features such as monitoring of a directory for .torrent files, auto-move based on status (incoming, seeding, complete, etc), categories, magnet link handling and RSS capabilities.
I have a headless system where I seed some (legal) torrents and wanted rTorrent to start up when the system did. There are some examples of init scripts on the rTorrent site but I didn't like how these ran, specifically the su constantly needing a password and the odd way it started GNU Screen.
Download
Installation
It's very light on system resources but has great features such as monitoring of a directory for .torrent files, auto-move based on status (incoming, seeding, complete, etc), categories, magnet link handling and RSS capabilities.
I have a headless system where I seed some (legal) torrents and wanted rTorrent to start up when the system did. There are some examples of init scripts on the rTorrent site but I didn't like how these ran, specifically the su constantly needing a password and the odd way it started GNU Screen.
Download
Installation
- Put it at /etc/init.d/bittorrent and make it executable
chmod +x /etc/init.d/bittorrent - Add to chkconfig
chkconfig --add bittorrent - Set to start on system startup
chkconfig bittorrent on - If you want non-root users to be able to control it, put this in your visudo, replace "username" with your non-priveledged username
username localhost=NOPASSWD:/etc/init.d/bittorrent* - Setup an alias to the script in the user's ~/.bashrc
alias bittorrent='/etc/init.d/bittorrent'
alias bt='bittorrent'
- CentOS 6 or Fedora 12-14. Probably also works on CentOS 5.
- screen and rtorrent
- A non-priveledged user to run rtorrent, this can either be a new user just to run torrents, or an existing user
- Absolute session path in the .rtorrent.rc file
session = /home/rtorrent/.session ## this is good
session = ~/.session ## this will break
session = .session ## so will this - Paths for other actions such as monitoring directories for .torrent files and auto-moving complete downloads don't have to be absolute.
- Control the daemon withbittorrent start, bittorent stop, bittorrent restart
- Get information about the daemon with
bittorrent status, bittorrent info - Connect to the screen session with
bittorrent connect
(press ctrl+a then d to disconnect)
- Released under GNU GPL v3 - http://www.gnu.org/licenses/
starting minecraft server on system startup
This is an initscript to run a Minecraft or CraftBukkit server on CentOS, Fedora, and Ubuntu.
Features
Distros using systemd (Fedora 15+, Arch Linux, etc) will not work.
Get the script, view Requirements, Installation, Backups, Multiple Instances, and Usage
License
Released under GNU GPL v3 - http://www.gnu.org/licenses/
Credits
Thanks to these people whose work I have used in the making of this
Features
- Start, stop, restart CraftBukkit as a system service
- Automatic (via cron) and manual logfile rotation
- Automatic (via cron) and manual backups
- Backup compression and rotation (keeps 7 days worth of backups)
- Check latest Recommended Build and update to it if required
- Information display including Java path, current memory usage, current TCP connections
- Able to run multiple separate instances of the server at once
- CentOS 6, CentOS 5, Fedora 14
(probably works on Fedora Core 6 and later, untested) - Ubuntu Server 12.04 LTS
Distros using systemd (Fedora 15+, Arch Linux, etc) will not work.
Get the script, view Requirements, Installation, Backups, Multiple Instances, and Usage
License
Released under GNU GPL v3 - http://www.gnu.org/licenses/
Credits
Thanks to these people whose work I have used in the making of this
The State of the Linux
It's been a while since I've posted but I'm still around. I started this blog as a sharing of tips and fixes I which used and couldn't find anywhere else. I've easily found most things I've wanted to do in the last few years so I haven't had the need to make much noise.
Today I'm here to write about the state of desktop Linux as I see it, compared to when this blog started in 2008 and I was obviously using Ubuntu.
These days I am a Fedora user. This was mostly brought about by starting to use RHEL at work and the need to become familiar with regular use of the RPM-based distro. I've also changed all my own servers over to CentOS.
Fedora's okay. Some people may deny this, but I think it's obviously a dumping ground for features Red Hat want to try, or to have the Fedora community crowdsource most of the initial testing and bugfixing before doing their own QA for inclusion in RHEL.
Fedora's near-bleeding-edge package cycles, still with a focus on stability, is both its strength and its weakness. It's great having the kernel or Wine updated in-distro within a week of new major upstream releases. It's frustrating having to deal with "latest and greatest" features I dislike such as GNOME 3 and systemd. I generally find Yum/rpm to be greatly messy and inferior compared to apt.
GNOME 3 is probably the largest thing to happen to desktop Linux in the last few years. It was met with such a hugely negative response and such polarising opinions that sites are still running polls about it. I think the developers made a mistake, sticking their head in the sand and telling users that the forced paradigm shift to Gnome Shell was what they wanted, despite the fact many people said they didn't.
Many Linux users appear to be seeking alternatives to Gnome Shell and all the competing Desktop Environments (KDE, XFCE, LXDE) have probably seen an increase in user base. Linux Mint listened to their users and pursued the MATE fork of GNOME 2, then created the Cinnamon Desktop. This fulfilling of majority demand has pushed them to the top of Distrowatch and kept them there. Kudos to the Mint Team for putting their money where their mouth is.
Ubuntu chose Unity over Gnome Shell. I've tried Unity several times and have been disgusted by it. It's so un-useable that I consider it just plain broken. I think it's pushing Ubuntu into irrelevance and their declining rating on Distrowatch tends to agree.
Mint used to be based on Ubuntu and have now started offering a re-base on Debian. CrunchBang also used to be based on Ubuntu and has switched to Debian altogether. Ubuntu's own variants Kubuntu and Xubuntu have fallen from sponsored releases to just "officially recognised" community projects.
I considered Ubuntu to be the king of Linux distros for a long time, but its reign is now well and truly over.
One interesting distro I saw on a magazine is Fuduntu. The name is ridiculous but the product itself is worth a look. It's a fork of Fedora 14 which has gone rolling-release. It still runs GTK2 and SysV Init but incorporates the latest 3.x kernel series. It has a smaller package set than Fedora and runs a Mac-like desktop environment based around a top-of-screen Gnome Panel and Avant Window Navigator.
My own desktop these days is Openbox with tint2 panel, a minimalist throwback from my days as an Arch Linux user.
Arch itself is a good distro with a great community, full of positive helpful people, few (if any) jerks, and little politics. However, I found myself spending around a quarter of my computer time just maintaining my system, rather than using the computer to do other things. Arch is something all Linux enthusiasts should do once for a period of time, the amount of tinkering and low-level micro-management required is educational and improves your Linux skills, but I feel it also is very much "using Linux for the sake of using Linux".
There seems to be a big paradigm shift happening all over desktops these days. Unity and GNOME 3 led for Linux and seem to have failed, Microsoft have shown their Metro interface which must be a bad joke. It's as if the development world has suddenly and unanimously decided the "menu and taskbar" interface which began with Windows 95 is no longer good enough, but nobody's come up with a suitable replacement, except perhaps Cinnamon.
Which desktop eventually wins out is still anyone's guess. Mint has shown that it's not necessarily the big corporate-backed developers who set the trend. Maybe we will never see one true new standard environment and modular personalised DEs will be the order of the day.
Today I'm here to write about the state of desktop Linux as I see it, compared to when this blog started in 2008 and I was obviously using Ubuntu.
These days I am a Fedora user. This was mostly brought about by starting to use RHEL at work and the need to become familiar with regular use of the RPM-based distro. I've also changed all my own servers over to CentOS.
Fedora's okay. Some people may deny this, but I think it's obviously a dumping ground for features Red Hat want to try, or to have the Fedora community crowdsource most of the initial testing and bugfixing before doing their own QA for inclusion in RHEL.
Fedora's near-bleeding-edge package cycles, still with a focus on stability, is both its strength and its weakness. It's great having the kernel or Wine updated in-distro within a week of new major upstream releases. It's frustrating having to deal with "latest and greatest" features I dislike such as GNOME 3 and systemd. I generally find Yum/rpm to be greatly messy and inferior compared to apt.
GNOME 3 is probably the largest thing to happen to desktop Linux in the last few years. It was met with such a hugely negative response and such polarising opinions that sites are still running polls about it. I think the developers made a mistake, sticking their head in the sand and telling users that the forced paradigm shift to Gnome Shell was what they wanted, despite the fact many people said they didn't.
Many Linux users appear to be seeking alternatives to Gnome Shell and all the competing Desktop Environments (KDE, XFCE, LXDE) have probably seen an increase in user base. Linux Mint listened to their users and pursued the MATE fork of GNOME 2, then created the Cinnamon Desktop. This fulfilling of majority demand has pushed them to the top of Distrowatch and kept them there. Kudos to the Mint Team for putting their money where their mouth is.
Ubuntu chose Unity over Gnome Shell. I've tried Unity several times and have been disgusted by it. It's so un-useable that I consider it just plain broken. I think it's pushing Ubuntu into irrelevance and their declining rating on Distrowatch tends to agree.
Mint used to be based on Ubuntu and have now started offering a re-base on Debian. CrunchBang also used to be based on Ubuntu and has switched to Debian altogether. Ubuntu's own variants Kubuntu and Xubuntu have fallen from sponsored releases to just "officially recognised" community projects.
I considered Ubuntu to be the king of Linux distros for a long time, but its reign is now well and truly over.
One interesting distro I saw on a magazine is Fuduntu. The name is ridiculous but the product itself is worth a look. It's a fork of Fedora 14 which has gone rolling-release. It still runs GTK2 and SysV Init but incorporates the latest 3.x kernel series. It has a smaller package set than Fedora and runs a Mac-like desktop environment based around a top-of-screen Gnome Panel and Avant Window Navigator.
My own desktop these days is Openbox with tint2 panel, a minimalist throwback from my days as an Arch Linux user.
Arch itself is a good distro with a great community, full of positive helpful people, few (if any) jerks, and little politics. However, I found myself spending around a quarter of my computer time just maintaining my system, rather than using the computer to do other things. Arch is something all Linux enthusiasts should do once for a period of time, the amount of tinkering and low-level micro-management required is educational and improves your Linux skills, but I feel it also is very much "using Linux for the sake of using Linux".
There seems to be a big paradigm shift happening all over desktops these days. Unity and GNOME 3 led for Linux and seem to have failed, Microsoft have shown their Metro interface which must be a bad joke. It's as if the development world has suddenly and unanimously decided the "menu and taskbar" interface which began with Windows 95 is no longer good enough, but nobody's come up with a suitable replacement, except perhaps Cinnamon.
Which desktop eventually wins out is still anyone's guess. Mint has shown that it's not necessarily the big corporate-backed developers who set the trend. Maybe we will never see one true new standard environment and modular personalised DEs will be the order of the day.
Subscribe to:
Posts (Atom)