Pages - Menu

Thursday, January 16, 2020

Disable Bluetooth on by default in Ubuntu MATE 18.04

I'd like to have my laptop power up, have the Bluetooth system tray app (blueman) start, but not have the actual Bluetooth powered on by default.

This was an easy change in the terminal:

gsettings set org.blueman.plugins.powermanager auto-power-on false

Thanks to Arch Wiki for this: https://wiki.archlinux.org/index.php/Blueman

Monday, May 7, 2018

Wifi latency using RALink RT3090 (rt2800pci)

After updating to Ubuntu 18.04 Bionic, I found my RALink RT3090 wireless card using the rt2800pci driver introduced latency. This was most noticeable as a delay while typing over SSH.

This is because power management is enabled on the card, which puts it into sleep mode and it takes some time to wake up again.

You can see the power management state of the card with:

sudo iwconfig

Amongst other things, this will show:

Power Management:on

You can test whether disabling power management solves the problem with:

sudo iwconfig DEV power off

Where DEV is the device name (eg: wlan0, wlp3s0)

If this solves the problem, then apply the setting on boot. Edit the file /etc/NetworkManager/conf.d/default-wifi-powersave-on.conf and change:

[connection]
wifi.powersave = 3


So it reads:

wifi.powersave = 2

Thursday, May 4, 2017

Commandline auto-login in Ubuntu 16.04

This is based on how RetroPie does its autologin and autostart which you can find in scriptmodules/supplementary/autostart.sh in the RetroPie-Setup source.

This works in Ubuntu and Raspbian, it probably works in any systemd distro.

mkdir -p /etc/systemd/system/getty@tty1.service.d/

Create a file at /etc/systemd/system/getty@tty1.service.d/autologin.conf with contents:

[Service]
ExecStart=
ExecStart=-/sbin/agetty --autologin USERNAME --noclear %I \$TERM


Replace USERNAME with the actual username to login.

You can now reboot and test this.

If you want to start something on startup, then create a file /etc/profile.d/10-runthing.sh with contents like:

if [ "$(tty)" = "/dev/tty1" ]; then
    THING
fi


Replace THING with the command to run. You could run tmux so you never forget. You could compile DOSBox so it runs on the framebuffer and you appear to have a DOS computer, which is why I wanted to do this.

Friday, March 24, 2017

3D Modelling Software on Ubuntu 16.04 (OpenSCAD, FreeCAD, gCAD3D, Blender)

In addition to the previous post with 3D printer control software, you will likely want something to make models with!

OpenSCAD


This modeller uses a C-like programming language to create models. For example, to create a 20mm cube you type cube([20,20,20]);

Install with:

sudo add-apt-repository ppa:openscad/releases
sudo apt update
sudo apt install openscad


Run with openscad

I also find the scad-utils repository to be useful, the morphology plugin has a fillet tool: https://github.com/openscad/scad-utils

OpenSCAD website: http://www.openscad.org/
OpenSCAD documentation: https://en.wikibooks.org/wiki/OpenSCAD_User_Manual/The_OpenSCAD_Language

FreeCAD


A graphical modeller. This has many more features than OpenSCAD though I don't like the interface as much. A good workflow might be to make shapes in OpenSCAD and then import into FreeCAD to polish.

Install with:

sudo add-apt-repository ppa:freecad-maintainers/freecad-stable
sudo apt update
sudo apt install freecad

Run with freecad

FreeCAD website: http://www.freecadweb.org/
FreeCAD documentation: https://www.freecadweb.org/wiki/Main_Page

gCAD3D


This looks like a simpler version of FreeCAD

A deb package download is provided on the website for 32-bit and 64-bit.

Once downloaded, install with:

sudo dpkg -i gCAD3D*.deb

Run with gcad3d

gCAD3D website: http://www.gcad3d.org/
gCAD3D documentation: http://www.gcad3d.org/doc/html/index_en.htm

Blender


I have not played with this much yet, but it's very popular

Install with:

sudo add-apt-repository ppa:thomas-schiex/blender
sudo apt update
sudo apt install blender

Run with blender

Blender website: https://www.blender.org/
Blender support and docs: https://www.blender.org/support/

Saturday, March 4, 2017

3D Printing Software on Ubuntu 16.04 (Cura, Slic3r, Pronterface, Repetier Host)

I recently got a cheap 3D printer and wanted to get started printing with Ubuntu. Some of the software is not so obvious to install, so here's how I got these popular applications going:

Cura


Cura is available in the https://launchpad.net/~thopiekar/+archive/ubuntu/cura PPA. To install, just:

sudo add-apt-repository ppa:thopiekar/cura
sudo apt update
sudo apt install cura

Launch with the cura command.

Cura website: https://ultimaker.com/en/products/cura-software

Slic3r (Slicer)


The version in repos is the latest (1.2.9) at the time of writing, however the package is missing some dependencies. This installs all that's required:

sudo apt install slic3r wx-common libopengl-perl libwx-glcanvas-perl

Launch the GUI with the slic3r --gui command

Slic3r website: http://slic3r.org/

Pronterface (aka Printrun)


This requires python-wxgtk2.8 which is removed in Ubuntu Xenial 16.04, and the replacement python-wxgtk3.0 does not work well with Printrun.

There are some instructions around the internet suggesting to install the Wily 15.10 version, but that led to messy dependencies for me.

Luckily, WebUpd8 package python-wxgtk2.8 for 16.04 in their PPA, thanks!

First, install the dependencies Printrun requires with:

sudo add-apt-repository ppa:nilarimogard/webupd8
sudo apt update
sudo apt install python-serial python-wxgtk2.8 python-pyglet python-numpy cython python-libxml2 python-gobject python-dbus python-psutil python-cairosvg git


Now change to the directory where you want the application to be and fetch the software with:

git clone https://github.com/kliment/Printrun.git

To run, change to the Printrun directory with cd Printrun and run the python pronterface.py command.

Pronterface website: http://www.pronterface.com/

Repetier Host


This uses Mono to run the Windows binary, install requirements with:

sudo apt install monodevelop libmono-system-serviceprocess4.0-cil

Download and uncompress:

tar xf repetierHostLinux_1_6_2.tgz
cd RepetierHost

Run with:

mono RepetierHost.exe

Repetier Host website: https://www.repetier.com/

Saturday, August 13, 2016

DOSBox Full Screen with proper Scaling and Aspect Ratio

It's quite confusing to get DOSBox to display games properly on modern hardware, partially because CRT monitors often displayed in non-square pixel modes, and partially because modern LCD displays don't scale well or support all the resolutions that CRTs did.

This post discusses the settings required in dosbox.conf to get DOS games displaying well.

fullresolution

Set this to your screen's native resolution such as fullresolution=1920x1200

output

The default surface cannot scale, so set this to output=overlay which can scale. If you have a decent graphics card you could also try output=openglnb

aspect

Set aspect=true to ensure the game is not stretched

scaler

Simple scaler=none will work, but you can experiment with the other scalers to see which performs well and which you like the look of.

The full config options are described at http://www.dosbox.com/wiki/dosbox.conf

Sunday, August 7, 2016

Another Steam Update breaks Ubuntu Xenial 16.04


As of the recent Steam update, Steam will not launch and the following is displayed:

$ steam
Setting up Steam content in /home/username/.local/share/Steam
Running Steam on ubuntu 16.04 64-bit
STEAM_RUNTIME is enabled automatically
Installing breakpad exception handler for appid(steam)/version(0)
libGL error: unable to load driver: r600_dri.so
libGL error: driver pointer missing
libGL error: failed to load driver: r600
libGL error: unable to load driver: swrast_dri.so
libGL error: failed to load driver: swrast

You can fix it with:

mv .local/share/Steam/ubuntu12_32/steam-runtime/amd64/usr/lib/x86_64-linux-gnu/libstdc++.so.6 .local/share/Steam/ubuntu12_32/steam-runtime/amd64/usr/lib/x86_64-linux-gnu/libstdc++.so.6.old
mv .local/share/Steam/ubuntu12_32/steam-runtime/amd64/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19 .local/share/Steam/ubuntu12_32/steam-runtime/amd64/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19.old
mv .local/share/Steam/ubuntu12_32/steam-runtime/i386/usr/lib/i386-linux-gnu/libstdc++.so.6 .local/share/Steam/ubuntu12_32/steam-runtime/i386/usr/lib/i386-linux-gnu/libstdc++.so.6.old
mv .local/share/Steam/ubuntu12_32/steam-runtime/i386/usr/lib/i386-linux-gnu/libstdc++.so.6.0.19 .local/share/Steam/ubuntu12_32/steam-runtime/i386/usr/lib/i386-linux-gnu/libstdc++.so.6.0.19.old

This is yet another instance of the C++ libraries in the Steam Runtime not being compatible with the free software radeon driver for AMD (ATI) graphics cards:

https://github.com/ValveSoftware/steam-for-linux/issues/3273

I had a previous fix for this when they just packaged the i386 C++ library, but now they're packaging both the i386 and AMD64 C++ library, so my fix needed updating.

If Valve package one more C++ library that'll be 3 separate libraries: Half-Life 3 confirmed.

Monday, December 7, 2015

Disable Promise FastTrack FakeRAID on CentOS 7

I have a system with an onboard Promise FastTrack SATA RAID controller. This isn't a "real" RAID controller, but one which relies on the OS to perform some of the RAID work. This is commonly called "FakeRAID" and is not desirable.

Despite setting the controller into AHCI Mode in the BIOS, I still get a disk called  /dev/mapper/pdc_abcdef or similar and cannot use the disk /dev/sdb directly. I wanted the FakeRAID gone.

The FakeRAID is managed by Device-Mapper RAID called "dmraid".

An old CentOS 5 post suggests removing the dmraid package might work, but this wanted to remove anaconda and many other things, so didn't seem a good option.

There is a kernel boot parameter to disable Device-Mapper RAID so I applied this.

In /etc/default/grub add nodmraid to GRUB_CMDLINE_LINUX_DEFAULT:

GRUB_CMDLINE_LINUX_DEFAULT="quiet rhgb nodmraid"

Then re-create the GRUB2 configuration file:

grub2-mkconfig -o /boot/grub2/grub.cfg

Reboot, and no more FakeRAID drive. Success.

Saturday, November 7, 2015

Unknown symbol __dynamic_pr_debug

I was working with the default x86_64 + kvm kernel config and a kernel module which uses the pr_debug() printk logging method.

On load, modprobe prints:

FATAL: Error inserting modulename: Unknown symbol in module, or unknown parameter (see dmesg)

And dmesg contains:

Unknown symbol __dynamic_pr_debug

I do have the CONFIG_DYNAMIC_DEBUG=y parameter set in my .config.

I found reference to this upstream at: https://lkml.org/lkml/2012/8/30/378 where Jim Crombie said "Ok, transient error, went away with a clean build".

I'd done a make; make modules_install; make install and reboot but that didn't work for me.

Like Jim found, a make clean then recompile worked for me.

You need to mount debugfs to use pr_debug(), see dynamic-debug-howto.txt in the kernel Documentation directory.

Sunday, June 1, 2014

kernel panic scsi_wait_scan on elrepo kernel-ml

Browsing around ELRepo, you can see they have the elrepo-kernel repository which contains the long-life kernel-lt (currently 3.10) and mainline kernel-ml (currently 3.14) packages.

I'd always been curious about these, so I spun up a test VM and did a CentOS 6 Basic Server install, then added ELRepo and installed them.

kernel-lt works fine first time.

However, kernel-ml will not boot, and simply repeats:
FATAL: Module scsi_wait_scan not found.

Sunday, March 16, 2014

hosting http and https domains with lighttpd

I have two domains, one with an SSL certificate and one without. I wanted to host these on the same lighttpd server which only has one IP address.

There are several forum threads and lighty wiki articles about this, but none really cover a concise example of how to do it. Many parts of the documentation are confusing and non-obvious, the situation is not helped by the fact that you cannot do name-based SSL hosting, so what you'd expect to be the logical configuration doesn't work.

After staring at these for a few hours:
And a lot of trial-and-error, here's the config I ended up using:

Tuesday, February 4, 2014

realtek RTL8188CUS slow on Raspberry Pi

I recently had an adventure troubleshooting slow wifi on one of my Raspberry Pi systems. No matter what I did, I could not get more than 57 kilobytes per second transfer speed to it.

Thursday, November 21, 2013

yum grouplist doesn't display all groups

I've noticed in Fedora that the yum grouplist command doesn't actually display all the groups:

# yum grouplist | grep Virt
# yum grouplist | grep Virt | wc -l
0

However you can still see a group with groupinfo:
# yum groupinfo Virtualization
Loaded plugins: langpacks, list-data, presto, refresh-packagekit
Group: Virtualization
 Group-Id: virtualization
 Description: These packages provide a virtualization environment.

Apparently there can be a "hidden" tag on groups, as reported in this Bugzilla entry:

Bug 986531 - unhide all groups from yum grouplist

You can see all the groups with the yum grouplist hidden command:

# yum grouplist | wc -l
39
# yum grouplist hidden | wc -l
126

Wednesday, November 13, 2013

how does mke2fs decide automatic check mount count?

Whenever you create an ext3 or ext4 filesystem with mkfs or mke2fs, a message is printed saying "This filesystem will be automatically checked every X mounts or Y days, whichever comes first".

However, the number seems to vary depending on the filesystem. How is this value calculated? I used the source of e2fsprogs-1.42.8 and the cscope source code tool to find this out.

Thursday, August 29, 2013

finding the netdev_priv struct

In addition to the net_device struct within the kernel, network drivers also have their own private or device-specific struct which stores stats unique to the individual hardware.

The name of the struct varies for each device type, however the location remains the same, right after net_device:

linux-2.6.32-358.14.1.el6.x86_64/include/linux/netdevice.h
/**
 *      netdev_priv - access network device private data
 *      @dev: network device
 *
 * Get network device private data
 */
static inline void *netdev_priv(const struct net_device *dev)
{
        return (char *)dev + ALIGN(sizeof(struct net_device), NETDEV_ALIGN);
}
A search in cscope for netdev_priv will show you many functions within drivers which update the priv structure via pointers, for example this one in e1000_main.c:
struct e1000_adapter *adapter = netdev_priv(netdev);
So, given that we know what the device-specific struct is called, and where the device-specific private struct is, how do you find it and read it?

We'll need the ability to read kernel memory using crash, either run on a live system, or in a vmcore captured with kdump:
crash /usr/lib/debug/lib/modules/2.6.32-358.14.1.el6.x86_64/vmlinux /var/crash/2013-08-26/vmcore
We'll also need the debugging symbols for the driver in question:
crash> mod -s e1000 /usr/lib/debug/lib/modules/2.6.32-358.14.1.el6.x86_64/kernel/drivers/net/e1000/e1000.ko.debug
     MODULE       NAME                 SIZE  OBJECT FILE
ffffffffa01550e0  e1000              170678  /usr/lib/debug/lib/modules/2.6.32-358.14.1.el6.x86_64/kernel/drivers/net/e1000/e1000.ko.debug 
The net command will show you the network devices in the system:
crash> net
   NET_DEVICE     NAME   IP ADDRESS(ES)
ffff88007e76b820  lo     127.0.0.1
ffff8800372c0020  eth0   192.168.1.126
We can then cast net_device against this to see the in-kernel device struct:
crash> net_device 0xffff8800372c0020
struct net_device {
  name = "eth0\000\000\060:03.0\000\000\000",
...
But we want to get after this struct and cast it against the struct in the driver.

We need to know how big net_device is:
crash> struct -o net_device
struct net_device {
...
}
SIZE: 1728
We then find the net address plus the size of net_device:
crash> px 0xffff8800372c0020+1728
$1 = 0xffff8800372c06e0
We can now cast the device-specific private struct against this new address:
crash> e1000_adapter 0xffff8800372c06e0
struct e1000_adapter {
  vlgrp = 0x0,
  mng_vlan_id = 65535,
  bd_number = 0,
  rx_buffer_len = 1522,
...
and we're done!

Tuesday, July 9, 2013

akg perception 120 usb mic works on linux

I recently bought an AKG Perception 120 USB microphone to do some voiceover work with.

I couldn't find any reliable source which confirmed whether this works on Linux or not, so hopefully this post will help someone out.

This microphone works great on Linux.

Not only that, it works great full stop. I was using a cheap headset before, and the difference in sound quality is incredible. I'm not any sort of audio expert or audiophile, though I can tell you the constant "hiss" that my old headset produced is eliminated with this new microphone.

I tested using Fedora 18 with kernel 3.9.6-200 which worked fine.
I also tried Ubuntu 13.04 with kernel 3.8.0-19 and that worked too.
However, Ubuntu 12.04.3 with kernel 3.8.0-29 didn't work. It recognised the device but would not pick up sound.

If you wonder about any specific distro, preferably one with a LiveCD, just leave a comment and I'll give it a try if I can.

Here is the output of lsusb on F18 with the device connected:

Bus 001 Device 027: ID 074d:3556 Micronas GmbH Composite USB-Device

Here is the output of dmesg on F18 as the device is connected:

usb 1-1.2: new full-speed USB device number 27 using ehci-pci
usb 1-1.2: New USB device found, idVendor=074d, idProduct=3556
usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
usb 1-1.2: Product: AKG Perception 120 USB
usb 1-1.2: Manufacturer: AKG Acoustics
ALSA sound/usb/clock.c:243 current rate 0 is different from the runtime rate 48000
ALSA sound/usb/clock.c:243 current rate 0 is different from the runtime rate 48000
usbcore: registered new interface driver snd-usb-audio
ALSA sound/usb/clock.c:243 current rate 48000 is different from the runtime rate 44100
ALSA sound/usb/clock.c:243 current rate 48000 is different from the runtime rate 44100

Tuesday, July 2, 2013

how to display tcp rto

Calculating the TCP Retransmission Timeout always makes my head explode, yet it can be useful when troubleshooting TCP algorithms like FRTO or TIME_WAIT timeout when TCP Timestamps are in use.

This excellent post by sgros gives a very thorough overview of the RTO algorithms in the kernel and how RTO is actually calculated:
However, I don't want to do that, and I certainly don't want to do it for every socket in realtime. Is there an easy way to display the RTO for every socket?

There sure is:
ss -i
The -i option of ss is the flag to "Show internal TCP information", check man ss for more options.

Here's an example of a socket on my system including its RTO:

State   Recv-Q   Send-Q    Local Address:Port      Peer Address:Port
ESTAB   0        0           10.99.0.123:51308   192.168.34.111:http
cubic wscale:6,7 rto:660 rtt:396/38.75 ato:40 cwnd:10 send 272.3Kbps rcv_space:14600
                 ^^RTO^^
ss is one of the "next generation" iproute2 commands to replace the older netstat. Much like ip addr has replaced ifconfig, ip route has replaced route, and ip neigh has replaced arp.

Wednesday, January 16, 2013

how to network jumbo frames to a kvm guest

Getting a jumbo frame to a KVM guest is not something which works out of the box, but can be configured to work quite easily.

One way get jumbo frames to a KVM guest is to directly pass the guest a physical NIC through PCI Passthrough, or a Virtual Function through SR-IOV.

Another way is to use the traditional libvirt bridge. Create a bridge on the host, and bridge the guest's NICs into this. The reason this doesn't work by default is all the interfaces are created with the default Ethernet MTU of 1500 bytes.

You can set MTU of the physical NIC, the bridge, and the guest interface with the network-scripts, but you cannot automatically set the MTU of the tap interface with the network-scripts. The tap interface is how the virtual network interface connects to the "real world". You'll see these tap devices with names like "vnet0" in your bridge.

So how do we do this?

First, some requirements:
  • A physical interface which supports jumbo frames.
    The Linux bridge code will only allow the bridge to change to the smallest MTU of all its interfaces.
  • A bridge with the larger required MTU
    Technically a jumbo frame covers MTU from 1501 to 9000, but almost everyone just wants to set 9000
  • virtio-net or Intel e1000 network interface inside the guest
    The real-life Realtek 8139 does not support an MTU of 9000, therefore there is no driver support for large MTU in the emulated version either.
Our first step is to make all tap interfaces on the host have an MTU of 9000. We do this by adding a udev rule to the top of the network device creation as follows:

/etc/udev/rules.d/70-persistent-net.rules (on host)
SUBSYSTEM=="net", ACTION=="add", KERNEL=="vnet*", ATTR{mtu}="9000"
Then set your physical interface MTU to 9000:

/etc/sysconfig/network-scripts/ifcfg-eth0 (on host):
DEVICE=eth0
HWADDR=[physical MAC address]
TYPE=Ethernet
ONBOOT=yes
BRIDGE=br0
MTU=9000
And do the same inside the bridge:

/etc/sysconfig/network-scripts/ifcfg-br0 (on host)
DEVICE=br0
TYPE=Bridge
ONBOOT=yes
DELAY=0
MTU=9000
Now inside your guest, set your virtio or e1000 interface to also have the larger MTU:

/etc/sysconfig/network-scripts/ifcfg-eth0 (in guest)
DEVICE=eth0
HWADDR=[guest MAC address]
TYPE=Ethernet
ONBOOT=yes
MTU=9000
You'll need to perform a "service network restart" on the host, or at least bring the bridge and eth interfaces down and up. Start your guest and confirm all interfaces have been made with the correct MTU:
# ip link
2: eth0: mtu 9000 qdisc pfifo_fast state UP qlen 1000
    link/ether 01:23:45:67:89:0A brd ff:ff:ff:ff:ff:ff
3: br0: mtu 9000 qdisc noqueue state UNKNOWN
    link/ether 01:23:45:67:89:0A brd ff:ff:ff:ff:ff:ff
4: vnet0: mtu 9000 qdisc pfifo_fast state UNKNOWN qlen 500
    link/ether fe:54:00:01:23:45 brd ff:ff:ff:ff:ff:ff
This should be enough to send a "Do Not Fragment" 9000 byte frame over the network. Let's test with a ping:
# ping -c 4 -s 8972 -M do 172.16.0.2
PING 172.16.0.2 (172.16.0.2) 8972(9000) bytes of data.
8980 bytes from 172.16.0.2: icmp_seq=1 ttl=64 time=1.27 ms
8980 bytes from 172.16.0.2: icmp_seq=2 ttl=64 time=0.284 ms
8980 bytes from 172.16.0.2: icmp_seq=3 ttl=64 time=0.202 ms
8980 bytes from 172.16.0.2: icmp_seq=4 ttl=64 time=0.260 ms
Success!

Sunday, August 5, 2012

how to persist ethtool settings across reboot

You can use the ethtool command to read and change information about your network interfaces.

For example, ethtool -g ethX reads the size of the ring buffer on the NIC, and ethtool -G ethx rx A tx B changes it. Use man ethtool to discover more settings.

But these options don't persist across reboot, so how do you make sure your settings are kept permanent?

You can enter the ethtool commands in /etc/rc.local (or your distribution's equivalent) where commands are run after the current runlevel completes, but this isn't ideal. Network services may have started during the runlevel and ethtool commands tend to interrupt network traffic. It would be more preferable to have the commands applied as the interface is brought up.

The network service in CentOS has the ability to do this. The script /etc/sysconfig/network-scripts/ifup-post checks for the existence of /sbin/ifup-local, and if it exists, runs it with the interface name as a parameter (eg: /sbin/ifup-local eth0)

We can create this file with touch /sbin/ifup-local make it executable with chmod +x /sbin/ifup-local set its SELinux context with chcon --reference /sbin/ifup /sbin/ifup-local and then open it in an editor.

A simple script to apply the same settings to all interfaces would be something like:
#!/bin/bash
if [ -n "$1" ]; then
    /sbin/ethtool -G $1 rx 4096 tx 4096
    /sbin/ethtool -K $1 tso on gso on
fi
Keep in mind this will attempt to apply settings to ALL interfaces, even the loopback.

If we have different interfaces we want to apply different settings to, or want to skip the loopback, we can make a case statement:
#!/bin/bash
case "$1" in
 eth0)
  /sbin/ethtool -G $1 rx 16384 tx 16384
  /sbin/ethtool -K $1 gso on gro on
  ;;
 eth1)
  /sbin/ethtool -G $1 rx 64 tx 64
  /sbin/ethtool -K $1 tso on gso on
  /sbin/ip link set $1 txqueuelen 0
  ;;
esac
exit 0
Now ethtool settings are applied to interfaces as they start, all potential interruptions to network communication are done as the interface is brought up, and your server can continue to boot with full network capabilities.

Saturday, August 4, 2012

kvm virtual machine won't start

Recently I brought my KVM host down for an upgrade. I shut down the guests within their OSes, confirmed they were powered off with a virsh list --all, and everything went well with the upgrade.

However, upon returning the guests to service, one of them would not start up. Typing virsh start I got the helpful error message

Error restoring domain: cannot send monitor command
Connection reset by peer

Why was this one VM broken but the others were fine?

I played around with virsh autostart and virsh autostart --disable but that had no effect, nor did a reboot of the hypervisor.

After some searching around, it turns out libvirt has the capability to keep "managed save states" of guests, kinda like a sleep mode or snapshot, to save you fully powering a guest OS off.

For some reason, a managed save for this one guest had been created, perhaps it had not shut down properly, or perhaps there's an errorr in libvirt. I could view the saved state with

# virsh list --all --managed-save
 Id Name                 State
----------------------------------
  - guest1               shut off
  - guest2               saved


Now a virsh managedsave-remove guest2 returned it to the "shut off" state, and I could start it properly with virsh start as per usual.