Author: daemon

After recent crash of my laptop (hardware died) my QCOW2 images suffered minor damage and for the heck I couldn’t convince Fedora VM to auto-fsck itself so I had to go all manual after N-th prompt from dracut shell about problems starting up.

Turns out it’s not so complicated to get to the guts of the QCOW2 after all:

 #  modprobe nbd max_part=16
 #  qemu-nbd --connect=/dev/nbd0 /var/lib/libvirt/images/foo.qcow2
 #  fdisk -l /dev/nbd0
 #  pvscan
 #  vgchange -a y fedora
 #  fsck.ext4 -y /dev/fedora/rootfs 
 #  vgchange -a n fedora
 #  qemu-nbd --disconnect /dev/nbd0

In above snippet “fedora” is my VM’s VG name and rootfs was the offending LV

I’ve been using my laptop for development for a while now (Fedora, of course, and we’re at 19 now), but the trouble is – several times without paying much attetention I let battery run dry on it while VMs are running. In KDE it would normally result in automatic graceful shutdown, under LXDE however all you get is a notice and then whatever happens – happens. First time I’ve learned to disable caching on my “images” partition:

$ grep libvirt /etc/fstab
/dev/mapper/vg_delliquent-libvirtfs /var/lib/libvirt     ext4    defaults,sync,dirsync        1 2

That kind of helped with me using SSD and other things – I have eliminated double-caching of FS transactions and now my images should survive a bit of “oopsie”. BUT: I still don’t want to leave it to a chance and loose several month of my work inside those VMs. So I needed impromptu backup solution that is portable. That’s where QEMU/QCOW2 external snapshots come handy. First we need to lay out images, and so we do:

# qemu-img create -f qcow2 -b vm1-base.qcow2 vm1-staging.qcow2
# qemu-img create -f qcow2 -b vm1-staging.qcow2 vm1-running.qcow2

Great, now we have a chain of images like so: vm1-base <- vm1-staging <- vm1-running.  Our libvirt VM definitionwill use that last image: vm1-running .

# virsh edit vm1
... replace reference to vm1-base.qcow2 with vm1-running.qcow2 ...

Now we’re running off snapshot and whatever happens both vm1-base and vm1-staging will not be modified and those shouldn’t be affected if we have a power failure etc.

On with the libvirt! Under /etc/libvirt we need a directory called “hooks“:

# mkdir -p /etc/libvirt/hooks

now we create qemu hook:

$ cat > /etc/libvirt/hooks/qemu
#!/bin/sh

guest_name=$1
state=$2
phase=$3
other=$4

RESNAP_SCRIPT=/usr/local/sbin/resnap.sh

date >> /tmp/libvirt.log

echo "$0 $@" >> /tmp/libvirt.log
if [ -e "$LOCK_FILE" ] 
     then
      # we've been asked to skip resnapping. politely exit now
      echo "Lock found: $LOCK_FILE . Skipping resnap..." >> /tmp/libvirt.log
      exit 0
fi 
if [ $state == "release" -a $phase == "end" ]
  then
    # 1. grab qemu image name
    # 2. create a clone/snapshot

    ## http://libvirt.org/hooks.html#names
    ## "...A hook script must not call back into libvirt, 
    ##  as the libvirt daemon is already waiting for the script to exit."

    # very crude attempt to sidestep "virsh dump_xml"
    vm_definition_xml=$(grep -Fl "<name>$guest_name</name>" /etc/libvirt/qemu/*.xml)
    source_image=$(xmllint --xpath 'string(/domain/devices/disk/source/@file)' $vm_definition_xml)
    image_type=$(file -b $source_image | awk '{print $1, $2;}')
    echo "$source_image $image_type" >> /tmp/libvirt.log
    if [ "$image_type" == 'QEMU QCOW' ]
      then
        ( date; echo "$RESNAP_SCRIPT $source_image" ) >> /tmp/resnap.log
        $RESNAP_SCRIPT $source_image
    fi
fi
^D

And if we really need to skip re-snap we just do:

$ touch /tmp/qemu_vm1.lock

Now a small matter of getting RESNAP_SCRIPT done 😉

# cat > /usr/local/sbin/resnap.sh
#!/bin/sh

image=$1

base_img=$(qemu-img info --output json $image | python -c  'import simplejson,sys; i=simplejson.load(sys.stdin); print i.get("full-backing-filename",i.get("backing-filename","ITSABASE"))')
if [ "$base_img" == "ITSABASE" ]
then
   exit 1
fi
qemu-img commit $image
mv $image $image.commited
qemu-img create -f qcow2 -b $base_img $image

Most likely you’ll need to restart libvirtd (I had to, anyway):

systemctl restart libvirtd.service

and we’re all set.

End result will be: whenever I shutdown a VM – hook script will kick in and commit all the changes to “vm1-staging.qcow2”, then create a backup copy of my commited changes after which it will create brand new snapshot vm1-runnint.qcow2 for me to use next time I start up vm1.

Couple of notes:

1. depending on requirements resnap.sh script could be easily modified to create chain of commits that could be commited instead of committing the “vm1-staging” to the base, like so ” commit vm1-running.qcow2.commited.1; commit vm1-running.qcow2.commited 2 …” this should work if you want to have  backup to your backup and if you’re going to watch your disk usage carefully as those do add up.
2. resnap.sh will kick in for any QCOW2 image,  it is smart enough to bail on base images but it will commit any snapshots back to whatever is next up the “backing-chain”, so be warned. To avoid this – extend the script and have either a whitelist or a blacklist of VMs you’ll be starting up.
3. we create files in /tmp and not /var as my intent was to not have those files around upon reboot.