Recovering filesystems within qcow2

After recent crash of my laptop (hardware died) my QCOW2 images suffered minor damage and for the heck I couldn’t convince Fedora VM to auto-fsck itself so I had to go all manual after N-th prompt from dracut shell about problems starting up.

Turns out it’s not so complicated to get to the guts of the QCOW2 after all:

 #  modprobe nbd max_part=16
 #  qemu-nbd --connect=/dev/nbd0 /var/lib/libvirt/images/foo.qcow2
 #  fdisk -l /dev/nbd0
 #  pvscan
 #  vgchange -a y fedora
 #  fsck.ext4 -y /dev/fedora/rootfs 
 #  vgchange -a n fedora
 #  qemu-nbd --disconnect /dev/nbd0

In above snippet “fedora” is my VM’s VG name and rootfs was the offending LV

Using libvirt hooks and qcow2 for safety net on laptop

I’ve been using my laptop for development for a while now (Fedora, of course, and we’re at 19 now), but the trouble is – several times without paying much attetention I let battery run dry on it while VMs are running. In KDE it would normally result in automatic graceful shutdown, under LXDE however all you get is a notice and then whatever happens – happens. First time I’ve learned to disable caching on my “images” partition:

$ grep libvirt /etc/fstab
/dev/mapper/vg_delliquent-libvirtfs /var/lib/libvirt     ext4    defaults,sync,dirsync        1 2

That kind of helped with me using SSD and other things – I have eliminated double-caching of FS transactions and now my images should survive a bit of “oopsie”. BUT: I still don’t want to leave it to a chance and loose several month of my work inside those VMs. So I needed impromptu backup solution that is portable. That’s where QEMU/QCOW2 external snapshots come handy. First we need to lay out images, and so we do:

# qemu-img create -f qcow2 -b vm1-base.qcow2 vm1-staging.qcow2
# qemu-img create -f qcow2 -b vm1-staging.qcow2 vm1-running.qcow2

Great, now we have a chain of images like so: vm1-base <- vm1-staging <- vm1-running.  Our libvirt VM definitionwill use that last image: vm1-running .

# virsh edit vm1
... replace reference to vm1-base.qcow2 with vm1-running.qcow2 ...

Now we’re running off snapshot and whatever happens both vm1-base and vm1-staging will not be modified and those shouldn’t be affected if we have a power failure etc.

On with the libvirt! Under /etc/libvirt we need a directory called “hooks“:

# mkdir -p /etc/libvirt/hooks

now we create qemu hook:

$ cat > /etc/libvirt/hooks/qemu
#!/bin/sh

guest_name=$1
state=$2
phase=$3
other=$4

RESNAP_SCRIPT=/usr/local/sbin/resnap.sh

date >> /tmp/libvirt.log

echo "$0 $@" >> /tmp/libvirt.log
if [ -e "$LOCK_FILE" ] 
     then
      # we've been asked to skip resnapping. politely exit now
      echo "Lock found: $LOCK_FILE . Skipping resnap..." >> /tmp/libvirt.log
      exit 0
fi 
if [ $state == "release" -a $phase == "end" ]
  then
    # 1. grab qemu image name
    # 2. create a clone/snapshot

    ## http://libvirt.org/hooks.html#names
    ## "...A hook script must not call back into libvirt, 
    ##  as the libvirt daemon is already waiting for the script to exit."

    # very crude attempt to sidestep "virsh dump_xml"
    vm_definition_xml=$(grep -Fl "<name>$guest_name</name>" /etc/libvirt/qemu/*.xml)
    source_image=$(xmllint --xpath 'string(/domain/devices/disk/source/@file)' $vm_definition_xml)
    image_type=$(file -b $source_image | awk '{print $1, $2;}')
    echo "$source_image $image_type" >> /tmp/libvirt.log
    if [ "$image_type" == 'QEMU QCOW' ]
      then
        ( date; echo "$RESNAP_SCRIPT $source_image" ) >> /tmp/resnap.log
        $RESNAP_SCRIPT $source_image
    fi
fi
^D

And if we really need to skip re-snap we just do:

$ touch /tmp/qemu_vm1.lock

Now a small matter of getting RESNAP_SCRIPT done 😉

# cat > /usr/local/sbin/resnap.sh
#!/bin/sh

image=$1

base_img=$(qemu-img info --output json $image | python -c  'import simplejson,sys; i=simplejson.load(sys.stdin); print i.get("full-backing-filename",i.get("backing-filename","ITSABASE"))')
if [ "$base_img" == "ITSABASE" ]
then
   exit 1
fi
qemu-img commit $image
mv $image $image.commited
qemu-img create -f qcow2 -b $base_img $image

Most likely you’ll need to restart libvirtd (I had to, anyway):

systemctl restart libvirtd.service

and we’re all set.

End result will be: whenever I shutdown a VM – hook script will kick in and commit all the changes to “vm1-staging.qcow2”, then create a backup copy of my commited changes after which it will create brand new snapshot vm1-runnint.qcow2 for me to use next time I start up vm1.

Couple of notes:

  1. depending on requirements resnap.sh script could be easily modified to create chain of commits that could be commited instead of committing the “vm1-staging” to the base, like so ” commit vm1-running.qcow2.commited.1; commit vm1-running.qcow2.commited 2 …” this should work if you want to have  backup to your backup and if you’re going to watch your disk usage carefully as those do add up.
  2. resnap.sh will kick in for any QCOW2 image,  it is smart enough to bail on base images but it will commit any snapshots back to whatever is next up the “backing-chain”, so be warned. To avoid this – extend the script and have either a whitelist or a blacklist of VMs you’ll be starting up.
  3. we create files in /tmp and not /var as my intent was to not have those files around upon reboot.

 

Android, adb and “no permission”

I’ve got tired of seeing:

$ adb devicesList of devices attached 
???????????? no permissions

So I had to dig a bit. Combining StackOverflow article and official docs I arrived at somewhat simple solution that avoids overly enthusiastic suggestion of making device “anybody-writable” and made things “just right” for myself (fixing old-school Udev rules sytax along the way):

  1. plug-in your device and using lsusb locate attributes like iVendor and iProduct (I just went with iVendor…)
  2. add /etc/udev/rules.d/51-android.rules :
    SUBSYSTEM=="usb",ATTR{idVendor}=="18d1",MODE="0660",GROUP="users"
  3. restart udev:
    udevadm control --reload-rules

End result: my any of my Nexus7 devices whenever attached will be owned by users group (which includes me and my family) with adb now properly responding:

$ adb devices
List of devices attached 
07bdc25b        unauthorized

which only means I have tap OK on N7 upon connect to allow adb debugging from this PC.

Neat trick I’ve spotted is that you can poke some of the rules without re-plug via:

udevadm trigger --attr-match=vendor='Yoyodyne' --attr-match=model='Frobnicator 300'

Didn’t have to use it this time – but may come handy next time.

Firefox + TabMixPlus and the case of lost session

Firefox 20 came with a handy feature of having parallel “private” windows a-la Chrome. Hurray. Now while playing with it I ended up with 2 Firefox windows and while closing them I did something out of sequence and lost my “last session” saved by TabMixPlus. While there was no definitive information on how to restore it – I did find useful piece:

$ killall firefox # or do it gracefully
$ cd $HOME/.firefox/blahblah/
$ cp session.rdf session.rdf.orig
$ cat sessionbackups/tabmix_sessions-2013-04-08.rdf > session.rdf
$ firefox

Linux and MBR

Today while installing another RedHat box via KickStart ended up with KickStart flaw which installed MBR onto my USB installation disk. While the rest of the data remained intact I did want to get rid of the unwanted passenger on my USB disk. Solution turned out to be simple:

# dd if=/dev/sdX of=/tmp/mbr count=1 bs=512
# file /tmp/mbr
/tmp/mbr: x86 boot sector; GRand Unified Bootloader, stage1 version 0x3, boot drive 0x81, 1st sector stage2 0x62c02, GRUB version 0.94; partition 1: ID=0x7, starthead 32, startsector 2048, 209715200 sectors; partition 2: ID=0x5, starthead 32, startsector 209717248, 767055920 sectors, extended partition table, code offset 0x48

From here on you can either save that copy (/tmp/mbr) in case things go wrong, and launch

# dd if=/dev/zero of=/dev/sdX bs=446 count=1

This will zero the MBR and get rid of grub (or other unwanted stuff you dropped there).

Note that we zero only 446 bytes and not whole 512 to preserve the partition table.

That’s it, done.

64bit Fedora + NVidia + Wine + 3D Games = WTF

I’ve been running Fedora for a while now with NVidia drivers packaged by rpmfusion folks. However only recently my Wine games quit working. Note: it’s a 64bit box and if I were running 32bit box probably wouldn’t have been a problem as it turns out, but all in it’s own time… First I though my ageing F16 was at fault so I have upgraded. While Nouveau drivers becoming better they are still not suitable for 3D gaming (at least under Wine).

What I have discovered is that:

  • NVidia drivers are really picky about certain parameters
  • For unknown to me reason they throw display DPI in the garbage so I’m left with elephant-sized fonts
  • You have to practically tiptoe around NVidia configs, while it’s own configurator (nvidia-settings) will gladly screw it up for you.

Certain combinations leave you with no 3D acceleration and crashing apps, like Chromium browser, glxinfo etc. After some experimenting and tinkering I’ve finally got something “close to normal”:

Section "Device"
        Identifier  "Videocard0"
        Driver      "nvidia"
EndSection

Section "Monitor"

    # HorizSync source: edid, VertRefresh source: edid
    Identifier     "Monitor0"
    VendorName     "Unknown"
    ModelName      "Samsung SyncMaster"
    DisplaySize     510    290
    HorizSync       26.0 - 81.0
    VertRefresh     24.0 - 75.0
    Option         "DPMS"
EndSection

Section "Screen"
    Identifier     "Screen0"
    Device         "Device0"
    Monitor        "Monitor0"
    Option         "UseEdidDpi" "False"
    Option         "ConstantDPI" "True"
    Option         "TwinViewOrientation" "DFP-1 LeftOf DFP-0"
EndSection

However that wasn’t the end of it. Now glxinfo was happy but Windows apps won’t see acceleration. Snooping around found the culprit: missing package:

xorg-x11-drv-nvidia-libs.i686

that was enough to bring 3D in Wine back. Now we’re back to playing Hitman 2, courtecy of gog.com 🙂

Jelly Bean, multiuser and lost security patterns

Finally got my N7 setup with multiuser goodness from JB. However my son in attack of curiosity locked himself out with security pattern. Well… I didn’t 😉 Thanks to always helpful XDA: [GUIDE][HOW-TO]Crack android pattern lock! solution was really quite simple:

# find / -name gesture.key 
# mv /data/system/users/11/gesture.key /data/system/users/11/gesture.key.bak 

bash color goodness (yes, I do miss Gentoo’s colorful CLI)

I keep on repeating the same setup over and over and every time I have to go and look.

# Black            \e[0;30m
# Blue             \e[0;34m
# Green            \e[0;32m
# Cyan             \e[0;36m
# Red              \e[0;31m
# Purple           \e[0;35m
# Brown            \e[0;33m
# Gray             \e[0;37m
# Dark Gray        \e[1;30m
# Light Blue       \e[1;34m
# Light Green      \e[1;32m
# Light Cyan       \e[1;36m
# Light Red        \e[1;31m
# Light Purple     \e[1;35m
# Yellow           \e[1;33m
# White            \e[1;37m

grey='\[\e[0;37m\]'
dgrey='\[\e[1;30m\]'
red='\[\e[1;31m\]'
green='\[\e[1;32m\]'
yellow='\[\e[1;33m\]'
blue='\[\e[1;34m\]'
magenta='\[\e[1;35m\]'
cyan='\[\e[1;36m\]'
white='\[\e[1;37m\]'
underline='\[\e[4m\]'
eofmt='\[\e[m\]'
eoc='\[\e[0m\]'

. ${HOME}/.bash_color
PS1_DIR_COLOR=${PS1_DIR_COLOR:-yellow}
eval PS1_DIR_COLOR=\$$PS1_DIR_COLOR
if [ "$TERM" == "xterm" ]; then
export PS1=$white'\! '$dgrey'\t '$white'\u'$eoc'@'$underline'\h'$eofmt' '$PS1_DIR_COLOR'\w '$white'\$ '$eoc
else
export PS1=$white'\! '$white'\u'$eoc'@'$underline'\h'$eofmt' '$PS1_DIR_COLOR'\w '$white'\$ '$eoc
fi
export HISTCONTROL="ignorespace:ignoredups"
export HISTTIMEFORMAT=" %F-%H:%M "

Now add to your ~/.bash_color:

#!/bin/bash

PS1_DIR_COLOR="green"

and you have per-machine colors with easy management.