Playing with Fedora on AWS I ended up needing HVM image yet all available images were Paravirt so I needed to “create” HVM image.

Thanks to helpful advise from ServerFault it turned out to be relatively simple (however I can’t say it was straightforward) using web console:

• using PV image create EBS-backed instance instanceA
• adjust instance config as necessary
• take a snapshot of instanceA – snapA
• from snapA create image specifying HVM as virt type
• done

I have also found Pyhon-based CLI which has proved somewhat entertaining. One thing not mentioned is that to use it you have to create group for admins (it’ll ask for type of permissions for that group), then create/assign user to that group. Use “aws ec2 configure”.

I have decided to try out UniFi AP for my home WiFi setup and either because of my “consumer-grade” switch etc. or some other reason I just couldn’t get it recognized on my “Controller” … so I had to look. Winning combination was:

1. write down MAC of UniFi AP
2. on my DHCP server track down UniFi AP’s acquired IP
3. SSH into UniFi AP unit (factory defaults creds are ubnt/ubnt)
4. start mca-cli
5. Issue command set-inform http://controllerIP:8080/inform
6. Adopt on controller (simple drop on the map is what worked for me)
7. Reissue command – set-inform http://controllerIP:8080/inform
8. AP is adopted and workable now

As a side-note controller software that runs on Linux required some iptables mashing as well since I was using remote browser:

# firewall-cmd --zone=home --add-port 8080/tcp
# firewall-cmd --zone=home --add-port 8443/tcp
# firewall-cmd --zone=home --add-port 8880/tcp
# firewall-cmd --zone=home --add-port 8843/tcp

This opens up ports only for the current session (runtime), after reboot those changes will be gone and if we need to retain them “–permanent” should be added and commands re-run.

I’m playing a lot with Ansible those days and one of the topics that does come up all the time is how do I set certain values in config files that are not plain-text or .ini style? Special mention goes to Apache configs that are neither plain-text nor XML (ugh) and tomcat (well at least it’s XML…).

Augeas is a swiss-army-knife for a sysadmin as it “understands” multiple configuration formats (lens’) and can deal with them in it’s own way.

So here’s what I needed to do: Setting up ownCloud VM on Fedora 20 I needed to edit /etc/httpd.d/conf.d/owncloud.conf file to override default restriction for localhost, in other words adding Require all granted to the end of <Directory "/usr/share/owncloud"> section. Perl and Python are fun but you practically need to build your own parser. Not me. Not today. So here’s how we handle things with augeas:

create command file /tmp/ocloud_aug, like so:

defvar conf /files/etc/httpd/conf.d/owncloud.conf
load

# Get <Directory "/usr/share/owncloud"> subtree
defvar ocloud_dir $conf/Directory[arg="/usr/share/owncloud/"]

touch $ocloud_dir/directive[last()+1]
defvar last_stmt $ocloud_dir/directive[last()]
set $last_stmt Require
set $last_stmt/arg[1] all
set $last_stmt/arg[2] granted
save

Hint: before going around modifying your live system consider one of two testing methods:

• setup $AUGEAS_ROOT to whatever directory you want to play in:

  # mkdir /tmp/aug
  # export AUGEAS_ROOT=/tmp/aug
  # cp -r /etc $AUGEAS_ROOT

• adding “-n” flag to augtool invocation:

  # augtool -n ...

So now from:

...
<Directory "/usr/share/owncloud">
...
      Require local
...
</Directory>

we get

...
<Directory "/usr/share/owncloud">
...
      Require local
...
Require all granted
</Directory>

after we run:

# augtool -f /tmp/ocloud_aug

That just about solves our problem of programmatically adding configuration lines. We could also remove offending “Require local”:

rm $ocloud_dir/*/*[self::directive="Require"][arg='local']

After recent crash of my laptop (hardware died) my QCOW2 images suffered minor damage and for the heck I couldn’t convince Fedora VM to auto-fsck itself so I had to go all manual after N-th prompt from dracut shell about problems starting up.

Turns out it’s not so complicated to get to the guts of the QCOW2 after all:

 #  modprobe nbd max_part=16
 #  qemu-nbd --connect=/dev/nbd0 /var/lib/libvirt/images/foo.qcow2
 #  fdisk -l /dev/nbd0
 #  pvscan
 #  vgchange -a y fedora
 #  fsck.ext4 -y /dev/fedora/rootfs 
 #  vgchange -a n fedora
 #  qemu-nbd --disconnect /dev/nbd0

In above snippet “fedora” is my VM’s VG name and rootfs was the offending LV

I’ve been using my laptop for development for a while now (Fedora, of course, and we’re at 19 now), but the trouble is – several times without paying much attetention I let battery run dry on it while VMs are running. In KDE it would normally result in automatic graceful shutdown, under LXDE however all you get is a notice and then whatever happens – happens. First time I’ve learned to disable caching on my “images” partition:

$ grep libvirt /etc/fstab
/dev/mapper/vg_delliquent-libvirtfs /var/lib/libvirt     ext4    defaults,sync,dirsync        1 2

That kind of helped with me using SSD and other things – I have eliminated double-caching of FS transactions and now my images should survive a bit of “oopsie”. BUT: I still don’t want to leave it to a chance and loose several month of my work inside those VMs. So I needed impromptu backup solution that is portable. That’s where QEMU/QCOW2 external snapshots come handy. First we need to lay out images, and so we do:

# qemu-img create -f qcow2 -b vm1-base.qcow2 vm1-staging.qcow2
# qemu-img create -f qcow2 -b vm1-staging.qcow2 vm1-running.qcow2

Great, now we have a chain of images like so: vm1-base <- vm1-staging <- vm1-running.  Our libvirt VM definitionwill use that last image: vm1-running .

# virsh edit vm1
... replace reference to vm1-base.qcow2 with vm1-running.qcow2 ...

Now we’re running off snapshot and whatever happens both vm1-base and vm1-staging will not be modified and those shouldn’t be affected if we have a power failure etc.

On with the libvirt! Under /etc/libvirt we need a directory called “hooks“:

# mkdir -p /etc/libvirt/hooks

now we create qemu hook:

$ cat > /etc/libvirt/hooks/qemu
#!/bin/sh

guest_name=$1
state=$2
phase=$3
other=$4

RESNAP_SCRIPT=/usr/local/sbin/resnap.sh

date >> /tmp/libvirt.log

echo "$0 $@" >> /tmp/libvirt.log
if [ -e "$LOCK_FILE" ] 
     then
      # we've been asked to skip resnapping. politely exit now
      echo "Lock found: $LOCK_FILE . Skipping resnap..." >> /tmp/libvirt.log
      exit 0
fi 
if [ $state == "release" -a $phase == "end" ]
  then
    # 1. grab qemu image name
    # 2. create a clone/snapshot

    ## http://libvirt.org/hooks.html#names
    ## "...A hook script must not call back into libvirt, 
    ##  as the libvirt daemon is already waiting for the script to exit."

    # very crude attempt to sidestep "virsh dump_xml"
    vm_definition_xml=$(grep -Fl "<name>$guest_name</name>" /etc/libvirt/qemu/*.xml)
    source_image=$(xmllint --xpath 'string(/domain/devices/disk/source/@file)' $vm_definition_xml)
    image_type=$(file -b $source_image | awk '{print $1, $2;}')
    echo "$source_image $image_type" >> /tmp/libvirt.log
    if [ "$image_type" == 'QEMU QCOW' ]
      then
        ( date; echo "$RESNAP_SCRIPT $source_image" ) >> /tmp/resnap.log
        $RESNAP_SCRIPT $source_image
    fi
fi
^D

And if we really need to skip re-snap we just do:

$ touch /tmp/qemu_vm1.lock

Now a small matter of getting RESNAP_SCRIPT done 😉

# cat > /usr/local/sbin/resnap.sh
#!/bin/sh

image=$1

base_img=$(qemu-img info --output json $image | python -c  'import simplejson,sys; i=simplejson.load(sys.stdin); print i.get("full-backing-filename",i.get("backing-filename","ITSABASE"))')
if [ "$base_img" == "ITSABASE" ]
then
   exit 1
fi
qemu-img commit $image
mv $image $image.commited
qemu-img create -f qcow2 -b $base_img $image

Most likely you’ll need to restart libvirtd (I had to, anyway):

systemctl restart libvirtd.service

and we’re all set.

End result will be: whenever I shutdown a VM – hook script will kick in and commit all the changes to “vm1-staging.qcow2”, then create a backup copy of my commited changes after which it will create brand new snapshot vm1-runnint.qcow2 for me to use next time I start up vm1.

Couple of notes:

1. depending on requirements resnap.sh script could be easily modified to create chain of commits that could be commited instead of committing the “vm1-staging” to the base, like so ” commit vm1-running.qcow2.commited.1; commit vm1-running.qcow2.commited 2 …” this should work if you want to have  backup to your backup and if you’re going to watch your disk usage carefully as those do add up.
2. resnap.sh will kick in for any QCOW2 image,  it is smart enough to bail on base images but it will commit any snapshots back to whatever is next up the “backing-chain”, so be warned. To avoid this – extend the script and have either a whitelist or a blacklist of VMs you’ll be starting up.
3. we create files in /tmp and not /var as my intent was to not have those files around upon reboot.

http://ddosattackprotection.org/blog/cyber-security-blogs/

Today while installing another RedHat box via KickStart ended up with KickStart flaw which installed MBR onto my USB installation disk. While the rest of the data remained intact I did want to get rid of the unwanted passenger on my USB disk. Solution turned out to be simple:

# dd if=/dev/sdX of=/tmp/mbr count=1 bs=512
# file /tmp/mbr
/tmp/mbr: x86 boot sector; GRand Unified Bootloader, stage1 version 0x3, boot drive 0x81, 1st sector stage2 0x62c02, GRUB version 0.94; partition 1: ID=0x7, starthead 32, startsector 2048, 209715200 sectors; partition 2: ID=0x5, starthead 32, startsector 209717248, 767055920 sectors, extended partition table, code offset 0x48

From here on you can either save that copy (/tmp/mbr) in case things go wrong, and launch

# dd if=/dev/zero of=/dev/sdX bs=446 count=1

This will zero the MBR and get rid of grub (or other unwanted stuff you dropped there).

Note that we zero only 446 bytes and not whole 512 to preserve the partition table.

That’s it, done.