http://lttng.org/blog/2015/02/04/web-request-latency-root-cause/
Category Archives: Rants of a Sysadmin
Fedora & AWS
Playing with Fedora on AWS I ended up needing HVM image yet all available images were Paravirt so I needed to “create” HVM image.
Thanks to helpful advise from ServerFault it turned out to be relatively simple (however I can’t say it was straightforward) using web console:
- using PV image create EBS-backed instance instanceA
- adjust instance config as necessary
- take a snapshot of instanceA – snapA
- from snapA create image specifying HVM as virt type
- done
I have also found Pyhon-based CLI which has proved somewhat entertaining. One thing not mentioned is that to use it you have to create group for admins (it’ll ask for type of permissions for that group), then create/assign user to that group. Use “aws ec2 configure”.
Linux Security Distros Compared: Tails vs. Kali vs. Qubes
Boostraping FreeBSD pkgng non-interactively
Building ansible playbooks for freebsd setup I came to a roadblock – pkgng demands to be bootstrapped interactively. However looking at it’s source code I’ve found the backdoor:
ASSUME_ALWAYS_YES=yes pkg bootstrap -f
and I’ve got things rolling. Overall “ASSUME_ALWAYS_YES” seems to go further then some “-y” switches.
UniFi AP setup
I have decided to try out UniFi AP for my home WiFi setup and either because of my “consumer-grade” switch etc. or some other reason I just couldn’t get it recognized on my “Controller” … so I had to look. Winning combination was:
- write down MAC of UniFi AP
- on my DHCP server track down UniFi AP’s acquired IP
- SSH into UniFi AP unit (factory defaults creds are ubnt/ubnt)
- start
mca-cli
- Issue command
set-inform http://controllerIP:8080/inform
- Adopt on controller (simple drop on the map is what worked for me)
- Reissue command –
set-inform http://controllerIP:8080/inform
- AP is adopted and workable now
As a side-note controller software that runs on Linux required some iptables mashing as well since I was using remote browser:
# firewall-cmd --zone=home --add-port 8080/tcp # firewall-cmd --zone=home --add-port 8443/tcp # firewall-cmd --zone=home --add-port 8880/tcp # firewall-cmd --zone=home --add-port 8843/tcp
This opens up ports only for the current session (runtime), after reboot those changes will be gone and if we need to retain them “–permanent” should be added and commands re-run.
Augeas for configuration modification
I’m playing a lot with Ansible those days and one of the topics that does come up all the time is how do I set certain values in config files that are not plain-text or .ini style? Special mention goes to Apache configs that are neither plain-text nor XML (ugh) and tomcat (well at least it’s XML…).
Augeas is a swiss-army-knife for a sysadmin as it “understands” multiple configuration formats (lens’) and can deal with them in it’s own way.
So here’s what I needed to do: Setting up ownCloud VM on Fedora 20 I needed to edit /etc/httpd.d/conf.d/owncloud.conf file to override default restriction for localhost, in other words adding Require all granted
to the end of <Directory "/usr/share/owncloud">
section. Perl and Python are fun but you practically need to build your own parser. Not me. Not today. So here’s how we handle things with augeas:
create command file /tmp/ocloud_aug, like so:
defvar conf /files/etc/httpd/conf.d/owncloud.conf load # Get <Directory "/usr/share/owncloud"> subtree defvar ocloud_dir $conf/Directory[arg="/usr/share/owncloud/"] touch $ocloud_dir/directive[last()+1] defvar last_stmt $ocloud_dir/directive[last()] set $last_stmt Require set $last_stmt/arg[1] all set $last_stmt/arg[2] granted save
Hint: before going around modifying your live system consider one of two testing methods:
- setup $AUGEAS_ROOT to whatever directory you want to play in:
# mkdir /tmp/aug # export AUGEAS_ROOT=/tmp/aug # cp -r /etc $AUGEAS_ROOT
- adding “-n” flag to augtool invocation:
# augtool -n ...
So now from:
... <Directory "/usr/share/owncloud"> ... Require local ... </Directory>
we get
... <Directory "/usr/share/owncloud"> ... Require local ... Require all granted </Directory>
after we run:
# augtool -f /tmp/ocloud_aug
That just about solves our problem of programmatically adding configuration lines. We could also remove offending “Require local”:
rm $ocloud_dir/*/*[self::directive="Require"][arg='local']
Recovering filesystems within qcow2
After recent crash of my laptop (hardware died) my QCOW2 images suffered minor damage and for the heck I couldn’t convince Fedora VM to auto-fsck itself so I had to go all manual after N-th prompt from dracut shell about problems starting up.
Turns out it’s not so complicated to get to the guts of the QCOW2 after all:
# modprobe nbd max_part=16 # qemu-nbd --connect=/dev/nbd0 /var/lib/libvirt/images/foo.qcow2 # fdisk -l /dev/nbd0 # pvscan # vgchange -a y fedora # fsck.ext4 -y /dev/fedora/rootfs # vgchange -a n fedora # qemu-nbd --disconnect /dev/nbd0
In above snippet “fedora” is my VM’s VG name and rootfs was the offending LV
Using libvirt hooks and qcow2 for safety net on laptop
I’ve been using my laptop for development for a while now (Fedora, of course, and we’re at 19 now), but the trouble is – several times without paying much attetention I let battery run dry on it while VMs are running. In KDE it would normally result in automatic graceful shutdown, under LXDE however all you get is a notice and then whatever happens – happens. First time I’ve learned to disable caching on my “images” partition:
$ grep libvirt /etc/fstab /dev/mapper/vg_delliquent-libvirtfs /var/lib/libvirt ext4 defaults,sync,dirsync 1 2
That kind of helped with me using SSD and other things – I have eliminated double-caching of FS transactions and now my images should survive a bit of “oopsie”. BUT: I still don’t want to leave it to a chance and loose several month of my work inside those VMs. So I needed impromptu backup solution that is portable. That’s where QEMU/QCOW2 external snapshots come handy. First we need to lay out images, and so we do:
# qemu-img create -f qcow2 -b vm1-base.qcow2 vm1-staging.qcow2 # qemu-img create -f qcow2 -b vm1-staging.qcow2 vm1-running.qcow2
Great, now we have a chain of images like so: vm1-base <- vm1-staging <- vm1-running. Our libvirt VM definitionwill use that last image: vm1-running .
# virsh edit vm1 ... replace reference to vm1-base.qcow2 with vm1-running.qcow2 ...
Now we’re running off snapshot and whatever happens both vm1-base and vm1-staging will not be modified and those shouldn’t be affected if we have a power failure etc.
On with the libvirt! Under /etc/libvirt we need a directory called “hooks“:
# mkdir -p /etc/libvirt/hooks
now we create qemu hook:
$ cat > /etc/libvirt/hooks/qemu #!/bin/sh guest_name=$1 state=$2 phase=$3 other=$4 RESNAP_SCRIPT=/usr/local/sbin/resnap.sh date >> /tmp/libvirt.log echo "$0 $@" >> /tmp/libvirt.log if [ -e "$LOCK_FILE" ] then # we've been asked to skip resnapping. politely exit now echo "Lock found: $LOCK_FILE . Skipping resnap..." >> /tmp/libvirt.log exit 0 fi if [ $state == "release" -a $phase == "end" ] then # 1. grab qemu image name # 2. create a clone/snapshot ## http://libvirt.org/hooks.html#names ## "...A hook script must not call back into libvirt, ## as the libvirt daemon is already waiting for the script to exit." # very crude attempt to sidestep "virsh dump_xml" vm_definition_xml=$(grep -Fl "<name>$guest_name</name>" /etc/libvirt/qemu/*.xml) source_image=$(xmllint --xpath 'string(/domain/devices/disk/source/@file)' $vm_definition_xml) image_type=$(file -b $source_image | awk '{print $1, $2;}') echo "$source_image $image_type" >> /tmp/libvirt.log if [ "$image_type" == 'QEMU QCOW' ] then ( date; echo "$RESNAP_SCRIPT $source_image" ) >> /tmp/resnap.log $RESNAP_SCRIPT $source_image fi fi ^D
And if we really need to skip re-snap we just do:
$ touch /tmp/qemu_vm1.lock
Now a small matter of getting RESNAP_SCRIPT done 😉
# cat > /usr/local/sbin/resnap.sh #!/bin/sh image=$1 base_img=$(qemu-img info --output json $image | python -c 'import simplejson,sys; i=simplejson.load(sys.stdin); print i.get("full-backing-filename",i.get("backing-filename","ITSABASE"))') if [ "$base_img" == "ITSABASE" ] then exit 1 fi qemu-img commit $image mv $image $image.commited qemu-img create -f qcow2 -b $base_img $image
Most likely you’ll need to restart libvirtd (I had to, anyway):
systemctl restart libvirtd.service
and we’re all set.
End result will be: whenever I shutdown a VM – hook script will kick in and commit all the changes to “vm1-staging.qcow2”, then create a backup copy of my commited changes after which it will create brand new snapshot vm1-runnint.qcow2 for me to use next time I start up vm1.
Couple of notes:
- depending on requirements resnap.sh script could be easily modified to create chain of commits that could be commited instead of committing the “vm1-staging” to the base, like so ” commit vm1-running.qcow2.commited.1; commit vm1-running.qcow2.commited 2 …” this should work if you want to have backup to your backup and if you’re going to watch your disk usage carefully as those do add up.
- resnap.sh will kick in for any QCOW2 image, it is smart enough to bail on base images but it will commit any snapshots back to whatever is next up the “backing-chain”, so be warned. To avoid this – extend the script and have either a whitelist or a blacklist of VMs you’ll be starting up.
- we create files in /tmp and not /var as my intent was to not have those files around upon reboot.
Top 100+ Cyber Security Blogs By Page Rank | @StopDDOSAttacks
Linux and MBR
Today while installing another RedHat box via KickStart ended up with KickStart flaw which installed MBR onto my USB installation disk. While the rest of the data remained intact I did want to get rid of the unwanted passenger on my USB disk. Solution turned out to be simple:
# dd if=/dev/sdX of=/tmp/mbr count=1 bs=512 # file /tmp/mbr /tmp/mbr: x86 boot sector; GRand Unified Bootloader, stage1 version 0x3, boot drive 0x81, 1st sector stage2 0x62c02, GRUB version 0.94; partition 1: ID=0x7, starthead 32, startsector 2048, 209715200 sectors; partition 2: ID=0x5, starthead 32, startsector 209717248, 767055920 sectors, extended partition table, code offset 0x48
From here on you can either save that copy (/tmp/mbr) in case things go wrong, and launch
# dd if=/dev/zero of=/dev/sdX bs=446 count=1
This will zero the MBR and get rid of grub (or other unwanted stuff you dropped there).
Note that we zero only 446 bytes and not whole 512 to preserve the partition table.
That’s it, done.