Remus on Xen 4.5 and Ubuntu 14.04

Installing Xen 4.5-rc from source, setting up remus-drbd, and problems encountered

Posted by Anthony Korzan on December 01, 2014
Backdated from old dokuwiki blog on January 01, 2015

Links to the installation steps

  1. Compilation of Xen
  2. Creating a regular DomU (VM) for Xen
  3. Compiling Linux for Remus-DRBD
  4. Compilation of DRBD for Remus
  5. Creating a DRBD DomU for Remus

High Level Overview

The following documents my steps to install the Xen Hypervisor and Remus on Ubuntu 14.04. Xen is a hypervisor--most popular virtualization platforms, such as VMWare Player, run as an application in the operating system; Xen instead is a virtualization platform that runs inbetween the operating system and the hardware allowing greater flexibility. It is used by many providers of Virtual Private Servers, VPSes, such as Amazon's EC2.

Remus is a high availability suite for Xen that effectively creates a “RAID 1” setup of virutal machines, VMs. In simpler terms it guarantees that if one machine fails, the end user cannot tell a failure has occured as his session is transferred to the safe clone.

My Thoughts

Remus was originally a research project at the University of British Columbia. The issue is that it carries over a lot of the research code--code written at a lower standard than that of production code. As such, at times it can be unstable depending on your system setup--Linux configuration, Operating System, or just buggy code. Setting up remus and then having a running remus setup crash for no reason is frustrating, and your only hope is to either search the xen-devel mailing lists for a similar problem or send an email to the developers.

Compiling Xen 4.5 on Ubuntu 14.04

Install the dependencies.

sudo apt-get install build-essential bcc bin86 gawk bridge-utils iproute libcurl3 libcurl4-openssl-dev bzip2 module-init-tools transfig tgif texinfo texlive-latex-base texlive-latex-recommended texlive-fonts-extra texlive-fonts-recommended pciutils-dev mercurial libjpeg-dev make gcc libc6-dev-i386 zlib1g-dev python python-dev python-twisted libncurses5-dev patch libvncserver-dev libsdl-dev libpixman-1-dev iasl libbz2-dev e2fslibs-dev git-core uuid-dev ocaml ocaml-findlib libx11-dev bison flex xz-utils libyajl-dev gettext markdown libaio-dev pandoc

For remus you will also need the following packages.

sudo apt-get install libnl-3-dev libnl-utils libnl1 libnl-cli-3-dev

You can download the most current Xen source from git, and checkout a respective branch. At the time of this writing, Xen 4.5-rc is the master branch. Downloading prepackaged source code in tar balls might lead to compilation errors as they may be lacking patches. Patches are available if you scavange the mailing lists. A problem I ran into is compiling any Xen Distribution 4.3.X and below required a patch to work with GCC4.8.

git clone git://xenbits.xen.org/xen.git

Compiling

cd xen
./configure –libdir=/usr/lib
make world -j8

If you receive the following error:
sys/cdefs.h: No such file or directory, you can run the following commands

apt-get purge libc6-dev
apt-get update
apt-get install libc6-dev-i386

Installation

sudo make install
sudo update-rc.d xencommons defaults
sudo update-rc.d xendomains defaults
sudo /sbin/ldconifg

Modify /etc/default/grub to boot Xen by default. At the time of this writing the master branch was at 4.5.0-rc. Modify respectively.
Please note that any Linux kernel greater than 3.4.X does not work with remus-drbd. We'll cover that later.

GRUB_DEFAULT="Advanced options for Ubuntu GNU/Linux (with Xen hypervisor)>Xen hypervisor, version 4.5.0-rc>Ubuntu GNU/Linux, with Xen 4.5.0-rc and Linux 3.13.0-32-generic"

Finally Network setup so that your VMs also known as DomUs have internet. Edit /etc/network/interfaces

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet manual

auto xenbr0
iface xenbr0 inet dhcp
        bridge_ports eth0

Creating a non-Remus DomU (VM)

TBD
For now you can follow the remus instructions and not run remus.

Compiling and Installing Linux for Remus-DRBD

Remus on Xen 4.5 currently requires a special version of DRBD 8.3.11. As such, Remus on Xen 4.5 is only compatible with Linux Kernels 3.0-3.4 due to incompatible headers.

TBD

Compiling and Installing DRBD for Remus

The special version of DRBD is available from one of the project maintainers Github repository.

git clone https://github.com/rshriram/remus-drbd

The following dependencies are required in addition to those of Xen:

sudo apt-get install xsltproc autoconf

To compile user-land tools

cd remus-drbd
./autogen.sh
./configure --prefix=/usr --localstatedir=/var --sysconfdir=/etc --with-km
make
sudo make install

To compile the DRBD kernel module

cd ../remus-drbd/drbd
make clean all
sudo make install

Add drbd to /etc/modules

Configuring and creating a Remus-DRBD DomU

For this portion you will require two servers both running Xen and Linux 3.0-3.4.
For my purposes there are two servers: 80556K and E6400. I will be using 80566K as the primary. You can name them whatever you'ld like. Those are the hostnames.

On the first server, create a logical volume on your volume group. In this case I installed my Ubuntu distribution into a LVM volume group name 80556K-vg.

sudo lvcreate -L 8192 -n ubuntu_vm_1 80556K-vg

On the second server, same thing different volume group name.

sudo lvcreate -L 8192 -n ubuntu_vm_1 E6400-vg

On both servers copy over the configuration file found in the source folder of remus-drbd.

sudo cp /usr/src/remus-drbd/scripts/global_common.conf.protoD /etc/drbd.d/global_common.conf

On the primary server, create and edit this file: /etc/drbd.d/ubuntu_vm_1.res

resource ubuntu_vm_1 {
        device /dev/drbd1;
        disk /dev/80556K-vg/ubuntu_vm_1;
        meta-disk internal;
        on "80556K" {
                address 192.168.1.201:7791;
        }
        on "E6400" {
                address 192.168.1.202:7791;
        }
}

On the secondary server, create and edit this file: /etc/drbd.d/ubuntu_vm_1.res. Note how the volume group has changed.

resource ubuntu_vm_1 {
        device /dev/drbd1;
        disk /dev/E6400-vg/ubuntu_vm_1;
        meta-disk internal;
        on "80556K" {
                address 192.168.1.201:7791;
        }
        on "E6400" {
                address 192.168.1.202:7791;
        }
}

On both servers run

sudo drbdadm create-md ubuntu_vm_1
sudo drbdadm up ubuntu_vm_1

Now on the primary server run

sudo drbdadm -- --overwrite-data-of-peer primary ubuntu_vm_1

Important: You will need to run this command each time you reboot the primary Dom0 if you want to use the DomU installed on this volume we are creating.

Wait for the transfer to complete. To check run this command on either server:

cat /proc/drbd

This may take an hour.

After it has mirrored the volume, we can set up the volume for DomU installation on the primary server.

pvcreate /dev/drbd1
vgcreate ubuntu_vm_1 /dev/drbd1

And finally to install the VM's (DomU's) operating system we can use netboot.
Create somewhere memorable a configuration file for your DomU on the primary server: ~/ubuntu1.cfg

name = "ubuntu1"
memory = 256
vcpus = 1
disk = ['phy:/dev/drbd1,xvda,w']

vif = ['mac=00:16:3e:00:00:01']

# Run our own kernels
#kernel = "/boot/vmlinuz-3.13.0-32-generic"
#ramdisk = "/boot/initrd.img-3.13.0-32-generic"

# Run VMs own kernels
#bootloader = "pygrub"

# For installation
kernel = "/nfs/imgs/ubuntu-netboot/vmlinuz"
ramdisk = "/nfs/imgs/ubuntu-netboot/initrd.gz"
extra = "debian-installer/exit/always_halt=true -- console=hvc0"