Comprehensive Debian
Wheezy Xen Tutorial
Table of Contents
Wheezy Installation
Wheezy Configuration
Compiling a Custom Linux Kernel
Compiling Xen
The following guide is comprised of my personal
documentation and excessive filtering for human
consumption. It details the series of steps I took to
successfully install and compile a custom Linux Kernel,
Xen 4.2 unstable, and run three virtual machines to
perform unique tasks all on a single physical computer.
To achieve the desired functionality, I used IOMMU for
PCI Passthrough with a multimedia operating system,
granting complete access to a graphics card for GPU
Acceleration.
I am a college student, not a trained professional, and I
am sharing this documentation for educational purposes.
Blind use of this document for a production environment
in a business setting would be ill-advised.
To quickly summarize my experiences, I decided to try
Xen and began researching in early January 2012.
I purchased equipment in March 2012, and began what I
thought would take "at most two weeks".
By late April I finally had a (mostly) functional system.
I have been fine tuning the system and my process since
then to produce this guide for others.
The purpose of this guide is to turn a 6 month project into
a series of steps that can be reproduced inside the time
frame of 1-2 days.
I wanted a single physical computer, that could handle
three computers worth of separated activities at all times,
including these four specifics:
Router/Firewall (PFSense)
DNS & Web Development (Debian Squeeze)
Application Development (Debian Wheezy)
Multimedia & Gaming (Windows 7)
I investigated alternative software, including VMWare's
ESXi and Citrix's XenServer.
I had come from a VMWare platform having used both
VMware Server 2 and VMWare Workstation 8 for the
same key objectives previously but with a Windows
Host, subject to Windows Updates which let to my
investigation.
ESXi was easy to install, but missing numerous drivers
for hardware components. I quickly ruled it out as I
wanted a home-use system, and there was no
convenient or well-documented method to installing or
even launching a virtual machine from their on-server
interface.
Citrix's XenServer was easily my favorite of the options,
it's design made for a great user experience. Everything
was easy to find and do. However it came with a
minimum price tag of $1000, and if I wanted PCI
passthrough for graphics cards $2500. This was beyond
my reach as a college student.
I chose Xen since it had support for my hardware, was a
free open-source project, and had a great community
with lots of activity.
Things you Will Need:
Compatible Hardware
ATI Graphics Card
Motherboard with UEFI & VT-d & Onboard
Graphics
Latest Ubuntu Live DVD
Debian Wheezy Beta1 (Or Newer) Installer
nVidia Cards can be made to work, with extensive
patching, in Windows XP and supposedly Windows 8
Preview, for more information visit
If you plan to pass your graphics card to a virtual
machine, you will need either a second graphics card,
onboard graphics, or a second computer to manage your
Dom0 system and install virtual machines over VNC.
UEFI compatible boot DVD will save you an
undocumented step for setting up a UEFI boot loader.
Xen is picky, different hardware may yield different
results, both at compile time and runtime. If you want to
save yourself some hassle, here is my hardware list and
some suggestions to avoid:
Motherboard:
ASRock Z68 Extreme4 Gen3
CPU:
Intel Core i2600
RAM:
12GB 1333Mhz Corsair XMS (2x2G + 2x4G)
Boot Disk:
240GB OCZ Vertex 3
GPU:
ATI Radeon HD 6870
LAN:
Onboard Broadcom BCM57781
PCIe EXPI9301CTBLK
Hardware and Configurations to Avoid:
NF200 Chipsets are not IOMMU compatible
nVidia Graphics Cards
RAID5 yields horrible performance
NF200 is a PCI Switch for motherboards sporting SLI
and CrossFire, avoid it if you want be able to use those
PCI slots for passthrough.
These instructions are for an Intel CPU and ASRock
UEFI Motherboard, and may vary depending on your
manufacturer as well as your choice of CPU. If you are
unfamiliar with motherboard configuration, you may want
to watch the video for a visual walk through.
Before starting reset your CMOS so you have a clean
slate to work from.
List of key settings:
Turn Legacy USB 3.0 off
Turn on VT-x and VT-d
Set drives to ACHI Mode
Change default video to Onboard
Justification:
Consider yourself warned, if you leave Legacy USB on
and leave a backup USB drive connected, your system
will fail to boot. To my surprise that is because it is trying
to boot from that USB drive (even if that drive is USB
2.0), which threw me off so I recommend disabling that
feature.
Most motherboards will have Intel Virtualization enabled
by default (VT-x), but that is not the same as IOMMU, be
sure to look for VT-d or check the manual for details. For
my system VT-x was in the CPU configuration area, and
VT-d was in the Northbridge configuration area.
ACHI is a superior choice for performance with modern
hard drives, and while RAID is an alternative that uses
ACHI as its underlying type, it increases boot time by
checking RAID configurations, and most onboard RAID
is software RAID that is built for Windows and rarely
helpful for Linux.
ASRock boards will use a PCI GPU by default if it is
plugged into a PCI slot, regardless of if any video cables
are connected, and if you plan to pass that device to a
virtual machine, you do not want to use it for video.
Select Advanced Options from the menu, and then
Advanced Install.
To expedite this process we can select defaults for most
options (such as for networking).
Once we reach Hard Drive Partitioning options, select
Manual at the bottom.
Select the disk(s) we will be using by title not partition
space and hit enter, we can now select gpt from the
partition tables instead of msdos.
We will then create three partitions:
256MB FAT32 Partition mounted to /boot/efi
256MB Ext4 Partition mounted to /boot
Remainder LVM Partition
Select Configure the Logical Volume Manager and
create a volume group.
Create three Logical Volumes:
8GB or More for Linux
20B or More for Home
2GB or More for Swap
Complete LVM Configuration, and then back to our
partition settings we can now select formats and mount
points:
8GB LV Ext4 for root (/)
20GB LV Ext4 for /home
2GB LV swap
Optionally add noatime flags to all the Ext4 partitioned
disks.
Complete the Partition steps by selecting done and
create the partitions.
Continue through the options with defaults, until we get
to the main Packages.
De-select Debian Desktop Environment and
continue
Do not install a boot loader, select the Continue
without a bootloader option and be sure to jott down
the information it provides us for later.
Complete the installation and restart our computer.
Insert the Ubuntu Live DVD and press whatever key
opens the boot options menu at startup.
Select the UEFI boot option from that menu.
When presented with an options screen, press c on the
keyboard to access the Emergency Grub Menu.
Emergency Grub allows us to manually boot our Linux,
since we did not install a boot loader.
Type ls to list the disks, we should have an (hd0) and
three gpt partitions.
Assuming the same partitioning as mine, in the below
example tab indicates hit the tab key to auto-complete,
and we can either use the data provided when we
selected Continue without a bootloader or just
substitute xen for the name we gave our Volume Group
name, and linux for our root (/)Logical Volume:
Gibberish may be displayed when tabbing to auto-
complete, this is normal and can be ignored. After
entering boot it will begin starting Linux.
If done successfully a terminal login will be presented,
which concludes this segment of the guide.
In the next segment we will start by enter our credentials
and we can start installing some key packages to get
UEFI working.
References
Note:
I have done my best to separate my crazier desires from
the average persons needs.
To do this I created bold headers with line breaks for
special sub-sections, whose subsequent contents you
can skip if said bold text does not apply to your situation.
Assuming you have just installed Debian Wheezy and
are at the login prompt, you want to login as root so you
can install packages (alternatively you can use
su
to
grub> set root=(hd0,gpt2)
grub> linux /vmlinu_tab_ ro root=/dev/mapper/xen-linux
grub> initrd /initrd_tab_
grub> boot
change users if you prefer).
UEFI STEPS
Assuming you came from my previous guide and have
just manually booted, your first objective should be
getting an efi bootloader installed:
aptitude install grub-efi-amd64
Once installed you will want to run the install and update
scripts:
grub-install
update-grub
END UEFI STEPS
Now let's install a couple more basic packages:
aptitude install ssh sudo
With ssh installed we can, optionlly, access remotely for
the remainder of the guide. With sudo we can add our
user to the sudoers group like so (cdelorme is my
username):
usermod -aG sudo cdelorme
If you omit the
a
you will replace all existing groups your
user belongs to, so be careful.
I recommend rebooting now to get ssh working, and also
so you can login as your user to use the sudo command
(if you used
su
you may need to logout and log back in
for
sudo
to work).
Next we want to prime our server with some general
utility packages:
parted
samba
ntp
screen
p7zip-full
Delicious Copy Pasta:
Note: I used to install
ntfsprogs
but the package is
currently bugged and eats CPU cycles, so I would
avoid it if possible.
Configuration of samba and ntp will vary by system, I
recommend using other guides online for this since as
long as they work they will be fine when we start using
Xen. If you want my configurations specifically, ask and I
may consider adding them to a future edition of this
guide.
Development Environment Steps
Since I intend to use Debian Wheezy as a development
environment, I also installed these packages:
openjdk-7-jdk
ant
g++
libboost-dev
gnustep
python
python-dev
Delicious Copy Pasta:
sudo aptitude install parted samba ntp screen p7zip-full
sudo aptitude install openjdk-7-jdk ant g++ libboost-dev gnustep python python-dev
End Development Environment Steps
GUI Environment Steps
To be honest, I don't care for the CD Installation of
Gnome3, it's less work but the cost is 30 minutes of
installation time, and a whole slew of packages I never
use, so I went minimalist and found exactly what I
needed:
gnome-session
gnome-terminal
gnome-screensaver
xserver-xorg-core
gdm3
gksu
ia32-libs
ia32-libs-gtk
libc6-dev-i386
binfmt-support
Delicious Copy Pasta:
If you intend to use Xen with SDL later on, I recommend
replacing
xserver-xorg-core
with
xorg
and
xorg-dev
packages, as these are what we will install later, and
they contain the
xserver-xorg-core
.
As a bare-bones install of Gnome3, it should take
roughly 5 minutes as opposed to 30.
At this stage you can run
startx
to launch the GUI
environment, but I generally reboot to test the login
screen.
End GUI Environment Steps
sudo aptitude install gnome-session gnome-terminal gnome-screensaver xserver-xorg-core gdm3 gksu ia32-libs ia32-libs-gtk libc6-dev-i386 binfmt-support
Optional GUI Additions
Optional additions include enabling audio, installing
some basic multimedia, utilities, and important software
like Guake terminal, Google Chrome, and Sublime Text.
Starting with Audio:
sudo aptitude install alsa-base
sudo alsactl init
Now with GUI utilities:
python-xdg
font-manager
gnome-disk-utility
gnome-screenshot
guake
Delicious Copy Pasta:
Note: Guake has a bug with Gnome3 that prevents it
from starting at boot, here is how you can fix it:
This is actually a very simple edit, in this specific version
of guake (located at
/usr/bin/guake
) line 653 contains
a
notification.show()
, we just have to wrap it in a
simple try-catch so it doesn't crash when it is run before
the GUI is ready to accept a pynotify event:
try:
notification.show()
except Exception:
pass
Next we want to set Guake to start at login. You can
make an auto-start file via:
sudo aptitude install python-xdg font-manager gnome-disk-utility gnome-screenshot guake
For google chrome we need to add a new debian source
to
/etc/apt/sources.list
, and download their package
key:
Next we can install these packages:
google-chrome-unstable
eog
gnash
vlc
gtk-recordmydesktop
Delicious Copy Pasta:
Note: I am aware that google source is for Squeeze, I
have had no problems running it in Wheezy.
After google-chrome has been installed, it will create its
own sources.list file entry, so we want to go back in and
remove the line we added from
/etc/apt/sources.list
,
mkdir ~/.config/autostart
cd ~/.config/autostart
echo "[Desktop Entry]" >> guake.desktop
echo "Name=Guake Terminal" >> guake.desktop
echo "Name[pt]=Guake Terminal" >> guake.desktop
echo "Name[pt_BR]=Guake Terminal" >> guake.desktop
echo "Comment=Use the command line in a Quake-like terminal" >> guake.desktop
echo "TryExec=guake" >> guake.desktop
echo "Exec=guake" >> guake.desktop
echo "Icon=/usr/share/pixmaps/guake/guake.png" >> guake.desktop
echo "Type=Application" >> guake.desktop
echo "Categories=GNOME;GTK;Utility;TerminalEmulator;" >> guake.desktop
su root
echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
aptitude update
exit
sudo aptitude install google-chrome-unstable eog gnash vlc gtk-recordmydesktop
and re-run
sudo aptitude update
to make sure it works.
Finally, I am in love with Sublime Text 2, so I went to
their website for the download, and went through these
steps for a complete installation.
Once downloaded, let's extract it and move it to
/usr/local
, and create a new command to run it from
terminal:
Next let's create a desktop file so we can run it as an
application and set default mimetypes from the GUI.
Let's navigate to
/usr/share/applications/
and create
a new file
subl.desktop
with these contents:
Next to set the mimetypes we add these lines to
/usr/share/applications/defaults.list
:
text/plain=subl.desktop
text/css=subl.desktop
text/htm=subl.desktop
text/javascript=subl.desktop
text/x-c=subl.desktop
text/csv=subl.desktop
text/x-java-source=subl.desktop
text/java=subl.desktop
There are user-account alternatives that may seem more
tar xf sublime-text.tar.bz2
sudo mv sublime-text /usr/local/
cd /usr/local/sublime-text
sudo ln -s /usr/local/sublime-text/sublime_text /usr/bin/subl
[Desktop Entry]
Name=Sublime Text 2
Comment=The Best Text Editor in the World!
TryExec=subl
Exec=subl
Icon=/usr/local/sublime-text/Icon/256x256/sublime_text.png
Type=Application
appropriate, but I wanted these to apply system-wide.
End Optional GUI Additions
Thus concludes our Debian Wheezy configuration, and
we are now ready to move onto compiling a Custom
Linux Kernel.
References
Missing ia32-libs-gtk for Gnome3
Compiling a Custom Linux
Kernel
Compiling a custom Linux Kernel will make PCI
passthrough much easier, but it also allows us to
improve performance in various areas for Dom0.
For starters let's create a folder for compiling our
packages in our home directory:
mkdir -p ~/src/linux
cd ~/src/linux
Next we will install the required packages:
wget
tar
bzip2
build-essential
libncurses-dev
kernel-package
fakeroot
Delicious copy pasta:
To improve compiling speed we can set the concurrency
to double the number of physical CPU cores by running
these commands:
IMPORTANT NOTE: Kernel 3.5 and 3.5.2 are bugged
and passthrough with USB Controllers fails (though
VGA Passthrough works for some odd reason). That is
why this guide is using 3.4.9, however 3.3.8 has also
been tested and works.
Our next step is downloading, extracting, and inserting
the current kernel configuration before adjusting the
kernel we are about to compile:
Configuring a kernel is not as complex as it seems, the
menu is just a hierarchy of options, and thankfully you
can search for settings with
/
and get details on
individual settings with
h
.
Here is a tree-map list of all the important settings we
need to change:
sudo aptitude install wget tar bzip2 build-essential libncurses-dev kernel-package fakeroot
su root
echo "CONCURRENCY_LEVEL=8" >> /etc/kernel-pkg.conf
exit
wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.4.9.tar.bz2
tar jxf linux-3.4.9.tar.bz2 && rm linux-3.4.9.tar.bz2
cd linux-3.4.9/
cp /boot/config-3.2.0-3-amd64 .config
make menuconfig
Processor Type and Features
Preemption Model (Voluntary Kernel
Preemption (Desktop))
Preemptible Kernel (Low-Latency
Desktop)
Enable cleancache driver to cache clean
pages if tmem is present
Timer frequency (250 HZ)
1000 HZ
Bus options (PCI etc.)
PCI Stub driver
Xen PCI Frontend
Device Drivers
Block devices
Xen virtual block device support
Xen block-device backend driver
Network device support
Xen network device frontend driver
Xen backend network device
Ethernet driver support
QLogic devices
NetXen Multi port (1/10)
Gigabit Ethernet NIC
Xen driver support
Xen memory balloon driver
Dynamically self-balloon kernel
memory to target
Memory hotplug support for Xen
balloon driver
Scrub pages before returning them to
system
Xen /dev/xen/evtchn device
Backend driver support
Xen filesystem
Create compatibility mount point
/proc/xen
Create xen entries under
/sys/hypervisor
userspace grant access device driver
User-space grant reference allocator
driver
Xen PCI-device backend driver
Firmware Drivers
EFI Variable Support via sysfs
For SSH Users
For users using ssh, you will want to use
screen
, this
way if your machine goes to sleep or your connection is
lost while compiling the process will not halt in the
middle.
Starting screen couldn't be easier:
screen
End For SSH Users
Now we can clean the system and then compile our
kernel:
Note that in later versions of gcc the revision must begin
with a number.
On my machine it takes roughly 20 minutes to compile a
kernel, so feel free to take a break.
Once the compilation is complete, we can back up a
directory where we should now find a portable .deb
installation package. I highly recommend making a
backup of this file if you want to reinstall that kernel
without all this effort in the future:
make-kpkg clean
fakeroot make-kpkg --initrd --revision=3.4.9.custom kernel_image
At the end of the installation, it should automatically run
update-grub
, at which point you can reboot your
system.
Once the machine has rebooted and you are logged in,
type
uname -r
and if the value it spits out is 3.4.9 (or
whatever kernel version you compiled) then you are
successful!
Strap in, because the next segment will cover compiling
the latest Xen from source.
References:
Amazingly Simple Kernel Compiling Guide
Kernel Flag Database for Lookup
Performance Regarding Paravirt Spinlocks
I have great news, the latest Xen source, RC3 currently,
was the first time I have been able to compile the source
without debugging make errors since I started back in
March.
Before we proceed I should run through the fact that Xen
can be affected by:
cd ..
sudo dpkg -i linux-image-3.4.9_3.4.9.custom_amd64.deb
Hardware
Environment (Dom0 OS)
Software
Xen Source
Dependent Packages
Xen is a spiderweb of dependencies simply due to the
number of facets it has to fill, and we have control over
barely half of these major factors.
My point is primarily that you can use the same
hardware, platform, and revision I use and still encounter
brand new errors due to changes in dependent
packages downloaded during compilation. Be prepared.
The following set of packages is a bare-minimum set to
compile and run Xen successfully, and omits well over a
gigabyte of the recommended dependencies:
Note: libpci-dev replaces pciutils-dev (which routes to
said package), just an FYI.
These packages appear to be on the requirements list,
but were already on my system at this stage. If you have
not been following from the previous segment you may
want to run this for good measure;
GUI Xen Packages
For those of us using Gnome or a GUI environment, you
may want to include these packages for vnc access
and/or sdl:
sudo aptitude install libsdl-dev gvncviewer
sudo aptitude install bridge-utils build-essential libncurses-dev python-dev uuid uuid-dev libglib2.0-dev libyajl-dev bcc gcc-multilib iasl libpci-dev mercurial flex bison libaio-dev
sudo aptitude install bcc patch libncurses5-dev python python-dev libglib2.0-dev bin86 bzip2 module-init-tools make gcc libc6-dev libcurl3 iproute xz-utils
Obviously you have the choice to use only one, you do
not need both, but if you intend to use the GUI
environment in Debian I highly recommend you give
SDL a spin, it's much faster than VNC, though VNC
does offer remote access.
End GUI Xen Packages
Complete The Dependencies
The list I already provided is a minimalists list,
containing only exactly what was required to compile
and run Xen for my own purposes, which includes
PVHVM systems of Windows, FreeBSD, and Linux, as
well as PCI Passthrough, and both VNC and SDL. It has
worked for me since I started tinkering back in March,
with minor modifications since.
For those who want the whole package and have a
spare gigabyte of OS Disk space available, I have put
together this sub-section.
I should also note that some of these packages will
change flags marked as no from
./configure
to a yes,
but that did not stop me from compiling and running the
system:
zlib1g-dev
libbz2-dev
liblzo2-dev
e2fslibs-dev
ocaml
ocaml-nox
ocaml-findlib
lzma
lzma-dev
liblzma-dev
markdown
libcurl4-openssl-dev
transfig
tgif
libvncserver-dev
python-twisted
libjpeg62-dev
gawk
git-core
texinfo
texlive-latex-base
texlive-latex-recommended
texlive-fonts-extra
texlive-fonts-recommended
If anyone knows exactly what any of these
dependencies do or are used for and can shed some
light on them I would greatly appreciate it.
End Complete The Dependencies
Next we can visit xenbits to get the latest revision. In our
case we will use revision 25777, and use mercurial to
download the source:
I actually saw revision 25784 was out, but when I tried it I
got an error for a potentially uninstantiated variable,
assuming gcc compatibility bug that won't get caught for
a few more revisions.
EFI Source Modification
Given that I encountered no make errors, I was shocked
to find that they still hadn't addressed the grub efi
mkdir -p ~/src/xen && cd ~/src/xen
rev=25777
hg clone -r $rev http://xenbits.xen.org/hg/xen-unstable.hg/ xen-unstable.hg-rev-${rev}
cd xen-unstable.hg-rev-${rev}
compatibility bug where it fails to recognize available
system memory.
The fix for this is a source modification, though they have
supposedly added a build process for xen.efi, which
would replace the debian.efi generated by grub. I have
yet to find adequate instructions to make that work, and
even if I did may break adding
xen-pciback.hide
to the
grub configuration, which we need if we are planning to
do the easy method of PCI Passthrough.
So, let's open up
xen/arch/x86/setup.c
with our
favorite text editor, and find the line containing
(
e820_raw_nr != 0 )
, and make that area look like this:
Notice we are commenting out a section of code using "if
0", which always evaluates to false, to eliminate the
section of e801 mapping which breaks our ram
recognition.
I have no idea what kind of adverse affects this has on
other systems, but it has worked for me since Xen 4.1.2.
#if 0
else if ( e820_raw_nr != 0 )
{
memmap_type = "Xen-e820";
}
else if ( bootsym(lowmem_kb) )
{
memmap_type = "Xen-e801";
e820_raw[0].addr = 0;
e820_raw[0].size = bootsym(lowmem_kb) << 10;
e820_raw[0].type = E820_RAM;
e820_raw[1].addr = 0x100000;
e820_raw[1].size = bootsym(highmem_kb) << 10;
e820_raw[1].type = E820_RAM;
e820_raw_nr = 2;
}
#endif
else if ( mbi->flags & MBI_MEMMAP )
End EFI Source Modification
SSH Users
Those of us using SSH to follow along should now run
screen
so we don't loose our footing while compiling.
End SSH Users
We can now run the configuration script and compile the
source:
./configure && make -j4 world && make -j4 deb
Notice the
-j4
flag which sets concurrency for us, with
that setting I have compiled Xen ins just under 7
minutes. It will consume roughly 1GB of space once fully
compiled.
Instead of
make install
I used
make deb
to create a
debian package which we can backup for future
installations, and to simplify removal. This file will be
placed in the
dist/
folder.
At this stage if you don't see any make errors and the
debian file is in the dist folder, you have successfully
compiled Xen and are ready to move to the next phase
of installing and configuring Xen!
References:
In the previous segment we had just finished compiling
Xen to the
dist/
folder, which is where we will go now
to install it!
cd dist/
sudo dpkg -i xen-upstream-4.2.0-rc4-pre.deb
Our work is not done yet, we have a lot of post-install
configuration to take care of before we can reboot and
verify that Xen is working.
Let's start with a trip to
/boot
to delete extra symbolic
links and related files:
Next we want to have Xen boot before the Linux Dom0
kernel, Xen will load that after. To do this we need to
move the grub configuration file forward and update grub
via:
Finally, before we reboot the system we need to make
sure we initialize the Xen toolstack and related features
at boot time via:
cd /boot
sudo rm xen.gz xen-4.gz xen-4.2.gz xen-syms-4.2.0-rc4-pre
sudo mv /etc/grub.d/20_linux_xen /etc/grub.d/09_linux_xen
sudo update-grub
sudo update-rc.d xencommons defaults 20 19
sudo update-rc.d xendomains defaults 21 22
sudo update-rc.d xen-watchdog defaults 23 22
At this stage we reboot our system, and if all goes well
the first option in grub will be Xen, which will bring us to
our linux login.
Our next step is logging in and testing the xl toolstack
with
sudo xl dmesg
, if the output you receive is a not an
error, then you have successfully installed Xen.
Next we need to setup a virtual bridge for our virtual
machines going forward. Here is a simple recommended
configuration you can use to replace the contents of
/etc/network/interfaces
(See my PFSense Segment
for a more complex multi-bridge configuration):
auto lo xenbr0
iface lo inet loopback
iface eth0 inet manual
iface xenbr0 inet dhcp
bridge_ports eth0
To make this change take affect you can restart the
network service, ifconfig, or take my approach and
reboot.
Making Things Easier
As many are aware, security comes at the cost of
usability.
So this section is a way to make using Xen easier at the
slight cost of some security.
Add these lines to the end of
/etc/sudoers
to omit
entering a password when running the xl command:
# Add XL Access
%sudo ALL=(ALL:ALL) NOPASSWD: /usr/sbin/xl
If using
vi
you may have to use
:wq!
to force write quit
when saving.
Next let's add an alias so whenever we type
xl
it
assumes
sudo xl
by adding an alias to
/etc/bash.bashrc
:
# XL Alias
alias xl='sudo xl'
Now when we login next instead of using
sudo xl
and
entering a password, we can just use
xl
and no
password is necessary!
End Making Things Easier
Your Xen configuration is now primed and ready to
begin. You can move onto any of these segments at your
discretion:
Installing Windows HVM
Installing PFSense HVM
I recommend changing the table of contents for details
on managerial activities, which includes backing up and
restoring, LVM and partitioning, and a few other helpful
related activities.
References
While you are welcome to follow along, I highly
recommend you spend a few hours reviewing the Xen
Man Pages.
Our first step is preparing a Logical Volume partition for
Xen.
With Xen, this partition will be treated as a whole hard
drive, and will create a sub-partition-table and partitions
inside of it, hence you cannot simply mount the logical
volume without an offset going forward. For more
information on this, check out the managerial tasks
section of my documentation.
I recommended Samba because there are drive
limitations with Xen, if you intend to share a partition
between multiple machines mounting with write
permissions can ruin your data. Xen will allow multiple
mounts, but neither system will inform the other of
changes, hence conflicting read and write operations.
Other disk limitations include only up to 4
hdX
drives,
which are treated as Parallel ATA, and only up to 15
sdX
drives which are treated as SCSI can be passed at
any time.
We will
You will need a Windows 7 installation DVD, I
recommend a disk image copy, which we can use to
install Windows 7.
Networking is also somewhat specific, while I have not
run into a problem with the mac address myself, many
others have. Xen documentation says to prefix your
virtualized mac addresses with
00:16:3e
. We will be
using our virtual bridge to create a virtual e1000 gigabit
ethernet adapter for our DomU.
Here is my Windows 7 configuration file for installation:
Once Windows has been installed we can change
boot='dc'
to
boot='c'
, and remove the iso from the
drives.
Passthrough is an important and moderately complex
topic. First you need to understand what the BDF is,
BDF is Bus, Device, Function, commonly in the format of
BB:DD.f
, and it is how the linux system identifies and
refers to PCI devices.
To check your systems PCI device identifiers you can
use
lspci
.
Once you know the BDF of the devices you wish to
name='windows'
builder='hvm'
vcpus=4
memory=6144
disk=[
'/dev/mapper/xen-win2,,hda,w',
'/media/samba/docs/win7.iso,,hdc,r,devtype=cdrom'
]
vif=[
'bridge=xenbr0,model=e1000,mac=00:16:3e:14:b1:1c'
]
boot='dc'
pae=1
nx=1
videoram=16
stdvga=1
sdl=1
vnc=0
usb=1
usbdevice="tablet"
localtime=1
pass, you need to prevent Dom0 from using them via
pciback. By default pciback is a module, and you can
use a late-binding method to hide a device from Dom0,
but for the best experience I find a custom kernel with
pciback built-in and modified grub does the trick much
better.
In my case, I have five items, the two functions of my
graphics card, two USB 3.0 controllers, and a USB 2.0
controller. The identifiers are:
00:1d.0 (USB 2.0)
01:00.0 (Graphics Card Video Function)
01:00.1 (Graphics Card Audio Function)
04:00.0 (USB 3.0)
05:00.0 (USB 3.0)
If you have followed this guide, then you probably
compiled a custom kernel, so our next step is adding a
line to grub!
Find this section of your grub configuration in the file
/boot/grub/grub.cfg
:
The line we want to modify starts with
module
, and here
is what we add:
submenu "Xen 4.2.0-rc4-pre" {
menuentry 'Debian GNU/Linux, with Xen 4.2.0-rc4-pre and Linux 3.4.9' --class debian --class gnu-linux --class gnu --class os --class xen {
insmod part_gpt
insmod ext2
set root='(hd0,gpt2)'
search --no-floppy --fs-uuid --set=root 9b2dde6d-5888-429c-a6dd-35e7fa11f632
echo 'Loading Xen 4.2.0-rc4-pre ...'
multiboot /xen-4.2.0-rc4-pre.gz placeholder
echo 'Loading Linux 3.4.9 ...'
module /vmlinuz-3.4.9 placeholder root=/dev/mapper/xen-dom0 ro quiet
echo 'Loading initial ramdisk ...'
module /initrd.img-3.4.9
}
We will have to force write-quit to save the file (
:wq!
),
then when we reboot the system our devices will be
available for passthrough!
Next we add this to our HVM Configuration File:
pci = [
'00:1d.0',
'04:00.0',
'05:00.0',
'01:00.0',
'01:00.1'
]
Important Note for Graphics Passthrough
nVidia cards do not work out of the box, but ATI does.
However, most cards do not support FLR (Function
Level Reset), which is important because it prevents the
card from operating properly without the right steps.
Essentially, unlike a physical computer, when the virtual
machine is shut down or restarted, it does not change
the power supplied to the graphics card. This means the
graphics card is never re-initialized, so FLR was
invented. However, because most graphics cards do not
support FLR you have to mimic this effect by ejecting the
card after every reboot, otherwise you will see a severe
performance degradation.
In addition, if you do not reset the card, and it is not fresh
when you attempt to install or uninstall drivers, the
process may fail leaving your system crippled, plus a
BSOD is likely to appear.
So, my recommendation when dealing with GPU drivers
and passthrough, always reboot the entire machine, or
module /vmlinuz-3.4.9 placeholder root=/dev/mapper/xen-dom0 ro quiet xen-pciback.hide=(00:1d.0)(01:00.0)(01:00.1)(04:00.0)(05:00.0)
take extra care to reset the card before making any
changes.
Another important fact is PV on HVM drivers, windows
has GPLPV and these drivers can dramatically improve
your hard drive and network performance, among others.
After installing GPLPV drivers, you may want to add this
line to your HVM configuration:
xen_platform_pci=1
The final configuration after my devices are passed and
PVHVM Drivers are installed looks like this:
name='windows'
builder='hvm'
vcpus=4
memory=6144
disk=[
'/dev/mapper/xen-win2,,hda,w'
]
vif=[
'bridge=xenbr0,model=e1000,mac=00:16:3e:14:b1:1c'
]
pci = [
'00:1d.0',
'04:00.0',
'05:00.0',
'01:00.0',
'01:00.1'
]
boot='c'
pae=1
nx=1
videoram=16
stdvga=1
sdl=1
vnc=0
usb=1
usbdevice="tablet"
localtime=1
xen_platform_pci=1
Also, important to note that if you do not have
ntp
on
Dom0, or do not set
localtime=1
in the configuration,
windows will time drift at every boot, which can cause
severe problems. At one point my system would boot a
month out of date, and even with a scheduled time sync
it would fail to properly reset the time. Icons would be
missing from my taskbar and SSL certificates would
claim they expired. It was a scary experience.
References
There are a few tricks I can share with regards to
managing Xen virtual machines.
However, having an understanding of the fundamentals
is also helpful.
Here are the key areas:
Drive Partitioning
LVM
Using mount With LVM Virtual Partitions
dd
Drive Partitioning
This is a fundamental, you are welcome to skip ahead if
you want the how and not the why.
I find a lot of emails floating around the xen-users email
list due to misunderstandings in how Xen recognizes
and managed partitions.
It is important to understand how Xen Virtual Machines
treat partitions.
It sees them as entire hard drives, and will create a
brand new partition table and partitions inside of it.
The result is sub-partitioning, and this causes an offset
between the start of the actual partition and the start of
the sub-partition containing all of the data on the virtual
machine.
In more detail
Computers typically segment hard drives into three
components.
Partition Tables
Partitions
File Systems
A partition table tells the disk where the beginning and
end of a partition exists and usually the file system.
The Partitions are simple blocks of storage.
The File System tells the operating system how to
access the partition.
A normal installation of Windows for example, will look
like:
Drive Block 0
Partition Table:
Starts at Block 256 File System NTFS
Drive Block 256
Partition
Data
LVM
Linux has LVM handle all the tough parts of managing
the positions of data blocks, and makes everything else
easier for the user.
Hence it is called the Logical Volume Manager, because
it manages logical volumes.
What this means is when we create a LVM Partition, we
are actually using an LVM "File System", which looks
like this:
Drive Block 0
Partition Table:
Starts at Block 256 File System LVM
Drive Block 256
Partition
Data
We can then create partitions using LVM which
automatically handles the location of the data inside the
LVM Partition.
For example, we can create two LVM Partitions for a
Windows and Linux Virtual Machine, which would look
like this:
Drive Block 0
Partition Table:
Starts at Block 256 File System LVM
Drive Block 256
Partition
Data
LV Windows
LV Linux
Only an operating system that recognizes LVM can see
"LV Windows" or "LV Linux" though, so if you plug that
drive into a basic Windows machine, it won't be able to
do anything with the data.
'''Finally we have Xen'''
With Xen we give one of those partitions to our virtual
machine, but as stated it sees that as a hard drive.
If we use the previous drive configuration and just add a
Windows Virtual Machine to
LV Windows
our
configuration looks like this:
Drive Block 0
Partition Table:
Starts at Block 256 File System LVM
Drive Block 256
Partition
Data
LV Windows
LV Block 0
Partition Table
Partition at Block
256 File System
NTFS
LV Block 256
Partition
Data
LV Linux
At this point, we cannot simple access the data on our
Xen partition because it has been sub-partitioned.
If you try simply using
mount
it will fail, because it
requires the offset to the start of the NTFS sub-partition!
Using mount With LVM Virtual
Partitions
To get the offset of the partition, we will use a tool like
fsdisk
or
parted
:
Using
unit B
we indicate bytes which will be used by
the mount command, and we get this:
Important to note that Windows 7 creates a 100MB Boot
Partition, so the first item with the boot flag is actually the
boot partition, we want the
Start
value of the second,
which is
105906176B
.
We can now mount the partition using:
That is how we can access the data inside our sub-
partitions.
Obviously it is ill-advised to access the data while the
operating system is running, mounted disks will NOT
inform the running platform of changes, so data can get
overwritten and cause all kinds of problems.
dd
The utility
dd
is probably one of my favorites among the
sudo parted /dev/mapper/xen-windows unit B print
Number Start End Size Type File system Flags
1 1048576B 105906175B 104857600B primary ntfs boot
2 105906176B 171796594687B 171690688512B primary ntfs
sudo mount -o offset=105906176 /dev/mapper/xen-windows /path/to/mount/dir
tools that come with most linux/unix systems.
It can be used to create an exact copy of data, and is not
subject to a number of limitations with higher level
software.
It is not the fastest utility, but supplemented with tools
like gzip you can compress a complete backup into a
size similar to that you might see with professional
software.
Here is a backup Example that compressed an 80GB
Windows LV with 70GB of Data into a 43GB GZipped
Image File:
Alternative one-liner (Takes a lot longer since it has to
compress "on the fly"):
I can also use this command to restore my system from
that compressed image:
Once again, one-liner alternative:
There are plenty of other uses I have found for the
dd
command, but backups and restoration make Linux
amazingly easy to restore after terrible failure, and
believe me in writing this guide I encountered that plenty
of times.
sudo dd if=/dev/mapper/xen-windows of=/path/to/backup/windows.img bs=1M
sudo gzip -9 windows.img
sudo dd if=/dev/mapper/xen-windows bs=1M | sudo gzip -9 > /path/to/backup/windows.img.gz
sudo gzip -d /path/to/backup/windows.img.gz
sudo dd if=/path/to/backup/windows.img of=/dev/mapper/xen-windows bs=1M
sudo gzip -dc /path/to/backup/windows.img.gz | sudo dd of=/dev/mapper/xen-windows bs=1M
You've reached the end of my guide!
Thanks for taking your time to read through my
documentation, if you found it at all helpful please feel
free to shoot me an email on the Xen Users email list.
Any feedback or corrections are also welcomed.
I am still working on the PFSense segment and will have
that up as soon as I can.
Personal Configuration
Data
This section contains documentation on my own
personal configurations and is not fit for third-party
consumption.
Configurations for these components will vary by user
and system.
SSH Configuration
For my personal SSH configuration, I like to change the
default port from 22 to something different. This helps
with security and eliminates conflicting port-forwards for
external mapping.
To do this we simply need to open up
/etc/ssh/sshd_config
and change the 22 in
port 22
to another number.
After that you need to restart
ssh
with
sudo service ssh
restart
.
Samba Configuration
My Samba configuration consists of three 1TB Hard
drives setup with LVM and multiple partitions for an
organized file storage system.
Samba configuration consists of:
Preparing mount points for each partitions
Setting up automatic drive mounting at boot time
Adding a samba user and changing permissions
on said mount points
As of this point, Samba should already be installed, and
a group called
sambashare
should exist on the system.
Our first objective will be adding our user to that group,
and adding our user to the samba users with:
sudo usermod -aG sambashare cdelorme
sudo smbpasswd -a cdelorme
When prompted for a password, this is the new
password you will be able to use to access shares over
the network, and can be different than your login
password.
Note: We must logout and back in before the system
recognizes us as part of the
sambashare
group.
Next we need to prepare a mount point where all of our
shared folders can sit, and by default the system uses
/media
so I will create
/media/samba
:
cd /media
sudo mkdir samba
Currently I have two volume groups named
media
and
data
, and a set of volume groups for each, my goal will
be to create a set of (optionally matching) directories for
each as a mounting point:
/dev/mapper/data-backups
/dev/mapper/data-docs
/dev/mapper/data-software
/dev/mapper/media-games
/dev/mapper/media-multi
/dev/mapper/media-music
So next I run the following command to create
appropriate folders:
Now we need to think about mounting the partitions, but
we need this to happen automatically at boot time, so we
turn to the
/etc/fstab
file.
This file is how we can add new partitions to mount at
boot time, and we have a decision to make.
Drives can be mounted by UUID or root path (eg.
/dev/mapper/...
). I go with UUID, each approach has
its own pros and cons.
To get the UUID of each of my partitions I use this
command:
sudo blkid /dev/mapper/VolumeGroup-VolumeName
Note: No UUID will be provided if a partition has not
been given a file system.
Here is what my
/etc/fstab
file contains:
cd /media/samba
sudo mkdir backups docs software games multi music
# /etc/fstab: static file system information.
#
For simplicity I reboot the system instead of running half
a dozen mount commands, which also verifies my fstab
file entries are valid.
Now for one last step we just need to set permissions on
our share folders and files:
sudo chown -R cdelorme:sambashare /media/samba/
Next we can finally configure Samba itself to share our
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc defaults 0 0
# Originals
#/dev/mapper/xen-dom0 / ext4 defaults,noatime,discard,errors=remount-ro 0 1
#/dev/mapper/xen-home /home ext4 defaults,noatime,discard,errors=remount-ro 0 2
#/dev/mapper/xen-swap /swap swap sw 0 0
# Revised UUID Listing
UUID=da46053a-c733-4505-ab87-3f510b08e338 /boot ext4 defaults,noatime,discard,errors=remount-ro 0 2
UUID=AB5A-DB78 /boot/efi vfat utf8 0 0
UUID=dfcc1ff9-8f21-44d0-b224-447c66de7dc0 / ext4 defaults,noatime,discard,errors=remount-ro 0 1
UUID=15833a6d-488b-46ac-adfc-e1db4929d096 /home ext4 defaults,noatime,discard,errors=remount-ro 0 2
UUID=d1e67556-cf2e-4166-81b2-b9c8da10af06 none swap sw 0 0
# Samba Shared LVM Drives
UUID=0bbfb482-ca39-43c5-a1f3-39749c9dadaa /media/samba/backups ext4 defaults,errors=remount-ro 0 0
UUID=30097f44-5978-4fc4-b5cf-663dca259363 /media/samba/docs ext4 defaults,errors=remount-ro 0 0
UUID=4fe0435a-7041-402e-abb3-708e258d39cd /media/samba/software ext4 defaults,errors=remount-ro 0 0
UUID=6c612d46-f0b9-405f-8a62-f77bdc266237 /media/samba/music ext4 defaults,errors=remount-ro 0 0
UUID=3f9f7128-d004-4b1a-81d7-e752631127a1 /media/samba/multi ext4 defaults,errors=remount-ro 0 0
UUID=b56f7d04-6037-4bfe-854e-5f7ee9a7da4a /media/samba/games ext4 defaults,errors=remount-ro 0 0
# DVDRom
/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0
folders, which is a whole different ballgame of its own.
My documentation is scarce, because the default samba
configuration file (located at
/etc/samba/smb.conf
) had
terrible default settings that shared my entire home
folder, so I had to gut it, and after much online reading
and a bit of testing I ended up with this simplified
version:
#======================= Global Settings =======================
[global]
workgroup = WORKGROUP
server string = %h server
dns proxy = no
log file = /var/log/samba/log.%m
max log size = 1000
syslog = 0
panic action = /usr/share/samba/panic-action %d
encrypt passwords = true
passdb backend = tdbsam
obey pam restrictions = yes
unix password sync = yes
passwd program = /usr/bin/passwd %u
passwd chat = *Enter\snew\s*\spassword:* %n\n *Retype\snew\s*\spassword:* %n\n *password\supdated\ssuccessfully* .
pam password change = yes
map to guest = bad user
usershare allow guests = yes
#======================= Share Definitions =======================
[Backups]
create mask = 660
directory mask = 770
path = /media/samba/backups
write list = cdelorme
[Docs]
create mask = 660
directory mask = 770
path = /media/samba/docs
write list = cdelorme
[Software]
create mask = 660
directory mask = 770
path = /media/samba/software
Note: For write and read lists, to specify a group, prefix
with
@
, otherwise use a username. The create and
directory masks are just defaults I found in many
examples.
After that file is saved I simply restart samba with
sudo
service samba restart
and voila everytime the system
boots it will automatically share my folders, which can
now be accessed over the network by any operating
system and with excellent speeds.
I have measured speeds in excess of 100MB over the
network from Xen virtual machines, without interruptions
to other running network services.
References
Samba Config
Input Reference
Keyboard Reference
read list = @sambashare
write list = cdelorme
[Music]
create mask = 660
directory mask = 770
path = /media/samba/music
write list = @sambashare
[Games]
create mask = 660
directory mask = 770
path = /media/samba/games
read list = @sambashare
write list = cdelorme
[Multi]
create mask = 660
directory mask = 770
path = /media/samba/multi
read list = @sambashare
write list = cdelorme