Ganeti is a cluster virtual server management software tool built on top of existing virtualization technologies such as Xen or KVM and other Open Source software. It is written by Google, and is available from the Google code site.
After googling for a while, I found a japanese site, http://jfut.integ.jp/linux/ganeti, which had source RPMs for Redhat 5 & 6, as well as written instructions in english. These instructions, while mostly complete, still assumes that the user do some tasks without being told.
During my work, I had to install Ganeti many times, and ended up writing a script to do the entire install, starting with a minimal or base desktop install on Scientific Linux 6. While not tested, the script should work on Redhat 6 as well. My script incorporates the instructions from jfut.integ.jp, as well as other steps and some code of my own to setup the bridges. The only other requirement is access to the ‘net.
All files are available here, as well as the original Japan locations:
bridge.sh (1.5 KiB, 887 hits)
calculate-address (1,013 bytes, 1,019 hits)
ganeti-2.4.2-1.el6.noarch.rpm (1.1 MiB, 492 hits)
ganeti-instance-debootstrap-0.7-2.el5.noarch.rpm (19.0 KiB, 521 hits)
|ganeti install script (updated 6/20/11)||
ganeti.sh (32.1 KiB, 852 hits)
Using the script
I like using both the screen program and Free-NX. If they aren’t installed my script asks if you want them installed.
In all questions, if there are answers in parentheses, pressing <return> will enter the default answer, which will be inside the parentheses. If there are two choices in side the parentheses, the default will be in caps.
The first two questions will be about the screen program and Free-NX, answer as you like, they are not needed for the install.
If you are using Scientific Linux, you will be asked about the Fastbug repo. I usually answer yes; again, this is not required.
After a number of RPMs are installed the following questions are asked:
Enter primary network interface (eth0):
The following question does not have a default answer. You must answer either M or S (case is not important):
Is this the Master server or a Slave node (M/S):
The hostname is obtained from the system, and presented here for confirmation:
Enter hostname (localhost.localdomain):
Enter cluster name:
Enter cluster address (must be unused):
Enter cluster netmask (255.255.255.0):
The cluster network is derived from the address and netmask, and presented here for confirmation:
Enter cluster network (10.50.10):
The default prefix for the node name is: “node”:
Enter node name prefix (node):
Enter node starting address:
Enter number of nodes you want:
The default prefix for the instances is: “instance”:
Enter instance name prefix (instance):
Enter instance starting address:
Enter number of instances you want:
Some more modules are installed, then:
Enter device partition for the LVM partition (ie: /dev/sda4, press <CR> to automatically find the LVM partition):
If the volume group is already created, you will get the following error:
Can't initialize physical volume "/dev/sda4" of volume group "vmvg" without -ff Error running: pvcreate /dev/sda4 Please fix the problem and try again Press <CR> to retry, or C to continue with the next step:
Enter volume group name, must be identical on all nodes (vmvg):
Ganeti will be downloaded and installed now.
There is a bug/incompatibility with the certificates which Ganeti generates which causes an error at this point. The error message will be:
Failure: command execution error: Node daemon on dev-ha1.regcolo.com didn't answer queries within 10.0 seconds Rc: 1 Do you need to fix the error now (Y/n):
Press <return> to have the script regenerate the certificates automatically. Assuming you do, you will be presented with the following query, two times. Just press <return> when asked:
Enter PEM pass phrase:
Finally, the last question will be about doing a full system update:
Would you like to do a full system update (Y/n):
At the end, the system is rebooted.
Manual Steps, if you like doing it yourself
Start off by doing a clean install of Scientific Linux 6. Make the OS partition between 20 and 40 gig in size, swap as needed, and create an LVM partition where the virtual systems will reside. Do not initialize the LVM partition, just create it and mark it as LVM.
Install the repos for epel, atrpms, elrepo and rpmforge. You can use this command:
yum install epel-release atrpms-repo elrepo-release rpmforge-release
If you want to use my scripts to create the bridges, you will also need to install ksh, since one of the scripts is currently written in ksh instead of bash:
yum install ksh
Download and install the following two scripts in /usr/local/sbin:
Turn off iptables:
chkconfig iptables off
and add the following to /etc/rc.local:
iptables --flush /usr/local/sbin/bridge.sh br0 eth0
Don’t forget to make the scripts executable:
chmod +x /usr/local/sbin/bridge.sh /usr/local/sbin/calculate-address
yum --enablerepo=elrepo install drbd kmod-drbd83
Make sure that a root .ssh directory. You can do this either by making the directory, or just sshing to another system.
Add a drbd.conf file to /etc/modprobe.d as follows:
echo "options drbd minor_count=128 usermode_helper=/bin/true" >/etc/modprobe.d/drbd.conf
install kvm python-virtinst
and some other packages:
yum --enablerepo=elrepo install kmod-drbd83
yum --enablerepo=epel install pyOpenSSL python-simplejson pyparsing python-inotify python-ctypes python-pycurl python-paramiko debootstrap socat
The volume group is required to be at least 20GiB.
If you haven’t configured your LVM volume group at install time you need to do it before trying to initialize the Ganeti cluster. This is done by formatting the devices/partitions you want to use for it and then adding them to the relevant volume group::
pvcreate /dev/sda3 vgcreate vmvg /dev/sda3 or: pvcreate /dev/sdb1 pvcreate /dev/sdc1 vgcreate vmvg /dev/sdb1 /dev/sdc1
Finally, download and install Ganeti:
wget http://jfut.integ.jp/linux/ganeti/noarch/ganeti-2.4.2-1.el6.noarch.rpm wget http://jfut.integ.jp/linux/ganeti/noarch/ganeti-instance-debootstrap-0.7-2.el5.noarch.rpm rpm -ivh ganeti-2.4.2-1.el6.noarch.rpm ganeti-instance-debootstrap-0.7-2.el5.noarch.rpm
At this point, Ganeti is installed. The final step is to fix an SSL error by regenerating the certificates:
openssl req -new -x509 -days 1825 -keyout server-key.pem -out server-cert.pem
openssl rsa -in server-key.pem -out server-key-nopass.pem
cat server-key-nopass.pem server-cert.pem > /var/lib/ganeti/server.pem
openssl req -new -x509 -days 1825 -keyout rapi-key.pem -out rapi-cert.pem
openssl rsa -in rapi-key.pem -out rapi-key-nopass.pem
cat rapi-key-nopass.pem rapi-cert.pem > /var/lib/ganeti/rapi.pem
and finally start Ganeti:
service ganeti start gnt-cluster modify --hypervisor-parameters kvm:kernel_path=