Home | Download/Install | Documentation | Packages | Screenshots | News | Forum/Mailing-lists | Contact | GForge

Virtual Plants Continuous Integration

The status can be seen here on http://vp-continuous.cirad.fr

Continuous integration for all of VirtualPlant's projects is handled by BuildBot. Buildbot is structured in a master and slaves way.

  • Master : Decides which tasks (aka builder) is to be executed by whom (aka slave) and when (aka sheduler)
  • Slave : Only in charge of registering itself to the master and then executing the tasks the master sends to it.

The slave can by installed on any system supported python. This way we can use it to run compilation/tests/… on many different systems.

Slaves

The slave can be any machine that can see the master through a network connection, and any OS that supports Python, on a physical on a virtual machine.

Creating the daemon

On the machine hosting the slave:

buildbot create-slave basedir host:port slave passwd.

basedir is the working dir of the slave. It's usually something like ~/Devlp/buildbot/slave. Sometimes Devlp is replaced by Devel and buildbot is omitted as the machine (or user account hosting the slave) is dedicated to this task.
host, port, slave and passwd are given by the buildbot administrator. All this data is in vplants/vplants/devel/continuous_integration/master/common.py. If you need to add a slave, edit this file! Put sensible names and passwords please!

Starting the slaves

The command is the same as for the server except we might not have virtual python environments.

cd Devlp/slave
buildbot [re]start .

Starting slaves as daemons

It is a good idea to automatically start the buildbot slave when the system starts up. The following sections show how to setup boot scripts for various systems. Be careful with the paths, fix them as needed!

Ubuntu and Fedora (with Upstart)

Create the /etc/init/buildbot_slave.conf file:

sudo vi /etc/init/buildbot_slave.conf

Then type in:

# We want the slave to start after ''network-interface'' 
# is started (so that it can contact the master).
start on started network-interface
 
# Print errors to the console
console output
 
# start-stop-daemon will ''start'' the executable given by ''--exec''  
# as a background (''-b'') service with the identity of the 
# user given by ''--chuid''
script
	start-stop-daemon --start --chuid buildmaster -b --exec /usr/bin/buildbot -- restart ~/devlp/slave/
end script

Windows

About the virtual machines

We have a build server running VirtualBoxed systems called Mango. This section will not present how to install an OS on a VB but how to make easier to use as a build system.

To avoid side-effects, Mango itself is not a build slave.

Automatically starting the virtual machines upon Host (eg: Mango) startup

Like we did here, we will extend or create the /etc/init/vplants_vboxes.conf file.

# Start one the network in running
start on started network-interface
 
# print things to console
console output
 
# Start the virtual machines as headless (no gui) background services.
# Each VM will watch a different host port for RDP connections.
script
        start-stop-daemon --start --chuid buildmaster -b --exec /usr/bin/VBoxHeadless -- -startvm "FC15-i686" -v on -e "TCP/Ports"=3389
 
        start-stop-daemon --start --chuid buildmaster -b --exec /usr/bin/VBoxHeadless -- -startvm "Ubuntu-11.04x64" -v on -e "TCP/Ports"=3390
 
        start-stop-daemon --start --chuid buildmaster -b --exec /usr/bin/VBoxHeadless -- -startvm "win7_32" -v on -e "TCP/Ports"=3391
end script
You need to open incoming TCP traffic on the host for the ports that are given as “TCP/Ports” argument.

Remote access

RDP

Since the machines are started user VBoxHeadless, they are accessible using rdesktop on linux or Remote Desktop Connxion (Connexion Bureau à distance) on Windows.

In the following examples, PORT refers to the port numbers given to the machines in the previous section.

For Linux

rdesktop mango.cirad.fr:PORT

For Windows

Start>>Programs>>Accessories>>Remote Desktop Connection
# Now type mango.cirad.fr:PORT in the server (or computer) field.
SSH

You can also setup ssh forwarding from host to guest like on this blog post. If you set the forwarding from host port PORT to UbuntuXX guest and want an ssh shell to UbuntuXX:

ssh username@mango.cirad.fr:PORT''

Windows can support SSH using freesshd or CYGWIN (quick tuto).

Master

Creating the daemon

You probably don't need to do this as the server is already running! Check http://vp-continuous.cirad.fr. If the page loads then everything is fine. If it doesn't that means the server died. In that case, you don't need to recreate it : just identify the problem, fix it and restart the server.
For VirtualPlant's projects we already have a set of preconfigured buildbot configuration scripts in the vplants/vplants/devel/continuous_integration/master gforge repository. For extensive documentation about buildbot deamons look at the buildbot installation documentation.

svn co –username svnuser –password svnpass https://scm.gforge.inria.fr/svn/vplants/vplants/devel/continuous_integration/master basedir.

svnuser and svnpass are given is the common.py file in that same repository. You need to be a developer to see it. basedir is the working dir for the master, eg: ”/home/buildmaster/continuous_integration/master”.

About the master's installation

The master is installed on vp-continuous.cirad.fr. Use the master login and password to log onto the machine via ssh.

The host OS is CentOs 5. The provided python is too old to make BuildBot run confortably so Python 2.6 is compiled (in ~/Devlp/Python-2.6) and the master is running in a virtualenv.

Starting the master server

Traditionally web clients send requests to servers' port 80. However, this port requires root privileges and it is NOT recommended to run Buildbot master as root. Instead we run Buildbot as a normal user and it will listen to port 8080 and we forward port 80 to 8080 using iptables.

Do NOT run buildbot as sudo.

cd Devlp/buildbot
source bin/activate 
cd master
buildbot [re]start .

Installing missing packages on the server

Missing Python packages can be installed using easy_install in the virtual environment.

cd Devlp/buildbot
source bin/activate 
easy_install <package>

Server configuration

As stated above the server configuration is stored on our VPlants subversion repository in vplants/vplants/devel/continuous_integration/master/common.py. You can use a cron task to update the server configuration from the svn at regular intervals (crontab -e).

You can force an update like this:

cd Devlp/buildbot
source bin/activate 
cd master
svn up
sudo buildbot restart .
Configuration files

A typical buildbot installation has a master.cfg file which describes the how schedulers/repositories/slaves/… are coordinated. It is actually a python file with a few special structures. For our usage we have created another python file named common.py which is imported by master.cfg and declares the exact same thing but in a more declarative way (This file is inline documented). If you need to modify the server behaviour, first see if you can't do the change in common.py. If you're sure you can't then modify master.cfg.

Server status

To know what's going on with the server and see if it complains about anything have a look at the Devlp/buildbot/master/twistd.log file. Use the tail -n 50 twistd.log command to what the 50 latest logs.

 
documentation/tools/continuous_integration.txt · Last modified: 2011/12/13 11:33 by user   Back to top
INRIA   INRA     CIRAD     AGROPOLIS
INRIA GForge RSS feed Valid XHTML 1.0 Valid CSS Driven by DokuWiki