[Pybots] Experiences on Setting up the Win32 slave
Sidnei da Silva
sidnei at enfoldsystems.com
Wed Sep 27 07:28:26 PDT 2006
Since Grig asked, I'm writing this up.
Setting up the Win32 slave wasn't as hard as I expected. We have Win32
slaves running at Enfold Systems  for building our products in various
different setups. The trick though is that Mark Hammond set those up
so I had no idea how much of an effort it would be.
In fact, the whole process involved touching several things that I had
never touched before. And even then it wasn't so hard.
So, it all started by setting up a Virtual Machine. Since I had a
couple VMs already running on my development box this was a
- I've installed Windows Server 2003 Enterprise Edition from the
- Then the Python 2.4.3 MSI installer, pywin32 2.0.9, Twisted
2.2.0 using the installer as well.
- buildbot 0.7.4 tarball. Extract, run setup.py install.
- Inside the buildbot tarball there's a contrib\windows
directory. There's a script to run the buildbot as a service
there. You run the script and it gives you the options. I've setup a
separate 'buildbot' user for running the service.
- At this point I decided to move the VM out of my development machine
at home into a dedicated machine at the office (2 blocks from my
house). That was the most troublesome part, because I've tried
copying the VM to a DVD but the DVD drive at the office wouldn't
work. So I've ran back home and copied it into my laptop instead.
- Next step was booting the old machine we had there, that was turned
off for months. It was running a mix of Ubuntu Dapper and Edgy so I
updated to latest Edgy. That's where things got worse. Accidentaly
I've removed the 'upstart' package. Ups, no /sbin/init in root
drive. :) To make it worse, the only rescue floppy was damaged. And
the CD drive wouldn't work. Well, almost. It worked once I've put it
in *vertical* position. Odd.
- Machine rescued, next step installing VMware Server. On
Linux. That's something I had never done either. Fortunately it was
not hard. I had to install the kernel-headers package for the
current running kernel. Other than that it just worked. I couldn't
start the vmware-server-console though. It would just sit there at
100% CPU usage and never show up. Quick searching in the VMware KB
revealed a workaround. Setting some LD_PRELOAD variable. Blergh. Ok,
now it works.
- I've also installed the Web Management Console or whatever that is
called. It comes with it's own version of Apache, and depended on
bdb3 or something. I had to find out the right package, I think it
was libdb3-compat or something. But it wouldn't start. I looked at
the init script, and set the DEBUG option. It was complaining that
'trap SIGHUP' was not valid or something. Took out the 'trap', and
then it started.
- Last step was to enable 'remote desktop' connection on the Win32
VM. This so I could connect to it from home. Of course it's not
available externally. I use ssh port-forwarding to port
- Grig then setup Win32-specific steps for this slave, by copying the
example from the official buildbot. That was the greatest
win. Because a Python checkout already has the .bat files for
build/test/clean inside Tools\buildbot I didn't had to do anything
extra. Well almost.
- Initially I tried using the Visual C++ Toolkit 2003. That wasn't
enough though. The .bat scripts from Python expected to find
'devenv' which is not available with the Visual C++ Toolkit 2003. So
I had to install Visual Studio .NET 2003.
- I had a copy of it, also from the MSDN DVDs, but for some reason it
wouldn't install. It would ask for the original DVD mid-way through
the installation. A quick search in the MSDN documentation revealed
that installing from a common share is common practice and the steps
involved were very simple. Copy the VS directory to a folder, edit a
.ini file. Bang. It works.
- Next I had to figure out a couple of batch file trickery. The build
step for testing external applications calls
'..\..\python-tool\test.bat'. So that means it's not inside
'python-tool' when it runs the script. So that means I have to get
the current dir and the script dir to do something useful.
- Turns out that wasn't so hard. We have to admit, when it comes to
documentation Microsoft is light years ahead. Everything is so much
easier to find, with examples and all most of the time. Getting the
current directory: '%CD%'. Getting the parent directory of the
current script from inside: '%~pd0'.
- I've then downloaded the dependencies for compiling lxml. This is
clearly described  fortunately. Deps in place, I've got the lxml
1.1.1 tarball, compiled python trunk, tried running 'setup.py
build_ext -i'. Ops, needs path to the includes. Add a couple '-I
..\this\that\theother\include'. Almost works. Needs to link against
'python26.lib'. But that doesn't exist. Since I'm running setup.py
with a debug build of python. I ask Mark Hammond and he tells me to
add '--debug' after 'setup.py'. And that does the trick. Almost. Now
it's complaining about symbols. It needs to link against libxml2's &
friends. There's a place to put that in 'setup.cfg'. But I don't
want to change that. So I look for command-line options for
setup.py. None of them works. I ask Mark again. 'Put them in the
LINK environment variable'. Sure that works.
- At this point it failed with missing symbol 'WSAgetLastError' or
similar. Quick googling. It's about WinSock. The lib is
'Ws2_32.lib'. Append that to 'LINK' and now it works.
- With everything finally built, I thought. Now it's just running the
tests. Wrong. Ran 'test.py'. Bang. Negative refcount on a
tuple. Fatal error. Open debugger or continue? I open the
debugger. Mark helps me trying to find the culprit in the stack
trace. It seems to have something to do with Pyrex. At this point I
gave up on test.py. Running selftest.py and selftest2.py worked so I
left test.py disabled. I will get the full stack trace and mail it
to lxml-dev soon.
That was it. I hope you enjoyed the ride. And didn't get bored. Next
thing I'm going to try setting up some other test runners, like
sqlobject and cherrypy. And avoid C extensions where I can :)
Sidnei da Silva
Enfold Systems http://enfoldsystems.com
Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214
More information about the Pybots