[Pybots] Experiences on Setting up the Win32 slave

Sidnei da Silva sidnei at enfoldsystems.com
Wed Sep 27 07:28:26 PDT 2006


Since Grig asked, I'm writing this up.

Setting up the Win32 slave wasn't as hard as I expected. We have Win32
slaves running at Enfold Systems [1] for building our products in various
different setups. The trick though is that Mark Hammond set those up
so I had no idea how much of an effort it would be.

In fact, the whole process involved touching several things that I had
never touched before. And even then it wasn't so hard.

So, it all started by setting up a Virtual Machine. Since I had a
couple VMs already running on my development box this was a
breeze.

- I've installed Windows Server 2003 Enterprise Edition from the
  MSDN DVDs.

- Then the Python 2.4.3 MSI installer, pywin32 2.0.9, Twisted
  2.2.0 using the installer as well.

- buildbot 0.7.4 tarball. Extract, run setup.py install.

- Inside the buildbot tarball there's a contrib\windows
  directory. There's a script to run the buildbot as a service
  there. You run the script and it gives you the options. I've setup a
  separate 'buildbot' user for running the service.

- At this point I decided to move the VM out of my development machine
  at home into a dedicated machine at the office (2 blocks from my
  house). That was the most troublesome part, because I've tried
  copying the VM to a DVD but the DVD drive at the office wouldn't
  work. So I've ran back home and copied it into my laptop instead.

- Next step was booting the old machine we had there, that was turned
  off for months. It was running a mix of Ubuntu Dapper and Edgy so I
  updated to latest Edgy. That's where things got worse. Accidentaly
  I've removed the 'upstart' package. Ups, no /sbin/init in root
  drive. :) To make it worse, the only rescue floppy was damaged. And
  the CD drive wouldn't work. Well, almost. It worked once I've put it
  in *vertical* position. Odd.

- Machine rescued, next step installing VMware Server. On
  Linux. That's something I had never done either. Fortunately it was
  not hard. I had to install the kernel-headers package for the
  current running kernel. Other than that it just worked. I couldn't
  start the vmware-server-console though. It would just sit there at
  100% CPU usage and never show up. Quick searching in the VMware KB
  revealed a workaround. Setting some LD_PRELOAD variable. Blergh. Ok,
  now it works.

- I've also installed the Web Management Console or whatever that is
  called. It comes with it's own version of Apache, and depended on
  bdb3 or something. I had to find out the right package, I think it
  was libdb3-compat or something. But it wouldn't start. I looked at
  the init script, and set the DEBUG option. It was complaining that
  'trap SIGHUP' was not valid or something. Took out the 'trap', and
  then it started.

- Last step was to enable 'remote desktop' connection on the Win32
  VM. This so I could connect to it from home. Of course it's not
  available externally. I use ssh port-forwarding to port
  3389.

- Grig then setup Win32-specific steps for this slave, by copying the
  example from the official buildbot. That was the greatest
  win. Because a Python checkout already has the .bat files for
  build/test/clean inside Tools\buildbot I didn't had to do anything
  extra. Well almost.

- Initially I tried using the Visual C++ Toolkit 2003. That wasn't
  enough though. The .bat scripts from Python expected to find
  'devenv' which is not available with the Visual C++ Toolkit 2003. So
  I had to install Visual Studio .NET 2003.

- I had a copy of it, also from the MSDN DVDs, but for some reason it
  wouldn't install. It would ask for the original DVD mid-way through
  the installation. A quick search in the MSDN documentation revealed
  that installing from a common share is common practice and the steps
  involved were very simple. Copy the VS directory to a folder, edit a
  .ini file. Bang. It works.

- Next I had to figure out a couple of batch file trickery. The build
  step for testing external applications calls
  '..\..\python-tool\test.bat'. So that means it's not inside
  'python-tool' when it runs the script. So that means I have to get
  the current dir and the script dir to do something useful.

- Turns out that wasn't so hard. We have to admit, when it comes to
  documentation Microsoft is light years ahead. Everything is so much
  easier to find, with examples and all most of the time. Getting the
  current directory: '%CD%'. Getting the parent directory of the
  current script from inside: '%~pd0'.

- I've then downloaded the dependencies for compiling lxml. This is
  clearly described [2] fortunately. Deps in place, I've got the lxml
  1.1.1 tarball, compiled python trunk, tried running 'setup.py
  build_ext -i'. Ops, needs path to the includes. Add a couple '-I
  ..\this\that\theother\include'. Almost works. Needs to link against
  'python26.lib'. But that doesn't exist. Since I'm running setup.py
  with a debug build of python. I ask Mark Hammond and he tells me to
  add '--debug' after 'setup.py'. And that does the trick. Almost. Now
  it's complaining about symbols. It needs to link against libxml2's &
  friends. There's a place to put that in 'setup.cfg'. But I don't
  want to change that. So I look for command-line options for
  setup.py. None of them works. I ask Mark again. 'Put them in the
  LINK environment variable'. Sure that works.

- At this point it failed with missing symbol 'WSAgetLastError' or
  similar. Quick googling. It's about WinSock. The lib is
  'Ws2_32.lib'. Append that to 'LINK' and now it works.

- With everything finally built, I thought. Now it's just running the
  tests. Wrong. Ran 'test.py'. Bang. Negative refcount on a
  tuple. Fatal error. Open debugger or continue? I open the
  debugger. Mark helps me trying to find the culprit in the stack
  trace. It seems to have something to do with Pyrex. At this point I
  gave up on test.py. Running selftest.py and selftest2.py worked so I
  left test.py disabled. I will get the full stack trace and mail it
  to lxml-dev soon.

That was it. I hope you enjoyed the ride. And didn't get bored. Next
thing I'm going to try setting up some other test runners, like
sqlobject and cherrypy. And avoid C extensions where I can :)

[1] http://buildbot.enfoldsystems.com/
[2] http://codespeak.net/lxml/build.html

-- 
Sidnei da Silva
Enfold Systems                http://enfoldsystems.com
Fax +1 832 201 8856     Office +1 713 942 2377 Ext 214



More information about the Pybots mailing list