Neil Hopcroft

A digital misfit

Octave Continuous Integration build

The nature of the Octave project is that it is a large project with lots of dependencies, this is one of the aspects that attracts me to it as a project to consider contributing toward. This does, however, mean that building it is quite complicated.
My aim was to set up a CI build in my Jenkins server, running on the AWS cloud, to launch a build whenever there was a code change. There are a few challenges here, firstly not all of the dependencies are available within the AWS yum repository, secondly not all of the dependencies are correctly checked by the configure script, leaving unexpected build failures, finally the versions of those things available with the yum repository are not necessarily recent enough to satisfy the dependencies.
My approach to missing yum dependencies has been to obtain the source code for the missing items and build and install them as required, arguably it would have been better to find a more complete yum repository, and perhaps that is what I would do next time – I didn’t know the extent of the missing dependencies when I set out.
As for the unexpected build failures, there isn’t really much you can do except to fix them as you encounter them. There is probably some scope for updating the configure dependencies to better reflect the actual dependencies, but that is beyond what I am aiming to do here.
The build itself is quite simple and takes the form:

./bootstrap
./configure
make
make check
make install

The process I used was to get the build steps into Jenkins as early as possible, then run the builds there – that way there is a repeatable process and a good way of recording build outcomes. The following are some of the dependencies I encountered while getting the build set up.
Things available in the AWS yum repository:

  • gperf
  • flex
  • bison
  • gnuplot
  • gcc-gfortran
  • libpcre
  • fftw3
  • glpk
  • curl
  • blas
  • lapack
  • suitesparse
  • ncurses
  • readline
  • freetype

Things built from source:

  • icotool
  • makeinfo
  • texi2dvi
  • librsvg
  • qhull
  • hdf5
  • arpack
  • mesa
  • qt
  • qrupdate
  • fltk

For most of these packages the default installation path was /usr/local, which highlighted some configuration problems in the AWS system I was using, the newly built version weren’t being picked up by pkg-config and the libraries weren’t being found. I hacked around these problems by copying the .pc files to a pkgconfig directory on the path and specifying /usr/local/lib/ as the LD_LIBRARY_PATH when launching the builds.


Maven Continuous Integration builds

Jenkins CI includes Maven integration, to take advantage of it I have added a main Maven pom.xml build file. To build a set of projects add them as modules and select ‘pom’ packaging. The Maven build then resolves the dependencies among the included projects and builds them in the appropriate order, publishing the results to the repository as they are built.

When setting up a CI build with deployment goals I publish them to the SNAPSHOT repository, which automates handling of version identifiers for deployed items without overwriting existing versions. This will need a little more thinking to make real releases.

These builds didn’t pick up the JUnit tests I had liberally scattered through the source code, but a quick reorganisation of the test code to put it all under the test/ directory, and appropriate change to the pom.xml file, makes the tests build and run. Of course they are still failing, but at least they are running.

Adding some plugins to Jenkins helps to visualise how the project is going, I’m using JUnit Plugin and Test Results Analyzer Plugin, which give some nice analysis of how the tests are going and graphs to show progress over recent builds.


Mavenification

It turns out that it is not that difficult to convert a project to be built using Maven. Here are the steps I went through:

  1. Set up a repository – I’m using Artifactory
  2. Set permissions on the repository to allow a user to write to it
  3. Get the settings.xml file for Maven configuration
  4. Add the user authentication information to the downloaded settings file
  5. For each project in Eclipse
  6. Add a Maven nature
  7. Add the repository to the distributionManagement section of the generated pom.xml file
  8. Add a debug configuration for ‘Maven build…’
  9. Give a goal of ‘deploy’ and the path to the settings file from the repository
  10. Attempt a build and resolve any missing dependencies
  11. When the build is successful the output will be published to the repository

The two most complicated parts of this process were getting the user permissions right and resolving dependencies. The permissions is the usual dance of trying to use the right username and setting the correct permissions for all of the repositories, it is documented and fairly clear once you’ve got it all set up right. Atrifactory has an ‘access.log’ (which you can see in System Logs admin screen) which shows permission results, it is a good place to check if you are getting errors when deploying. During the setup phase I was building again with the same version number, which requires the ‘delete’ permission.

More interesting is resolving the dependencies, mostly searching Google for the package that reported an error gave a link on Maven Repository which contains an XML snippet to add the dependency information to the pom.xml file. I didn’t encounter any conflicts during this process as I was, for the most part, using versions already known to work but obtained from other sources. I’ll be writing a bit more about versions shortly, when I try to bring everything up to date.


Event finder – a revival

Some months ago I did quite a lot of work on a Java EE server to look for events at venues. This was going quite well but being significantly hampered by my poor internet connection – the next stage was to be a massive search and web crawl to collect a lot of data.

Now that I have an AWS heavy utilisation instance available it is time to revive this project. The plan for this looks roughly like this:

  • Get the code building in a new environment
  • Bring all the components up to date
  • Maven-ify the build and dependency management
  • Write some integration tests
  • Set up a continuous integration build
  • Get project deployable
  • Set up staging and live environments

The original work was done in a Linux (Ubuntu) environment, but my new laptop runs Windows 8.1 (yuck!), so I’m using that for getting the build working again from a clean machine. It is times like this that you are thankful you had the foresight to split everything platform dependent in the code into separate Settings and Implementation classes. Of course there are still a lot of failures when I try to run anything, but most of those are because I haven’t got PostgreSQL set up correctly yet.


Building Octave

Having spent the last few months working with Matlab, Octave caught my eye. It is GNU project intended to be ‘quite similar to Matlab’. It looks like they have a bunch of unmaintained packages that could use a bit of refreshing.

These are the steps I’ve taken to build and run Octave under Cygwin:

hg clone http://www.octave.org/hg/octave
cd octave
./bootstrap
./configure –enable-shared –with-x –without-framework-opengl –disable-docs CFLAGS=”-Dtimezone=_timezone”
make

Dependencies I encountered to get the build to complete:
– autoconf
– automake
– gcc-g++
– gcc-fortran
– make
– lib-pcre-devel
– openblas
– liblapack
– libreadline-devel
– flex
– bison
– gperf

Further dependencies that can be included:
– qt5-devel-tools
– libqtopengl4-devel
– libamd-devel
– libarpack-devel
– libcamd-devel
– libccolamd-devel
– libcolamd-devel
– libcurl-devel
– libfftw3-devel
– libglpk-devel
– libhdf5-devel
– libqhull5-devel
– libqhull-deverl
– libqrupdate-devel
– libumfpack-devel
– terminfo-extra
– zlib-devel
– lib-cxsparse
– glu-debuginfo
– libglu-devel
– libglut-devel
– libncurses-devel
– qt5-base-examples
– pkg-config
– fltk
– libfltk-devel
– libqscintilla-devel
– libgl2ps-devel


Another new era – it’s different this time

I’ve been looking around for a while for an open source project to get involved with, until now nothing has really piqued my interested. After heartbleed I took a look at OpenSSL but figured that had a lot of eyes on it especially then and I’d get lost in the crowd.

Now my life is changing and my time is going to be available more as little snippets of an hour or two here and there rather than full days to dedicate to work. Some of this could be used for helping out an open source project.

Open source is essential. Working with a bunch of jerks is not

Of course anything I do will be in the context of some of the other projects I have going on, which mainly revolve around forex markets and geo-spatial data analysis, but extend to include OBD vehicle diagnostics and natural language understanding. More of those over the next few weeks, as I come to them.