Neil Hopcroft

A digital misfit

Dolfin success

It has taken a bit of wrangling but I’ve finally gotten dolfin to build. The final set of adjustments I have made to the build are:
mv dolfin/la/PETScKrylovSolver.cpp dolfin/la/PETScKrylovSolver.cpp.orig
sed 's/^.*KSPSetNull/\/\/ Line removed to fix build problem/' < dolfin/la/PETScKrylovSolver.cpp.orig > dolfin/la/PETScKrylovSolver.cpp
mv dolfin/la/CholmodCholeskySolver.cpp dolfin/la/CholmodCholeskySolver.cpp.orig
echo '#include "SuiteSparse_config.h"' > dolfin/la/CholmodCholeskySolver.cpp
cat dolfin/la/CholmodCholeskySolver.cpp.orig >> dolfin/la/CholmodCholeskySolver.cpp
mv dolfin/la/UmfpackLUSolver.cpp dolfin/la/UmfpackLUSolver.cpp.orig
echo '#include "SuiteSparse_config.h"' > dolfin/la/UmfpackLUSolver.cpp
cat dolfin/la/UmfpackLUSolver.cpp.orig >> dolfin/la/UmfpackLUSolver.cpp

This gets the package building with a few optional dependencies missing:

  • — (**) SLEPC
  • — (**) SLEPC4PY
  • — (**) TRILINOS
  • — (**) PASTIX
  • — (**) SCOTCH
  • — (**) HDF5

SLEPc and SLEPc4Py don’t make any sense since those are building and installing apparently correctly.
Trilinos doesn’t clone correctly in the context of the automated build, but I can clone it if I do it manually. However, even when I do that it doesn’t build.
Pastix and Scotch are both built and installed, but not found.
HDF5 is built without parallel support.


Octave build break: SuiteSparse version

The recent update to the latest SuiteSparse has caused a build error, undefined references to SuiteSparse_realloc, SuiteSparse_free, SuiteSparse_time, SuiteSparse_calloc, SuiteSparse_config and SuiteSparse_malloc. This appears to be because they have moved to a new library, libsuitesparse_config.a and the octave dependencies haven’t yet been updated. Indeed, I suspect there are a number of other projects which will suffer a similar break.
After a little hacking around I have added a build hack to include the library in the dependency list whenever cxsparse is used. This isn’t the right solution, but should get the build working for the moment.
Of course, the frustrating thing here is that the SuiteSparse upgrade doesn’t resolve the error I was seeing in the dolfin build. That is still giving me the same undefined symbol errors. Again, I have made a hack to get around them, but they need a proper solution. What I have done is add the SuiteSparse_config.h header to the files causing errors, CholmodCholeskySolver.cpp and UmfpackLUSolver.cpp
This, however, shows there might be another problem with the build – there is some overlap between SuiteSparse_config.h and UFConfig.h, suggesting there could be a version clash between different suitesparse versions. (which, indeed, may be the cause of the original error in the octave build)
For the moment I will just worry about getting a build working again, and trust that the developers of suitesparse have done a reasonable job of maintaining compatibility.


fem-fenics octave package

As I mentioned before the fem-fenics package requires its own whole set of dependencies, a number of which are available on the fenics website. The octave package is an octave interface onto the fenics project, so to test it we need to install fenics.

  • To install the octave fem-fenics package we first need to build the code in the fem-fenics/src directory. This causes a configure error
    • “error: dolfin required to install Fem-fenics Package”.
    • build dolfin package

  • Building the dolfin package:
    • “Unable to find the requested Boost libraries.”
    • sudo yum install boost-devel

    • “Could not find a configuration file for package UFC that is compatible with requested version 1.6.0.”
    • build ffc package, which includes the now-deprecated UFC package

    • “Could NOT find Eigen3”
    • build eigen package

    • “Could NOT find PythonLibs”
    • I changed python symlink to point to 2.7.9 while python2 symlink remained linked to 2.6.9.

    • “ImportError: No module named ufl.log”
    • build ufl package

    • “ImportError: No module named FIAT.functional”
    • build fiat package

    • “ImportError: No module named instant”
    • build instant package

    • “CholmodCholeskySolver.cpp:97:17: error: ‘SuiteSparse_long’ was not declared in this scope”
    • Update SuiteSparse to the latest version

  • Building the ffc package:
    • “Python 2.7 or higher required, please upgrade.”
    • Switched symlinks to 2.7

    • “Unable to find SWIG version 3.0.3 or higher.”
    • build the swig package

  • Building the swig package:
    • “yodl2man: Command not found”
    • Install yodl, which requires icmake to build

  • Building the eigen3 package:
  • …there are a number of optional packages, which may be required for other packages, so we’ll need a rebuild once everything else is building

  • Building the ufl package:
  • …builds without error

  • Building the fiat package:
  • …builds without error

  • Building the instant package:
  • …builds without error

  • Building the SuiteSparse package:
    • “/usr/bin/ld: cannot find -lopenblas”
    • Install the latest OpenBLAS package


Build errors in the stk octave package

I’ve taken a look at the stk package following comments from Julien Bect on the maintainers mailing list. The Octave code I have comes from http://www.octave.org/hg/octave and stk code from http://hg.code.sf.net/p/octave/stk, my build is configured to poll for SCM changes and rebuild when a change happens. It’s running on a small AWS instance, so builds can take quite a long time.

The octave package build performs the same steps for all packages:

  • update from source control
  • uninstall package
  • adjust build files
  • build c++ code
  • build package
  • install package
  • test package

It seems like this sequence isn’t appropriate for stk. The build error I see is during the build package stage, where it complains “package is missing file: DESCRIPTION”.
Looking more closely there are some instructions in the README file describing the installation process, run skt_init.m in octave. Running this the package installs, but it doesn’t follow the conventions of the other packages and doesn’t show in the list of installed packages.
The stk_init.m file has a setting STK_OCTAVE_PACKAGE = false; which appears to be a mechanism to set the code to install as an uninstallable package, but that isn’t working for me.


Response from the octave maintainers mailing list

There were a number of comments to my initial mail to the maintainers mail list.

  • “reviving the fixed-point arithmetic package would be great. It’s gotten badly bit-rotten, and is of interest to people working with certain digital signal processing hardware.”
  • “Having regularly updated reports on the status of Octave packages would be great. Even better would be to have some code coverage statistics associated to the unit tests, but we’re not there yet.”
  • “In your post “Octave Packages CI build dependencies “, you report “Build error” for stk, but you don’t provide the version of stk (nor the version of Octave) that you have tested. Could you please tell me more about this “Build error”? A release was made very recently (2.3.0), could you try it and see if the build error is still there?”
  • “Neat. Would you like your blog to be syndicated at http://planet.octave.org ?”
  • “Btw, we already have code coverage for core Octave. It’s part of our Hydra builds: http://hydra.nixos.org/jobset/gnu/octave-default#tabs-jobs”…”But it covers C++ code, not m-files, right ?”

Which boils down to a few useful things for me to do:

  • Get all the package tests working and reporting results in a sensible way
    • Reporting any problems I find along the way to the current maintainers where there are any
  • Get some .m file coverage tests working for the packages
  • Pick up the fixed package and bring it up to releasable standard

octave build break – interp1q

My octave CI build has failed with this error:

make[2]: Entering directory `/usr/share/tomcat7/.jenkins/workspace/octave/scripts’
make[2]: *** No rule to make target `deprecated/interp1q.m’, needed by `all-am’. Stop.

This appears to be caused by

Changeset 19526:ccef60b2a058 by Rik:
maint: Remove functions deprecated in 3.8.

* scripts/deprecated/module.mk: Remove deprecated scripts from build system.

* scripts/deprecated/default_save_options.m,
scripts/deprecated/gen_doc_cache.m, scripts/deprecated/interp1q.m,
scripts/deprecated/isequalwithequalnans.m,
scripts/deprecated/java_convert_matrix.m, scripts/deprecated/java_debug.m,
scripts/deprecated/java_invoke.m, scripts/deprecated/java_new.m,
scripts/deprecated/java_unsigned_conversion.m, scripts/deprecated/javafields.m,
scripts/deprecated/javamethods.m,
scripts/deprecated/re_read_readline_init_file.m,
scripts/deprecated/read_readline_init_file.m,
scripts/deprecated/saving_history.m:
Remove scripts deprecated in 3.8 ahead of 4.2 release.

Which appears to remove a number of deprecated scripts, but maybe not completely.

It looks like it should be fixed by this change:

Changeset 19527:cd5ae0f080e8 by Rik:
maint: Remove interp1q deprecated in 3.8.

* scripts/deprecated/module.mk: Remove interp1q.m from build system.
The file was modified scripts/deprecated/module.mk (diff)


Review of the octave package build errors

Some of the octave packages are causing build errors, here is a quick review of how things are:

  • ad: gives “error: ‘DECLARE_OCTAVE_ALLOCATOR’ does not name a type”
  • ann: there appears to be a couple of problems here, delving a bit deeper into the build, this package can be built from the command line, allowing the setting of some options to the configure script. It seems like there have been some changes to types in some headers which break the build, also the configuration in Makeconf is not correctly extracting names for a few of the variables it requests from Octave, leaving some control characters on the beginning of their strings – this is quite confusing since they are invisible characters leading to errors claiming ‘ar not found’ even when ‘ar’ is available. Removing these control characters removes a couple of errors but does not fix the build.
  • communications: Gives “warning: autoload: ‘gf.oct’ is not an absolute file name”, there is some C++ source that appears to create gf.oct, but it doesn’t compile.
    database: Missing pq_interface.oct file. There is a configure script and some C++ source code which isn’t built as part of the package installation. Building this code creates the pq_interface.oct file. I have added this to the build script. This appears to resolve the problem. Fix included in CI build.
  • fem-fenics C++ code is not built, attempting to build it yields error about dolfin missing – this is probably worth an entry all by itself.
  • fixed: Gives “error: no match for ‘operator==’ (operand types are ‘const volatile FixedPoint’ and ‘const volatile FixedPoint’)” amongst many other errors.
  • fl-core: Complains ” error: ‘_SC_NPROCESSORS_ONLN’ was not declared in this scope”.
  • galois: Gives “tar: field-0.0.1.tar: Cannot stat: No such file or directory”
  • graceplot: Fails with “error: ‘symbol_record’ was not declared in this scope”. Installing grace doesn’t improve this.
  • jhandles: Fails with “Java support not compiled”, it seems I need to install the java package first, but I haven’t yet found any source code for it.
  • mpi: “/bin/sh: mpic++: command not found” – installing openmpi should help.
  • multiprecision: “package is missing file: DESCRIPTION” – adding a description file leads to a build error.
  • NaN: Fails to install because of capital letters, renaming tgz file nan.tgz makes it work. I have adjusted the .tgz creation to use lower case file names.
  • ngspice: “/usr/bin/ld: cannot find -lngspice”, added libngspice build.
  • ocs: Reports a number of missing directories, but claims to install anyway
  • oct2mat: Fails with “unterminated character string constant”
  • octcdf: ncdap-config command is missing. Again, there is a configure script and some C++ code which isn’t built. Seems like we need netCDF library for the ncdap-config command. However, compiling netCDF doesn’t appear to yield the command, although it does create dap-config. I have tried making a symlink of this as ncdap-config, hoping that it is the same thing renamed. Indeed, it seems better, although I am now seeing “undefined symbol: uuid_unparse”. This looks like it might be a missing version of uuid-devel.
  • optim: “invalid doc file format make: *** [../doc/optim.texi] Error 25”
  • optiminterp: “mkoctfile lacks Fortran 90 support”. This error can be resolved by making the src code first, but that fails because of missing Makeconf.in. Attempting to copy this file from another project gives me syntax errors. More investigation neeeded.
  • pdb: Gives “error: ‘Octave_map’ was not declared in this scope.”
  • perl: README says: “This is a perl package and is part of CPAN. It makes no sense to make and octave package of this.”
  • prony: “The keyword ‘description’ of the package ‘prony’ has an empty value” – indeed, there are a number of things missing from the DESCRIPTION file. It installs with these updates.
  • queueing: Complains about misplaced {}, but installs anyway.
  • real2rgb: Complains “the DESCRIPTION file must have a Categories field, when no INDEX file is given” and fails to install, even adding a Categories field doesn’t resolve this.
  • secs1d: Complains “error: movefile: no files to move”
  • secs2d: Complains “error: ‘Octave_map’ does not name a type”
  • sparsersb: Complains “octave_allocator class has been deprecated” and “relocation R_X86_64_32 against `.text’ can not be used when making a shared object; recompile with -fPIC”, the former will require some code rewriting while the latter might be resolved with a rebuild of something. Rebuilding librsb resolves the relocation problem.
  • system-identification: package name ‘system identification’ doesn’t correspond to its filename ‘system-identification’. Fixing the package DESCRIPTION file gets past that problem, but leads to errors for missing files system-identification/inst/tisean_wrapper/src/* and system-identification-0.1.0/tisean_wrapper. The former is an empty directory in the source control system and can be resolved by placing a file in the directory, the latter I don’t yet know how to resolve.
  • tk_octave: Complains “fatal error: tk.h: No such file or directory”. This implies tk is not installed, and there is no package in the yum repository. Building the latest tk requires the latest tcl too. These resolve the tk.h error, but now blt.h is missing.
  • triangular: Commplains “error: ‘DECLARE_OCTAVE_ALLOCATOR’ does not name a type”
  • tsa: Complains “make: *** No rule to make target `covm_mex.cpp’, needed by `covm_mex.mex’. Stop.”
  • video: “Package is empty”. Attempting to build the src directory leads to missing libavformat, from ffmpeg.
  • xraylib: Complains “error: no matching function for call to ‘Array::Array(int)’”

I will go into a bit more detail on these over the next few posts.


Octave build break – griddata

My CI build failed with this error:

GEN griddata.eps
error: print: no axes object in figure to print
error: called from
print at line 301 column 5
geometryimages at line 69 column 5

in octave/doc/interpreter. This appears to have been caused by this change:

Changeset 19511:561af1ab6099 by Mike Miller:
griddata.m: Return values instead of plotting for Matlab compatibility (bug #45125)

* griddata.m: Return interpolated values instead of plotting a mesh for
compatibility with Matlab.  Adjust %!demos to call mesh on the output.
* NEWS: Mention change to griddata for 4.2.

Although it is not immediately clear to me from the diffs why this would cause an error.

Edit: This break is fixed by:
Changeset 19512:4e15a4c331e7 by Mike Miller:
doc: Fix griddata example to new calling convention

* geometry.txi (Interpolation on Scattered Data): Fix griddata example.
* geometryimages.m: Fix griddata call to match documented example and
produce the correct figure.


Current state of unmaintained octave packages

For the moment I’m just looking at the list of packages to try to understand what would be needed to get them up to a releasable standard.

/td>

Name Install Licence Texinfo doc Required files
actuarial Yes GPL No Yes
ad No GPL Yes Yes
ann No GPL No Yes
audio Yes GPL/Public domain No Yes
benchmark Yes GPL No Yes
bioinfo Yes GPL No Yes
civil-engineering Yes GPL No Yes
engine Yes GPL No Yes
fem-fenics No GPL No Yes
fixed No GPL Yes Yes
fl-core No GPL No Yes
galois No GPL No Yes
gnuplot Yes GPL No Yes
graceplot No GPL No Yes
gsl Yes GPL Scripts Yes
information-theory Yes GPL No Yes
integration Yes GPL No Yes
irsa Yes GPL No Yes
jhandles No GPL/LGPL No Yes
mapping Yes GPL No Yes
missing-functions Yes Public domain No Yes
mpi No GPL No Yes
multicore Yes GPL No Yes
multi-precision No GPL No No
nan No GPL No Yes
ncarray No GPL No Yes
ngspice No GPL No Yes
nlwing2 Yes GPL No Yes
nnet Yes GPL No Yes
oct2mat No GPL No Yes
octcdf No GPL No Yes
octgpr Yes GPL No Yes
odebvp Yes GPL No Yes
optim No GPL Yes Yes
optiminterp No GPL No Yes
outliers Yes GPL No Yes
pdb No GPL No Yes
perl No Not stated No No
plot Yes GPL No Yes
project-web No N/A N/A N/A
prony No GPL No Yes
real2rgb No GPL No Yes
secs1d No GPL Yes Yes
secs2d No GPL No Yes
simp Yes GPL No Yes
soctcl No Public domain No Yes
sparsersb No GPL No Yes
special-matrix Yes Public Domain No Yes
stk No GPL No Yes
symband Yes GPL Yes Yes
system-identification No GPL No Yes
tk_octave No Public domain No Yes
triangular No GPL Yes Yes
video No ? No Yes
wavelet No GPL No Yes
xraylib No GPL No Yes
zenity Yes GPL No Yes

After some replies to my volunteering to be a maintainer email, it seems like the project would first prefer to see some other patches from me for currently maintained packages or core code. So, while this is interesting, it might not be immediately useful.
Edit: Julien Bect points out that stk is erroneously in this list, he is one of the maintainers thereof.


Building Android OBD Reader

I have, for a while, been interested in OBD automotive diagnostics and have tried using several OBD applications on Android and Windows phone. One of them is Android OBD Reader. To add this to the Jenkins build server I have installed the Android SDK for Linux and the Gradle build system.
I tried using the Jenkins Gradle plugin, but that didn’t allow me to set the ANDROID_HOME environment variable for the Gradle build, so I have used a shell command instead. This lead me to an error “/lib/ld-linux.so.2: bad ELF interpreter: No such file or directory”. This can be fixed by sudo yum install glibc.i686
Running $ANDROID_HOME/build-tools/22.0.1/aapt gives a little more information about the actual errors encountered during the build. Cycling around filling in all the missing libraries:

Library Package
ld-linux.so.2 glibc.i686
libstdc++.so.6 libstdc++48.i686

I think these errors are because I’m trying to run a 32-bit program on a 64-bit operating system, which doesn’t have a complete set of 32-bit libraries installed. It seems a bit odd to me that the Android SDK would be 32-bit, considering that the operating system itself runs on processors capable of 64-bit operation (even if that is not actually being used).