Neil Hopcroft

A digital misfit

Response from the octave maintainers mailing list

There were a number of comments to my initial mail to the maintainers mail list.

  • “reviving the fixed-point arithmetic package would be great. It’s gotten badly bit-rotten, and is of interest to people working with certain digital signal processing hardware.”
  • “Having regularly updated reports on the status of Octave packages would be great. Even better would be to have some code coverage statistics associated to the unit tests, but we’re not there yet.”
  • “In your post “Octave Packages CI build dependencies “, you report “Build error” for stk, but you don’t provide the version of stk (nor the version of Octave) that you have tested. Could you please tell me more about this “Build error”? A release was made very recently (2.3.0), could you try it and see if the build error is still there?”
  • “Neat. Would you like your blog to be syndicated at http://planet.octave.org ?”
  • “Btw, we already have code coverage for core Octave. It’s part of our Hydra builds: http://hydra.nixos.org/jobset/gnu/octave-default#tabs-jobs”…”But it covers C++ code, not m-files, right ?”

Which boils down to a few useful things for me to do:

  • Get all the package tests working and reporting results in a sensible way
    • Reporting any problems I find along the way to the current maintainers where there are any
  • Get some .m file coverage tests working for the packages
  • Pick up the fixed package and bring it up to releasable standard

octave build break – interp1q

My octave CI build has failed with this error:

make[2]: Entering directory `/usr/share/tomcat7/.jenkins/workspace/octave/scripts’
make[2]: *** No rule to make target `deprecated/interp1q.m’, needed by `all-am’. Stop.

This appears to be caused by

Changeset 19526:ccef60b2a058 by Rik:
maint: Remove functions deprecated in 3.8.

* scripts/deprecated/module.mk: Remove deprecated scripts from build system.

* scripts/deprecated/default_save_options.m,
scripts/deprecated/gen_doc_cache.m, scripts/deprecated/interp1q.m,
scripts/deprecated/isequalwithequalnans.m,
scripts/deprecated/java_convert_matrix.m, scripts/deprecated/java_debug.m,
scripts/deprecated/java_invoke.m, scripts/deprecated/java_new.m,
scripts/deprecated/java_unsigned_conversion.m, scripts/deprecated/javafields.m,
scripts/deprecated/javamethods.m,
scripts/deprecated/re_read_readline_init_file.m,
scripts/deprecated/read_readline_init_file.m,
scripts/deprecated/saving_history.m:
Remove scripts deprecated in 3.8 ahead of 4.2 release.

Which appears to remove a number of deprecated scripts, but maybe not completely.

It looks like it should be fixed by this change:

Changeset 19527:cd5ae0f080e8 by Rik:
maint: Remove interp1q deprecated in 3.8.

* scripts/deprecated/module.mk: Remove interp1q.m from build system.
The file was modified scripts/deprecated/module.mk (diff)


Review of the octave package build errors

Some of the octave packages are causing build errors, here is a quick review of how things are:

  • ad: gives “error: ‘DECLARE_OCTAVE_ALLOCATOR’ does not name a type”
  • ann: there appears to be a couple of problems here, delving a bit deeper into the build, this package can be built from the command line, allowing the setting of some options to the configure script. It seems like there have been some changes to types in some headers which break the build, also the configuration in Makeconf is not correctly extracting names for a few of the variables it requests from Octave, leaving some control characters on the beginning of their strings – this is quite confusing since they are invisible characters leading to errors claiming ‘ar not found’ even when ‘ar’ is available. Removing these control characters removes a couple of errors but does not fix the build.
  • communications: Gives “warning: autoload: ‘gf.oct’ is not an absolute file name”, there is some C++ source that appears to create gf.oct, but it doesn’t compile.
    database: Missing pq_interface.oct file. There is a configure script and some C++ source code which isn’t built as part of the package installation. Building this code creates the pq_interface.oct file. I have added this to the build script. This appears to resolve the problem. Fix included in CI build.
  • fem-fenics C++ code is not built, attempting to build it yields error about dolfin missing – this is probably worth an entry all by itself.
  • fixed: Gives “error: no match for ‘operator==’ (operand types are ‘const volatile FixedPoint’ and ‘const volatile FixedPoint’)” amongst many other errors.
  • fl-core: Complains ” error: ‘_SC_NPROCESSORS_ONLN’ was not declared in this scope”.
  • galois: Gives “tar: field-0.0.1.tar: Cannot stat: No such file or directory”
  • graceplot: Fails with “error: ‘symbol_record’ was not declared in this scope”. Installing grace doesn’t improve this.
  • jhandles: Fails with “Java support not compiled”, it seems I need to install the java package first, but I haven’t yet found any source code for it.
  • mpi: “/bin/sh: mpic++: command not found” – installing openmpi should help.
  • multiprecision: “package is missing file: DESCRIPTION” – adding a description file leads to a build error.
  • NaN: Fails to install because of capital letters, renaming tgz file nan.tgz makes it work. I have adjusted the .tgz creation to use lower case file names.
  • ngspice: “/usr/bin/ld: cannot find -lngspice”, added libngspice build.
  • ocs: Reports a number of missing directories, but claims to install anyway
  • oct2mat: Fails with “unterminated character string constant”
  • octcdf: ncdap-config command is missing. Again, there is a configure script and some C++ code which isn’t built. Seems like we need netCDF library for the ncdap-config command. However, compiling netCDF doesn’t appear to yield the command, although it does create dap-config. I have tried making a symlink of this as ncdap-config, hoping that it is the same thing renamed. Indeed, it seems better, although I am now seeing “undefined symbol: uuid_unparse”. This looks like it might be a missing version of uuid-devel.
  • optim: “invalid doc file format make: *** [../doc/optim.texi] Error 25”
  • optiminterp: “mkoctfile lacks Fortran 90 support”. This error can be resolved by making the src code first, but that fails because of missing Makeconf.in. Attempting to copy this file from another project gives me syntax errors. More investigation neeeded.
  • pdb: Gives “error: ‘Octave_map’ was not declared in this scope.”
  • perl: README says: “This is a perl package and is part of CPAN. It makes no sense to make and octave package of this.”
  • prony: “The keyword ‘description’ of the package ‘prony’ has an empty value” – indeed, there are a number of things missing from the DESCRIPTION file. It installs with these updates.
  • queueing: Complains about misplaced {}, but installs anyway.
  • real2rgb: Complains “the DESCRIPTION file must have a Categories field, when no INDEX file is given” and fails to install, even adding a Categories field doesn’t resolve this.
  • secs1d: Complains “error: movefile: no files to move”
  • secs2d: Complains “error: ‘Octave_map’ does not name a type”
  • sparsersb: Complains “octave_allocator class has been deprecated” and “relocation R_X86_64_32 against `.text’ can not be used when making a shared object; recompile with -fPIC”, the former will require some code rewriting while the latter might be resolved with a rebuild of something. Rebuilding librsb resolves the relocation problem.
  • system-identification: package name ‘system identification’ doesn’t correspond to its filename ‘system-identification’. Fixing the package DESCRIPTION file gets past that problem, but leads to errors for missing files system-identification/inst/tisean_wrapper/src/* and system-identification-0.1.0/tisean_wrapper. The former is an empty directory in the source control system and can be resolved by placing a file in the directory, the latter I don’t yet know how to resolve.
  • tk_octave: Complains “fatal error: tk.h: No such file or directory”. This implies tk is not installed, and there is no package in the yum repository. Building the latest tk requires the latest tcl too. These resolve the tk.h error, but now blt.h is missing.
  • triangular: Commplains “error: ‘DECLARE_OCTAVE_ALLOCATOR’ does not name a type”
  • tsa: Complains “make: *** No rule to make target `covm_mex.cpp’, needed by `covm_mex.mex’. Stop.”
  • video: “Package is empty”. Attempting to build the src directory leads to missing libavformat, from ffmpeg.
  • xraylib: Complains “error: no matching function for call to ‘Array::Array(int)’”

I will go into a bit more detail on these over the next few posts.


Octave build break – griddata

My CI build failed with this error:

GEN griddata.eps
error: print: no axes object in figure to print
error: called from
print at line 301 column 5
geometryimages at line 69 column 5

in octave/doc/interpreter. This appears to have been caused by this change:

Changeset 19511:561af1ab6099 by Mike Miller:
griddata.m: Return values instead of plotting for Matlab compatibility (bug #45125)

* griddata.m: Return interpolated values instead of plotting a mesh for
compatibility with Matlab.  Adjust %!demos to call mesh on the output.
* NEWS: Mention change to griddata for 4.2.

Although it is not immediately clear to me from the diffs why this would cause an error.

Edit: This break is fixed by:
Changeset 19512:4e15a4c331e7 by Mike Miller:
doc: Fix griddata example to new calling convention

* geometry.txi (Interpolation on Scattered Data): Fix griddata example.
* geometryimages.m: Fix griddata call to match documented example and
produce the correct figure.


Current state of unmaintained octave packages

For the moment I’m just looking at the list of packages to try to understand what would be needed to get them up to a releasable standard.

/td>

Name Install Licence Texinfo doc Required files
actuarial Yes GPL No Yes
ad No GPL Yes Yes
ann No GPL No Yes
audio Yes GPL/Public domain No Yes
benchmark Yes GPL No Yes
bioinfo Yes GPL No Yes
civil-engineering Yes GPL No Yes
engine Yes GPL No Yes
fem-fenics No GPL No Yes
fixed No GPL Yes Yes
fl-core No GPL No Yes
galois No GPL No Yes
gnuplot Yes GPL No Yes
graceplot No GPL No Yes
gsl Yes GPL Scripts Yes
information-theory Yes GPL No Yes
integration Yes GPL No Yes
irsa Yes GPL No Yes
jhandles No GPL/LGPL No Yes
mapping Yes GPL No Yes
missing-functions Yes Public domain No Yes
mpi No GPL No Yes
multicore Yes GPL No Yes
multi-precision No GPL No No
nan No GPL No Yes
ncarray No GPL No Yes
ngspice No GPL No Yes
nlwing2 Yes GPL No Yes
nnet Yes GPL No Yes
oct2mat No GPL No Yes
octcdf No GPL No Yes
octgpr Yes GPL No Yes
odebvp Yes GPL No Yes
optim No GPL Yes Yes
optiminterp No GPL No Yes
outliers Yes GPL No Yes
pdb No GPL No Yes
perl No Not stated No No
plot Yes GPL No Yes
project-web No N/A N/A N/A
prony No GPL No Yes
real2rgb No GPL No Yes
secs1d No GPL Yes Yes
secs2d No GPL No Yes
simp Yes GPL No Yes
soctcl No Public domain No Yes
sparsersb No GPL No Yes
special-matrix Yes Public Domain No Yes
stk No GPL No Yes
symband Yes GPL Yes Yes
system-identification No GPL No Yes
tk_octave No Public domain No Yes
triangular No GPL Yes Yes
video No ? No Yes
wavelet No GPL No Yes
xraylib No GPL No Yes
zenity Yes GPL No Yes

After some replies to my volunteering to be a maintainer email, it seems like the project would first prefer to see some other patches from me for currently maintained packages or core code. So, while this is interesting, it might not be immediately useful.
Edit: Julien Bect points out that stk is erroneously in this list, he is one of the maintainers thereof.


Building Android OBD Reader

I have, for a while, been interested in OBD automotive diagnostics and have tried using several OBD applications on Android and Windows phone. One of them is Android OBD Reader. To add this to the Jenkins build server I have installed the Android SDK for Linux and the Gradle build system.
I tried using the Jenkins Gradle plugin, but that didn’t allow me to set the ANDROID_HOME environment variable for the Gradle build, so I have used a shell command instead. This lead me to an error “/lib/ld-linux.so.2: bad ELF interpreter: No such file or directory”. This can be fixed by sudo yum install glibc.i686
Running $ANDROID_HOME/build-tools/22.0.1/aapt gives a little more information about the actual errors encountered during the build. Cycling around filling in all the missing libraries:

Library Package
ld-linux.so.2 glibc.i686
libstdc++.so.6 libstdc++48.i686

I think these errors are because I’m trying to run a 32-bit program on a 64-bit operating system, which doesn’t have a complete set of 32-bit libraries installed. It seems a bit odd to me that the Android SDK would be 32-bit, considering that the operating system itself runs on processors capable of 64-bit operation (even if that is not actually being used).


Octave packages test results

Having set up a build of octave packages, I have now added a mechanism to test them. This is currently quite crude and probably doesn’t really reflect the actual state of the packages.

Package Maintained Version Successful Run Pass
actuarial No 1.1.0 Yes 16 0
ad No No 0 0
ann No No 0 0
audio No 1.1.4 Yes 12 3
benchmark No 1.1.1 Yes 16 0
bim Yes 1.1.4 Yes 53 9
bioinfo No 0.1.2 Yes 9 4
cgi Yes 1.0.1 Yes 5 0
civil-engineering No 1.0.7 Yes 7 0
communications Yes 1.2.1 Yes 100 52
control Yes 2.8.0 Yes 130 0
data-smoothing Yes 1.3.0 Yes 9 1
database Yes 2.3.1 Yes 13 0
dataframe Yes 1.1.0 Yes 6 0
dicom Yes 0.1.1 Yes 4 0
divand Yes 1.1.2 Yes 75 0
econometrics Yes 1.1.1 Yes 35 0
engine No 1.0.9 Yes 6 0
fem-fenics Yes No 0 0
fenv No 0.1.0 Yes 9 3
financial Yes 0.4.0 Yes 75 2
fits Yes 1.0.6 Yes 4 0
fixed No 0 0
fl-core Yes No 0 0
fpl Yes 1.3.4 Yes 15 2
fuzzy-logic-toolkit Yes 0.4.5 Yes 61 17
ga Yes 0.10.0 Yes 27 6
galois No 0 0
general Yes 1.3.4 Yes 14 3
generate_html Yes 0.1.7 Yes 13 0
generate_latex 0.0.1 Yes 8 0
geometry Yes 1.7.0 Yes 13 0
gnuplot No 1.0.3 Yes 32 2
graceplot No 0 0
graph 0.0.0 Yes 5 0
gsl No 1.1.0 Yes 11 0
image-acquisition Yes 0.2.1 Yes 12 0
image Yes 2.3.0 Yes 132 8
informationtheory No 0.1.8 Yes 32 26
instrument-control Yes 0.2.1 Yes 13 0
integration No 1.0.7 Yes 37 1
interval Yes 0.1.5 Yes 14 0
io Yes 2.2.7 Yes 42 0
irsa No 1.0.7 Yes 19 7
java Yes No 0 0
jhandles No 0 0
level-set Yes No 0 0
linear-algebra Yes 2.2.2 Yes 25 2
lssa Yes 0.1.2 Yes 14 0
ltfat Yes No 0 0
macosx No 0 0
mapping No 1.2.0 Yes 45 0
mechanics Yes 1.3.1 Yes 10 0
miscellaneous Yes 1.2.1 Yes 37 1
missing-functions No 1.0.2 Yes 9 0
mpi Yes No 0 0
msh Yes 1.0.10 Yes 25 7
multicore No 0.2.15 Yes 21 0
multi-precision No 0 0
mvn Yes 1.1.0 Yes 23 0
nan Yes No 0 0
ncarray Yes No 0 0
netcdf Yes 1.0.6 Yes 16 0
ngspice No 0 0
nlwing2 No 1.2.0 Yes 23 0
nnet No 0.1.13 Yes 35 0
nurbs Yes 1.3.10 Yes 73 18
ocs Yes 0.1.3 Yes 11 0
oct2mat No No 0 0
octcdf Yes No 0 0
octclip Yes 1.0.3 Yes 7 1
octgpr No 1.2.1 Yes 6 0
octproj Yes 1.1.2 Yes 11 0
ode 1.0.1 Yes 13 0
odebvp No 1.0.6 Yes 5 0
odepkg Yes 0.8.4 Yes 42 14
optics Yes 0.1.1 Yes 48 0
optim Yes No 0 0
optiminterp Yes No 0 0
outliers No 0.13.9 Yes 18 0
parallel Yes 2.2.1 Yes 19 0
pdb No No 0 0
perl No 0 0
plot No 1.1.0 Yes 14 4
project-web No 0 0
prony No 0 0
quaternion Yes 2.4.0 Yes 14 0
queueing Yes 1.2.3 Yes 108 3
real2rgb No 0 0
robotis 0.0.1 Yes 6 0
secs1d Yes No 0 0
secs2d Yes No 0 0
secs3d Yes 0.0.1 Yes 0 0
signal Yes 1.3.1 Yes 152 23
simp No 1.1.0 Yes 34 0
sockets Yes 1.2.0 Yes 4 0
soctcl No 0 0
sparsersb No 0 0
specfun Yes 1.1.0 Yes 28 4
special-matrix No 1.0.7 Yes 6 0
splines Yes 1.2.7 Yes 16 1
statistics Yes 1.2.4 Yes 119 8
stk Yes No 0 0
strings Yes 1.1.0 Yes 10 0
struct Yes 1.0.10 Yes 8 0
symband No 1.0.10 Yes 12 0
symbolic Yes No 0 0
system-identification No 0 0
tcl-octave No No 9 0
triangular No 0 0
tsa Yes No 0 0
video No No 0 0
vrml Yes 1.0.13 Yes 51 2
wavelet 0.1.0 Yes 9 0
windows Yes No 0 0
xraylib No No 0 0
zenity No No 14 0

Looking more closely at pkg(), it is possible to get the list of functions each package contains. With this list we can call test() on each, and collate the results. This should give more representative results. I will have a go at replacing the above mechanism.


Becoming a maintainer of octave packages

I have asked the octave-maintainers@gnu.org mailing list which packages I should pick up if I wanted to maintain some packages. Any of the unmaintained packages could be a candidate, but I figured that it would be better to ask which anyone cared about rather than just wade in and start maintaining some of them. These are the unmaintained packages I’ve made successful builds of:

  • actuarial
  • audio
  • benchmark
  • bioinfo
  • civil-engineering
  • engine
  • fenv
  • gnuplot
  • gsl
  • information-theory
  • integration
  • irsa
  • mapping
  • missing-functions
  • multicore
  • nlwing2
  • nnet
  • octgpr
  • odebvp
  • outliers
  • plot
  • simp
  • special-matrix
  • symband

While I realise that making a successful build does not mean something is easy to get to a maintainable state, it is at least a good place to start from. I will take a closer look at these packages over the next few days.


Improving the octave build

Looking further at config.log and the output from configure, there are a few further improvements that could be made to make the build better.
Add –enable-jit option, although this causes a warning claiming to be a missing LLVM include file, but closer examination of the config.log file shows actually is caused by failure to use the required ‘–std=gnu++11’ option to the compiler. Including this option, however, causes other build failures. For the moment I will keep this option but not worry about the warning it generates.
Add –with-openssl option. This opened a can of worms. Starting with disabling the GUI, with a claim that it couldn’t find the OpenGL library. This was actually not the case, instead the OpenGL library was missing a TLS function (it remains unclear to me so far why a general purpose graphics library should use encryption, but such are the mysteries of open source software). Adding the –enable-glx-tls option to the mesa build should resolve that. Only, it doesn’t.
Turns out that what I am building is 64-bit software, so the libraries land in /usr/local/lib64 rather than the search path I had given of /usr/local/lib. Confusingly there appears to be duplicates of a number of libraries between the two directories and their versions are not necessarily in sync. Most of this is because I have built things, but I don’t think I’m responsible for all of it. I don’t yet understand how Linux handles thunking around between 32 and 64 bit. I’m guessing 64-bit software can easily thunk down to use 32-bit libraries, but that it is either difficult or impossible to go the other way. From what I remember, also, dependency handling for different versions is a bit of a mess, meaning it would be possible to have both 32-bit and 64-bit versions of the same library loaded by the same program and there would be no guarantee that they would be of the same code version and configuration. This could lead to some entertaining bugs where doing the same thing in slightly different ways within a program might lead to different behaviours. But that is not what I am considering here, I just want to build it at the moment.
Then using static data in 64-bit libraries causes a build error unless you are building with the -fPIC option – which was not the case for fltk, adding the –enable-shared option to the fltk build resolves that.
Examining the build configuration shows OSMesa and Sndfile are missing.
OSMesa can be enabled as part of the mesa build, but again there is a dependency on TLS support, which is not fulfilled – looking into this a bit more there are again two versions of libOSMesa.so in my system, this time I know it to be my fault as one is in /usr/lib, the other /usr/local/lib. The first step to resolving this is to ensure that there is only one version of libGL.so on the library path, then rebuilding mesa.
And I have set up a new Jenkins project for libSndFile, this in turn depends upon FLAC.


Fixing the octave build break

As I noted before, the ‘clean build’ option was not set in the Octave build, setting this option and resolving the errors caused by it – I had manually copied groff.enc into the build tree, which now needs to be done automatically – makes the build work. This brings up a couple of points, it seems that the Jenkins default configuration for ‘freestyle’ projects is not to clean when launching a new build. I will need to check the other projects to make sure they are set to clean too. It also makes me think that Jenkins ‘freestyle’ build configurations are not under change control. The way to resolve this is to have a build script which is under change control and call that and only that from the Jenkins build. I don’t think what I am doing here is significant enough to warrant that, so I will leave my configuration as is, but it is something to think about on more complex configurations.