Ubuntu PPA

[staticfloat]

EDIT: For the impatient ones, Skip down to my comment below with directions on how to install all dependencies for Julia on Ubuntu 12.04.

It'd be great to have an Ubuntu PPA that collects all the software that Julia needs to compile. This could significantly cut down on compile times, and would be a nice first step toward having actual binary packages of Julia for debian.

From my tests, the following packages work with/are necessary to build Julia:

gfortran
llvm-3.1-dev (ppa:dr-graef/llvm-3.1.precise)
libblas-dev
liblapack-dev
libarpack2
libpcre3-dev
libglpk-dev
lighttpd
libgmp-dev
libunwind7-dev
libreadline-dev

This leaves SuiteSparse (Only 3.4.0 is available, no 4.0.2, as far as I can see), FFTW (Installing libfftw3-dev didn't work for me; I got the dreaded LoadError, so I figured I'd just let Julia compile her own which works), zlib (It doesn't come with large file support by default), and whatever else I'm missing.

[ViralBShah]

There is very little difference from SuiteSparse 3.4.0 to SuiteSparse 4.0.2. Does it not just work with 3.4.0?

[dmbates]

libblas-dev depends on the libblas3 which is the package for the reference blas. Installing reference blas and reference lapack will result in slower linear algebra code that installing OpenBLAS or Atlas. There is a libopenblas-dev package but the current version is 0.1.1 whereas the version specified in julia/deps/Versions.make is 0.2.4. I don't know why the Ubuntu version is so old - there may be problems with the license for later versions.

Similarly, libpcre3-dev on Ubuntu 12.10 is version 8.30 whereas Version.deps asks for 8.31. I'm not sure if that is important. For libglpk-dev version 4.45 is available but 4.47 is requested.. For libarpack2-dev version 3.1.1 is available and 3.1.2 is requested. ...

Viral has already suggested that an earlier SuiteSparse may be acceptable. It would be good to determine which available Ubuntu/Debian packages are acceptable for compiling Julia. When the currently released version is not acceptable it may be possible to get the source package and upgrade it.

For example, the sources for the debian subdirectory of suitesparse packages are maintained in a git repository at http://anonscm.debian.org/git/pkg-scicomp/suitesparse.git/ but they haven't been updated in 2 years.

[staticfloat]

@ViralBShah: SuiteSparse 3.4.0 from the ubuntu repository does not work for me. When just running make (Because I forget to first run repackage_system_suitesparse3.make) I get the following error:

SuiteSparse_wrapper.c:109:25: error: use of undeclared identifier 'SuiteSparse_long'
        isize =  sizeof(SuiteSparse_long); break;
                        ^
SuiteSparse_wrapper.c:118:9: error: use of undeclared identifier 'SuiteSparse_long'
        SuiteSparse_long *spt = (SuiteSparse_long *) cs->p;
        ^

Which does on for a while, but I think you get the gist. Even when I run repackage_system_suitesparse3.make first, (even though I don't believe it would fix the above error) I get the following:

/usr/bin/ld: libamd.a(amd_i_control.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
libamd.a(amd_i_control.o): could not read symbols: Bad value
collect2: ld returned 1 exit status

I'm not entirely sure what this error is about, though.

@dmbates: I'm not sure how to go about creating deb's. Never done it before, although I am looking into it.

[staticfloat]

Because OpenBLAS has such a finely-grained hardware compilation process (e.g. it specializes on CPU architecture) I'm not sure dpkg has methods for choosing the proper deb for any given computer. I just compiled a .deb of OpenBLAS 0.2.4, but it's specialized for my processor.

[ViralBShah]

As @staticfloat said, some of these will not "just work", but it should be easy to make them work. I cannot imagine us depending on new features of any of these. The exceptions are openblas, which is old, and also that we compile lapack into it with the openblas patches. At some point, perhaps debian/ubuntu can have an openblas package that provides the lapack dependency as well - but I think we will have to take it up with the Debian Science Maintainers.

The other one is Rmath, where we have patched it to use julia's librandom (basically dsfmt+randmtzig). For the rest, we should be able to use existing debian/ubuntu packages with some small modifications.

[ViralBShah]

@staticfloat if you build openblas with DYNAMIC_ARCH=1, it should build for all the cpu types it knows about. The julia make install does this, and you end up with a libopenblas that is twice as large.

[dmbates]

I have successfully built and tested Julia under Ubuntu 12.10 amd64 with the flags

USE_SYSTEM_LLVM=1 USE_SYSTEM_READLINE=1 USE_SYSTEM_ZLIB=1 USE_SYSTEM_ARPACK=1 USE_SYSTEM_FFTW=1 USE_SYSTEM_PCRE=1 USE_SYSTEM_GMP=1 USE_SYSTEM_LIBUNWIND=1 USE_SYSTEM_LIGHTTPD=1

I did remove the contents of the julia/usr/lib directory before checking that Julia would pass testall with these system libraries. This cuts down on the number of libraries in julia/usr/lib but of course the biggest gains could be realized by getting system packages of openblas and suitesparse.

May I suggest that the USE_DEBIAN stanza in julia/Make.inc be replaced with only these flags and possibly renamed to USE_UBUNTU_12.10 or something like that.

Also, I think it would be a good idea to change to

OPENBLAS_VER = v0.2.4
LAPACK_VER = 3.4.2

in julia/deps/Versions.make

Both of these versions have been out for some time and it would be good to keep current.

[staticfloat]

@dmbates: Great, that's exactly what I've experienced with the exception of LLVM on 12.04. (I had to use a third-party backport to 12.04 to get it)

Also, I've been using Homebrew for so long, I had no idea that Julia natively built those OpenBLAS and LAPACK versions! I'll go open a pull request.

[staticfloat]

And I am delirious, and did not remember that OpenBLAS 0.2.4 is the one I tested and didn't work because of this bug.

[staticfloat]

I've started a "Julia Dependencies" PPA, but I'm still ironing out some issues. LLVM 3.1 and FFTW should pretty much work as I simply backported them from Quantal (Ubuntu 12.10).

[svillemot]

Hi,

I am a Debian Developer, and I am interested in getting Julia into Debian (and therefore into Ubuntu).

Before this can happen, some work is needed on dependencies. AFAICT, here are the dependencies that are not yet packaged in Debian:

• SuiteSparse 4.0.2 (should happen soon, 4.0.2 is already committed in Git)
• double-conversion
• openlibm
• libuv
• amos (though we may decide to not use an independent package and reuse the files in julia's tree)
• random/dsfmt

OpenBLAS is indeed at version 0.1.1 in unstable (0.2.4 in experimental), but this is not an issue. FFTW and zlib seems to work fine, unless I am missing something. LLVM 3.1 works fine as long as you set LLVM_CONFIG=llvm-config-3.1 on the command line.

I don't know exactly when I will be able to do this packaging work. Of course, any help is welcome. In particular, I am willing to sponsor uploads into Debian.

[staticfloat]

Great to hear, Sébastien! As I pointed out above, I've started putting together an Ubuntu PPA, but as this is my first time packaging software for Ubuntu, I'm certain I've done a rather shoddy job (For instance, I haven't been packaging SuiteSparse or OpenBLAS as libraries, but rather as a single binary!). Someone with your experience will help a lot in this regard.

I believe the reason zlib wasn't working for me was that zlib must be compiled with large file support which the 32 bit version available in Ubuntu is not, but the 64-bit version is. (And by 32 vs. 64-bit versions, I am referring to lib32z vs. lib64z)

I'm still not certain why the FFTW available in 12.04 didn't work. It's only a few versions behind, but Julia didn't want to link with it. Backporting the 12.10 version worked, however, and it's of the version we want (>= 3.3).

[nolta]

We need FFTW 3.3.2 because of this fix (from the release notes):

Make libfftw{threads,mpi} depend upon libfftw when using libtool; this is consistent with most other libraries and simplifies the life of various distributors of GNU/Linux.

You should be able to use an earlier version if you dlopen "libfftw3" instead of "libfftw3_threads".

[svillemot]

Here is the Intent to Package (ITP) that I opened today in Debian: http://bugs.debian.org/691912

I'll be commenting there when progress is being made.

[ViralBShah]

@sebastien-villemot - Thanks a lot for the Debian ITP. This will be incredibly useful for julia. I am happy to help in any way possible to get julia accepted into debian.

I also notice that you are one of the Dynare developers. It would be interesting to hear on views on dynare and julia at some point.