Installing netcdf fortran in docker container

[niravshah241]

Hello,

I am trying to install netcdf-fortran 4.6.2 in a Docker image. I am using ubuntu:latest as base image. I installed netcdf-c 4.9.3 using

CC=$(which mpicc) CPPFLAGS="-I${H5DIR}/include -I${PNDIR}/include -I${ZLDIR}/include" \
LDFLAGS="-L${H5DIR}/lib -L${PNDIR}/lib -L${ZLDIR}/lib" \
LIBS="-L${H5DIR}/lib -lhdf5_hl -lhdf5 -L${PNDIR}/lib -lpnetcdf -L${ZLDIR}/lib -lz -lm" \
./configure --enable-pnetcdf --enable-parallel-tests --disable-libxml2 --disable-shared \
--enable-static --prefix=${NCDIR} && \
make && \
make install

The H5DIR is HDF5 installation directory and similarly PNDIR, ZLDIR are PnetCDF and Zlib installation directories. The netcdf-c installation location is NCDIR .

CC=$(which mpicc) FC=$(which mpif90) F77=$(which mpif77) \
CPPFLAGS="-I${NCDIR}/include -I${H5DIR}/include -I${PNDIR}/include -I${ZLDIR}/include" \
LDFLAGS=$(nc-config --libs) \
./configure --disable-shared --enable-parallel-tests --prefix=${NFDIR} && \
make && \
make install

However, it gives me error

 Could not link to netcdf C library. Please set LDFLAGS; for static builds set LIBS to the results of nc-config --libs.

I tried removing LDFLAGS and setting LIBS=$(nc-config --libs). I got the same error again. Can some one help?

[WardF]

Try modifying LDFLAGS=$(nc-config --libs) to LDFLAGS=$(nc-config --libs --static) and see if that works. If not, can you enter the docker container, or otherwise modify your Dockerfile, so that we can see the output from nc-config --libs --static?

[niravshah241]

Hello. I tried modifying LDFLAGS to nc-config --libs --static but I get same error. Here is output of nc-config --libs --static

-L${NCDIR}/lib -L${H5DIR}/lib -L${PNDIR}/lib -L${ZLDIR}/lib -lnetcdf -lpnetcdf -lm -lcurl -L${H5DIR}/lib -lhdf5_hl -lhdf5 -L${PNDIR}/lib -lpnetcdf -L${ZLDIR}/lib -lz -lm

Additionally, here is how I installed HDF5 1.10.11

CC=$(which mpicc) ./configure --enable-parallel --enable-fortran --enable-hl --with-szlib=${ZLDIR} --prefix=${H5DIR}
make
make install

[WardF]

What is the output from running env | sort ?

[niravshah241]

Here is env | sort output:

CUDIR=/usr/bin/curl
H5DIR=/path/to/hdf5/installation
HOME=/root
HOSTNAME=some_host_name
LD_LIBRARY_PATH=${NCDIR}:${NCDIR}/include:${NCDIR}/lib:${NCDIR}/bin
LESSCLOSE=/usr/bin/lesspipe %s %s
LESSOPEN=| /usr/bin/lesspipe %s
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90:
NCDIR=/path/to/netcdf-c
NFDIR=/path/for/netcdf-fortran
OLDPWD=/project_directory
PATH=${NCDIR}:${NCDIR}/include:${NCDIR}/lib:${NCDIR}/bin:${PNDIR}/bin:${PNDIR}/include:${PNDIR}/lib:${H5DIR}/include:${H5DIR}/lib:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PNDIR=/path/to/pnetcdf
PWD=/project_directory/netcdf-fortran-4.6.2
SHLVL=1
TERM=xterm
ZLDIR=${ZLDIR}
_=/usr/bin/env

[WardF]

Hello. I tried modifying LDFLAGS to nc-config --libs --static but I get same error. Here is output of nc-config --libs --static

-L${NCDIR}/lib -L${H5DIR}/lib -L${PNDIR}/lib -L${ZLDIR}/lib -lnetcdf -lpnetcdf -lm -lcurl -L${H5DIR}/lib -lhdf5_hl -lhdf5 -L${PNDIR}/lib -lpnetcdf -L${ZLDIR}/lib -lz -lm

Additionally, here is how I installed HDF5 1.10.11

CC=$(which mpicc) ./configure --enable-parallel --enable-fortran --enable-hl --with-szlib=${ZLDIR} --prefix=${H5DIR}
make
make install

Looking at this, it is curious to me that the output from nc-config is not expanding the environmental variables. It is printing -L${NCDIR}/lib instead of -L/path/to/netcdf-c/lib. This suggests to me that the environmental variable expansion is not happening for some reason.

Do you have the Dockerfile you are working from? Perhaps I can make sense of what's going on if I look at that directly.

Alternatively, you might examine how I build netCDF-Fortran (via a script inside a docker container) as part of our CI. If you run

$ docker run --rm -it --entrypoint bash docker.unidata.ucar.edu/nctests

You can look through the run_netcdf_tests.sh script present in the directory you will find yourself in. Perhaps that will provide some hints.

[niravshah241]

Hello. Thank you for your reply. The nc-config is printing absolute paths. I modified in my answer for ease of readability. Please find here my docker file.

# Linux base image
FROM ubuntu:latest

# Set Work directory
RUN mkdir -p /home/TIMBER
WORKDIR /home/TIMBER
RUN mkdir -p /home/TIMBER/hdf5
RUN mkdir -p /home/TIMBER/pnetcdf

# Install MPI
RUN apt-get update
RUN apt-get install -y cmake
RUN apt-get install -y python3
RUN apt-get install -y python3-pip
RUN apt-get clean
RUN apt install -y build-essential
RUN apt-get install -y curl
RUN apt-get install -y libcurl4-openssl-dev
RUN apt-get install libxml2
RUN apt-get install -y libcurl4-gnutls-dev

# Install git
RUN apt-get install -y git

# Install wget
RUN apt-get install wget -y

# Install zlib
RUN mkdir -p /home/TIMBER/zlib
RUN wget https://github.com/madler/zlib/archive/refs/tags/v1.3.tar.gz
RUN tar -xzvf v1.3.tar.gz
RUN cd zlib-1.3/ && ./configure --prefix=/home/TIMBER/zlib && make && make install

# Install szip

# Install MPI
RUN apt-get install -y openmpi-bin openmpi-doc libopenmpi-dev

# Install HDF5
# Gives error in make check but make check-install seems ok (??)
RUN wget https://github.com/HDFGroup/hdf5/archive/refs/tags/hdf5-1_10_11.tar.gz
RUN tar xzf hdf5-1_10_11.tar.gz
RUN cd hdf5-hdf5-1_10_11/ && CC=$(which mpicc) ./configure \
--enable-parallel --enable-fortran --enable-hl --with-szlib=/home/TIMBER/zlib \
--prefix=/home/TIMBER/hdf5 && make && make install && make check-install
# RUN make check 
# RUN make install
# RUN make check-install

ENV PATH=/home/TIMBER/hdf5/lib:$PATH
ENV PATH=/home/TIMBER/hdf5/include:$PATH

# Install PnetCDF
RUN wget https://parallel-netcdf.github.io/Release/pnetcdf-1.13.0.tar.gz
RUN tar xzf pnetcdf-1.13.0.tar.gz
RUN cd pnetcdf-1.13.0 && \
./configure --prefix=/home/TIMBER/pnetcdf \
MPICC=$(which mpicc) MPICXX=$(which mpicxx) MPIF77=$(which mpif77) MPIF90=$(which mpif90) \
&& make -j8 && make check && make install prefix=/home/TIMBER/pnetcdf
# make ptest
ENV PATH=/home/TIMBER/pnetcdf/lib:$PATH
ENV PATH=/home/TIMBER/pnetcdf/include:$PATH
ENV PATH=/home/TIMBER/pnetcdf/bin:$PATH
RUN which ncmpidump
RUN which ncmpidiff

# Install netCDF
# netCDF c
ENV NCDIR=/home/TIMBER/netcdf/c
ENV NFDIR=/home/TIMBER/netcdf/fortran
ENV H5DIR=/home/TIMBER/hdf5
ENV ZLDIR=/home/TIMBER/zlib
ENV PNDIR=/home/TIMBER/pnetcdf
RUN mkdir -p /home/TIMBER/netcdf/c
RUN wget https://github.com/Unidata/netcdf-c/archive/refs/tags/v4.9.3.tar.gz
RUN tar xzf v4.9.3.tar.gz
RUN cd netcdf-c-4.9.3/ && \
    CC=$(which mpicc) CPPFLAGS="-I${H5DIR}/include -I${PNDIR}/include -I${ZLDIR}/include" \
    LDFLAGS="-L${H5DIR}/lib -L${PNDIR}/lib -L${ZLDIR}/lib" \
    LIBS="-L${H5DIR}/lib -lhdf5_hl -lhdf5 -L${PNDIR}/lib -lpnetcdf -L${ZLDIR}/lib -lz -lm" \
    ./configure --enable-pnetcdf --enable-parallel-tests --disable-libxml2 --disable-shared \
    --enable-static --prefix=${NCDIR} && make && make install
# make test failing on 7 tests (??)
ENV PATH=/home/TIMBER/netcdf/c/bin:$PATH
ENV PATH=/home/TIMBER/netcdf/c/lib:$PATH
ENV PATH=/home/TIMBER/netcdf/c/include:$PATH
ENV PATH=/home/TIMBER/netcdf/c:$PATH

ENV LD_LIBRARY_PATH=/home/TIMBER/netcdf/c/bin
ENV LD_LIBRARY_PATH=/home/TIMBER/netcdf/c/lib:$LD_LIBRARY_PATH
ENV LD_LIBRARY_PATH=/home/TIMBER/netcdf/c/include:$LD_LIBRARY_PATH
ENV LD_LIBRARY_PATH=/home/TIMBER/netcdf/c:$LD_LIBRARY_PATH

# netCDF Fortran
ENV NFDIR=/home/TIMBER/netcdf/fortran
ENV CUDIR=/usr/bin/curl
RUN mkdir -p /home/TIMBER/netcdf/fortran
RUN wget https://downloads.unidata.ucar.edu/netcdf-fortran/4.6.2/netcdf-fortran-4.6.2.tar.gz
RUN tar xzf netcdf-fortran-4.6.2.tar.gz

# RUN cd netcdf-fortran-4.6.2 && \
# CC=$(which mpicc) FC=$(which mpif90) F77=$(which mpif77) \
# CPPFLAGS="-I${NCDIR}/include -I${H5DIR}/include -I${PNDIR}/include -I${ZLDIR}/include" \
# LDFLAGS="-L${NCDIR}/lib -L${H5DIR}/lib -L${PNDIR}/lib -L${ZLDIR}/lib" \
# LD_LIBRARY_PATH=${NCDIR}/lib:${H5DIR}/lib:${PNDIR}/lib:${ZLDIR}/lib \
# LIBS="-L${NCDIR}/lib -lnetcdf -L${H5DIR}/lib -lhdf5_hl -lhdf5 -L${PNDIR}/lib -lpnetcdf -L${ZLDIR}/lib -lz" \
# ./configure --disable-shared --prefix=${NFDIR} && make && make install

# RUN cd netcdf-fortran-4.6.2 && \
# CC=$(which mpicc) FC=$(which mpif90) F77=$(which mpif77) \
# CPPFLAGS="-I${NCDIR}/include -I${H5DIR}/include -I${PNDIR}/include -I${ZLDIR}/include" \
# LDFLAGS="-L${NCDIR}/lib -L${H5DIR}/lib -L${PNDIR}/lib -L${ZLDIR}/lib" \
# LD_LIBRARY_PATH=${NCDIR}/lib:${H5DIR}/lib:${PNDIR}/lib:${ZLDIR}/lib \
# LIBS="-L${NCDIR}/lib -lnetcdf -L${H5DIR}/lib -lhdf5_hl -lhdf5 -L${PNDIR}/lib -lpnetcdf -L/lib -lz -lm" \
# ./configure --disable-shared --prefix=${NFDIR} && make && make install

# RUN cd netcdf-fortran-4.6.2 && \
# CC=$(which mpicc) FC=$(which mpif90) F77=$(which mpif77) \
# CPPFLAGS="-I${NCDIR}/include -I${H5DIR}/include -I${PNDIR}/include -I${ZLDIR}/include" \
# ./configure --disable-shared --enable-parallel-tests --prefix=${NFDIR} && make && make install

# RUN cd netcdf-fortran-4.6.2 && \
# CC=$(which mpicc) FC=$(which mpif90) F77=$(which mpif77) \
# CPPFLAGS="-I${NCDIR}/include -I${H5DIR}/include -I${PNDIR}/include -I${ZLDIR}/include" \
# LDFLAGS=$(nc-config --libs) \
# ./configure --disable-shared --enable-parallel-tests --prefix=${NFDIR} && \
# make && make install

# # RUN make check
# # RUN make check install
# ENV PATH=/home/TIMBER/netcdf/fortran/bin:$PATH
# ENV PATH=/home/TIMBER/netcdf/fortran/lib:$PATH
# ENV PATH=/home/TIMBER/netcdf/fortran/include:$PATH
# ENV PATH=/home/TIMBER/netcdf/fortran:$PATH

# Install XIOS(1 or 2)

# Install Nemo

# Install Planktom

# Keep container alive
CMD ["sleep", "infinity"]

[niravshah241]

I found a work-around which seems to let netcdf-fortran is below. After installing netcdf-c, I did apt-get installation:
apt-get install -y libnetcdff-dev . This installs netcdf-fortran 4.5.4.

and then when I run below, which lets netcdf-fortran 4.6.2 be installed:

CC=$(which mpicc) FC=$(which mpif90) F77=$(which mpif77) \
CPPFLAGS="-I${NCDIR}/include -I${H5DIR}/include -I${PNDIR}/include -I${ZLDIR}/include" \
./configure --disable-shared --enable-parallel-tests --prefix=${NFDIR} && \
make && \
make install

and add relevant directories to PATH

ENV PATH=/home/TIMBER/netcdf/fortran/bin:$PATH
ENV PATH=/home/TIMBER/netcdf/fortran/lib:$PATH
ENV PATH=/home/TIMBER/netcdf/fortran/include:$PATH
ENV PATH=/home/TIMBER/netcdf/fortran:$PATH

Afterwards, when I do nf-config --all, I get following:

This netCDF-Fortran 4.6.2 has been built with the following features:

  --cc        -> /usr/bin/mpicc
  --cflags    -> -I/home/TIMBER/netcdf/fortran/include -I/home/TIMBER/netcdf/c/include -I/home/TIMBER/hdf5/include -I/home/TIMBER/pnetcdf/include -I/home/TIMBER/zlib/include

  --fc        -> /usr/bin/mpif90
  --fflags    -> -I/home/TIMBER/netcdf/fortran/include -I/home/TIMBER/netcdf/fortran/include
  --flibs     -> -L/home/TIMBER/netcdf/fortran/lib -lnetcdff  -lnetcdf -lnetcdf -lm
  --has-f90   ->
  --has-f03   -> yes

  --has-nc2   -> yes
  --has-nc4   -> yes

  --prefix    -> /home/TIMBER/netcdf/fortran
  --includedir-> /home/TIMBER/netcdf/fortran/include
  --version   -> netCDF-Fortran 4.6.2

This approach requires 2 netcdf installs, the one with apt-get and one with netcdf-fortran 4.6.2. While, this is not ideal, I think this can be a temporary fix. I guess this also helps to understand the issue better.