Proj with Python bindings installation confusion in Dockerfile

[justb4]

I see that through the pygeoapi Dockerfile both python3-proj and the PyPi package pyproj (via requirements.txt) are installed. My question arises, since I think one gets two full and even different versions installed at different locations /usr/ (python3-proj) and /usr/local (pyproj). At least I did a similar install in a MapProxy Docker Image.

"Bashing" into that running MapProxy image I see:

$  /usr/bin/pyproj -v
pyproj info:
    pyproj: 3.4.1
      PROJ: 9.1.1
  data dir: /usr/share/proj
user_data_dir: /tmp/proj
PROJ DATA (recommended version): 1.12
PROJ Database: 1.2
EPSG Database: v10.076 [2022-08-31]
ESRI Database: ArcGIS Pro 3.0 [2022-07-09]
IGNF Database: 3.1.0 [2019-05-24]

System:
    python: 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0]
executable: /usr/bin/python3
   machine: Linux-5.4.0-182-generic-x86_64-with-glibc2.36

Python deps:
   certifi: 2022.9.24
    Cython: None
setuptools: None
       pip: None

And (the default)

$ /usr/local/bin/pyproj -v
pyproj info:
    pyproj: 3.6.1
      PROJ: 9.3.0
  data dir: /usr/local/lib/python3.11/site-packages/pyproj/proj_dir/share/proj
user_data_dir: /tmp/proj
PROJ DATA (recommended version): 1.15
PROJ Database: 1.2
EPSG Database: v10.094 [2023-08-08]
ESRI Database: ArcGIS Pro 3.1 [2023-19-01]
IGNF Database: 3.1.0 [2019-05-24]

System:
    python: 3.11.9 (main, May 14 2024, 08:15:15) [GCC 12.2.0]
executable: /usr/local/bin/python
   machine: Linux-5.4.0-182-generic-x86_64-with-glibc2.36

Python deps:
   certifi: 2024.2.2
    Cython: None
setuptools: 65.5.1
       pip: 24.0

The latter is the default:

$ which pyproj
/usr/local/bin/pyproj

The PyPi version pyproj also is more recent. Can I assume that we could leave out python3-proj? Or do we have to install a libproj-dev ? Maybe @sebastic knows?

[sebastic]

In my opinion you should use only Debian packages for your (Python) dependencies, or get everything from PyPI.

But if the Python environments are separate, it shouldn't be an issue to have pyproj installed for both.

Just don't link to differently versions of libproj in your address space through dependencies, this happens to QGIS users relatively often when they install different versions of dependencies locally or use packages from PPAs that aren't properly integrated.

[justb4]

Thanks @sebastic ! Yes, that is my usual approach: use Debian python-* packages, unless the lib is Python-only. Maybe what is also not clear to me: looks like pyproj is a complete install, including PROJ binaries itself, not just Python bindings (?) like with GDAL. As I remember, I first used python3-proj-only. Then got errors that disappeared when adding PyPi pyproj.

But I will revisit the install, maybe it is some variable like PROJ_DATA or PYTHONPATH. Searching the web gives so many issues for different versions and installs of Proj. I will let know here as in the best case we can leave out pyproj also in the pygeoapi Dockerfile (and promote this Discussion to an Issue)....

Edit: the official MapProxy Dockerfile, based on Debian Bookworm, uses python3-proj only, but applies this nasty fix:

# fix potential issue finding correct shared library libproj (fixed in newer releases)
RUN ln -s /usr/lib/`uname -m`-linux-gnu/libproj.so /usr/lib/`uname -m`-linux-gnu/liblibproj.so

[justb4]

TL;DR
After trial/error:

• pip install pyproj can be omitted.
• the nasty dynlink fix above is not required.
• Need two ENV vars in the Dockerfile:

PYTHONPATH=/usr/lib/python3/dist-packages
PROJ_DATA=/usr/share/proj

The PYTHONPATH "does not feel right" though...

Expanded with Logs

First

  File "/usr/local/lib/python3.11/site-packages/mapproxy/config/loader.py", line 62, in load_globals
    self.globals = GlobalConfiguration(conf_base_dir=conf_base_dir,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mapproxy/config/loader.py", line 347, in __init__
    self.preferred_srs = preferred_srs(self.conf.get('srs', {}))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mapproxy/config/loader.py", line 323, in preferred_srs
    from mapproxy.srs import SRS, PreferredSrcSRS
  File "/usr/local/lib/python3.11/site-packages/mapproxy/srs.py", line 26, in <module>
    from mapproxy.proj import USE_PROJ4_API
  File "/usr/local/lib/python3.11/site-packages/mapproxy/proj.py", line 282, in <module>
    res = try_import()
          ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mapproxy/proj.py", line 155, in try_libproj_import
    libproj = init_libproj()
              ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mapproxy/proj.py", line 66, in init_libproj
    libproj.pj_init_plus.argtypes = [c_char_p]
    ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ctypes/__init__.py", line 389, in __getattr__
    func = self.__getitem__(name)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ctypes/__init__.py", line 394, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: /lib/x86_64-linux-gnu/libproj.so.25: undefined symbol: pj_init_plus

This can be fixed by setting the ENV var PYTHONPATH=/usr/lib/python3/dist-packages.

Then next:

/usr/lib/python3/dist-packages/pyproj/__init__.py:91: 
UserWarning: Valid PROJ data directory not found. 
Either set the path using the environmental variable 
PROJ_DATA (PROJ 9.1+) | PROJ_LIB (PROJ<9.1) or with `pyproj.datadir.set_data_dir`.

This can be fixed by setting the ENV var PROJ_DATA=/usr/share/proj.

So now only /usr/bin/pyproj is present giving:

pyproj info:
    pyproj: 3.4.1
      PROJ: 9.1.1
  data dir: /usr/share/proj
user_data_dir: /home/mapproxy/.local/share/proj
PROJ DATA (recommended version): 1.12
PROJ Database: 1.2
EPSG Database: v10.076 [2022-08-31]
ESRI Database: ArcGIS Pro 3.0 [2022-07-09]
IGNF Database: 3.1.0 [2019-05-24]

System:
    python: 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0]
executable: /usr/bin/python3
   machine: Linux-6.6.22-linuxkit-aarch64-with-glibc2.36

Python deps:
   certifi: 2022.9.24
    Cython: None
setuptools: None
       pip: None