autotest: footprint test fails if run after mdim convert test

[dbaston]

What is the bug?

Running autotest/utilities/test_gdalalg_pipeline.py::test_gdalalg_pipeline_footprint after autotest/utilities/test_gdalalg_mdim_convert.py::test_gdalalg_mdim_convert_creation_option fails on OSX. The tests succeed if run independently.

Steps to reproduce the issue

pytest -s autotest/utilities/test_gdalalg_mdim_convert.py::test_gdalalg_mdim_convert_creation_option autotest/utilities/test_gdalalg_pipeline.py::test_gdalalg_pipeline_footprint

fails with

________________________________________________________________________________________ test_gdalalg_pipeline_footprint _________________________________________________________________________________________

    def test_gdalalg_pipeline_footprint():

>       with gdal.Run(
            "pipeline",
            input="../gcore/data/byte.tif",
            output_format="MEM",
            output="",
            pipeline="read ! footprint ! write",
        ) as alg:

/Users/dan/dev/gdal/build/autotest/utilities/test_gdalalg_pipeline.py:358:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/Users/dan/dev/gdal/build/swig/python/osgeo/gdal.py:3593: in Run
    if not alg.Run(progress):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <osgeo.gdal.Algorithm; proxy of <Swig Object of type 'GDALAlgorithmHS *' at 0x1139ce730> >, args = (None,)

    def Run(self, *args):
        r"""Run(self, callback=0, callback_data=None) -> bool"""

        self.has_run = True


>       return _gdal.Algorithm_Run(self, *args)
E       RuntimeError: Terminating translation prematurely after failed
E       translation of layer footprint (use -skipfailures to skip errors)
E       May be caused by: stream.get_next() failed
E       May be caused by: disk I/O error

Versions and provenance

master, 2bde98d

Additional context

I'm guessing it has something to do with both tests using byte.tif.

[dbaston]

I think the "disk I/O error" message originates from SQLite. This environment has a handful of test failures related to SQLite so this may be related.

[rouault]

I think the "disk I/O error" message originates from SQLite.

I assume this comes from the use of a temporary GeoPackage file in gdalalg_raster_footprint.cpp. Can you try to comment line 350 "VSIUnlink(outputFilename.c_str());" in it ? I'm wondering if some SQLite versions might not like working on a deleted file. What is your SQLite version ? Is it one coming from the operating system itself (I know Apple patches / applies some non default build options), or through other packaging system?

[dbaston]

Yes, the test passes after commenting line 350. I am trying to figure out where SQLite is coming from but it doesn't seem simple (to me) on OS X. The linker command is using /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib/libsqlite3.tbd, the contents of which start with

--- !tapi-tbd
tbd-version:     4
targets:         [ x86_64-macos, x86_64-maccatalyst, x86_64h-macos, x86_64h-maccatalyst,
                   arm64-macos, arm64-maccatalyst, arm64e-macos, arm64e-maccatalyst ]
install-name:    '/usr/lib/libsqlite3.dylib'
current-version: 380
compatibility-version: 9
exports:

But it doesn't seem possible that a 1-year-old computer is using SQLite 3.8. /usr/bin/sqlite3 shows version 3.43.2.

[rouault]

@dbaston try ogrinfo :memory: -sql "select sqlite_version()" and ogrinfo :memory: -sql "PRAGMA compile_options" -al -q | grep compile_options

[dbaston]

dan@mini build % ogrinfo :memory: -sql "select sqlite_version()"
ERROR 1: OGR2SQLITE_Setup() failed due to sqlite3_api == nullptr
INFO: Open of `:memory:'
      using driver `SQLite' successful.

Layer name: SELECT
Geometry: None
Feature Count: 1
Layer SRS WKT:
(unknown)
sqlite_version(): String (0.0)
OGRFeature(SELECT):0
  sqlite_version() (String) = 3.43.2

dan@mini build % ogrinfo :memory: -sql "PRAGMA compile_options" -al -q | grep compile_options
ERROR 1: OGR2SQLITE_Setup() failed due to sqlite3_api == nullptr
  compile_options (String) = ATOMIC_INTRINSICS=1
  compile_options (String) = BUG_COMPATIBLE_20160819
  compile_options (String) = CCCRYPT256
  compile_options (String) = COMPILER=clang-17.0.0
  compile_options (String) = DEFAULT_AUTOVACUUM
  compile_options (String) = DEFAULT_CACHE_SIZE=2000
  compile_options (String) = DEFAULT_CKPTFULLFSYNC
  compile_options (String) = DEFAULT_FILE_FORMAT=4
  compile_options (String) = DEFAULT_JOURNAL_SIZE_LIMIT=32768
  compile_options (String) = DEFAULT_LOOKASIDE=1200,102
  compile_options (String) = DEFAULT_MEMSTATUS=0
  compile_options (String) = DEFAULT_MMAP_SIZE=0
  compile_options (String) = DEFAULT_PAGE_SIZE=4096
  compile_options (String) = DEFAULT_PCACHE_INITSZ=20
  compile_options (String) = DEFAULT_RECURSIVE_TRIGGERS
  compile_options (String) = DEFAULT_SECTOR_SIZE=4096
  compile_options (String) = DEFAULT_SYNCHRONOUS=2
  compile_options (String) = DEFAULT_WAL_AUTOCHECKPOINT=1000
  compile_options (String) = DEFAULT_WAL_SYNCHRONOUS=1
  compile_options (String) = DEFAULT_WORKER_THREADS=0
  compile_options (String) = DQS=3
  compile_options (String) = ENABLE_API_ARMOR
  compile_options (String) = ENABLE_BYTECODE_VTAB
  compile_options (String) = ENABLE_COLUMN_METADATA
  compile_options (String) = ENABLE_DBSTAT_VTAB
  compile_options (String) = ENABLE_FTS3
  compile_options (String) = ENABLE_FTS3_PARENTHESIS
  compile_options (String) = ENABLE_FTS3_TOKENIZER
  compile_options (String) = ENABLE_FTS4
  compile_options (String) = ENABLE_FTS5
  compile_options (String) = ENABLE_LOCKING_STYLE=1
  compile_options (String) = ENABLE_MATH_FUNCTIONS
  compile_options (String) = ENABLE_NORMALIZE
  compile_options (String) = ENABLE_PREUPDATE_HOOK
  compile_options (String) = ENABLE_RTREE
  compile_options (String) = ENABLE_SESSION
  compile_options (String) = ENABLE_SNAPSHOT
  compile_options (String) = ENABLE_SQLLOG
  compile_options (String) = ENABLE_STMT_SCANSTATUS
  compile_options (String) = ENABLE_UNKNOWN_SQL_FUNCTION
  compile_options (String) = ENABLE_UPDATE_DELETE_LIMIT
  compile_options (String) = HAS_CODEC_RESTRICTED
  compile_options (String) = HAVE_ISNAN
  compile_options (String) = MALLOC_SOFT_LIMIT=1024
  compile_options (String) = MAX_ATTACHED=10
  compile_options (String) = MAX_COLUMN=2000
  compile_options (String) = MAX_COMPOUND_SELECT=500
  compile_options (String) = MAX_DEFAULT_PAGE_SIZE=8192
  compile_options (String) = MAX_EXPR_DEPTH=1000
  compile_options (String) = MAX_FUNCTION_ARG=127
  compile_options (String) = MAX_LENGTH=2147483645
  compile_options (String) = MAX_LIKE_PATTERN_LENGTH=50000
  compile_options (String) = MAX_MMAP_SIZE=1073741824
  compile_options (String) = MAX_PAGE_COUNT=1073741823
  compile_options (String) = MAX_PAGE_SIZE=65536
  compile_options (String) = MAX_SQL_LENGTH=1000000000
  compile_options (String) = MAX_TRIGGER_DEPTH=1000
  compile_options (String) = MAX_VARIABLE_NUMBER=500000
  compile_options (String) = MAX_VDBE_OP=250000000
  compile_options (String) = MAX_WORKER_THREADS=8
  compile_options (String) = MUTEX_UNFAIR
  compile_options (String) = OMIT_AUTORESET
  compile_options (String) = OMIT_LOAD_EXTENSION
  compile_options (String) = STMTJRNL_SPILL=131072
  compile_options (String) = SYSTEM_MALLOC
  compile_options (String) = TEMP_STORE=1
  compile_options (String) = THREADSAFE=2
  compile_options (String) = USE_URI

[rouault]

I've tried to replicate a few of the likely relevant build parameters in a custom SQLite3 build, but didn't manage to reproduce the issue. So #13824 will just skip the early VSIUnlink() on Mac