The same request is handled differently when repeating it several times

[christophfriedrich]

Description

With my WCS hack in place (see #1041), I get the result I want for the first (few) request(s) after starting up the Docker container that's running OWS, but when issuing the same query repeatedly, at some point it fails with a 500 Internal Server Error and the message Unexpected server error: 'index_function' appears. This is the traceback:

file /home/datacube/datacube_ows/ogc.py, line 152 in ogc_svc_impl
file /home/datacube/datacube_ows/utils.py, line 29 in log_wrapper
file /home/datacube/datacube_ows/wcs1.py, line 31 in handle_wcs1
file /home/datacube/datacube_ows/utils.py, line 29 in log_wrapper
file /home/datacube/datacube_ows/wcs1.py, line 122 in get_coverage
file /home/datacube/datacube_ows/wcs1_utils.py, line 403 in get_coverage_data
file /home/datacube/datacube_ows/utils.py, line 29 in log_wrapper
file /home/datacube/datacube_ows/data.py, line 250 in data
file /usr/local/lib/python3.10/dist-packages/datacube/model/__init__.py, line 605 in lookup_measurements
file /usr/local/lib/python3.10/dist-packages/datacube/model/__init__.py, line 605 in <genexpr>

I inserted a few debugging lines into my codebase, so the line numbers don't align with the version on here, so these are the actual lines of interest:

in wcs1.py -> get_coverage:

datacube-ows/datacube_ows/wcs1.py

Line 114 in aa67e0b

n_datasets, data = get_coverage_data(req, qprof)

in wcs1_utils.py -> get_coverage_data:

datacube-ows/datacube_ows/wcs1_utils.py

Line 395 in aa67e0b

output = stacker.data(datasets, skip_corrections=True)

in data.py -> data:

datacube-ows/datacube_ows/data.py

Line 245 in aa67e0b

measurements = pbq.products[0].lookup_measurements(pbq.bands)

So apparently OWS looks up the measurements and of course there is none with the name index_function, because that band has just been dynamically created. I would somewhat understand if this error would appear all the time. But why does it work at first and then the behaviour changes? Does get_coverage_data run into a different if-branch because of cache stuff or something?

Live example

https://owsdev.eo2cube.org/wcs?SERVICE=WCS&VERSION=1.0.0&REQUEST=GetCoverage&FORMAT=GeoTIFF&COVERAGE=s2_vi_ndvi&TIME=2020-01-02T00:00:00.000Z&BBOX=1383632,7143434,1388857,7138548&CRS=EPSG:3857&RESPONSE_CRS=EPSG:3857&WIDTH=10&HEIGHT=10

Let me know if you want me to reset it to test for yourself, or whatever other information might be helpful; I just wanted to get this question out in case someone has an obvious answer to it.

Context (Environment)

datacube-ows version (datacube-ows-update --version):

Open Data Cube Open Web Services (datacube-ows) version 1.8.36.dev0+g67003f3.d20240812

ows_config.py file (link, sample code)

https://github.com/eo2cube/ows_config/tree/bc4b795a93f6478a40ba37cc782bd1d2137ffdca

datacube product metadata (datacube product show product_name)

see #1041

[SpacemanPaul]

This is a really interesting problem and I would love to help you sort this out, but my schedule is absolutely crazy at the moment and I can't really give you a time-frame.

[SpacemanPaul]

I suspect it's related to over-writing the product band index in the product/layer configuration object (from #1041):

req.product.band_idx = FakeBandIndex()  # very hacky...

These objects are created on startup when parsing the config file and are assumed to be immutable.