DM-43712: Add configurable buffer for template and refcat preload #204
Conversation
There was a problem hiding this comment.
Many thanks for the well-organized changes! However, I'm a bit worried that they're eroding the (already shaky) coherence of MiddlewareInterface, specifically with the region state variable and the splitting of queries across multiple _filter_datasets calls. I tried to offer some specific suggestions, but I'd like to take another look at the final result.
hsinfang
force-pushed
the
tickets/DM-43712
branch
3 times, most recently
from
6839a4b
to
5653980
Compare
hsinfang
force-pushed
the
tickets/DM-43712
branch
14 times, most recently
from
319061d
to
423db09
Compare
tests/test_middleware_interface.py
Outdated
| def test_compute_region(self): | ||
| """Test preload region computation.""" | ||
| region = self.interface._compute_region() | ||
| self.assertTrue(isinstance(region, lsst.sphgeom.Region)) |
There was a problem hiding this comment.
Since the tests don't include pointing/slew errors, the nextVisit boresight and exposure boresight are identical. Is it possible to test the values based on that?
There was a problem hiding this comment.
I thought of that, but was not sure how to get the "truth" that is not from the exactly same code as in _compute_region(). Do you mean a comparison with a directly instantiated polygon?
There was a problem hiding this comment.
I was thinking of using the visit-detector records, with matched padding. That should give similar results, shouldn't it?
There was a problem hiding this comment.
I'm taking too long to make it work, so I decided to file a new ticket DM-47460 for it instead of delaying this PR.
I moved this test to MiddlewareInterfaceWriteableTest where visit-detector records are available (after ingesting raws and defining visits).
| src_butler.query_datasets.assert_called_once_with( | ||
| "bias", instrument="LSSTComCamSim", explain=False) | ||
| existing_butler.query_datasets.assert_called_once_with( | ||
| "bias", instrument="LSSTComCamSim", explain=False) |
There was a problem hiding this comment.
The explain=False is an implementation detail. Unfortunately, I don't know how to write a test without it other than inspecting Mock.call_args directly.
hsinfang
force-pushed
the
tickets/DM-43712
branch
from
423db09
to
2117f98
Compare
hsinfang
force-pushed
the
tickets/DM-43712
branch
5 times, most recently
from
2fb0771
to
fca73e8
Compare
This should have been together with 674a4b1 Unlike the old DECam test dataset, the new test LSSTComCamSim dataset does not have crosstalk. So, use another dataset type to test. Also fix a time stamp missed in the previous commit.
The optimal value likely depends on the instrument and the skymap choice.
Instead of determining the instrument's wcsFlipX here, it is more robust to use its formatter, which knows its camera orientation from its obs package.
This changes the APIs of _export_skymap_and_templates and _export_refcats to take a lsst.sphgeom.Region directly. Note that the centroid of the region is not the same of the detector center, but it should not matter. Because htm7 can be too coarse compared to the patch size, using htm7 indices to search for templates may lead to preloading more patches than necessary and wasting time. This feature of using htm7 to search for overlapping templates is also about to be deprecated and replaced by the arbitrary spatial region query in Butler. The usage will be replaced when switching to the new butler query system.
This reduces the number of calib dataset types to loop through.
Querying butler one dataset type at a time is not necessary with the butler.registry.queryDatasets. But this is a preparation step before switching to the new query system which can only queries one dataset type at a time. Currently we can preload more types of calibs/refcats/templates than the actual pipelines really need. It's possible that some types are not preloaded but it's okay.
The new Butler query systems supports spatial-constraint query via lsst.sphgeom.Region directly. With this change, we use it in template and refcat search. This needs stack w_2024_38 or newer. make_export.py uses _filter_datasets so it needs to adjust to the new underlying API too.
Some unit tests were temporarily marked expectedFailure in 674a4b1. Now that we switch to the new query system, make them work again. The test repo was put together using middleware tools, which intrinsically uses butler repo's visit-detector regions with its padding from defineVisits config. That padding config is not the same as the preload region padding in prompt processing. This explains the patch differences in template selection.
hsinfang
force-pushed
the
tickets/DM-43712
branch
from
fca73e8
to
7a4a239
Compare
No description provided.