Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DM-43712: Add configurable buffer for template and refcat preload #204

Merged
merged 13 commits into from
Nov 13, 2024
Merged

DM-43712: Add configurable buffer for template and refcat preload #204

merged 13 commits into from
Nov 13, 2024
+192 −149

Commits on Nov 12, 2024

  1. Update unit tests with the test data change 

    This should have been together with 674a4b1

Unlike the old DECam test dataset, the new test LSSTComCamSim dataset
does not have crosstalk. So, use another dataset type to test.

Also fix a time stamp missed in the previous commit.
    @hsinfang
    hsinfang committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    fd5b828 View commit details
    Browse the repository at this point in the history

  2. Fix a missing space

    @hsinfang
    hsinfang committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    c9b5dcd View commit details
    Browse the repository at this point in the history

  3. Pad the detector region in the template search

    @hsinfang
    hsinfang committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    432ed91 View commit details
    Browse the repository at this point in the history

  4. Make the preload padding configurable at the service level 

    The optimal value likely depends on the instrument and the
skymap choice.
    @hsinfang
    hsinfang committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    e576636 View commit details
    Browse the repository at this point in the history

  5. Use the instrument's formatter to get the sky wcs 

    Instead of determining the instrument's wcsFlipX here, it is more
robust to use its formatter, which knows its camera orientation
from its obs package.
    @hsinfang
    hsinfang committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    f8c0c3f View commit details
    Browse the repository at this point in the history

  6. Use the actual template coadd types instead of wildcarding

    @hsinfang
    hsinfang committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    7f3e400 View commit details
    Browse the repository at this point in the history

  7. Factor out the region computation

    @hsinfang
    hsinfang committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    1fd83f7 View commit details
    Browse the repository at this point in the history

  8. Use regionTimeInfo's region to export template/refcat datasets 

    This changes the APIs of _export_skymap_and_templates and
_export_refcats to take a lsst.sphgeom.Region directly.

Note that the centroid of the region is not the same of the
detector center, but it should not matter.

Because htm7 can be too coarse compared to the patch size, using
htm7 indices to search for templates may lead to preloading more
patches than necessary and wasting time. This feature of using
htm7 to search for overlapping templates is also about to be
deprecated and replaced by the arbitrary spatial region query in
Butler. The usage will be replaced when switching to the new
butler query system.
    @hsinfang
    hsinfang committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    cc4248a View commit details
    Browse the repository at this point in the history

  9. Warn if preload region padding is smaller than defineVisits padding

    @hsinfang
    hsinfang committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    59e1ae1 View commit details
    Browse the repository at this point in the history

  10. Constrain the calib types to those existing in the collection 

    This reduces the number of calib dataset types to loop through.
    @hsinfang
    hsinfang committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    58e8b8a View commit details
    Browse the repository at this point in the history

  11. Change _filter_datasets to take an iterable of dataset types 

    Querying butler one dataset type at a time is not necessary with
the butler.registry.queryDatasets. But this is a preparation step
before switching to the new query system which can only queries
one dataset type at a time.

Currently we can preload more types of calibs/refcats/templates
than the actual pipelines really need. It's possible that some types
are not preloaded but it's okay.
    @hsinfang
    hsinfang committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    367cb8a View commit details
    Browse the repository at this point in the history

  12. Use the new Butler query system in _filter_datasets for region query 

    The new Butler query systems supports spatial-constraint query
via lsst.sphgeom.Region directly. With this change, we use it
in template and refcat search. This needs stack w_2024_38 or newer.

make_export.py uses _filter_datasets so it needs to adjust to the
new underlying API too.
    @hsinfang
    hsinfang committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    52e2f2b View commit details
    Browse the repository at this point in the history

  13. Enable prep_butler unit tests again 

    Some unit tests were temporarily marked expectedFailure in 674a4b1.

Now that we switch to the new query system, make them work again.

The test repo was put together using middleware tools, which
intrinsically uses butler repo's visit-detector regions with its
padding from defineVisits config.

That padding config is not the same as the preload region padding
in prompt processing.

This explains the patch differences in template selection.
    @hsinfang
    hsinfang committed Nov 12, 2024
    Configuration menu
    Copy the full SHA
    7a4a239 View commit details
    Browse the repository at this point in the history