Consider alternative healpix libraries

[delucchi-cmu]

We're currently using healpy, which is a somewhat bloated library (also contains matplotlib, cfitsio, etc), and has installation issues on macOS, and is incompatible with Windows.

Alternatives suggested so far:

• https://hpgeom.readthedocs.io/en/latest/index.html
• cdshealpix
• astropy.healpix

If any of these are a little behind healpy in terms of raw performance, but otherwise a good replacement, it may be worthwhile to contribute improvements to their code bases.

[hombit]

Performance comparison for healpy, hpgeom, cdshealpix and astropy-healpix on my laptop:

• healpy 28.4 ms ± 78.7 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
• hpgeom 37.1 ms ± 377 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
• cdshealpix single thread 26.1 ms ± 202 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
• cdshealpix 8 threads 12.8 ms ± 75 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
• astropy-healpix 132 ms ± 261 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

import healpy
import hpgeom
import cdshealpix
import astropy_healpix

import astropy
import numpy as np


n = 1_000_000

rng = np.random.default_rng(0)
ra = rng.uniform(0, 360, n)
dec = np.degrees(np.arccos(rng.uniform(-1, 1, n))) - 90

%timeit healpy.ang2pix(1 << 29, ra, dec, lonlat=True, nest=True)
%timeit hpgeom.angle_to_pixel(1 << 29, ra, dec, lonlat=True, degrees=True, nest=True)
%timeit cdshealpix.nested.lonlat_to_healpix(astropy.coordinates.Longitude(ra, 'deg'), astropy.coordinates.Latitude(dec, 'deg'), 29, num_threads=1)
%timeit cdshealpix.nested.lonlat_to_healpix(astropy.coordinates.Longitude(ra, 'deg'), astropy.coordinates.Latitude(dec, 'deg'), 29, num_threads=8)
%timeit astropy_healpix.lonlat_to_healpix(ra * astropy.units.deg, dec * astropy.units.deg, nside=1 << 29, order='nested')

[delucchi-cmu]

Keeping this for posterity. If we can get a Windows-compatible healpix library, we can enable a windows-file-system checking regression test workflow:

# This workflow will install Python dependencies and run tests in a Windows environment.
# This is intended to catch any file-system specific issues, and so runs less
# frequently than other test suites.

name: Windows unit test

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build:

    runs-on: windows-latest
    strategy:
      matrix:
        python-version: ['3.10']

    steps:
    - uses: actions/checkout@v3
    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v4
      with:
        python-version: ${{ matrix.python-version }}
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install .
        pip install .[dev]
    - name: Run unit tests with pytest
      run: |
        python -m pytest tests

[delucchi-cmu]

Ugh. Another ding.

#65

[maxwest-uw]

going through all the available healpix implementations and checking to see if they've ported all the healpy functions that we need to handle margin caching.

hpgeom

• has everything!

cdshealpix

• no boundaries function 😭
• however they do have external_neighbors, which is essentially our get_margin code but implemented in rust 👀

astropy-healpix

• has everything!
• has boundaries_lonlat which does the conversion of boundary coords into ra/dec, which healpy annoyingly doesn't do by default so that's kinda nice.

was really rooting for cdshealpix but it unfortunately doesn't have all the necessary functions ported over. if they do end up implementing healpy.boundaries we should consider using it as it's fast and could even up allowing us to delete some code.

[maxwest-uw]

small update on this, since the margin cache code has been significantly changed: we're still reliant on the healpy.boundaries function, probably more so now in fact 😭

[eteq]

Coming back here after some out-of-band discussion at a hipscat WG meeting: am I reading it right that astropy-healpix is compatible with everything, but the main restriction is performance? (basing that on #53 (comment)) And if so, any insight on where the performance hotspot is @delucchi-cmu ?

[eteq]

(to be clear, I also am interested in cdshealpix given the rust under-layer, since it seems to support a lot of the same user-level constructs that I like about astropy-healpix, but if I am reading the above it seems like it's out of the running without boundaries?)

[hombit]

@eteq @delucchi-cmu I slightly updated the original benchmarking comment #53 (comment), basically fixing typos in the code. The performance of the packages hasn't change since I tested it last time.

[DaftPict]

I just happened to need cdshealpix for another project and in reading the docs noticed that another package it can use (mocpy) had a get_boundaries function. As a old-school astronomer rather than a modern developer I just wondered if this would solve the missing boundaries issue mentioned earlier and allow a switch to cdshealpix? I'm stuck on windows and would love a version I can use locally.

[fxpineau]

Hi,
I am the developer of cds-healpix-rust, @tboch talked to me about this issue.
I think that the equivalent of boundaries in cdshealxpix is path_along_cell_edge.
If needed, we can add it in the cdshealpix python wrapper (cc @bmatthieu3, @ManonMarchand).

In such cases (feature requests), do not hesitate to open issues.

[ManonMarchand]

Yes, don't hesitate to interact either on mocpy or cdshealpix repos. You're more than welcome to ask for features, point places in the docs where we're not clear, point to bugs or slow methods... or even contribute 🙂
If github issues are not your preferred communication canal, our emails are also in the documentation

[fxpineau]

I just checked the code.
Using vertices_skycoord with step>1 corresponds to the boundaries method: when step is >1 the Rust method path_along_cell_edge is used internally. See this line of code.

[nevencaplar]

As discussed today, @delucchi-cmu will make a list of all tickets that need to completed before we make make this switch.

[delucchi-cmu]

• Update cone search demo notebook plotting #398
• Read/write fits files with cds-healpix methods #399
• Migrate margin code to use cds-healpix #400
• Migrate box filter to use cds-healpix utilities #401
• Migrate polygon filter to use cds-healpix utilities #402
• Implement healpix math methods #423
• Update margin tutorial notebook plotting methods lsdb#502

[delucchi-cmu]

As of the most recent releases, we are no longer depending on healpy for hats operations.