ENH: SIA should raise exception for non-existing collection

[bsipocz]

[Edit:] This has started as an astroquery issue, but looking more into it, the exception should be raised already on the pyvo level, so I transferred the issue.

>>> import pyvo as vo
>>> sia2 = vo.dal.sia2.SIA2Service('https://irsa.ipac.caltech.edu/SIA')
>>> sia2.search((0,0,10), collection='foobar')
<DALResultsTable length=0>
  s_ra   s_dec  facility_name instrument_name dataproduct_subtype ... pol_xel cloud_access o_ucd  upload_row_id
  deg     deg                                                     ...                                          
float64 float64     object         object            object       ...  int64     object    object     int64    
------- ------- ------------- --------------- ------------------- ... ------- ------------ ------ -------------

============

Currently it's possible to query a collection that doesn't exist on the server and receive an empty result table. I would think a better UX would be to receive an exception in these cases.

E.g. currently this is what I get:

In [49]: Irsa.query_sia(pos=(coord, 10), collection='foobar')
Out[49]: 
<DALResultsTable length=0>
  s_ra   s_dec  facility_name instrument_name dataproduct_subtype ... pol_xel cloud_access o_ucd  upload_row_id
  deg     deg                                                     ...                                          
float64 float64     object         object            object       ...  int64     object    object     int64    
------- ------- ------------- --------------- ------------------- ... ------- ------------ ------ -------------

[bsipocz]

This is actually an upstream issue, as we should get the exception on the pyvo level already.

[msdemlei]

On Thu, Feb 13, 2025 at 11:10:18AM -0800, Brigitta Sipőcz wrote: bsipocz created an issue (astropy/pyvo#651) Currently it's possible to query a collection that doesn't exist on the server and receive an empty result table. I would think a better UX would be to receive an exception in these cases.

Frankly, I don't think this is possible with reasonable effort, because outside of special cases you wouldn't know if the empty result set is because the collection does not exist or because there is just nothing matching the other constraints within that collection. Somewhat relatedly, I have been arguing for having advanced column metadata in the tableset <http://ivoa.net/documents/Notes/colstatnote/index.html>, which eventually would include the possible values of enumerated columns (like obs_collection); DaCHS actually already implements that. Unfortunately, these efforts have not gained much traction yet. *If* in this way we had a way to discover the legal values of obs_collection, my objection "impossible without reasonable effort" would not stand. But even then I suspect by outright rejecting queries that, judging by the column metadata, would return nothing, we'd cause more harm than benefit. We *might* have a "preflight" mode, though, where the query methods first check against the ranges and don't bother running the query if parameters are out of range. This might be clever in, for instance, global dataset discovery. But we're years away from having enough buy-in to better column metadata, so I don't think we need to plan for this now.

[bsipocz]

Thank you Markus. Yes, I got to the same conclusion that this would be very hard even if possible on the pyvo level. I know that e.g. at IRSA we have a metadata query response for the special case of maxrec=0, but that's not universal behaviour from the services, and we cannot even handle the IRSA case (#519).

So I may just reopen this issue in astroquery where I could do the sanity check for the collection parameter using a cached list of possible collection names.

[msdemlei]

On Mon, Feb 17, 2025 at 07:50:02PM -0800, Brigitta Sipőcz wrote: at IRSA we have a metadata query response for the special case of `maxrec=0`, but that's not universal behaviour from the services, and we cannot even handle the IRSA case

Metadata on MAXREC=0 is standard (DALI) behaviour, but that does not include column statistics, so that won't help you here.

So I may just reopen this issue in astroquery where I could do the sanity check for the collection parameter using a cached list of possible collection names.

Frankly, *if* we think this is worthwhile behaviour (and while I'm obviously a big fan of column statistics, I'm not sure about that particular use case), let's push for an interoperable solution. Having more columns statistics would be a great thing for so many other purposes, too. And they're not horribly complex to do either.

[bsipocz]

An interoperable solution would be indeed best!