HDU Hierarchy for event level data

[kosack]

Hierarchy

Using the FITS hierarchical grouping standard, you can think of a set of HDUs of as a "folder-like" file structure, where the groups are like directories and the HDUs are like files within them. The nature of the GROUPING tables allows the actual HDUs to be stored in the same file as the group, or in a separate file or even a distant file with a URL. Think of this like each HDU can be physically in the folder, or just a symbolic link to it is in the folder. As a folder structure this looks like:

• PRIMARY
• Observation/
  • EVENTS
  • GTI
  • MONITORING
  • IRF/
    • IRF1
    • IRF2
    • ...
    • IRFn

In more detail, this would look like:

• PRIMARY (contains common metadata)
• GROUPING (GRPNAME=observation, EXTVER=0) : Observation Grouping table
  • EVENTS
  • GTI (optional)
  • MONITORING* tables (optional), e.g. POINTING
  • PROVENANCE (optional)
  • GROUPING (GRPNAME=IRF, EXTVER=1): the extra columns in the group should correspond to the hierarchy of HDUCLASn (maybe even name them HDUCLASn to be consistent) This is instrument dependent.
    • IRF1_SOI0†
    • IRF1_SOI_n_
    • IRF2_SOI0
    • IRF2_SOI_n_
    • ....
    • IRF1_SOI0_UNCERT1 (optional)
    • IRF_1_SOIn+IRF_UNCERT2 (optional)
    • ...
    • IRF_VALIDITY1 (optional)
    • IRF_VALIDITY1 (optional)

†: Note that the HDU names (EXTNAME,EXTVER) for all the IRFs need some thought, as they must be unique. One idea is to use the IRF component name and use HDUVERS to disambiguate them, just incrementing. For the format, it doesn't matter, but the implementer has to decide on a convention

The different sub-types of IRF HDUs: IRF, IRF_UNCERT, IRF_VALIDITY are separated for memory reasons: you would not want to load all that in one HDU each time, since the validity/uncertainty/etc are no all needed.

Grouping the IRFs for CTAO

in CTAO, we have IRFs for each SOI and for each EventType.

So the IRF grouping table looks something like, or we say exactly like (open question):

• HDUCLAS1 is "RESPONSE"
• HDUCLAS2 is the SOI number [int], and there must be extra columns of START, STOP associated with it
• HDUCLAS3 is the irf component: [EFF_AREA, EDISP, PSF, BGK, RAD_MAX*]
• HDUCLAS4 is the event_type_group ["event_type_group_n"]
• HDUCLAS5 is the storage type ([BG3D, ..., RAD_MAX]) (get from gammapy)
• HDUCLAS6 is the spatial selection ['NONE', 'RAD_MAX'].

(*) Note that RAD_MAX is not really an IRF, but we want to be able to associate it with each SOI bin, so having it in the hierarchy and IRF grouping table is convenient, at the expense of "purity of data model"

The the IRF grouping table would look something like this (but with 7 HDUCLASn columns, and TSTART, TSTOP for the SOI)

MEMBER_NAME	MEMBER_VERSION	...	HDUCLAS2	TSTART	TSTOP	HDUCLAS3	HDUCLAS4	HDUCLAS5	HDUCLAS6
effective_area	0	...	0	523323.121	523323.256	EFF_AREA	2	EffArea3D	None
effective_area	1	...	0	523323.121	523323.256	EFF_AREA	2	EffArea3D	RAD_MAX
effective_area	2	...	1	523323.121	523323.256	EFF_AREA	2	EffArea3D	None

[kosack]

Note that the motivation for separating HDUs in the IRFs is for memory efficiency. Using a single N-D IRF would work, but you would have to read it all for each observation.

[kosack]

After discussion with @LauraOlivera , there is another structure possibility (or rather 2):

What SWGO Does is:

• Observation/ (for SWGO on a longer time-scale)
  • EventType_1/
    • EVENTS
    • IRFs/
      • IRF1
      • IRF2
      • IRF3
  • EventType_n/
    • EVENTS
    • IRFs/
      • IRF1
      • IRF2
      • IRF3

Another optoin would be to use flattened groups as we did in the top-level description, but do the same for EventLists... However that then doesn't show the connection between the events and the IRFs..

• Observation/
  • EventLists/
    • EVENTS
    • EVENTS
    • EVENTS
  • IRFs/
    • IRF1
    • IRF2
    • IRF3

I think this deeper hierarchy may not be necessary - in the original description, we encode the different IRF dimensions into the group itself. The only issue is how to also include the EVENTS in that scheme, if youo want to have one EVENTS hdu per event type?

[bkhelifi]

one should reconcile the SWGO view with the fact that one can have events without any SOI, or without any IRFs

[LauraOlivera]

I don't see why that is an issue: you just have events with no IRFs, so in Karl's example above for example IRF3 is None or similar.

[LauraOlivera]

Some thoughts I had when looking at the graph in #5 but for which perhaps a discussion about rather belongs here

I find that having the StableObservationInterval as subset of Observation to be odd and CTA jargon. I would propose a more generic approach where we consider the smallest unit of data that has a distinct set of IRFs (which can be no IRFs). In CTA that'd be the SOIs, which you then bundle through some observatory criteria (e.g. how you organize your data taking) and call an Observation internally. But at its simplest, you always have a link between a set of EVENTS and an IRF (or lack of thereof). How observatories choose to group them for bookkeeping might not be the thing we should focus on.

In a way it's funny: in SWGO the concept "Observation" was a meaningless because it doesn't really relate to like, starting and stopping the data recording. I find that now, with the SOI, it might also be meaningless in CTA because the real relevant partitions are the SOI 😄 but I don't necessarily mean to start a deep discussion now, maybe just to warn that I will object to this structure 😅

[LauraOlivera]

In a sense, an event either has IRFs or it doesn't. That CTA would have "high quality" IRFs and "medium quality" IRFs (to say an example) and what to distinguish between them if conditions change halfway through data-taking, is irrelevant: I'd say nothing connects those events anymore if they don't share an IRF (other than, perhaps, internal bookkeeping).

[bkhelifi]

• "in SWGO the concept "Observation" was a meaningless" is not true. Event SWGO makes observation of a portion of the sky at a given time. So, observation is full of meaning: we are making astrophysics. The choice of the TSTART and TSTOP of a file is up to the observatory, and creates a minimal unit of observation.
• SOI: why it is odd to consider the observation process is stable and under control? Can you elaborate? Any name is a jargon, but the concept behind is not, even for SWGO. I know that it happened that HAWC was not stable or not under control. So, even for a WCD, SOI has a sense, even if this time interval is 6 months.

[bkhelifi]

I think that one should take care that the format does not impose some data partition to observatories. The choose scheme should be general: being able to have event_type into an event list and the IRFs in the same file, and also being able to have only 1 event type and the associated IRFs into the same file.

[bkhelifi]

In a sense, an event either has IRFs or it doesn't. That CTA would have "high quality" IRFs and "medium quality" IRFs (to say an example) and what to distinguish between them if conditions change halfway through data-taking, is irrelevant: I'd say nothing connects those events anymore if they don't share an IRF (other than, perhaps, internal bookkeeping).

There are connected by : an instrument working with the same conditions (beyond the atmosphere in the case that you mention), the same s/w workflow, the same RoI (for such pointing instrument), the same proposal.
The concept of observation does exist, for CTAO, SWGO and KM3NeT!

And, the whole Data Model of CTAO relies on this concept. VODF can not exclude the CTAO DM.
One should converge towards wording that defines the same concepts, and to find a format that is generic enough to handle the different analyses concept, different data releases types. And this is doable because we are making observations of the sky in the VHE range and we have the same factorisation of the IRFs.