Use unique run numbers

[maxnoe]

The current LSTProd2 is restarting run numbers for each pointing node of each declination line from 1.

While it's always bad to have non-unique run id / event_id combinations, there is a very practical issue:
This makes it impossible to merge the runs of one declination line into a single dataset for training using ctapipe tools,
as the ctapipe dataformat requires the primary key of (obs_id, event_id) to be unique in the file.

[jsitarek]

We also had problems with it in MCP chain, and had to group the events by (obs_id, event_id, true_alt, true_az)

[maxnoe]

As a workaround, I used the following to add offsets to obs ids in the merged dl1 files of one pointing node before merging for the whole declination line:

import tables

nodes = [
    "GammaDiffuse/node_corsika_theta_16.087_az_108.090__merged.dl1.h5",
    "GammaDiffuse/node_corsika_theta_52.162_az_82.384__merged.dl1.h5",
    ...,
    "Protons/node_theta_37.661_az_89.359__merged.dl1.h5",
    "Protons/node_theta_66.446_az_284.017__merged.dl1.h5",
]

for i, node in enumerate(nodes):
    factor = 10000
    offset = i * factor

    with tables.open_file(node, mode="r+") as f:
        if "fixed_obs_ids" in f.root._v_attrs and bool(f.root._v_attrs["fixed_obs_ids"]) is True:
            print(f"{node} already fixed")
        else:
            print(f"Fixing {node}")
            
        for table in f.root._f_walknodes("Table"):
            if "obs_id" in table.colnames:
                print(f"Updating table {table._v_pathname}")
                obs_id = (table.col("obs_id") - 1)  // 100
                assert obs_id.max() < factor

                table.modify_column(colname="obs_id", column=offset + obs_id)

        f.root._v_attrs["fixed_obs_ids"] = True

We are also adding an override_obsid option to SimTelEventSource in ctapipe.

[Voutsi]

Hi @maxnoe,

thanks a lot for pointing it out. It will be fixed for all new declination bands production.

[Voutsi]

Hi @maxnoe ,

couple of clarifications in order to conclude on this issue:

• From the code snippet I understand it doesn't matter for ctapipe if the run_numbers aren't sequential? So it would work if each node's numbering starts with a unique identifier concatenated to the run_number, e.g. for 100 runs per node:
  • node1: run11, run12 ... run 199, run1100
  • node2: run21, run22 ... run 299, run2100
• I suppose we shouldn't only care having unique run numbers within a declination band only, but between declination bands as well, right?

[maxnoe]

From the code snippet I understand it doesn't matter for ctapipe if the run_numbers aren't sequential?

Correct, they must only be unique within data that is merged into a single file or a single table for analysis. However, for traceability, it would of course be better to have a large as possible scope where the run ids are unique.

I suppose we shouldn't only care having unique run numbers within a declination band only, but between declination bands as well, right?

Identifiers should be as unique as possible. EVB for example uses a file with exclusive access to store the last run number of the real data runs. You could implement something similar.

In ctapipe, we have a utility for this ctapipe.utils.FileLock, it doesn't have dependencies on other parts of ctapipe, so you could just copy the code into your project:

https://github.com/cta-observatory/ctapipe/blob/main/ctapipe/utils/filelock.py

It's basically an inter-process mutex based on exclusive write access of a lock file.

For your use case, you could use it like this:

run_id_path = Path("/fefs/aswg/data/mc/CURRENT_RUN_ID")


def get_next_run_id():
    with FileLock(run_id_path.with_suffix(".lock")):
        if not run_id_path.exists():
            next_run_id = 1
        else:
            next_run_id = int(run_id_path.read_text()) + 1

        run_id_path.write_text(str(next_run_id))

        return next_run_id