Add full cut optimization as introduced in pyirf 0.13

[LukasBeiske]

This changes the PointSourceSensitivityOptimizer to use the full cut optimization introduced in pyirf 0.13. The previous EventDisplay-like optimization can now be used via the PointSourceSensitivityGhOptimizer.

I did a quick comparison of the three optimizers using prod6 files (multiplicity >= 2 for gh opt and percentile cuts, as the HillasReconstructer is used):

I am not sure, why the EventDisplay-like optimization results in a better sensitivity at high energies.

Fixes #2771

[maxnoe]

How fine did you make the scanning of the cuts? In principle, the full cut opt should always be at least as good as the one that is restricted,.if it is allowed to find the same cuts.

[LukasBeiske]

How fine did you make the scanning of the cuts? In principle, the full cut opt should always be at least as good as the one that is restricted,.if it is allowed to find the same cuts.

I kept everything at the default values, as we have them in here right now. So, if I'm not mistaken, the only difference would be that the EventDisplay-like optimization has a theta cut with 68% efficiency, while the full optimization only tries 60% and 70%.
But I doubt that this makes such a difference. I'll test that and take a closer look again tomorrow.

And, I think, there is no check for a minimum number of events per bin in the full cut optimization, while the 68% theta cut for the EventDisplay-like optimization has a minimum of 10 events per bin. However, this only seems to play a role for the two lowest energy bins, if at all.

[LukasBeiske]

There was still an error with the application of the cuts in the irf tool. It should be correct now and this improved the sensitivity situation a lot, but there are still some bins where the EventDisplay-like optimization outperforms the full optimization, even though the first uses a theta cut with 70% efficiency for this plot, which also gets tested in the full optimization.

[ctao-dpps-sonarqube]

Analysis Details

1 Issue

• 0 Bugs
• 0 Vulnerabilities
• 1 Code Smell

Coverage and Duplications

• 89.10% Coverage (94.30% Estimated after merge)
• 0.00% Duplicated Code (0.70% Estimated after merge)

Project ID: cta-observatory_ctapipe_AY52EYhuvuGcMFidNyUs

View in SonarQube

[maxnoe]

If I remember correctly, these percentiles cannot be compared directly, as the eventdisplay-like optimization computes the percentile on the events surviving the initial cut, whereas the full optimization computes it on all events.

[LukasBeiske]

If I remember correctly, these percentiles cannot be compared directly, as the eventdisplay-like optimization computes the percentile on the events surviving the initial cut, whereas the full optimization computes it on all events.

Ah, good point, I forgot about that. I'll run the full optimization again with a finer gridding. I guess, these underperformance for some bins will disappear then.

[LukasBeiske]

Running both optimizations with finer grids:

PointSourceSensitivityOptimizer:
  gh_cut_efficiency_step=0.02
  theta_cut_efficiency_step=0.02
  
PointSourceSensitivityGhOptimizer:
  gh_cut_efficiency_step=0.02

gets the performance of the full optimization closer, but some bins are still worse then with EventDisplay-like optimization.

[kosack]

gh_cut_efficiency_step=0.02
theta_cut_efficiency_step=0.02

If a smaller step is important to get a good sensitivity, please also update the ctapipe-quickstart configurations to provide the best values for users, and also the default values in the tool itself.

[LukasBeiske]

gh_cut_efficiency_step=0.02
theta_cut_efficiency_step=0.02

If a smaller step is important to get a good sensitivity, please also update the ctapipe-quickstart configurations to provide the best values for users, and also the default values in the tool itself.

I just noticed this (similar for the full optimization):

The config is exactly the same (default values) besides the stepsize. This doesn't make sense. I'm re-running everything now and, if this behavior is still there, I'll convert this PR to draft until I figure out whats going on here.