OCPBUGS-62269 - Fix race condition causing missing audit log entries for rapid commands

[BhargaviGudi]

What type of PR is this?

/kind bug

What this PR does / why we need it:

This PR fixes a race condition in the JSON enricher that causes audit log entries to be missing or incomplete when commands are executed in rapid succession (e.g., touch test1 && touch test2 && touch test3).

Problem:
When short-lived processes execute quickly, audit events arrive at the JSON enricher before the corresponding BPF events have been processed from the ring buffer. This results in:

• Empty cmdLine fields
• Missing requestUID
• Null resource information (pod/namespace/container)

Solution:
Implements a retry mechanism in dispatchSeccompLine() that attempts to fetch missing information multiple times with progressive delays (0ms, 10ms, 50ms):

1. BPF Cache Retry: Retries cmdLine and requestUID lookup from BPF process cache, giving the BPF ring buffer poller time to process events
2. Container Info Retry: Retries container information lookup if initially null

Since LogBuckets remain in cache for 60 seconds before being written to the audit log, there's ample time for retries to succeed while BPF events are processed asynchronously.

Testing Results:

• Before fix: ~33% success rate for rapid consecutive commands
• After fix: 100% success rate - all commands captured with complete information

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/OCPBUGS-62269

Does this PR have test?

Yes - The fix was validated through:

• Manual testing with rapid consecutive commands (oc exec and oc rsh)
• Existing unit tests continue to pass
• Build verification completed successfully

No new automated tests added as this is a timing-dependent race condition that's difficult to reliably reproduce in unit tests.

Special notes for your reviewer:

• This fix is scoped exclusively to jsonenricher.go - no modifications to BPF components or ring buffer configuration
• The retry mechanism only activates when information is missing, avoiding unnecessary delays for normal cases
• Progressive delays (0ms → 10ms → 50ms) balance responsiveness with reliability
• Early break on success minimizes latency impact

Does this PR introduce a user-facing change?

Fixed race condition in JSON enricher causing audit log entries to be missing or incomplete when commands are executed in rapid succession. Audit logs now reliably capture cmdLine, requestUID, and resource
information for all executed commands, even during bursts of activity.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: BhargaviGudi
Once this PR has been reviewed and has the lgtm label, please assign ccojocar for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Hi @BhargaviGudi. Thanks for your PR.

I'm waiting for a github.com member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 28.88889% with 32 lines in your changes missing coverage. Please review.
✅ Project coverage is 24.17%. Comparing base (11d77f4) to head (b75d4d8).
⚠️ Report is 1072 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #3052       +/-   ##
===========================================
- Coverage   45.50%   24.17%   -21.33%     
===========================================
  Files          79      125       +46     
  Lines        7782    17822    +10040     
===========================================
+ Hits         3541     4309      +768     
- Misses       4099    13229     +9130     
- Partials      142      284      +142     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[saschagrunert]

/ok-to-test

[BhargaviGudi]

/retest-required

[saschagrunert]

@BhargaviGudi do you mind a rebase?

[ngopalak-redhat]

@BhargaviGudi Thanks for the PR. Can you try to test if e.retryContainerInfo(ctx, logBucket, processID, nodeName) alone can fix the issue without the retryBpfCmdLine and retryBpfRequestUID? I feel missing container info is the cause of the issue.

[BhargaviGudi]

I tested with only retryContainerInfo as suggested, but it doesn't fix the issue for short-lived processes.

• The short-lived processes exit so fast that by the time the LogBucket expires (30 seconds later), their /proc/ is gone and are missing from audit log

[BhargaviGudi]

@ngopalak-redhat Could you please help me review the PR? Thanks

[BhargaviGudi]

/test pull-security-profiles-operator-verify

[saschagrunert]

@BhargaviGudi thank you for the PR! Please rebase to fix the CI.

[BhargaviGudi]

@ngopalak-redhat Could you please help me review the PR? Thanks