Add CodeQL check to detect cause of issue #12014

[ryao]

Motivation and Context

This is a simple check that would detect instances of the cause of issue #12014.

Description

This check is currently limited to checking mismatches that occur in the same stack frame. It does not detect across stack frames.

I had originally intended not to send this in favor of developing a more comprehensive check and sending the patch for that, but @behlendorf asked for it in #17340, so I am opening a PR with what I currently have. Note that I had made an initial attempt at making the more comprehensive check on the weekend, but I am not well versed in the CodeQL language, so it will take a while to implement.

How Has This Been Tested?

I verified it identified the issue on ryao/zfs.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[amotin]

I wonder if we could/should assert that flags passed to the both functions are the same. Or at least this specific flag.

[ryao]

I wonder if we could/should assert that flags passed to the both functions are the same. Or at least this specific flag.

We could make a different check for that. I will keep it in mind as something else to try to check after I have improved this. I am spending a few hours on this every few days, so I do not expect to have a revision of this for at least a week.

That said, this API strikes me as being naturally error prone. Changing it to use a cookie approach like spl_fstrans_mark()/spl_fstrans_unmark() does would give us correctness by design. That is because we would have only 1 release function and then pass the "cookie" structure into it. This would come at the expense of additional stack usage and I am not yet of the opinion that we should do this over adopting custom static analysis checks.

[behlendorf]

@ryao if you're still iterating on this can your mark it as a draft until you're ready. That said, I'm also fine with merging this functional, but less comprehensive, version until you come up with something better.