Aggressive Asserts in Runtime lead to un-necessary server shutdowns

[keirafadams]

Describe the bug

Currently, there are asserts in the runtime path (e.g. within the descriptor set commands) to verify certain return values are present in metadata queries. However, the use of these asserts can lead to abrupt server shutdowns, disrupting client use.

It looks like the intent of this was to prevent data corruption, however its use in later parts of the query flow means that any opportunity to prevent metadata inconsistency (in particular, for PMGD based handlers) has already passed, or is not actually relevant as it we are only returning information, not creating any new entries within VDMS.

To Reproduce

Create a FindDescriptor query with an aggressive set of constraints to limit the number of returned entities to zero. This will potentially trigger the assert failure, despite not actually leading to the system being in a dangerous state.

Expected behavior

VDMS should return a relevant error to the client, instead of fully shutting down the server in response.

[keirafadams]

Digging into this a little more, will update if I find some more concrete stuff, but we see this most commonly in the DescriptorCommands.cc implementations of descriptor set commands.

Generally speaking we should just look for where asserts are occurring and bubble up errors rather than collapsing the entire server.