Protect the machine controller against unexpected BareMetalHost state

[dtantsur]

In Kubernetes, we cannot expect that a host that CAPM3 has just chosen won't change in an unexpected way. Some of the ways may be transient, some a permanent. Examples include:

• Re-inspection: host entering inspecting. Hopefully transient, but we may need to handle the situation when inspection fails consistently.
• Detachment. A detached host cannot be provisioned. So if it's not already provisioned, CAPM3 cannot proceed. Technically, a transient state, but requires user input to be resolved.
• OperationalStatus==Error. Hopefully transient.
• Externally Provisioned. Permanent state for now, the host cannot be used any more.
• Unexpected provisioning settings: mismatch in image, user data, etc. A decision to make: should CAPM3 deprovision the host and provision it with the right settings?

What's the expected course of actions? At the very least, we need to report the issue to a user somehow.

/kind bug

[Rozzii]

/triage accepted
So the plan so far is :
As discussed on slack and community meeting, a HostSelector with similar functionality to this https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#resources-that-support-set-based-requirements combined with metal3-io/baremetal-operator#2774 would resolve the immediate "Externally Provisioned issue" and would open the door for supporting many different use-cases.