Operator handle rotating Kibana EncryptionKeys

[stefnestor]

Follow-up to #8150 (comment)

But in general I believe we should not try to squeeze the key rotation topic into this TIP. Because just doing what you are describing here will not rotate the encryption keys it will just duplicate the key into a decpryption-only key and lead to a crash looping Kibana pod, because the primary encryption key cannot also be a decryption-only key. So the primary key has also to be changed to a new valid (!) value. I believe key rotation is something ECK should orchestrate not users manually. This would be a new operator feature.

The business problem to solve is users not knowing to originally set Kibana *.encryptionKey values and then needing to override either those or xpack.encryptedSavedObjects.keyRotation.decryptionOnlyKeys in order to later decrypt snapshot restored (and reindexed?) Kibana Saved Objects. I expect this comes up infrequently but the hope is to align ECK to easier full cluster migrations/restores. Or at least forewarn users of general manual flow in ECK docs if it doesn't end up automated.

[kaykhan]

I came across this issue a few days ago installing fresh ECK and had no idea i needed to set encryptionKey (before we have never had to). Now i updated the operator config to set it although i'm not entirely sure if i needed to do anything else.

      config:
        xpack.encryptedSavedObjects:
            encryptionKey: <key>

[stefnestor]

@kaykhan kindly reference Scale out Kibana (master) which includes changes from #8150. Per doc, ECK by default sets these encryptionKey values for you, although you can override as needed (which'd be same as bare metal process). The purpose of this Github issue is to have ECK automate encryption key rotations, but this is an infrequent process. If you have further questions, kindly raise them in Discuss or a Support case where we're happy to help.

[kaykhan]

@kaykhan kindly reference Scale out Kibana (master) which includes changes from #8150. Per doc, ECK by default sets these encryptionKey values for you, although you can override as needed (which'd be same as bare metal process). The purpose of this Github issue is to have ECK automate encryption key rotations, but this is an infrequent process. If you have further questions, kindly raise them in Discuss or a Support case where we're happy to help.

I see so it seems like i should not need to set this manually? The reason i did this is because when i killed the Kibana pod and it rebooted everything was encrypted and it did not work