Improve the key validation in secret identifier. #17351

mashhurs · 2025-03-19T06:12:31Z

Release notes

The keystore has improved validation, and will no longer allow the creation of secrets that cannot be accessed with config replacement parameters (${ ... }).

What does this PR do?

Before this change, key-value can be added to keystore but it might impossible to reference the key because ConfigVariableExpander ignores if key name doesn't align with SUBSTITUTION_PLACEHOLDER_REGEX
Improves user experience with the secret store CLI that when providing key name, LS validates to accept a value which can be used in pipeline (aligns with ConfigVariableExpander#SUBSTITUTION_PLACEHOLDER_REGEX).
If already invalid key(s) were added, it warns to remove/replace the keys.

Why is it important/What is the impact to the user?

No user impact. Keystore CLI restricts adding meaningless keys to keystore.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~[] I have made corresponding changes to the documentation~~
~~[] I have made corresponding change to the default configuration files (and/or docker env variables)~~
I have added tests that prove my fix is effective or that my feature works

Author's Checklist

[ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

mergify · 2025-03-19T06:13:05Z

This pull request does not have a backport label. Could you fix it @mashhurs? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
backport-8.x is the label to automatically backport to the 8.x branch.
If no backport is necessary, please add the backport-skip label

yaauie · 2025-03-19T22:03:51Z

No users should be impacted unless they are using restricted symbols (["?", "..", "/", "\", "'", """, "$", "*", "|", "<", ">", " "]) in the key.

This sounds like a breaking change.

What was the prior experience when using a key that includes now-restricted characters?
What determined that these specific characters are restricted (while allowing a bunch of non-printing characters in the lower-ascii plane and the gamut of unicode characters above the lower-ascii plane)?

github-actions · 2025-03-22T03:20:20Z

📃 DOCS PREVIEW ✨ https://logstash_bk_17351.docs-preview.app.elstc.co/diff

mashhurs · 2025-03-23T03:42:56Z

No users should be impacted unless they are using restricted symbols (["?", "..", "/", "", "'", """, "$", "*", "|", "<", ">", " "]) in the key.

This sounds like a breaking change.

What was the prior experience when using a key that includes now-restricted characters?

What determined that these specific characters are restricted (while allowing a bunch of non-printing characters in the lower-ascii plane and the gamut of unicode characters above the lower-ascii plane)?

Yeah, didn't think of the case if keystore had invalid keys already included.
a3372c5 commit changes the approach:

to restrict adding invalid keys to keystore with CLI. Invalid means here outside of ConfigVariableExpander#SUBSTITUTION_PLACEHOLDER_REGEX ;
warns/guides users if their keystore already contains invalid keys;

yaauie

Mechanically, this works, and safely guards against adding keys that cannot be referenced in the pipelines without breaking users whose keystores include offending keys.

I've left suggestions about DRY-ing it up a bit, and improving the admitedly-difficult description of the effective key pattern requirement.

logstash-core/src/main/java/org/logstash/secret/SecretIdentifier.java

docs/reference/keystore.md

logstash-core/src/main/java/org/logstash/secret/SecretIdentifier.java

github-actions · 2025-04-09T16:38:58Z

📃 DOCS PREVIEW ✨ https://logstash_bk_17351.docs-preview.app.elstc.co/diff

github-actions · 2025-04-09T17:30:13Z

📃 DOCS PREVIEW ✨ https://logstash_bk_17351.docs-preview.app.elstc.co/diff

github-actions · 2025-04-09T18:16:56Z

📃 DOCS PREVIEW ✨ https://logstash_bk_17351.docs-preview.app.elstc.co/diff

…ey names to the keystore through keystore CLI. Keysotre now warns on invalid keys if they were already added.

[doc] better explanation of accepting key format. Co-authored-by: Rye Biesemeyer <[email protected]>

…ription. Co-authored-by: Rye Biesemeyer <[email protected]>

mashhurs · 2025-04-09T20:33:45Z

Rebased against main as we were having CI issues, fixed by #17531

github-actions · 2025-04-09T20:37:05Z

📃 DOCS PREVIEW ✨ https://logstash_bk_17351.docs-preview.app.elstc.co/diff

yaauie

👍🏼 looking on-track.

I've left a comment about message format, and suggested more useful random key generation for tests.

logstash-core/src/main/java/org/logstash/secret/cli/SecretStoreCli.java

logstash-core/src/test/java/org/logstash/secret/cli/SecretStoreCliTest.java

logstash-core/src/main/java/org/logstash/secret/SecretIdentifier.java

Warning and error messages well formatted. Co-authored-by: Rye Biesemeyer <[email protected]>

github-actions · 2025-04-11T01:29:11Z

📃 DOCS PREVIEW ✨ https://logstash_bk_17351.docs-preview.app.elstc.co/diff

github-actions · 2025-04-11T18:08:43Z

📃 DOCS PREVIEW ✨ https://logstash_bk_17351.docs-preview.app.elstc.co/diff

yaauie

Works as advertised.

I've left one nit-pick about making the validation step more cohesive, but you're free to decide whether to resolve it or not.

logstash-core/src/main/java/org/logstash/secret/SecretIdentifier.java

elastic-sonarqube · 2025-04-24T00:23:47Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2025-04-24T00:38:24Z

💚 Build Succeeded

Buildkite Build
Commit: 700e596

History

💚 Build #2845 succeeded 6e816bc
💔 Build #2840 failed 8e0136a
💚 Build #2823 succeeded bfea29d
💔 Build #2820 failed 6ac79ce
💔 Build #2819 failed 9139460
💔 Build #2818 failed 0d9c5af

cc @mashhurs

github-actions · 2025-04-24T04:14:31Z

@Mergifyio backport 8.17 8.18 8.19 9.0

mergify · 2025-04-24T04:14:56Z

backport 8.17 8.18 8.19 9.0

✅ Backports have been created

#17585 [8.17] (backport #17351) Improve the key validation in secret identifier. has been created for branch 8.17 but encountered conflicts
#17586 [8.18] (backport #17351) Improve the key validation in secret identifier. has been created for branch 8.18 but encountered conflicts
#17587 [8.19] (backport #17351) Improve the key validation in secret identifier. has been created for branch 8.19 but encountered conflicts
#17588 [9.0] (backport #17351) Improve the key validation in secret identifier. has been created for branch 9.0

* Improve the key validation in secret identifier. * Restrict adding invalid (from ConfigVariableExpander point of view) key names to the keystore through keystore CLI. Keystore now warns on invalid keys if they were already added. * Key pattern is moved to a central config expander class with its description. Co-authored-by: Rye Biesemeyer <[email protected]> (cherry picked from commit d24675a) # Conflicts: # docs/reference/keystore.md

* Improve the key validation in secret identifier. * Restrict adding invalid (from ConfigVariableExpander point of view) key names to the keystore through keystore CLI. Keystore now warns on invalid keys if they were already added. * Key pattern is moved to a central config expander class with its description. Co-authored-by: Rye Biesemeyer <[email protected]> (cherry picked from commit d24675a)

* Improve the key validation in secret identifier. * Restrict adding invalid (from ConfigVariableExpander point of view) key names to the keystore through keystore CLI. Keystore now warns on invalid keys if they were already added. * Key pattern is moved to a central config expander class with its description. Co-authored-by: Rye Biesemeyer <[email protected]> (cherry picked from commit d24675a) Co-authored-by: Mashhur <[email protected]>

…ier. (#17585) * Improve the key validation in secret identifier. (#17351) * Improve the key validation in secret identifier. * Restrict adding invalid (from ConfigVariableExpander point of view) key names to the keystore through keystore CLI. Keystore now warns on invalid keys if they were already added. * Key pattern is moved to a central config expander class with its description. Co-authored-by: Rye Biesemeyer <[email protected]> (cherry picked from commit d24675a) # Conflicts: # docs/reference/keystore.md * Adjust the doc against 8.x --------- Co-authored-by: Mashhur <[email protected]> Co-authored-by: Mashhur <[email protected]>

…ier. (#17586) * Improve the key validation in secret identifier. (#17351) * Improve the key validation in secret identifier. * Restrict adding invalid (from ConfigVariableExpander point of view) key names to the keystore through keystore CLI. Keystore now warns on invalid keys if they were already added. * Key pattern is moved to a central config expander class with its description. Co-authored-by: Rye Biesemeyer <[email protected]> (cherry picked from commit d24675a) # Conflicts: # docs/reference/keystore.md * Adjust docs against 8.x --------- Co-authored-by: Mashhur <[email protected]> Co-authored-by: Mashhur <[email protected]>

…ier. (#17587) * Improve the key validation in secret identifier. (#17351) * Improve the key validation in secret identifier. * Restrict adding invalid (from ConfigVariableExpander point of view) key names to the keystore through keystore CLI. Keystore now warns on invalid keys if they were already added. * Key pattern is moved to a central config expander class with its description. Co-authored-by: Rye Biesemeyer <[email protected]> (cherry picked from commit d24675a) # Conflicts: # docs/reference/keystore.md * Adjust docs against 8.x --------- Co-authored-by: Mashhur <[email protected]> Co-authored-by: Mashhur <[email protected]>

* add applies_to badge to changes introduced in #17351 * add applies_to badge to changes introduced in #17759 * reorder list items * use 9.0.1 not 9.1.0 * Update docs/reference/keystore.md Co-authored-by: Karen Metts <[email protected]> --------- Co-authored-by: Karen Metts <[email protected]>

* add applies_to badge to changes introduced in #17351 * add applies_to badge to changes introduced in #17759 * reorder list items * use 9.0.1 not 9.1.0 * Update docs/reference/keystore.md Co-authored-by: Karen Metts <[email protected]> --------- Co-authored-by: Karen Metts <[email protected]> (cherry picked from commit aed28eb)

* add applies_to badge to changes introduced in #17351 * add applies_to badge to changes introduced in #17759 * reorder list items * use 9.0.1 not 9.1.0 * Update docs/reference/keystore.md --------- (cherry picked from commit aed28eb) Co-authored-by: Colleen McGinnis <[email protected]> Co-authored-by: Karen Metts <[email protected]>

mashhurs self-assigned this Mar 19, 2025

mashhurs added backport-active-9 Automated backport with mergify to all the active 9.[0-9]+ branches backport-active-8 Automated backport with mergify to all the active 8.[0-9]+ branches labels Mar 19, 2025

mashhurs marked this pull request as ready for review March 19, 2025 16:00

mashhurs requested a review from yaauie March 24, 2025 15:32

yaauie reviewed Apr 8, 2025

View reviewed changes

logstash-core/src/main/java/org/logstash/secret/SecretIdentifier.java Outdated Show resolved Hide resolved

docs/reference/keystore.md Outdated Show resolved Hide resolved

logstash-core/src/main/java/org/logstash/secret/SecretIdentifier.java Outdated Show resolved Hide resolved

mashhurs force-pushed the secret-identifier-key-validator branch from 9139460 to 6ac79ce Compare April 9, 2025 18:13

mashhurs and others added 4 commits April 9, 2025 13:33

Improve the key validation in secret indentifier.

2f205c1

Restrict adding invalid (from ConfigVariableExpander point of view) k…

440e894

…ey names to the keystore through keystore CLI. Keysotre now warns on invalid keys if they were already added.

Apply suggestions from code review

3f6f66d

[doc] better explanation of accepting key format. Co-authored-by: Rye Biesemeyer <[email protected]>

Key pattern is moved to a central config expander class with its desc…

bfea29d

…ription. Co-authored-by: Rye Biesemeyer <[email protected]>

mashhurs force-pushed the secret-identifier-key-validator branch from 6ac79ce to bfea29d Compare April 9, 2025 20:33

mashhurs requested a review from yaauie April 9, 2025 21:04

yaauie reviewed Apr 10, 2025

View reviewed changes

mashhurs commented Apr 11, 2025

View reviewed changes

logstash-core/src/main/java/org/logstash/secret/SecretIdentifier.java Outdated Show resolved Hide resolved

Apply suggestions from code review

8e0136a

Warning and error messages well formatted. Co-authored-by: Rye Biesemeyer <[email protected]>

Generate gandom key name based on ConfigVarExpender#KEY_PATTERN regex.

6e816bc

mashhurs requested a review from yaauie April 11, 2025 18:31

yaauie approved these changes Apr 23, 2025

View reviewed changes

logstash-core/src/main/java/org/logstash/secret/SecretIdentifier.java Outdated Show resolved Hide resolved

Absorb validateKey method into validateWithTransform.

700e596

mashhurs merged commit d24675a into elastic:main Apr 24, 2025
9 checks passed

mergify bot mentioned this pull request Apr 24, 2025

[8.17] (backport #17351) Improve the key validation in secret identifier. #17585

Merged

3 tasks

This was referenced Apr 24, 2025

[8.18] (backport #17351) Improve the key validation in secret identifier. #17586

Merged

[8.19] (backport #17351) Improve the key validation in secret identifier. #17587

Merged

mergify bot mentioned this pull request Apr 24, 2025

[9.0] (backport #17351) Improve the key validation in secret identifier. #17588

Merged

3 tasks

mashhurs deleted the secret-identifier-key-validator branch April 24, 2025 04:59

colleenmcginnis added a commit to colleenmcginnis/logstash that referenced this pull request Jul 28, 2025

add applies_to badge to changes introduced in elastic#17351

9894be8

colleenmcginnis mentioned this pull request Jul 28, 2025

[docs] Add applies_to labels for 9.1.0 #17864

Merged

mergify bot mentioned this pull request Aug 14, 2025

[9.1] (backport #17864) [docs] Add applies_to labels for 9.1.0 #17981

Merged

Improve the key validation in secret identifier. #17351

Improve the key validation in secret identifier. #17351

Uh oh!

Conversation

mashhurs commented Mar 19, 2025 • edited by yaauie Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Release notes

What does this PR do?

Why is it important/What is the impact to the user?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

Uh oh!

mergify bot commented Mar 19, 2025

Uh oh!

yaauie commented Mar 19, 2025

Uh oh!

github-actions bot commented Mar 22, 2025

Uh oh!

mashhurs commented Mar 23, 2025

Uh oh!

yaauie left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Apr 9, 2025

Uh oh!

github-actions bot commented Apr 9, 2025

Uh oh!

github-actions bot commented Apr 9, 2025

Uh oh!

mashhurs commented Apr 9, 2025

Uh oh!

github-actions bot commented Apr 9, 2025

Uh oh!

yaauie left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Apr 11, 2025

Uh oh!

github-actions bot commented Apr 11, 2025

Uh oh!

yaauie left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elastic-sonarqube bot commented Apr 24, 2025

Quality Gate passed

Uh oh!

elasticmachine commented Apr 24, 2025

💚 Build Succeeded

History

Uh oh!

Uh oh!

github-actions bot commented Apr 24, 2025

Uh oh!

mergify bot commented Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mashhurs commented Mar 19, 2025 •

edited by yaauie

Loading

yaauie left a comment •

edited

Loading

mergify bot commented Apr 24, 2025 •

edited

Loading