Cannot operate on existing KV entries that contains spaces in the key

[jhitt25]

Overview of the Issue

We've been running Consul for some time now. We have an automated script to place our ACME certs in the KV store for use with Traefik, and we opted for the key naming convention "acme - hostname". Since upgrading to v1.21.5, spaces are no longer allowed in key names. We recreated all the entries using an underscore instead, but now we cannot delete the existing entries containing spaces. We get the following error:

There was an error saving your key.
500: invalid key name, keys should respect the "^[a-zA-Z0-9,_./\\-?&=]+$" format.

This applies to both API calls and CLI usage.

---

Reproduction Steps

Steps to reproduce this issue, eg:

1. Create a cluster using v1.21.4
2. Create KV entries containing spaces in the key.
3. Upgrade to v1.21.5
4. Try to view, modify, or delete the key.

Consul info for both Client and Server

Server info

agent:
        check_monitors = 0
        check_ttls = 0
        checks = 3
        services = 3
build:
        prerelease =
        revision = 3261d11a
        version = 1.21.5
        version_metadata =
consul:
        acl = enabled
        bootstrap = false
        known_datacenters = 2
        leader = false
        leader_addr = 10.60.30.11:8300
        server = true
raft:
        applied_index = 25396266
        commit_index = 25396266
        fsm_pending = 0
        last_contact = 41.669233ms
        last_log_index = 25396266
        last_log_term = 26
        last_snapshot_index = 25396214
        last_snapshot_term = 26
        latest_configuration = [{Suffrage:Voter ID:2e96dfcc-f51b-84a9-44e9-d05151cc07f7 Address:10.60.30.11:8300} {Suffrage:Voter ID:b808679a-a180-e5ca-3f8f-486c2d108cbc Address:10.60.30.4:8300} {Suffrage:Voter ID:58c895df-a7b4-76ca-fbaa-aa06925b9bc2 Address:10.60.30.12:8300}]
        latest_configuration_index = 0
        num_peers = 2
        protocol_version = 3
        protocol_version_max = 3
        protocol_version_min = 0
        snapshot_version_max = 1
        snapshot_version_min = 0
        state = Follower
        term = 26
runtime:
        arch = amd64
        cpu_count = 2
        goroutines = 3976
        max_procs = 2
        os = linux
        version = go1.25.0

Operating system and Environment details

NAME="Oracle Linux Server" VERSION="9.6" ID="ol" ID_LIKE="fedora" VARIANT="Server" VARIANT_ID="server" VERSION_ID="9.6" PLATFORM_ID="platform:el9" PRETTY_NAME="Oracle Linux Server 9.6" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:oracle:linux:9:6:server" HOME_URL="https://linux.oracle.com/" BUG_REPORT_URL="https://github.com/oracle/oracle-linux"

ORACLE_BUGZILLA_PRODUCT="Oracle Linux 9"
ORACLE_BUGZILLA_PRODUCT_VERSION=9.6
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=9.6

[uduncanu]

You might also want to check that Traefik is still pulling the certs from Consul, as it's likely that it won't be able to get data from Consul after a restart, resulting in a high risk of an outage. Traefik has a healthcheck that checks if a key exists and considers Consul as down if it gets any errors, but unfortunately that key doesn't match the new regex so it always marks Consul as down.

Related Traefik issue: traefik/traefik#12092
Other related Consul issue to this: #22822

[dcarbone]

I mentioned this over on #22822, but you should be able to control whether the check runs or not via this agent config key: https://developer.hashicorp.com/consul/docs/reference/agent/configuration-file/general#disable_http_unprintable_char_filter

[jhitt25]

Thanks for mentioning this here as well! That did the trick and I was able to safely delete the old keys.

[ghthor]

Not sure if anyone else would like to chime in on the PR improving this KV validation feature

#22863

[alexleach]

This stung me badly yesterday! I have a vault instance backed by consul, with auth paths at e.g. auth/Entra ID, auth/Google Cloud. Spent well over an hour trying to figure out why Vault wouldn't unseal...

Thanks to @dcarbone for posting the link to the disable_http_unprintable_char_filter flag, which allowed me to boot up Vault / Consul and then run e.g. vault auth move "auth/Entra ID" auth/entra_id