Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RedPanda Public metrics #23661

Open
maksym-iv opened this issue Oct 7, 2024 · 4 comments
Open

RedPanda Public metrics #23661

maksym-iv opened this issue Oct 7, 2024 · 4 comments
Labels
kind/bug Something isn't working

Comments

@maksym-iv
Copy link

maksym-iv commented Oct 7, 2024 •
edited
Loading

Version & Environment

Redpanda version: v24.2.5

Helm: v24.2.5
Platform: K8s (GKE)

What went wrong?

We are using RedPanda for quite a while, and started the upgrade from v24.1.7 (helm 5.8.8) to v24.2.5 (helm 5.9.5).
Everything works fine, except the /public_metrics. We are using TLS, and apparently Admin API (from where metrics is served) switched to TLSv1.3 only, and this is a good thing for sure. However, all metrics disappeared from Google Monitoring (prometheus).
Since we are running on the GKE we are using GMP (Google Managed Service for Prometheus), and it looks like there is an issue in the GMP that prevents scraping TLSv1.3 only endpoints. Although in their docs it is stated that TLS13 can be used, in reality there is an error.

        lastError: 'Get "https://THE_IP:9644/public_metrics": tls: no supported
          versions satisfy MinVersion and MaxVersion'
        lastScrapeDurationSeconds: "0.001722733"

And I've tried to use the good old openssl s_client -connect 10.0.24.158:9644 -tls1_3/-tls1_2 to confirm it's TLS1.3, also tried local prometheus targeting endpoints in GKE with TLS13 set in the scrape config, all works.

So meanwhile GCP is registering the issue, I wonder, is there a way to launch Admin API with TLS1.2 support? Tried tls_min_version, but that didn't work for the Admin API.

What should have happened instead?

How to reproduce the issue?

  1. Deploy RedPanda v24.2.5 in the GCP
  2. Use GMP to scrape endpoints (GMP version v0.12.1, image gke.gcr.io/prometheus-engine/prometheus:v2.45.3-gmp.7-gke.0@sha256:8c8e35af7e2b92ac9d82ce640621c0d3aa10d7d62856681af3572d0a8fbb787b
  3. Observe the PodMonitoring status

Just to note, I don't believe it's the RedPanda issue, it's GMP fails to scrape, I just wonder if there is a way to have TLS1.2 within admin API and RedPanda v24.2.5
@maksym-iv maksym-iv added the kind/bug Something isn't working label Oct 7, 2024
@rockwotj
Copy link
Contributor

rockwotj commented Oct 9, 2024

Hello @maksym-iv - I am sorry to hear that the stats disappeared!

This change happened in #21372 and is indeed controlled by a cluster property min_tls_version. You should be able to downgrade that way. Please let us know if that doesn't work as intended.

Link to docs on how to change cluster properties: https://docs.redpanda.com/current/manage/kubernetes/k-cluster-property-configuration/

@rockwotj
Copy link
Contributor

rockwotj commented Oct 9, 2024

Also, we should also be supporting TLS 1.2 by default, if that's not the case than thats a larger bug.

cc: @michael-redpanda

@michael-redpanda
Copy link
Contributor

Hi @maksym-iv, sorry again you're experiencing issues. Would it be possible for you to send me the output of the openssl s_client -tls1_2 -connect and the -tls1_3 command? Can you also confirm the value from rpk cluster config get tls_min_version? Can you also try setting it to v1.0 and see if that changes anything? rpk cluster config set tls_min_version v1.0

Also feel free to DM me (@MichaelBoquard) on the Redpanda community slack channel.

@maksym-iv-elf
Copy link

Hey @michael-redpanda

Sure, here is the output, I've also added curl output to double confirm it works/not works

tls_min_version v1.2

rpk cluster config get tls_min_version
v1.2

TLSv1.2 openssl/curl:

  • Results of the openssl s_client -connect 10.0.24.158:9644 -tls1_2

    Connecting to 10.0.24.158
CONNECTED(00000003)
408F1EFD01000000:error:0A000410:SSL routines:ssl3_read_bytes:ssl/tls alert handshake failure:ssl/record/rec_layer_s3.c:907:SSL alert number 40
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 194 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1728564913
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---

  • Results of the curl -k --tls-max 1.2 -v https://10.0.24.158:9644/public_metrics

    *   Trying 10.0.24.158:9644...
* Connected to 10.0.24.158 (10.0.24.158) port 9644
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* LibreSSL/3.3.6: error:1404B410:SSL routines:ST_CONNECT:sslv3 alert handshake failure
* Closing connection
curl: (35) LibreSSL/3.3.6: error:1404B410:SSL routines:ST_CONNECT:sslv3 alert handshake failure

TLSv1.3 openssl/curl:

  • Results of the openssl s_client -connect 10.0.24.158:9644 -tls1_3

    Connecting to 10.0.24.158
CONNECTED(00000003)
Can't use SSL_get_servername
depth=1 O=Redacted Inc.
verify error:num=19:self-signed certificate in certificate chain
verify return:1
depth=1 O=Redacted Inc.
verify return:1
depth=0
verify return:1
---
Certificate chain
0 s:
  i:O=Redacted Inc.
  a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256
  v:NotBefore: Mar  5 15:28:48 2024 GMT; NotAfter: Mar  5 15:28:48 2025 GMT
1 s:O=Redacted Inc.
  i:O=Redacted Inc.
  a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256
  v:NotBefore: Mar  5 15:01:44 2024 GMT; NotAfter: Mar  4 15:01:44 2029 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIBwzCCAWigAwIBAgIRAJABdTZN0Lk2EUVF5i6poicwCgYIKoZIzj0EAwIwGjEY
REDACTED
TQZXguHXnLd3NkdtYrL3eZZd4uPWW5M=
-----END CERTIFICATE-----
subject=
issuer=O=Redacted Inc.
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1228 bytes and written 293 bytes
Verification error: self-signed certificate in certificate chain
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 256 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 19 (self-signed certificate in certificate chain)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
    Session-ID: 59A823131EF62AA544CCA648259E89494136E84F955439A558677F3BA45955AC
    Session-ID-ctx:
    Resumption PSK: 1749FC153575D6B85F597A322FA1A85840101E8C9E3021F9E98102CB631B0687
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - e7 62 c1 75 1f b4 3a 55-28 4f 23 5a 78 06 69 23   .b.u..:U(O#Zx.i#
    REDACTED
    00b0 - e7 16 80 95 1e e2 8f a5-f3 48 b2 22 dc 41 d7 8a   .........H.".A..

    Start Time: 1728564925
    Timeout   : 7200 (sec)
    Verify return code: 19 (self-signed certificate in certificate chain)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
    Session-ID: 252ACF60F42A6961A31BE705BDB926280123C45D4559372746D28E0DBAE746C0
    Session-ID-ctx:
    Resumption PSK: 058DAB389E2CCE58AEB7568B0917B23073982BD1570C6EA76D0EF4CC37A19613
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - e7 62 c1 75 1f b4 3a 55-28 4f 23 5a 78 06 69 23   .b.u..:U(O#Zx.i#
    REDACTED
    00b0 - d2 a4 20 2a e2 c4 d8 b0-03 3c 6d f5 30 d6 32 fe   .. *.....<m.0.2.

    Start Time: 1728564925
    Timeout   : 7200 (sec)
    Verify return code: 19 (self-signed certificate in certificate chain)
    Extended master secret: no
    Max Early Data: 0
---

  • Results of the curl -k --tls-max 1.3 -v https://10.0.24.158:9644/public_metrics

    *   Trying 10.0.24.158:9644...
* Connected to 10.0.24.158 (10.0.24.158) port 9644
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES128-GCM-SHA256 / [blank] / UNDEF
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: [NONE]
*  start date: Mar  5 15:28:48 2024 GMT
*  expire date: Mar  5 15:28:48 2025 GMT
*  issuer: O=Redacted Inc.
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* using HTTP/1.x
> GET /public_metrics HTTP/1.1
> Host: 10.0.24.158:9644
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
< Content-Type: text/plain
< Date: Thu, 10 Oct 2024 12:57:36 GMT
< Server: Seastar httpd
<
# HELP redpanda_application_build Redpanda build information
# TYPE redpanda_application_build gauge
...
...
...

tls_min_version v1.0

Set via values.yaml, restarted nodes

rpk cluster config status
NODE  CONFIG-VERSION  NEEDS-RESTART  INVALID  UNKNOWN
0     3               false          []       []
1     3               false          []       []
2     3               false          []       []
rpk cluster config get tls_min_version
v1.0

TLSv1.2 openssl/curl:

  • Results of the openssl s_client -connect 10.0.24.165:9644 -tls1_2

    Connecting to 10.0.24.165
CONNECTED(00000003)
408F1EFD01000000:error:0A000410:SSL routines:ssl3_read_bytes:ssl/tls alert handshake failure:ssl/record/rec_layer_s3.c:907:SSL alert number 40
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 194 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1728567379
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---

  • Results of the curl -k --tls-max 1.2 -v https://10.0.24.165:9644/public_metrics

    *   Trying 10.0.24.165:9644...
* Connected to 10.0.24.165 (10.0.24.165) port 9644
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* LibreSSL/3.3.6: error:1404B410:SSL routines:ST_CONNECT:sslv3 alert handshake failure
* Closing connection
curl: (35) LibreSSL/3.3.6: error:1404B410:SSL routines:ST_CONNECT:sslv3 alert handshake failure

TLSv1.3 openssl/curl:

  • Results of the openssl s_client -connect 10.0.24.165:9644 -tls1_3

    Connecting to 10.0.24.165
CONNECTED(00000003)
Can't use SSL_get_servername
depth=1 O=Redacted Inc.
verify error:num=19:self-signed certificate in certificate chain
verify return:1
depth=1 O=Redacted Inc.
verify return:1
depth=0
verify return:1
---
Certificate chain
0 s:
  i:O=Redacted Inc.
  a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256
  v:NotBefore: Mar  5 15:28:48 2024 GMT; NotAfter: Mar  5 15:28:48 2025 GMT
1 s:O=Redacted Inc.
  i:O=Redacted Inc.
  a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256
  v:NotBefore: Mar  5 15:01:44 2024 GMT; NotAfter: Mar  4 15:01:44 2029 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIBwzCCAWigAwIBAgIRAJABdTZN0Lk2EUVF5i6poicwCgYIKoZIzj0EAwIwGjEY
REDACTED
RgIhAIytVDUAfZFmRPPMGkh4rG3T9tmqMTXgDJ7+ky6o/ZhiAiEAyLokGPzMohDS
TQZXguHXnLd3NkdtYrL3eZZd4uPWW5M=
-----END CERTIFICATE-----
subject=
issuer=O=Redacted Inc.
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1228 bytes and written 293 bytes
Verification error: self-signed certificate in certificate chain
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 256 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 19 (self-signed certificate in certificate chain)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
    Session-ID: 3B38C54CD96E8ACB6DE7B0B7C88010A7C9DC454C3FADAB1782244D65A3C37256
    Session-ID-ctx:
    Resumption PSK: D232B6643395251780FB8E9CD8EA389FFDA3EEA7200E5C8CE7251E81FACBBEC4
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - 82 db c0 3a 70 1c e3 ef-a5 c2 f9 de cb 50 ec 2e   ...:p........P..
    REDACTED
    00b0 - 0f 5a 1d d5 03 76 ff ea-01 fc 59 89 73 ce 9a 0e   .Z...v....Y.s...

    Start Time: 1728567423
    Timeout   : 7200 (sec)
    Verify return code: 19 (self-signed certificate in certificate chain)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
    Session-ID: A5C1EC7646B0E0443D2FCC5BDCEF102A3CFE2AC0623E704CDA1356596A4EA998
    Session-ID-ctx:
    Resumption PSK: DBF2740D8BFA96274D41B5DF849E989372249D5222309F7176A87FC265E70E64
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - 82 db c0 3a 70 1c e3 ef-a5 c2 f9 de cb 50 ec 2e   ...:p........P..
    REDACTED
    00b0 - 3c 70 a2 2f 2a fc df 4a-15 f3 4b f5 71 af 44 46   <p./*..J..K.q.DF

    Start Time: 1728567423
    Timeout   : 7200 (sec)
    Verify return code: 19 (self-signed certificate in certificate chain)
    Extended master secret: no
    Max Early Data: 0
---

  • Results of the curl -k --tls-max 1.3 -v https://10.0.24.165:9644/public_metrics

    *   Trying 10.0.24.165:9644...
* Connected to 10.0.24.165 (10.0.24.165) port 9644
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES128-GCM-SHA256 / [blank] / UNDEF
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: [NONE]
*  start date: Mar  5 15:28:48 2024 GMT
*  expire date: Mar  5 15:28:48 2025 GMT
*  issuer: O=Redacted Inc.
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* using HTTP/1.x
> GET /public_metrics HTTP/1.1
> Host: 10.0.24.165:9644
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
< Content-Type: text/plain
< Date: Thu, 10 Oct 2024 13:37:59 GMT
< Server: Seastar httpd
<
# HELP redpanda_application_build Redpanda build information
# TYPE redpanda_application_build gauge
...
...
...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@maksym-iv @rockwotj @michael-redpanda @maksym-iv-elf