FLINK-5725: Add extra Flink details to paasta status #4063

nleigh · 2025-05-13T15:16:58Z

May be easier to review first per commit

https://jira.yelpcorp.com/browse/FLINK-5725

HACKATHON-39

To speed up gathering of details for a Flink cluster/job, this PR adds

yelpsoa and srv links to paasta status verbose output for flink
github and sourcegraph link
paasta log commands for flink components
grafana dashboard links
Flink Owner
Flink RB
Splunk costs
Formatting dividers for easier readability

When running paasta status, you are normally debugging something (something is broke)
You then may need to check related resources for the flink job such as yelpsoa/srv
Also when in a slack thread this information can useful to share in thread, reduce time to spot the issue

Output Example

Prod

paasta status -s acorn -i main -c pnw-devc -v


acorn.main in pnw-devc (EKS)
    Version:    6563d637 (desired)
    Config SHA: 00ce0453
    Flink Pool: flink-spot
    Owner: streaming-infrastructure
    Flink Runbook: y/rb-acorn
    Repo(git): https://github.yelpcorp.com/services/acorn
    Repo(sourcegraph): https://sourcegraph.yelpcorp.com/services/acorn
    Flink version: 1.13.5 0ff28a7 @ 2021-12-14T23:26:04+01:00
    URL: http://flink.eks.pnw-devc.paasta:31080/acorn-7f797b79f6/
    Yelpsoa configs: https://github.yelpcorp.com/sysgit/yelpsoa-configs/tree/master/acorn
    Srv configs: https://github.yelpcorp.com/sysgit/srv-configs/tree/master/ecosystem/devc/acorn
==================================================================
    Flink Log Commands:
      Service:     paasta logs -a 1h -c pnw-devc -s acorn -i main
      Taskmanager: paasta logs -a 1h -c pnw-devc -s acorn -i main.TASKMANAGER
      Jobmanager:  paasta logs -a 1h -c pnw-devc -s acorn -i main.JOBMANAGER
      Supervisor:  paasta logs -a 1h -c pnw-devc -s acorn -i main.SUPERVISOR
==================================================================
    Flink Monitoring:
      Job Metrics: https://grafana.yelpcorp.com/d/flink-metrics/flink-job-metrics?orgId=1&var-datasource=Prometheus-flink&var-region=uswest2-devc&var-service=acorn&var-instance=main&var-job=All&from=now-24h&to=now
      Container Metrics: https://grafana.yelpcorp.com/d/flink-container-metrics/flink-container-metrics?orgId=1&var-datasource=Prometheus-flink&var-region=uswest2-devc&var-service=acorn&var-instance=main&from=now-24h&to=now
      JVM Metrics: https://grafana.yelpcorp.com/d/flink-jvm-metrics/flink-jvm-metrics?orgId=1&var-datasource=Prometheus-flink&var-region=uswest2-devc&var-service=acorn&var-instance=main&from=now-24h&to=now
      Flink Cost: https://splunk.yelpcorp.com/en-US/app/yelp_computeinfra/paasta_service_utilization?form.service=acorn&form.field1.earliest=-30d%40d&form.field1.latest=now&form.instance=main&form.cluster=pnw-devc
==================================================================
    State: Running
    Pods: 5 running, 0 evicted, 0 other
    Jobs: 1 running, 0 finished, 0 failed, 0 cancelled
    3 taskmanagers, 8/30 slots available
    Jobs:
      Job Name                State       Started
      heartbeat_s3_checkpoint Running 2025-05-15 15:42:57 (11 hours ago)
    Pods:
      Pod Name                                       Host                                        Phase    Uptime
      acorn-7f797b79f6-jobmanager-5859456947-qv2ml   ip-10-81-17-237.us-west-2.compute.internal  Running  0d11h3m9s
      acorn-7f797b79f6-supervisor-p9l6n              ip-10-81-19-13.us-west-2.compute.internal   Running  0d17h11m23s
      acorn-7f797b79f6-taskmanager-5bf76bf45f-2js7q  ip-10-81-23-26.us-west-2.compute.internal   Running  0d2h54m15s
      acorn-7f797b79f6-taskmanager-5bf76bf45f-76x24  ip-10-81-23-26.us-west-2.compute.internal   Running  0d2h54m15s
      acorn-7f797b79f6-taskmanager-5bf76bf45f-rkh9w  ip-10-81-19-237.us-west-2.compute.internal  Running  0d7h42m18s

Full Output tests
May 29th output
paasta status -s acorn -i main -c pnw-devc -v
paasta status -s acorn -i main -c pnw-prod -v
paasta status -s acorn -i main_eks -c infrastage -v
paasta status -s acorn -i main_eks -c norcal-stageg -v
paasta status -s acorn -i main_eks -c norcal-stagef -v
https://fluffy.yelpcorp.com/i/TV5vKLJgjmHZpwqPcjFJ7frmCm9sf8mM.html

https://jira.yelpcorp.com/browse/FLINK-5725 HACKATHON-39 This PR adds yelpsoa and srv links to paasta status verbose output for flink to speed up gathering of details for the cluster/job When running paasta status, you are normally debugging something (something is broke) You then may need to check related resources for the flink job such as yelpsoa/srv Also when in a slack thread this information can useful to share in thread, reduce time to spot the issue Plan to add more in future PR's New paasta status output (ran locally) ``` paasta status -s sqlclient -i excluded_assignments_log_counts_by_dimensions -c pnw-prod -vvv Traceback (most recent call last): File "/nail/home/nathanleigh/source/paasta/.tox/py38-linux/bin/paasta ``` Output ``` paasta status -s sqlclient -i excluded_assignments_log_counts_by_dimensions -c pnw-prod -vvv sqlclient.excluded_assignments_log_counts_by_dimensions in pnw-prod (EKS) Version: c74dd64a (desired) Config SHA: 2ba0c242 Flink version: 1.17.2 c0027e5 @ 2023-11-09T13:24:38+01:00 URL: https://flink-eks-pnw-prod.yelpcorp.com/sqlclient-c9d8cdc84/ Yelpsoa configs: https://sourcegraph.yelpcorp.com/sysgit/yelpsoa-configs/-/tree/sqlclient Srv configs: https://sourcegraph.yelpcorp.com/sysgit/srv-configs/-/tree/ecosystem/pnw-prod/sqlclient State: Running Pods: 14 running, 0 evicted, 0 other Jobs: 1 running, 0 finished, 0 failed, 0 cancelled 12 taskmanagers, 12/72 slots available ... ```

nleigh · 2025-05-13T15:58:17Z

Testing copilot review out of curiosity (haven't used before)

Copilot

Pull Request Overview

This PR enhances paasta status output for Flink deployments by adding Yelpsoa and Srv configuration links to facilitate quicker diagnostics. The changes propagate through both the CLI status commands and the corresponding tests.

Introduces new output lines for Yelpsoa and Srv configs in status display.
Updates tests to validate the new output format.
Modifies function signatures in status commands to include an additional "cluster" parameter for proper URL generation.

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
tests/cli/test_cmds_status.py	Adds expected Yelpsoa and Srv configs to test output
paasta_tools/cli/cmds/status.py	Adds extra config links and updates function calls

Copilot · 2025-05-13T15:58:41Z

paasta_tools/cli/cmds/status.py

+        output.append(
+            f"    Yelpsoa configs: https://sourcegraph.yelpcorp.com/sysgit/yelpsoa-configs/-/tree/{service}"
+        )
+        cluster_without_pnw_dash = cluster.replace("pnw-", "")


Using .replace("pnw-", "") may remove multiple occurrences of the substring instead of only the prefix. Consider using a conditional check or a method that only removes the prefix if present.

Suggested change

cluster_without_pnw_dash = cluster.replace("pnw-", "")

cluster_without_pnw_dash = cluster.removeprefix("pnw-")

https://jira.yelpcorp.com/browse/FLINK-5725 HACKATHON-39 This commit adds git and sourcegraphj links to paasta status verbose output for flink to speed up gathering of details for the cluster/job When running paasta status, you are normally debugging something (something is broke) You then may need to check related resources for the flink job such as yelpsoa/srv Also when in a slack thread this information can useful to share in thread, reduce time to spot the issue Plan to add more in future PR's New paasta status output (ran locally) ``` paasta status -s sqlclient -i excluded_assignments_log_counts_by_dimensions -c pnw-prod -vvv Traceback (most recent call last): File "/nail/home/nathanleigh/source/paasta/.tox/py38-linux/bin/paasta ``` Output ``` paasta status -s sqlclient -i excluded_assignments_log_counts_by_dimensions -c pnw-prod -vvv sqlclient.excluded_assignments_log_counts_by_dimensions in pnw-prod (EKS) Version: c74dd64a (desired) Config SHA: 2ba0c242 Repo(git): https://github.yelpcorp.com/services/sqlclient Repo(sourcegraph): https://sourcegraph.yelpcorp.com/services/sqlclient Flink version: 1.17.2 c0027e5 @ 2023-11-09T13:24:38+01:00 URL: https://flink-eks-pnw-prod.yelpcorp.com/sqlclient-c9d8cdc84/ Yelpsoa configs: https://sourcegraph.yelpcorp.com/sysgit/yelpsoa-configs/-/tree/sqlclient Srv configs: https://sourcegraph.yelpcorp.com/sysgit/srv-configs/-/tree/ecosystem/prod/sqlclient State: Running Pods: 14 running, 0 evicted, 0 other Jobs: 1 running, 0 finished, 0 failed, 0 cancelled 12 taskmanagers, 12/72 slots available Jobs: Job Name State Job ID Started excluded_assignments_log_counts_by_dimensions Running d6c20dd9e9741c9646a41daa2f5b88aa 2025-05-13 05:11:30 (4 hours ago) https://flink-eks-pnw-prod.yelpcorp.com/sqlclient-c9d8cdc84/#/jobs/d6c20dd9e9741c9646a41daa2f5b88aa Pods: Pod Name Host Phase Uptime sqlclient-c9d8cdc84-jobmanager-5ff9cd5ff5-469m2 ip-10-69-177-166.us-west-2.compute.internal Running 0d4h59m1s ... ```

https://jira.yelpcorp.com/browse/FLINK-5725 HACKATHON-39 This commit adds flink logs commands to paasta status verbose output for flink to speed up gathering of details for the cluster/job When running paasta status, you are normally debugging something (something is broke) You then may need to check related resources for the flink job such as yelpsoa/srv Also when in a slack thread this information can useful to share in thread, reduce time to spot the issue New paasta status output (ran locally) ``` paasta status -s sqlclient -i excluded_assignments_log_counts_by_dimensions -c pnw-prod -vvv Traceback (most recent call last): File "/nail/home/nathanleigh/source/paasta/.tox/py38-linux/bin/paasta ``` Output ``` paasta status -s sqlclient -i excluded_assignments_log_counts_by_dimensions -c pnw-prod -vvv sqlclient.excluded_assignments_log_counts_by_dimensions in pnw-prod (EKS) Version: c74dd64a (desired) Config SHA: 2ba0c242 Repo(git): https://github.yelpcorp.com/services/sqlclient Repo(sourcegraph): https://sourcegraph.yelpcorp.com/services/sqlclient Flink version: 1.17.2 c0027e5 @ 2023-11-09T13:24:38+01:00 URL: https://flink-eks-pnw-prod.yelpcorp.com/sqlclient-c9d8cdc84/ Yelpsoa configs: https://sourcegraph.yelpcorp.com/sysgit/yelpsoa-configs/-/tree/sqlclient Srv configs: https://sourcegraph.yelpcorp.com/sysgit/srv-configs/-/tree/ecosystem/prod/sqlclient Flink Log Commands: Service: paasta logs -a 1h -c pnw-prod -s sqlclient -i excluded_assignments_log_counts_by_dimensions Taskmanager: paasta logs -a 1h -c pnw-prod -s sqlclient -i excluded_assignments_log_counts_by_dimensions.TASKMANAGER Jobmanager: paasta logs -a 1h -c pnw-prod -s sqlclient -i excluded_assignments_log_counts_by_dimensions.JOBMANAGER Supervisor: paasta logs -a 1h -c pnw-prod -s sqlclient -i excluded_assignments_log_counts_by_dimensions.SUPERVISOR State: Running Pods: 14 running, 0 evicted, 0 other Jobs: 1 running, 0 finished, 0 failed, 0 cancelled 12 taskmanagers, 12/72 slots available ... ```

https://jira.yelpcorp.com/browse/FLINK-5725 HACKATHON-39 This commit adds flink grafana monitoring links to paasta status verbose output for flink to speed up gathering of details for the cluster/job When running paasta status, you are normally debugging something (something is broke) You then may need to check related resources for the flink job such as yelpsoa/srv Also when in a slack thread this information can useful to share in thread, reduce time to spot the issue New paasta status output (ran locally) ``` paasta status -s sqlclient -i excluded_assignments_log_counts_by_dimensions -c pnw-prod -vvv Traceback (most recent call last): File "/nail/home/nathanleigh/source/paasta/.tox/py38-linux/bin/paasta ``` Output ``` paasta status -s sqlclient -i excluded_assignments_log_counts_by_dimensions -c pnw-prod -vvv sqlclient.excluded_assignments_log_counts_by_dimensions in pnw-prod (EKS) Version: c74dd64a (desired) Config SHA: 2ba0c242 Repo(git): https://github.yelpcorp.com/services/sqlclient Repo(sourcegraph): https://sourcegraph.yelpcorp.com/services/sqlclient Flink version: 1.17.2 c0027e5 @ 2023-11-09T13:24:38+01:00 URL: https://flink-eks-pnw-prod.yelpcorp.com/sqlclient-c9d8cdc84/ Yelpsoa configs: https://sourcegraph.yelpcorp.com/sysgit/yelpsoa-configs/-/tree/sqlclient Srv configs: https://sourcegraph.yelpcorp.com/sysgit/srv-configs/-/tree/ecosystem/prod/sqlclient Flink Log Commands: Service: paasta logs -a 1h -c pnw-prod -s sqlclient -i excluded_assignments_log_counts_by_dimensions Taskmanager: paasta logs -a 1h -c pnw-prod -s sqlclient -i excluded_assignments_log_counts_by_dimensions.TASKMANAGER Jobmanager: paasta logs -a 1h -c pnw-prod -s sqlclient -i excluded_assignments_log_counts_by_dimensions.JOBMANAGER Supervisor: paasta logs -a 1h -c pnw-prod -s sqlclient -i excluded_assignments_log_counts_by_dimensions.SUPERVISOR Flink Monitoring: Job Metrics: https://grafana.yelpcorp.com/d/flink-metrics/flink-job-metrics?orgId=1&var-datasource=Prometheus-flink&var-region=uswest2-prod&var-service=sqlclient&var-instance=excluded_assignments_log_counts_by_dimensions&var-job=All&from=now-24h&to=now Container Metrics: https://grafana.yelpcorp.com/d/flink-container-metrics/flink-container-metrics?orgId=1&var-datasource=Prometheus-flink&var-region=uswest2-prod&var-service=sqlclient&var-instance=excluded_assignments_log_counts_by_dimensions&from=now-24h&to=now JVM Metrics: https://grafana.yelpcorp.com/d/flink-jvm-metrics/flink-jvm-metrics?orgId=1&var-datasource=Prometheus-flink&var-region=uswest2-prod&var-service=sqlclient&var-instance=excluded_assignments_log_counts_by_dimensions&from=now-24h&to=now State: Running Pods: 14 running, 0 evicted, 0 other Jobs: 1 running, 0 finished, 0 failed, 0 cancelled ... ```

anohovsky · 2025-05-14T09:30:35Z

paasta_tools/cli/cmds/status.py

+    if verbose:
+        output.append(f"    Flink Monitoring:")
+        output.append(
+            f"      Job Metrics: https://grafana.yelpcorp.com/d/flink-metrics/flink-job-metrics?orgId=1&var-datasource=Prometheus-flink&var-region=uswest2-{cluster_without_pnw_dash}&var-service={service}&var-instance={instance}&var-job=All&from=now-24h&to=now"


I like the idea in general, the only concern is that these are yelp urls while changes are in the public repository. Is this repository in use outside of Yelp?

@anohovsky I'm not sure, I imagine the chances of people using it AND Flink would be very low

This paasta tools already has a lot of 'yelpy' stuff such as yelpsoa parsing and we already have yelpy url's in the code
https://github.com/Yelp/paasta/blob/1fb40c62b5199f8da2305f98619a2a28d506e1fe/paasta_tools/cli/cmds/status.py#L816C1-L819C1

# Annotation "flink.yelp.com/dashboard_url" is populated by flink-operator dashboard_url = metadata["annotations"].get("flink.yelp.com/dashboard_url") output.append(f" URL: {dashboard_url}/")

Some usages of yelpcorp

@anohovsky I have asked compute infra(paasta owners) just to double check
#4063 (comment)

no one outside of yelp uses paasta (nor are we expecting anyone to): it's more of a "developed in the open" type of thing than anything else

we do normally try to limit some of the yelpiness when possible by making things configurable through SystemPaastaConfig - but that's not really a hard line

e.g., if we wanted to stop hardcoding some of these URLs in paasta, we could do something like store the base urls in system paasta config and then build/append the query string here

nemacysts · 2025-05-14T12:40:49Z

paasta_tools/cli/cmds/status.py

@@ -809,6 +817,47 @@ def _print_flink_status_from_job_manager(
        dashboard_url = metadata["annotations"].get("flink.yelp.com/dashboard_url")
        output.append(f"    URL: {dashboard_url}/")

+    cluster_without_pnw_dash = cluster.replace("pnw-", "")


this is somewhat brittle and will not work for things like infrastage

i think we might want a mapping somewhere of cluster -> ecosystem + cluster -> region (perhaps system paasta config if there's not already something for this)

We have a very small amount of services running on (non) pnw clusters so thought I would omit adding for these cases. But have added to fully work for all mappings
56cdcc5

nemacysts · 2025-05-14T12:41:41Z

paasta_tools/cli/cmds/status.py

+        )
+
+    # Print Flink Log Commands
+    if verbose:


we have a bunch of if verbose: blocks in a row - should these be a single block?

Possibly, it was more for just formatting/separating
Tbf it's getting to the point where could maybe refactor whole function, will consider it in a new PR

we can separate with comments inside a single if verbose: block :)

that said ++ to more refactoring in another PR

nemacysts · 2025-05-14T12:43:58Z

paasta_tools/cli/cmds/status.py

+            f"    Yelpsoa configs: https://sourcegraph.yelpcorp.com/sysgit/yelpsoa-configs/-/tree/{service}"
+        )
+        output.append(
+            f"    Srv configs: https://sourcegraph.yelpcorp.com/sysgit/srv-configs/-/tree/ecosystem/"


i'd probably link to GH here since it's easier to go back to the PR that edited a line from there

i think we also never really type these as "srv configs" and "yelpsoa configs" on CI, but i guess whatever your users are expecting is fine 😔

Updated to github links in
02c4363

nemacysts · 2025-05-14T12:50:37Z

paasta_tools/cli/cmds/status.py

+    if verbose:
+        output.append(f"    Flink Monitoring:")
+        output.append(
+            f"      Job Metrics: https://grafana.yelpcorp.com/d/flink-metrics/flink-job-metrics?orgId=1&var-datasource=Prometheus-flink&var-region=uswest2-{cluster_without_pnw_dash}&var-service={service}&var-instance={instance}&var-job=All&from=now-24h&to=now"


no one outside of yelp uses paasta (nor are we expecting anyone to): it's more of a "developed in the open" type of thing than anything else

we do normally try to limit some of the yelpiness when possible by making things configurable through SystemPaastaConfig - but that's not really a hard line

nemacysts · 2025-05-14T12:53:09Z

paasta_tools/cli/cmds/status.py

+    if verbose:
+        output.append(f"    Flink Monitoring:")
+        output.append(
+            f"      Job Metrics: https://grafana.yelpcorp.com/d/flink-metrics/flink-job-metrics?orgId=1&var-datasource=Prometheus-flink&var-region=uswest2-{cluster_without_pnw_dash}&var-service={service}&var-instance={instance}&var-job=All&from=now-24h&to=now"


e.g., if we wanted to stop hardcoding some of these URLs in paasta, we could do something like store the base urls in system paasta config and then build/append the query string here

https://jira.yelpcorp.com/browse/FLINK-5725 HACKATHON-39 To address review comment ``` this is somewhat brittle and will not work for things like infrastage i think we might want a mapping somewhere of cluster -> ecosystem + cluster -> region (perhaps system paasta config if there's not already something for this) ```

https://jira.yelpcorp.com/browse/FLINK-5725 HACKATHON-39

Copilot

Pull Request Overview

This PR enriches the paasta status output for Flink clusters by adding additional Flink resource links and log commands to assist in debugging and troubleshooting.

Tests have been updated to reflect the new outputs including repo links, log commands, and monitoring URLs.
A mapping has been added for determining the ecosystem based on the Flink cluster, and the status printing functions have been updated to utilize it.

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
tests/cli/test_cmds_status.py	Updated expected output strings for enhanced Flink details.
paasta_tools/utils.py	Added mapping SUPPERREGION_TO_ECOSYSTEM_MAPPINGS for ecosystem inference.
paasta_tools/cli/cmds/status.py	Adjusted status functions to include extra Flink details using the new mapping.

Comments suppressed due to low confidence (1)

paasta_tools/utils.py:185

[nitpick] The mapping variable name 'SUPPERREGION_TO_ECOSYSTEM_MAPPINGS' appears to be inconsistent with the comment (which references 'superregion'). Consider renaming it to 'SUPERREGION_TO_ECOSYSTEM_MAPPINGS' for clarity.

SUPPERREGION_TO_ECOSYSTEM_MAPPINGS = {

nemacysts · 2025-05-14T15:08:14Z

paasta_tools/cli/cmds/status.py

+        )
+
+    # Print Flink Log Commands
+    if verbose:


we can separate with comments inside a single if verbose: block :)

that said ++ to more refactoring in another PR

nemacysts · 2025-05-14T15:08:48Z

paasta_tools/utils.py

+# https://github.yelpcorp.com/sysgit/srv-configs/tree/master/superregion
+SUPPERREGION_TO_ECOSYSTEM_MAPPINGS = {
+    "norcal-devc": "devc",
+    "norcal-stagef": "stagef",
+    "norcal-stageg": "stageg",
+    "nova-prod": "prod",
+    "pnw-devc": "devc",
+    "pnw-prod": "prod",
+}


this should probably be accessed through SystemPaastaConfig - i.e., live in puppet

also, this approach will still fail for infrastage - not all clusters are named after a superregion

@nemacysts we only have 1 service/instance in infrastage that we use for testing
https://sourcegraph.yelpcorp.com/search?q=repo:%5Esysgit/yelpsoa-configs%24+flinkeks-infrastage&patternType=keyword&sm=0

Infrastage would require a bit of extra logic,(srv and soa links don't follow pattern) so I decided to remove it

I am not sure what I would be accessing via SystemPaastaConfig
https://sourcegraph.yelpcorp.com/search?q=repo:%5EYelp/paasta%24+SystemPaastaConfigDict%28&patternType=keyword&sm=0
SystemPaastaConfigDict(
https://sourcegraph.yelpcorp.com/Yelp/paasta/-/blob/paasta_tools/utils.py?L1948-1950

nemacysts · 2025-05-14T15:10:01Z

tests/cli/test_cmds_status.py

@@ -2702,7 +2702,7 @@ def test_output_stopping_jobmanager(
        output = []
        mock_flink_status["status"]["state"] = "Stoppingjobmanager"
        print_flink_status(
-            cluster="fake_cluster",
+            cluster="pnw-devc",


once the mapping is moved to SPC, we can probably mock the getter and have a fake ecosystem for fake_cluster

@nleigh we still wanna do this: using a non-existent cluster ensures that if folks make a mistake wrt their mocks, we never actually hit a real paasta api

i guess for now this is still technically safe since we don't have any service called fake_service - but it would definitely make things safer (both for the existing tests and for folks copying this and potentially adding real service names/instances in new tests)

I see thanks
Updated in 2720594

https://jira.yelpcorp.com/browse/FLINK-5725 HACKATHON-39 To speed up gathering of details for a Flink cluster/job When running paasta status, you are normally debugging something (something is broke) You then may need to check related resources for the flink job such as yelpsoa/srv Also when in a slack thread this information can useful to share in thread, reduce time to spot the issue Paasta status output ``` paasta status -s sqlclient -i excluded_assignments_log_counts_by_dimensions -c pnw-devc -vvv sqlclient.excluded_assignments_log_counts_by_dimensions in pnw-devc (EKS) Version: c74dd64a (desired) Config SHA: c8ee0154 Owner: fxp Repo(git): https://github.yelpcorp.com/services/sqlclient Repo(sourcegraph): https://sourcegraph.yelpcorp.com/services/sqlclient Flink version: 1.17.2 c0027e5 @ 2023-11-09T13:24:38+01:00 URL: http://flink.eks.pnw-devc.paasta:31080/sqlclient-c9d8cdc84/ Yelpsoa configs: https://github.yelpcorp.com/sysgit/yelpsoa-configs/tree/master/sqlclient Srv configs: https://github.yelpcorp.com/sysgit/srv-configs/tree/master/ecosystem/devc/sqlclient Flink Log Commands: Service: paasta logs -a 1h -c pnw-devc -s sqlclient -i excluded_assignments_log_counts_by_dimensions Taskmanager: paasta logs -a 1h -c pnw-devc -s sqlclient -i excluded_assignments_log_counts_by_dimensions.TASKMANAGER Jobmanager: paasta logs -a 1h -c pnw-devc -s sqlclient -i excluded_assignments_log_counts_by_dimensions.JOBMANAGER Supervisor: paasta logs -a 1h -c pnw-devc -s sqlclient -i excluded_assignments_log_counts_by_dimensions.SUPERVISOR Flink Monitoring: Job Metrics: https://grafana.yelpcorp.com/d/flink-metrics/flink-job-metrics?orgId=1&var-datasource=Prometheus-flink&var-region=uswest2-devc&var-service=sqlclient&var-instance=excluded_assignments_log_counts_by_dimensions&var-job=All&from=now-24h&to=now Container Metrics: https://grafana.yelpcorp.com/d/flink-container-metrics/flink-container-metrics?orgId=1&var-datasource=Prometheus-flink&var-region=uswest2-devc&var-service=sqlclient&var-instance=excluded_assignments_log_counts_by_dimensions&from=now-24h&to=now JVM Metrics: https://grafana.yelpcorp.com/d/flink-jvm-metrics/flink-jvm-metrics?orgId=1&var-datasource=Prometheus-flink&var-region=uswest2-devc&var-service=sqlclient&var-instance=excluded_assignments_log_counts_by_dimensions&from=now-24h&to=now State: Running Pods: 7 running, 0 evicted, 0 other Jobs: 0 running, 0 finished, 8 failed, 0 cancelled 4 taskmanagers, 4/4 slots available ```

https://jira.yelpcorp.com/browse/FLINK-5725 HACKATHON-39

https://jira.yelpcorp.com/browse/FLINK-5725 HACKATHON-39 To speed up gathering of details for a Flink cluster/job When running paasta status, you are normally debugging something (something is broke) You then may need to check related resources for the flink job such as yelpsoa/srv Also when in a slack thread this information can useful to share in thread, reduce time to spot the issue Paasta status output ``` paasta status -s sqlclient -i ad_indexing_service_area_place_ids -c pnw-prod -vvv sqlclient.ad_indexing_service_area_place_ids in pnw-prod (EKS) Version: c74dd64a (desired) Config SHA: 3ac51a85 Flink Pool: flink Owner: ranking_ingestion Repo(git): https://github.yelpcorp.com/services/sqlclient Repo(sourcegraph): https://sourcegraph.yelpcorp.com/services/sqlclient Flink version: 1.17.2 c0027e5 @ 2023-11-09T13:24:38+01:00 URL: https://flink-eks-pnw-prod.yelpcorp.com/sqlclient-c6b979558/ Yelpsoa configs: https://github.yelpcorp.com/sysgit/yelpsoa-configs/tree/master/sqlclient Srv configs: https://github.yelpcorp.com/sysgit/srv-configs/tree/master/ecosystem/prod/sqlclient Flink Log Commands: Service: paasta logs -a 1h -c pnw-prod -s sqlclient -i ad_indexing_service_area_place_ids Taskmanager: paasta logs -a 1h -c pnw-prod -s sqlclient -i ad_indexing_service_area_place_ids.TASKMANAGER Jobmanager: paasta logs -a 1h -c pnw-prod -s sqlclient -i ad_indexing_service_area_place_ids.JOBMANAGER Supervisor: paasta logs -a 1h -c pnw-prod -s sqlclient -i ad_indexing_service_area_place_ids.SUPERVISOR Flink Monitoring: Job Metrics: https://grafana.yelpcorp.com/d/flink-metrics/flink-job-metrics?orgId=1&var-datasource=Prometheus-flink&var-region=uswest2-prod&var-service=sqlclient&var-instance=ad_indexing_service_area_place_ids&var-job=All&from=now-24h&to=now Container Metrics: https://grafana.yelpcorp.com/d/flink-container-metrics/flink-container-metrics?orgId=1&var-datasource=Prometheus-flink&var-region=uswest2-prod&var-service=sqlclient&var-instance=ad_indexing_service_area_place_ids&from=now-24h&to=now JVM Metrics: https://grafana.yelpcorp.com/d/flink-jvm-metrics/flink-jvm-metrics?orgId=1&var-datasource=Prometheus-flink&var-region=uswest2-prod&var-service=sqlclient&var-instance=ad_indexing_service_area_place_ids&from=now-24h&to=now State: Running Pods: 12 running, 0 evicted, 0 other Jobs: 1 running, 0 finished, 0 failed, 1 cancelled ```

https://jira.yelpcorp.com/browse/FLINK-5725 HACKATHON-39 To speed up gathering of details for a Flink cluster/job When running paasta status, you are normally debugging something (something is broke) You then may need to check related resources for the flink job such as yelpsoa/srv Also when in a slack thread this information can useful to share in thread, reduce time to spot the issue Paasta status output ``` paasta status -s sqlclient -i ad_indexing_service_area_place_ids -c pnw-prod -vvv sqlclient.ad_indexing_service_area_place_ids in pnw-prod (EKS) Version: c74dd64a (desired) Config SHA: 3ac51a85 Flink Pool: flink Owner: ranking_ingestion Flink Runbook: y/rb-ring-sqlclient Repo(git): https://github.yelpcorp.com/services/sqlclient Repo(sourcegraph): https://sourcegraph.yelpcorp.com/services/sqlclient Flink version: 1.17.2 c0027e5 @ 2023-11-09T13:24:38+01:00 URL: https://flink-eks-pnw-prod.yelpcorp.com/sqlclient-c6b979558/ Yelpsoa configs: https://github.yelpcorp.com/sysgit/yelpsoa-configs/tree/master/sqlclient Srv configs: https://github.yelpcorp.com/sysgit/srv-configs/tree/master/ecosystem/prod/sqlclient Flink Log Commands: Service: paasta logs -a 1h -c pnw-prod -s sqlclient -i ad_indexing_service_area_place_ids Taskmanager: paasta logs -a 1h -c pnw-prod -s sqlclient -i ad_indexing_service_area_place_ids.TASKMANAGER Jobmanager: paasta logs -a 1h -c pnw-prod -s sqlclient -i ad_indexing_service_area_place_ids.JOBMANAGER Supervisor: paasta logs -a 1h -c pnw-prod -s sqlclient -i ad_indexing_service_area_place_ids.SUPERVISOR Flink Monitoring: Job Metrics: https://grafana.yelpcorp.com/d/flink-metrics/flink-job-metrics?orgId=1&var-datasource=Prometheus-flink&var-region=uswest2-prod&var-service=sqlclient&var-instance=ad_indexing_service_area_place_ids&var-job=All&from=now-24h&to=now Container Metrics: https://grafana.yelpcorp.com/d/flink-container-metrics/flink-container-metrics?orgId=1&var-datasource=Prometheus-flink&var-region=uswest2-prod&var-service=sqlclient&var-instance=ad_indexing_service_area_place_ids&from=now-24h&to=now JVM Metrics: https://grafana.yelpcorp.com/d/flink-jvm-metrics/flink-jvm-metrics?orgId=1&var-datasource=Prometheus-flink&var-region=uswest2-prod&var-service=sqlclient&var-instance=ad_indexing_service_area_place_ids&from=now-24h&to=now State: Running Pods: 12 running, 0 evicted, 0 other Jobs: 1 running, 0 finished, 0 failed, 1 cancelled ```

https://jira.yelpcorp.com/browse/FLINK-5725 HACKATHON-39 To speed up gathering of details for a Flink cluster/job When running paasta status, you are normally debugging something (something is broke) You then may need to check related resources for the flink job such as yelpsoa/srv Also when in a slack thread this information can useful to share in thread, reduce time to spot the issue ========== Links to the paasta service utilisation dashboard Adds diveder to paasta status for easier readability Renames of function Paasta status output ``` paasta status -s acorn -i main -c pnw-devc -v acorn.main in pnw-devc (EKS) Version: 6563d637 (desired) Config SHA: 00ce0453 Flink Pool: flink-spot Owner: streaming-infrastructure Flink Runbook: y/rb-acorn Repo(git): https://github.yelpcorp.com/services/acorn Repo(sourcegraph): https://sourcegraph.yelpcorp.com/services/acorn Flink version: 1.13.5 0ff28a7 @ 2021-12-14T23:26:04+01:00 URL: http://flink.eks.pnw-devc.paasta:31080/acorn-7f797b79f6/ Yelpsoa configs: https://github.yelpcorp.com/sysgit/yelpsoa-configs/tree/master/acorn Srv configs: https://github.yelpcorp.com/sysgit/srv-configs/tree/master/ecosystem/devc/acorn ================================================================== Flink Log Commands: Service: paasta logs -a 1h -c pnw-devc -s acorn -i main Taskmanager: paasta logs -a 1h -c pnw-devc -s acorn -i main.TASKMANAGER Jobmanager: paasta logs -a 1h -c pnw-devc -s acorn -i main.JOBMANAGER Supervisor: paasta logs -a 1h -c pnw-devc -s acorn -i main.SUPERVISOR ================================================================== Flink Monitoring: Job Metrics: https://grafana.yelpcorp.com/d/flink-metrics/flink-job-metrics?orgId=1&var-datasource=Prometheus-flink&var-region=uswest2-devc&var-service=acorn&var-instance=main&var-job=All&from=now-24h&to=now Container Metrics: https://grafana.yelpcorp.com/d/flink-container-metrics/flink-container-metrics?orgId=1&var-datasource=Prometheus-flink&var-region=uswest2-devc&var-service=acorn&var-instance=main&from=now-24h&to=now JVM Metrics: https://grafana.yelpcorp.com/d/flink-jvm-metrics/flink-jvm-metrics?orgId=1&var-datasource=Prometheus-flink&var-region=uswest2-devc&var-service=acorn&var-instance=main&from=now-24h&to=now Flink Cost: https://splunk.yelpcorp.com/en-US/app/yelp_computeinfra/paasta_service_utilization?form.service=acorn&form.field1.earliest=-30d%40d&form.field1.latest=now&form.instance=main&form.cluster=pnw-devc ================================================================== State: Running Pods: 5 running, 0 evicted, 0 other Jobs: 1 running, 0 finished, 0 failed, 0 cancelled 3 taskmanagers, 8/30 slots available Jobs: Job Name State Started heartbeat_s3_checkpoint Running 2025-05-15 15:42:57 (11 hours ago) Pods: Pod Name Host Phase Uptime acorn-7f797b79f6-jobmanager-5859456947-qv2ml ip-10-81-17-237.us-west-2.compute.internal Running 0d11h3m9s acorn-7f797b79f6-supervisor-p9l6n ip-10-81-19-13.us-west-2.compute.internal Running 0d17h11m23s acorn-7f797b79f6-taskmanager-5bf76bf45f-2js7q ip-10-81-23-26.us-west-2.compute.internal Running 0d2h54m15s acorn-7f797b79f6-taskmanager-5bf76bf45f-76x24 ip-10-81-23-26.us-west-2.compute.internal Running 0d2h54m15s acorn-7f797b79f6-taskmanager-5bf76bf45f-rkh9w ip-10-81-19-237.us-west-2.compute.internal Running 0d7h42m18s ```

https://jira.yelpcorp.com/browse/FLINK-5725 HACKATHON-39 To speed up gathering of details for a Flink cluster/job When running paasta status, you are normally debugging something (something is broke) You then may need to check related resources for the flink job such as yelpsoa/srv Also when in a slack thread this information can useful to share in thread, reduce time to spot the issue ========== paasta_tools/flink_tools.py:114: error: Function is missing a return type annotation paasta_tools/flink_tools.py:125: error: TypedDict "FlinkDeploymentConfigDict" has no key 'spot'

Copilot

Pull Request Overview

This PR enhances paasta status output for Flink clusters by adding extra resource links, log commands, and owner/runbook details. Key changes include:

Updating tests to cover new Flink status details and configuration behaviors.
Adding new utility methods in both paasta_tools/utils.py and paasta_tools/flink_tools.py.
Updating the CLI status commands to include extra Flink resource links and formatted divider outputs.

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/test_utils.py	Added tests for retrieving ecosystem and runbook details.
tests/test_flink_tools.py	Added tests for new Flink pool determination logic.
tests/cli/test_cmds_status.py	Updated tests to reflect new output format and resource links.
requirements-minimal.txt	Added dependency on the environment-tools package.
paasta_tools/utils.py	Added get_runbook and get_ecosystem_for_cluster methods.
paasta_tools/flink_tools.py	Added a new get_pool method that selects between two Flink pools.
paasta_tools/cli/cmds/status.py	Updated Flink status output logic to include extra resource links.

Comments suppressed due to low confidence (1)

tests/cli/test_cmds_status.py:2470

[nitpick] There is an inconsistency in the tests between using 'fake_cluster' and 'fake-cluster' for the cluster name. Consider standardizing the naming for clarity and consistency.

cluster="fake_cluster",

paasta_tools/flink_tools.py

paasta_tools/utils.py

Co-authored-by: Copilot <[email protected]>

https://jira.yelpcorp.com/browse/FLINK-5725 HACKATHON-39 To speed up gathering of details for a Flink cluster/job When running paasta status, you are normally debugging something (something is broke) You then may need to check related resources for the flink job such as yelpsoa/srv Also when in a slack thread this information can useful to share in thread, reduce time to spot the issue ==========

…DetailsToFlinkPaastaStatus' into u/nathanleigh/FLINK-5725/AddMoreDetailsToFlinkPaastaStatus

paasta_tools/utils.py

nemacysts · 2025-06-05T17:53:48Z

tests/test_utils.py

@@ -591,6 +592,36 @@ def test_SystemPaastaConfig_get_cluster_fqdn_format():
    assert actual == expected


+@patch("paasta_tools.utils.convert_location_type", autospec=True)


for newer tests, we tend to prefer using the context manager form of patching rather than the decorator form

Thanks, updated in e0388bb

nemacysts · 2025-06-05T17:56:35Z

tests/test_utils.py

@@ -21,6 +21,7 @@
 from typing import Any
 from typing import Dict
 from typing import List
+from unittest.mock import patch


oh wow, we're behind on modernizing things - we're still using mock in this file/repo!

that said: can we do the "wrong" thing here and keep using mock so that we can swap this file to unittest.mock in one fell swoop later?

Sounds good, yes happy to modernise in another PR

nemacysts · 2025-06-05T17:58:29Z

tests/test_utils.py

+@patch("paasta_tools.utils.convert_location_type", autospec=True)
+def test_SystemPaastaConfig_get_ecosystem_for_cluster(mock_convert_location_type):
+    # Mock convert_location_type to return the expected ecosystem
+    mock_convert_location_type.return_value = ["devc"]


iirc, you can also set return_value in the patch() call - but this is also fine as-is

paasta_tools/cli/cmds/status.py

Co-authored-by: Luis Pérez <[email protected]>

@patch

https://jira.yelpcorp.com/browse/FLINK-5725 HACKATHON-39 To speed up gathering of details for a Flink cluster/job When running paasta status, you are normally debugging something (something is broke) You then may need to check related resources for the flink job such as yelpsoa/srv Also when in a slack thread this information can useful to share in thread, reduce time to spot the issue ========== Address review comment @patch("paasta_tools.utils.convert_location_type", autospec=True) for newer tests, we tend to prefer using the context manager form of patching rather than the decorator form

https://jira.yelpcorp.com/browse/FLINK-5725 HACKATHON-39 To speed up gathering of details for a Flink cluster/job When running paasta status, you are normally debugging something (something is broke) You then may need to check related resources for the flink job such as yelpsoa/srv Also when in a slack thread this information can useful to share in thread, reduce time to spot the issue ========== Address review comment ``` FAILED tests/cli/test_cmds_status.py::TestPrintFlinkStatus::test_error_no_flink_config - AttributeError: 'NoneType' object has no attribute 'get_pool' FAILED tests/cli/test_cmds_status.py::TestPrintFlinkStatus::test_error_no_flink_overview - AttributeError: 'NoneType' object has no attribute 'get_pool' FAILED tests/cli/test_cmds_status.py::TestPrintFlinkStatus::test_successful_return_value - AttributeError: 'NoneType' object has no attribute 'get_pool' ```

tests/test_utils.py

paasta_tools/cli/cmds/status.py

…linkPaastaStatus

Co-authored-by: Luis Pérez <[email protected]>

https://jira.yelpcorp.com/browse/FLINK-5725 HACKATHON-39 To speed up gathering of details for a Flink cluster/job When running paasta status, you are normally debugging something (something is broke) You then may need to check related resources for the flink job such as yelpsoa/srv Also when in a slack thread this information can useful to share in thread, reduce time to spot the issue ========== Address review comment

https://jira.yelpcorp.com/browse/FLINK-5725 HACKATHON-39 To speed up gathering of details for a Flink cluster/job When running paasta status, you are normally debugging something (something is broke) You then may need to check related resources for the flink job such as yelpsoa/srv Also when in a slack thread this information can useful to share in thread, reduce time to spot the issue ========== Address review comment flink_instance_config Should always be populated

https://jira.yelpcorp.com/browse/FLINK-5725 HACKATHON-39 To speed up gathering of details for a Flink cluster/job When running paasta status, you are normally debugging something (something is broke) You then may need to check related resources for the flink job such as yelpsoa/srv Also when in a slack thread this information can useful to share in thread, reduce time to spot the issue ========== Test failing with ``` > flink_pool = flink_instance_config.get_pool() E AttributeError: 'NoneType' object has no attribute 'get_pool' ``` Fix by populating object Remove duplication by moving object flink_instance_config to test ficture

tests/test_utils.py

nleigh requested a review from a team as a code owner May 13, 2025 15:16

nleigh requested review from marcos-sb and Copilot May 13, 2025 15:57

Copilot AI reviewed May 13, 2025

View reviewed changes

nleigh requested review from AshishKhatkar, omwbennett, anohovsky, abareghi and gkousouris May 13, 2025 15:59

nleigh added 3 commits May 13, 2025 10:09

anohovsky reviewed May 14, 2025

View reviewed changes

anohovsky previously approved these changes May 14, 2025

View reviewed changes

nleigh requested a review from ilkinmammadzada May 14, 2025 12:06

nemacysts reviewed May 14, 2025

View reviewed changes

nleigh dismissed anohovsky’s stale review via 56cdcc5 May 14, 2025 14:12

FLINK-5725: Update souregraph to github link

02c4363

https://jira.yelpcorp.com/browse/FLINK-5725 HACKATHON-39

nleigh requested a review from Copilot May 14, 2025 14:37

Copilot AI reviewed May 14, 2025

View reviewed changes

nemacysts reviewed May 14, 2025

View reviewed changes

nleigh added 6 commits May 14, 2025 14:16

FLINK-5725: Refactors

ba32de6

https://jira.yelpcorp.com/browse/FLINK-5725 HACKATHON-39

nleigh requested a review from Copilot June 5, 2025 16:53

Copilot AI reviewed Jun 5, 2025

View reviewed changes

paasta_tools/flink_tools.py Outdated Show resolved Hide resolved

paasta_tools/utils.py Outdated Show resolved Hide resolved

nleigh and others added 2 commits June 5, 2025 17:55

Update paasta_tools/flink_tools.py

994e4d6

Co-authored-by: Copilot <[email protected]>

Update paasta_tools/utils.py

33b03fa

Co-authored-by: Copilot <[email protected]>

nleigh requested a review from nemacysts June 5, 2025 17:32

nleigh added 2 commits June 5, 2025 10:38

Merge remote-tracking branch 'origin/u/nathanleigh/FLINK-5725/AddMore…

ffab533

…DetailsToFlinkPaastaStatus' into u/nathanleigh/FLINK-5725/AddMoreDetailsToFlinkPaastaStatus

nemacysts reviewed Jun 5, 2025

View reviewed changes

nleigh and others added 7 commits June 6, 2025 11:24

Update paasta_tools/utils.py

42fc583

Co-authored-by: Luis Pérez <[email protected]>

Update paasta_tools/utils.py

571a4cd

Co-authored-by: Luis Pérez <[email protected]>

Update paasta_tools/utils.py

377584d

Co-authored-by: Luis Pérez <[email protected]>

Update paasta_tools/cli/cmds/status.py

22d2921

Co-authored-by: Luis Pérez <[email protected]>

Update paasta_tools/utils.py

dbc5e66

Co-authored-by: Luis Pérez <[email protected]>

nleigh requested a review from nemacysts June 6, 2025 11:26

nemacysts reviewed Jun 6, 2025

View reviewed changes

tests/test_utils.py Outdated Show resolved Hide resolved

tests/test_utils.py Outdated Show resolved Hide resolved

paasta_tools/cli/cmds/status.py Outdated Show resolved Hide resolved

paasta_tools/cli/cmds/status.py Outdated Show resolved Hide resolved

nleigh and others added 8 commits June 10, 2025 15:28

Merge branch 'master' into u/nathanleigh/FLINK-5725/AddMoreDetailsToF…

7138c66

…linkPaastaStatus

Update paasta_tools/cli/cmds/status.py

399c0ad

Co-authored-by: Luis Pérez <[email protected]>

Update tests/test_utils.py

b71546c

Co-authored-by: Luis Pérez <[email protected]>

Update tests/test_utils.py

d5e6765

Co-authored-by: Luis Pérez <[email protected]>

Update paasta_tools/cli/cmds/status.py

5bf589d

Co-authored-by: Luis Pérez <[email protected]>

nemacysts reviewed Jun 10, 2025

View reviewed changes

tests/test_utils.py Outdated Show resolved Hide resolved

Update tests/test_utils.py

55b4371

nemacysts approved these changes Jun 11, 2025

View reviewed changes

nleigh merged commit 61a3ff7 into master Jun 11, 2025
10 checks passed

	cluster_without_pnw_dash = cluster.replace("pnw-", "")
	cluster_without_pnw_dash = cluster.removeprefix("pnw-")

		@@ -591,6 +592,36 @@ def test_SystemPaastaConfig_get_cluster_fqdn_format():
		assert actual == expected


		@patch("paasta_tools.utils.convert_location_type", autospec=True)

FLINK-5725: Add extra Flink details to paasta status #4063

FLINK-5725: Add extra Flink details to paasta status #4063

Uh oh!

Conversation

nleigh commented May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nleigh commented May 13, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI May 13, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

nleigh commented May 13, 2025 •

edited

Loading