Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Accessing stdout from Dask worker and scheduler pods #768

Open
TomAugspurger opened this issue Oct 1, 2020 · 8 comments
Open

Accessing stdout from Dask worker and scheduler pods #768

TomAugspurger opened this issue Oct 1, 2020 · 8 comments

Comments

@TomAugspurger
Copy link
Member

The problem: users can't see text printed to stdout on remote pods (primarily Dask's scheduler and workers). This makes debugging certain problems difficult or impossible if you can't track down someone with access to the logs (i.e. somewhen with access to the kubernetes API).

There are two primary sources of things being printed to stdout

  1. Python logging messages (primarily dask & distributed)
  2. stdout / stderr (directly via print)

I'd like to make it easy to get both of these. Problem 1 can be somewhat solved through conventional tools. We'd configure Dask's loggers to output somewhere (probably in-memory, maybe over the network to the scheduler?). Then we'd provide some method for collecting those to the client (either using client.get_worker_logs() or by client.run(custom_function) to collect the log output..

The second is harder. If a library is printing directly to stdout (say it's some C or fortran library that has no concept of Python's logging), it's essentially too late for Dask to catch it. Kubernetes, however, does catch it, and displays it in the pod's logs. With the default configuration, Dask's logs with a level of logging.WARNING or higher also appear there.

Given that solving the second problem also solves the first, let's focus on that. My proposal has two components:

  1. Use kubernetes RBAC authorization to grant read permission (get and list) to user's jupyter pods.
  2. Provide something (documentation or a small python package) to easily get the logs from all the pods associated with a Dask cluster. This will be a sequence of kubernetes API calls, and possibly some helpers to filter / display them nicely.

In the past we had concerns about exposing the kubernetes API to user pods. My hope is that we can be limited in what permissions we grant users. Ideally users would only be able to see (and not modify) objects related to their own pods. I suspect that isolating users from each other won't be feasible, and everyone will be able to read everyone else's pods. Is that a blocker for anyone?

@TomAugspurger
Copy link
Member Author

A small proof of concept on the GCP cluster.

These permissions are overly broad, since they grant read access to all the pods in the namespace. Ideally it would be filtered to just pods of a certain type / label (hopefully we can do this, I haven't checked).

---
# rbac-reader-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: prod
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "watch", "list"]

---
# rbac-reader-rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: prod
subjects:
- kind: ServiceAccount
  name: pangeo
  namespace: prod
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

With those permissions

import kubernetes.config
import kubernetes.client
from dask_gateway import GatewayCluster
kubernetes.config.load_incluster_config()

cluster = GatewayCluster()
client = cluster.get_client()
cluster.scale(1)

v1 = kubernetes.client.CoreV1Api()

ret = v1.list_namespaced_pod("prod", watch=False)
ret = v1.list_namespaced_pod("prod", watch=False)
pods = ret.items

import os


def filter_pods(cluster_name, pods):
    return [
        pod for pod in pods if
        pod.metadata.labels.get("app.kubernetes.io/name") == "dask-gateway" and
        pod.metadata.labels.get("hub.jupyter.org/username") == os.environ["JUPYTERHUB_USER"]
        and cluster_name.split(".")[-1] in set(pod.metadata.name.split("-"))
    ]
    

mypods = filter_pods(cluster.name, pods)

>>> logs = [v1.read_namespaced_pod_log(pod.metadata.name, "prod") for pod in mypods]
>>> logs
['dask_gateway.dask_cli - INFO - Requesting scale to 1 workers from 0\n', '']

I imagine things like fetching the pods and filtering them would be packaged into a small pangeo_cloud package, which would be included in the pangeo-notebook docker image. Then if people run into issues, we can instruct them to

import pangeo_cloud

pangeo_cloud.get_logs(cluster.name)

@TomAugspurger
Copy link
Member Author

These permissions are overly broad, since they grant read access to all the pods in the namespace. Ideally it would be filtered to just pods of a certain type / label (hopefully we can do this, I haven't checked).

Apparently, this is difficult. You can filter on resourceNames but our resource names are dynamic like dask-scheduler-6fe382b5cb7a4cb4980c095fbcf742a1. Support for this usecase has been proposed (selector for resourceNames: kubernetes/kubernetes#56582, based on labels: kubernetes/kubernetes#44703 (comment)) but not implemented.

So if I'm correct, then supporting this means we need to be comfortable with everybody reading everyone else's logs, including the logs from our pods (the hub, traefik, dask-gateway, etc.).

@scottyhq
Copy link
Member

scottyhq commented Oct 1, 2020

Thanks for looking into this Tom!

Ideally users would only be able to see (and not modify) objects related to their own pods. I suspect that isolating users from each other won't be feasible, and everyone will be able to read everyone else's pods. Is that a blocker for anyone?

Yes, this is partially what motivated the move to dask-gateway. Personally I hope there is an alternative, but don't mind if you go ahead with this.

Perhaps the long term solution, which might also address #693, is to give each user their own kubernetes namespace
(https://github.com/jupyterhub/kubespawner/pull/387/files).

One last idea, maybe instead of pangeo_cloud.get_logs(cluster.name) there is just a container running that pipes logs to a persistent storage space. Maybe the $PANGEO_SCRATCH/user/logs bucket? I'd be surprised if some k8s utilities didn't already exist for such a thing... @consideRatio any ideas here?

@jhamman
Copy link
Member

jhamman commented Oct 2, 2020

Thanks @TomAugspurger for organizing thoughts here. Quick question regarding this bit from above?

Then we'd provide some method for collecting those to the client (either using client.get_worker_logs() or by client.run(custom_function) to collect the log output..

Is anything like this possible today or is this a hypothetical api?

@TomAugspurger
Copy link
Member Author

TomAugspurger commented Oct 2, 2020 via email

@consideRatio
Copy link
Member

One last idea, maybe instead of pangeo_cloud.get_logs(cluster.name) there is just a container running that pipes logs to a persistent storage space. Maybe the $PANGEO_SCRATCH/user/logs bucket? I'd be surprised if some k8s utilities didn't already exist for such a thing... @consideRatio any ideas here?

There are plenty of k8s logging scrapers, but the problem becomes that suddenly you need to: a) scrape logs, b) store logs, c) manage access to logs with a separate auth system that still need to relate to the dask's auth system as there should be 1to1 mapping to grant the correct access.

@consideRatio
Copy link
Member

consideRatio commented Oct 2, 2020

I think the way to go is to stay within the code base of Dask in general. I think its too easy to end up developing something that is hard to maintain sustainable otherwise as it may lock in too many parameters (k8s, single namespace deployment, or similarly...) with a too small user base to motivate its maintenance.

Dask's code base can reasonably keep track of what goes on (logs), and when interacting with it you have already an established identity, and when you interact with it you can avoid locking in to something that becomes specific to the deployment method (k8s worker pods, same server workers processes, hpc jobs).

In my, these considerations are similar to a domain I'm more familiar with. Consider a feature request relating to KubeSpawner that creates the pods for JupyterHub where users get their own Jupyter server. One could implement this feature...

  • a) alongside but separate from KubeSpawner's code base
  • b) within KubeSpawner's code base
  • c) within JupyterHub's Spawner base class

Option c) here is the more robust long term solution, and I think for something like collecting logs from a Dask worker, it sounds like we should look for the equivalent of something like option c), or perhaps b), but not a).

@TomAugspurger
Copy link
Member Author

As a small proof of concept, with the following in a config file picked up by dask, all the logging messages (from distributed or other libraries) will be sent to a handler on the root logger that keeps the last distributed.admin.log-length messages in memory.

logging:
  version: 1
  handlers:
    deque_handler:
      class: distributed.utils.DequeHandler
      level: INFO
  loggers:
    "":
      level: INFO
      handlers:
        - deque_handler

That could be loaded when we start a pod (singleuser, scheduler, or worker) by including it in the Docker image. With some code, we can get the logs from the workers

def get_logs():
    import logging
    from distributed.utils import DequeHandler
    
    logger = logging.getLogger("")
    handler, = [handler for handler in logger.handlers if isinstance(handler, DequeHandler)]
    return [(msg.levelname, handler.format(msg)) for msg in handler.deque]

client.run(get_logs)

And from the scheduler with client.run_on_scheduler(get_logs).


But this workflow isn't the nicest. That logging config should be mostly harmless for others, but we aren't the only users of pangeo's docker images. And that get_logs code doesn't really have a home, other than the pangeo cloud docs perhaps.

So now I'm wondering if we'd be better served by a new page in the scheduler dashboard that aggregates all the logs from the cluster (right now the worker logs page just shows the logs from distributed.worker, not other modules). I might poke at that over the next couple days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants