Creating pangeo-eosc infrastructure based on elastic Kubernetes Virtual Cluster using IM-Dashboard

[tinaok]

This is an issue so that we can coordinate for creating pangeo-eosc infrastructure based on elastic Kubernetes Virtual Cluster using IM-Dashboard.

[tinaok]

Here is what I have in mind as to do list. Any thoughts @guillaumeeb & @j34ni ?

1. • We need resource to test the elastic options. I.e. need to shutdown some instances to free VCPUs for start testing.
2. • Feed back from former tests during August?
3. • Find a way to create the infrastructure using 'elastic' option on IM-Dashboard
4. • Creation (how do we set the limit : minimum and max? can we change the max time to time for hosting tutorials? for example?)
5. • Benchmarks

[guillaumeeb]

👍 for me! I'll let @j34ni answer the first two questions. For point 4, not sure we really need a maximum 🙂.

[j34ni]

@tinaok :
1- Feel free to remove all the nodes you want from the existing infrastructures (I did not manage to login into the IM dashboard lately)
2- The two of them worked great. The one without EGI check-in saved the day at the workshop since a lot of the participants had not enrolled. I have also used them after and they are quite "responsive", more than 8GB RAM would be nice for some applications though

[guillaumeeb]

@j34ni @tinaok, I think at one point the Elastic Kubernetes offer from IM Dashboard was tried? Was there any reason why it was not kept?

[j34ni]

@guillaumeeb I do not remember that, may be was it at the same time as other things which failed and we went back a few steps to get something working?

[guillaumeeb]

Yeah probably. I guess we just need to make some room on our VMs and try to redeploy an elastic version on Kubernetes to host our Pangeo platform.

[sebastian-luna-valero]

Ok, following up from #21

I think we never tried the elastic option before since we wanted to focus on other higher priority issues.

Have you tried the elastic option now? If so, could you please provide feedback?

Happy to help with this.

[guillaumeeb]

Have you tried the elastic option now? If so, could you please provide feedback?

Not yet for my part. I can put this on my todo-list for the next days/weeks if @j34ni or @tinaok hasn't time to do so.

[tinaok]

Hi @guillaumeeb thank you very much, I had no time (and will have no time at all this week neither) for trying out unfortunately. Your help will be super appreciated.
I think it can be named as 'pangeo-eosc'

[guillaumeeb]

In the process of creating a Pangeo deployment with elastic Kubernetes on IM Dashboard, following https://github.com/pangeo-data/pangeo-eosc/blob/main/EGI.md, I've got some questions/remarks (noting there as much as for other people as for myself):

• https://github.com/pangeo-data/pangeo-eosc/blob/main/EGI.md#step-1-dns-name says to create a new domain, but what we really need is a DNS name right, so a new host.
• I selected Kubernetes, then added Elastic option and Grafana. No other options. I'm planning to do all the Daskhub thing via Helm only.
• On IM Dashboard K8S cluster creation, HW Data tab, there are 1 front node, and several workers. Is the front node only used for K8S master component, or will it also host some user pods? This is important to know if a small VM is enough, or if we should use the same flavor as for workers.
• Which Kubernetes version should we used? I kept the 1.23 default (not sure if Dask gateway or Jupyterhub is up to date).
• Cloud Provider/ Select Site image: I chose Ubuntu Jammy. Is that OK?
• For Elastic K8S, we can only chose the maximum of Worker nodes (not the minimum, but this might be the number of workers entered on K8S tab).
• EGI.md lacks information on how to connect to the K8S front VM to issue commands (getting the pem file, connecting with cloudadm).
• I don't know how to retrieve EGI Check-in secrets other than by searching in previous deployment. Is there a way through https://aai.egi.eu/federation?
• My K8S deployment is marked in status "running", not "configured", but it looks like the deployment is terminated? Oh no, it took a long time to end up in "configured" state. I'm not sure all is well though, the prompt on the front node is not the same as on other deployments.

So after step 2 from EGI.md, I created my daskhub.yaml file as below:

dask-gateway:
  enabled: true
  gateway:
    auth:
      jupyterhub:
        apiToken: <token1>
      type: jupyterhub
    extraConfig:
      dasklimits: |
        c.ClusterConfig.cluster_max_cores = 6
        c.ClusterConfig.cluster_max_memory = "24 G"
        c.ClusterConfig.cluster_max_workers = 4
        c.ClusterConfig.idle_timeout = 1800
      optionHandler: |
        from dask_gateway_server.options import Options, Integer, Float, String

        def options_handler(options):
          if ":" not in options.image:
            raise ValueError("When specifying an image you must also provide a tag")
          return {
            "worker_cores": options.worker_cores,
            "worker_memory": int(options.worker_memory * 2 ** 30),
            "image": options.image,
          }

        c.Backend.cluster_options = Options(
          Integer("worker_cores", default=1, min=1, max=4, label="Worker Cores"),
          Float("worker_memory", default=2, min=2, max=8, label="Worker Memory (GiB)"),
          String("image", default="pangeo/ml-notebook:2022.09.21", label="Image"),
          handler=options_handler,
        )
    backend:
      worker:
        cores:
          limit: 4
        memory:
          limit: 8G
        threads: 2
dask-kubernetes:
  enabled: false
jupyterhub:
  hub:
    config:
      GenericOAuthenticator:
        client_id: <client>
        client_secret: <secret>
        oauth_callback_url: https://pangeo-elastic.vm.fedcloud.eu/hub/oauth_callback
        authorize_url: https://aai-dev.egi.eu/auth/realms/egi/protocol/openid-connect/auth
        token_url: https://aai-dev.egi.eu/auth/realms/egi/protocol/openid-connect/token
        userdata_url: https://aai-dev.egi.eu/auth/realms/egi/protocol/openid-connect/userinfo
        login_service: EGI Check-In
        scope:
          - openid
          - email
          - profile
          - eduperson_entitlement
        username_key: preferred_username
        userdata_params:
          state: state
        allowed_groups:
          - urn:mace:egi.eu:group:vo.pangeo.eu:role=member#aai.egi.eu
        claim_groups_key: eduperson_entitlement
      JupyterHub:
        authenticator_class: generic-oauth
    services:
      dask-gateway:
        apiToken: <token1>
  proxy:
    secretToken: <token2>
    service:
      type: ClusterIP
  singleuser:
    cpu:
      guarantee: 1
      limit: 2
    defaultUrl: /lab
    extraEnv:
      DASK_GATEWAY__CLUSTER__OPTIONS__IMAGE: '{JUPYTER_IMAGE_SPEC}'
    image:
      name: pangeo/ml-notebook
      tag: 2022.09.21
    memory:
      guarantee: 2G
      limit: 4G
    startTimeout: 600
    storage:
      capacity: 2Gi
      type: dynamic
rbac:
  enabled: true

and just issued the helm command:

sudo helm upgrade daskhub daskhub --repo=https://helm.dask.org --install --wait --cleanup-on-fail --create-namespace --namespace daskhub --version 2022.8.2 --values daskhub.yaml

Followed by reconfiguring ingress.

The helm command and kubectl for ingress worked with no error.
I can see Jupyterhub, but I get a login error:

Same as the other day when trying to access IM Dashboard.

I'll stop there for tonight. If someone can test the deployment at https://pangeo-foss4g.vm.fedcloud.eu and see if they can login?

---

Edit: correct link is https://pangeo-elastic.vm.fedcloud.eu/

[tinaok]

Thank you @guillaumeeb !!
I just tried to log on to https://pangeo-foss4g.vm.fedcloud.eu/jupyterhub/, but may be this one is the foss4g configuration (=old) one ( I have all my historical left to old one there, so I guess this is not the new one you are creating based on the elastic Kubernetes? )

https://github.com/pangeo-data/pangeo-eosc/blob/main/EGI.md#step-1-dns-name says to create a new domain, but what we really need is a DNS name right, so a new host.

In the example it was pangeo.vm.fedcloud.eu but I guess we can have it like pangeo-eosc.vm.fedcloud.eu ?

I selected Kubernetes, then added Elastic option and Grafana. No other options. I'm planning to do all the Daskhub thing via Helm only.

👍

I checked IP address of the cluster you made from swift dashboard and I could have the grafana login portal

EGI.md lacks information on how to connect to the K8S front VM to issue commands (getting the pem file, connecting with cloudadm).

If I remember right, the button in the IM dashboard indicating cluster configuration "configured" , could be clicked and there was information about cloudadm

[guillaumeeb]

Crap, I didn't indicate the right link.

The correct one is https://pangeo-elastic.vm.fedcloud.eu. This is only a test deployment for now. There is no need to add /jupyterhub/. Jupyterhub is available at /.

[tinaok]

Thanks for the new link!
I confirm I get error too

[guillaumeeb]

@sebastian-luna-valero do you have any idea why the auth might failed?

I'll have another look at it tonight.

[sebastian-luna-valero]

Hi,

Could you please double check that the Redirect UR in https://aai-dev.egi.eu/federation has the same value as oauth_callback_url in the values.yaml?