Unable to view GatewayCluster dashboard with dask-labextension, lots of network errors

[scottyhq]

I haven't used the dask labextension for a bit, and seem to have forgotten which URL to paste into the labextension URL box to get the widgets to work. @TomAugspurger can you advise here?

Note that the Dashboard link does take me to the standard web UI, but pasting these in the labextension window everything remains greyed out (/services/dask-gateway/clusters/icesat2-staging.3cc976d1113848f0ac2c8845b1da152b/status --> https://aws-uswest2.pangeo.io/services/dask-gateway/clusters/icesat2-prod.a3413571b0054e2592d0ef35f2684b3d/status).

In the gallery example, there is no need to manually past the URL, just clicking the search glass in labextension seems to automatically find the active cluster, so I'm guessing we're missing some important path configuration on the AWS hub...
https://gallery.pangeo.io/repos/TomAugspurger/pangeo-dask-gateway/using-dask-gateway.html

[TomAugspurger]

Hmm, so checking the console log in the browser shows errors like

XHRGET https://aws-uswest2.pangeo.io/user/tomaugspurger/services/dask-gateway/clusters/icesat2-prod.903c313ed02a412b9d9a5e87e748c764/individual-plots.json?1602675145245
[HTTP/2 404 Not Found 90ms]

That URL is incorrect. It's roughly <domain>/user/<JUPYTERHUB_USER>/services/dask-gateway/clusters/.... We don't want the /user/<JUPYTERHUB_USER> bit. @ian-r-rose do you know, is this the same as dask/dask-labextension#96? Is there any config we can set that controls where the labextension looks for individual-plots.json?

[scottyhq]

Also, @TomAugspurger just to clarify, same error seems to occur on GCP, but not your pangeo gallery example, so perhaps this is an issue coming up with more recent dask or jupyter library versions. Previously the links looked like this:

In the javascript console I see a few other errors when starting up, not sure if it is relevant...

Error: widget model not found
    at C.get_model (vendors~main.0696ed76febe5543ce98.js:2)
    at p.renderModel (vendors~main.0696ed76febe5543ce98.js:2)

And this error just keeps repeating:

GET https://us-central1-b.gcp.pangeo.io/user/scottyhq/services/dask-gateway/clusters/prod.7f60c199408d498283d72934034c527e/individual-plots.json?1602690823911 404

[scottyhq]

I'm also noticing the jupyterlab interface has a lot of lag, and even without connecting to a dask cluster, i'm seeing lots of repeated 404 errors in the javascript console (namely trying to get dask-logo.svg). Pinging @jcrist in case he has any insight ;)

[Error] Failed to load resource: A server with the specified hostname could not be found. (dask-logo.svg, line 0) https://hub.aws-uswest2.pangeo.io/services/dask-gateway/clusters/icesat2-prod.a3413571b0054e2592d0ef35f2684b3d/statics/images/dask-logo.svg?1602712769183

Also, for what it's worth it seems that dynamic=True for hvplot.image() is no longer updating, so there is something generally up with jupyterlab and widgets, the one other thing i notice in my jupyter pod log is [IPKernelApp] ERROR | No such comm target registered: hv-extension-comm... But really I'm just feeling around in the dark here, sharing things that seem out of sorts!

[scottyhq]

Ok, I think a lot of these network errors happening behind the scenes are maybe red herrings and fairly normal. I realized the main problem was I had some old dask config in /home/jovyan/.config/dask that was conflicting with our standard pangeo docker image config in /srv/conda/etc/dask.yml

Once I figured that out I can successfully launch a gateway cluster and then connect to it via the extension... but! I have to put in the entire url (https://aws-uswest2.pangeo.io/services/dask-gateway/clusters/icesat2-prod.3efefa2916d041728ffae1591d540734/status) and hit 'Enter', the search button doesn't work.