Support for Deepnote

[ochen1]

Is your feature request related to a problem? Please describe.
Deepnote just launched a few days ago, and it looks like they have 24/7 instances. I feel many people would love to be able to SSH & VNC into their 24/7 Deepnote servers.

Describe the solution you'd like
I tried to run remocolab on Deepnote, but nvida-smi not being present caused some errors.

Following the StackOverflow answer here, I was able to at least get the python-apt module working.

Would support for Deepnote be possible?

[demotomohiro]

I tried to run remocolab on deepnote but I cannot login to ssh.

I ran following code on the notebook cell:

!sudo apt update && sudo apt upgrade -q -y
!sudo apt install -q -y -o=Dpkg::Use-Pty=0 python3-apt
!pip3 install git+https://github.com/demotomohiro/remocolab.git@add-deepnote

with open("remocolab_run.py", "w") as f:
  f.write("""
import sys, os, tempfile
sys.path.append("/usr/lib/python3/dist-packages")
sys.path.append("/opt/venv/lib/python3.7/site-packages")

with open("/etc/ssh/sshd_config", "a") as f:
  f.write("\\n\\nListenAddress localhost:58387")

import remocolab
os.chdir(tempfile.mkdtemp(prefix = "remocolab"))
#remocolab.setupSSHD(check_gpu_available = False, tunnel = "argotunnel")
remocolab.setupVNC(check_gpu_available = False, tunnel = "argotunnel")

Then ran following command on deepnote's terminal:

sudo python3 remocolab_run.py

sshd worked on the server and setupVNC ran without errors but when I tried to connect the server, I got following error:

Connection closed by UNKNOWN port 65535

Note:

• apt update need to be executed before installing python3-apt.
• Unlike Google colab, deepnote runs cells as non-root user. So I need to use sudo when I install packages.
• When command line program wait for user input, deepnote cells cannot take user input and wait for forever.
  • That means I have to run remocolab from deepnote's terminal
• Files in home directory are kept after restarting machine. So I change working directory to temp directory before running setupVNC.
• sshd doesn't run in default config. It need to listen port other than 22.

It seems deepnote shutdown the instance if I didn't use it for long time like google colab.
I run python3 -c "import time;time.sleep(1111111111)" in deepnote terminal and closed the tab.
When I open deepnote again after few minutes, that command was running.
But when I reopen the tab after about 1 hour, that command was not running.

[ochen1]

Hi! Any updates on Deepnote support?

I noticed a new branch being created, which fortuately fixes the nvidia-smi FileNotFound exception.

I found out how to install remocolab and its dependencies onto Deepnote, but ngrok is giving me

"Connection refused" (on ssh client when trying to connect)

and

"There are no tunnels currently online in your account." (on ngrok dashboard)

(at the time of testing _(now))

argotunnel sorta works, but not really.

INFO[2020-12-23T01:07:37Z] Route propagating, it may take up to 1 minute for your new route to become functional
ERROR[2020-12-23T01:07:49Z] Cannot connect to remote: dial tcp 127.0.0.1:58387: connect: connection refused
ERROR[2020-12-23T01:07:49Z] CF-RAY: 605e30156af01bf4-SEA Proxying to ingress 0 error: websocket: bad handshake

It gets me closer than ever before to a connection, but at the last minute the above error message appears on the server.

[ochen1]

Hello! It's odd, but it appears my findings are inconsistent with your notes!

Notes:
^{(by "Deepnote" I may mean "Deepnote's default docker instance")}

• Deepnote provides a handy Terminal, which I prefer using over the notebook
  • The terminal allows spawning new interactive Python REPL processes
  • Programs like htop work and are great for debug
• The default version of Python on Deepnote is Python 3.7.9
  
  • Installing any other version of Python is very complicated (touch docker, get into a dependency hell with apt)
• Deepnote does not have apt cache updated by default, need to run apt update before anything
• python(3)-apt does not install successfully on Deepnote (pip and apt both fail installation)
  • pip fails due to ModuleNotFound: DistUtilsExtra error
    
    • If the notebook cell is not running as root (as per your speculations), then pip would essentially be installing to (3) different places, which might cause confusion and/or errors: /root/.local/lib/python3.7/ (does not exist on a fresh new instance), /home/jovyan/.local/lib/python3.7/ (does not exist on a fresh new instance), and /usr/lib/python3.7/
  • the package manager apt succeeds when installing python-apt
    • apt still can't be imported, raising a ModuleNotFound exception at import
    • apt_pkg can't be imported either
    • the import names above were tested (ie. raised a ModuleNotFound exception) in the folowing situations:
      1. restarting the notebook (NOT the instance), in a notebook cell
      2. using a new instance of the Python 3 REPL
  • It appears usage of the python-apt module is still possible though, this StackOverflow question has some useful answers. Also see my (semi-)working code block below.
• Deepnote runs cells as the user root. The terminal is also running as root.
  
  
• Deepnote has no support for stdin, nor support for ipywidgets (yet) (so must use terminal to run remocolab.py _{(for now)})
• The content inside Deepnote's default ~/work directory is persistent across runtimes / sessions / instances / reboots
• The $HOME for user root in the bash shell is /home/jovyan, but this is inconsistent with /etc/passwd
  • (my speculations are that) this may be due to some line overriding the $HOME variable in .bashrc or something similar
    
    

Here is the code block I used for testing:

# Install remocolab
%cd ~/
!git clone -b add-deepnote https://github.com/demotomohiro/remocolab.git
%cd remocolab/
!apt -qq update
!apt -qq install -y python3-dev python3-setuptools python3-distutils python3-distutils-extra python3-apt --reinstall
!mv /usr/lib/python3/dist-packages/apt_pkg.cpython-37m-x86_64-linux-gnu.so /usr/lib/python3/dist-packages/apt_pkg.so
!mv /usr/lib/python3/dist-packages/apt_inst.cpython-37m-x86_64-linux-gnu.so /usr/lib/python3/dist-packages/apt_inst.so
!echo "pyngrok" >> requirements.txt
!pip -q install -r requirements.txt
!cp -r /usr/lib/python3/dist-packages/* ~/remocolab/
!cd /tmp/ && wget "https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip" && unzip ngrok-stable-linux-amd64.zip && rm ngrok-stable-linux-amd64.zip && mv /tmp/ngrok /usr/bin && chmod +x /usr/bin/ngrok
# !cd ~/remocolab/ && wget "https://github.com/cloudflare/cloudflared/releases/download/2020.11.11/cloudflared-linux-amd64" && mv ./cloudflared-linux-amd64 ./cloudflared && chmod +x ./cloudflared

# Now run the following command in a terminal (so you can access stdin):
#!cd ~/remocolab/ && python -c "import remocolab; remocolab.setupSSHD()"
#!cd ~/remocolab/ && python -c "import remocolab; remocolab.setupSSHD(tunnel='argotunnel')"

Files in home directory are kept after restarting machine. So I change working directory to temp directory before running setupVNC.

I'm not so sure about this. From my experiments, anything inside the ~/work directory is persistent storage.

It appears that this is the case because ~/work/ is a directory symlink to -> /work/

/work is mounted as a remote filesystem, as seen here:

This makes sense, because the remote persistent filesystem caps free accounts to 5GB of storage. However, the local usable filesystem is much larger, at 100GB. Anything inside the local filesystem will, however, be purged during a hardware restart or instance stop. Note that this means every Python module installed must be reinstalled after a hardware restart (unless a virtualenv is used inside persistent storage), because the directory /usr/share/python3/ is not persistent.

It also appears Deepnote gives 750 hours of free uptime per month, meaning 100% uptime of a machine, so you can have a persistent instance! _{(wouldn't bet on it though, the instance might get powered off)}

I've tested this as my friend is now hosting a Discord bot on Deepnote, without issue (no, this is not against the ToS_{I don't think})

The bot is still online, even after over 8 hours. 😃

dashboard still displays online status

session running on standard runtime long after the tab was closed

cell running on standard runtime long after the tab was closed

and the bot is still online

and well and responding to events

[ochen1]

This is the error message I am getting with cloudflared:

[ochen1]

Is this expected? I am getting a <defunct> with ps aux | grep sshd 🤔

[rayanfer32]

Cloudflare takes 2 or more minutes sometimes to establish connection

…

On Wed, 23 Dec, 2020, 11:54 pm ochen1, ***@***.***> wrote: This is the error message I am getting with cloudflared: [image: cloudflared error message screenshot] <https://user-images.githubusercontent.com/59662605/103020560-d8d62d00-4505-11eb-920a-fe5ca09912ce.png> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#76 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI3MT5U4PWVIOEXAPM4B4LTSWIYUXANCNFSM4TIHX33A> .

[rayanfer32]

Wait for sometime....it will work.. Im also faving the delay to establish connection

…

On Wed, 23 Dec, 2020, 11:55 pm ochen1, ***@***.***> wrote: Is this expected? I am getting a <defunct> with ps aux | grep sshd 🤔 [image: defunct screenshot] <https://user-images.githubusercontent.com/59662605/103026555-73883900-4511-11eb-9a0c-1c4854076622.png> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#76 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI3MT5S5TYK2Z3JNBL6GZITSWIYYBANCNFSM4TIHX33A> .

[ochen1]

Nope, nvm, I was just being stupid lol.

For some reason running remocolab failed to start sshd, so I had to invoke it manually via /usr/sbin/sshd.

Also, remocolab did not automatically add port configuration to /etc/ssh/sshd_config for some reason. I added it manually using terminal + vim.

Apparently, the error I posted before (connection refused; bad handshake) was because cloudflared could not find the open port on localhost.

Now I am getting this error:

Connection closed by UNKNOWN port 65535

Looks like @demotomohiro also got that error!

[rayanfer32]

Ohh i see... plz let me know how you get that one fixed... same thing happening for me...within 10 mins of using remocolab

…

On Thu, 24 Dec, 2020, 12:06 am ochen1, ***@***.***> wrote: Nope, nvm, I was just being stupid lol. For some reason running remocolab failed to start sshd, so I had to invoke it manually via /usr/sbin/sshd. Also, remocolab did not automatically add port configuration to /etc/ssh/sshd_config for some reason. I added it manually using terminal + vim. Apparently, the error I posted before (connection refused; bad handshake) was because cloudflared could not find the open port on localhost. Now I am getting this error: Connection closed by UNKNOWN port 65535 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#76 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI3MT5QM33PQ2QUPOFS7SELSWI2DLANCNFSM4TIHX33A> .

[ochen1]

@rayanfer32 Are you running on Deepnote? I'm not sure if this issue is related to what you are trying to do.

[ochen1]

It looks like Deepnote is blocking port 22! Even root is unable to start a server on port 22.

Trying to use sshd with port 22 causes <defuct> in ps aux and trying to run SimpleHttpServer or http.server on port 22 causes a PermissionError (even as root). Please see attatched screenshot below:

[rayanfer32]

Its happening in colab too

…

On Thu, 24 Dec, 2020, 12:18 am ochen1, ***@***.***> wrote: @rayanfer32 <https://github.com/Rayanfer32> Are you running on Deepnote? I'm not sure if this issue is related to what you are trying to do. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#76 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI3MT5XCPJP5ZXLMF2NHA5LSWI3OHANCNFSM4TIHX33A> .

[ochen1]

This issue is for adding support for Deepnote #76

It is note related to Google Colab or anything happening there.

If you are experiencing an issue, feel free to open another new issue for it. Just make sure to look for duplicates 🤗

[rayanfer32]

Its fine... i didnt notice the title... btw support for tunnels seems to be getting blocked eventually

…

On Thu, 24 Dec, 2020, 12:26 am ochen1, ***@***.***> wrote: This issue is for adding support for Deepnote #76 <#76> It is note related to Google Colab or anything happening there. If you are experiencing an issue, feel free to open another new issue for it. Just make sure to look for duplicates 🤗 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#76 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI3MT5UVN2FXT4SDJKRHNPTSWI4MLANCNFSM4TIHX33A> .

[ochen1]

This is odd: sshd is started on port 8022 without error.

ngrok is forwarding to localhost (which 0.0.0.0 covers) port 8022.

Yet I am unable to connect to ssh:

EDIT:
I think I've found some mistakes in the above screenshots:

1. Logging in as a non-existent user: the user colab does not exist on Deepnote. What am I doing!
   It exists now, as it was created automatically by remocolab.

2. PermitRootLogin needs to be enabled in sshd_config in order to login as root.

But its still not working. (Connection closed)