Jupyterhub in docker container not working with nginx reverse proxy #390

abashkinas · 2020-08-19T13:16:16Z

Bug description

Hello, I have started jupyterhub as a service in dockerswarm. Without nginx reverse proxy it works without problems. When I configure nginx reverse proxy with the following configuration:

server {

    listen 80;

    server_name jupyter.ifora.hse.ru;


    index index.html index.htm;

    access_log /var/log/nginx/jupyter.ifora.hse.ru-access.log ;
    error_log /var/log/nginx/jupyter.ifora.hse.ru-error.log error;



    location / {
      proxy_pass http://172.18.207.29:8000;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header Host $host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      # websocket headers
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;
    }
}

The following messages begin to appear in the jupyterhub logs:

upyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:25:35.840 JupyterHub log:181] 200 POST /hub/api/users/admin/activity ([email protected]) 17.07ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:25:40.132 JupyterHub log:181] 302 GET / -> /hub/ (@172.18.207.29) 1.28ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:25:40.136 JupyterHub log:181] 302 GET /hub/ -> /hub/login (@172.18.207.29) 0.63ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:25:40.263 JupyterHub log:181] 200 GET /hub/login (@172.18.207.29) 37.34ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:25:55.101 JupyterHub log:181] 302 GET / -> /hub/ (@172.18.207.29) 0.97ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:25:55.106 JupyterHub log:181] 302 GET /hub/ -> /hub/login (@172.18.207.29) 0.66ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:25:55.134 JupyterHub log:181] 200 GET /hub/login (@172.18.207.29) 24.86ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:26:10.103 JupyterHub log:181] 302 GET / -> /hub/ (@172.18.207.29) 1.43ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:26:10.126 JupyterHub log:181] 302 GET /hub/ -> /hub/login (@172.18.207.29) 1.28ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:26:10.170 JupyterHub log:181] 200 GET /hub/login (@172.18.207.29) 38.67ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:26:25.102 JupyterHub log:181] 302 GET / -> /hub/ (@172.18.207.29) 1.14ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:26:25.107 JupyterHub log:181] 302 GET /hub/ -> /hub/login (@172.18.207.29) 0.89ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:26:25.147 JupyterHub log:181] 200 GET /hub/login (@172.18.207.29) 35.87ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:26:40.126 JupyterHub log:181] 302 GET / -> /hub/ (@172.18.207.29) 1.22ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:26:40.130 JupyterHub log:181] 302 GET /hub/ -> /hub/login (@172.18.207.29) 0.92ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:26:40.162 JupyterHub log:181] 200 GET /hub/login (@172.18.207.29) 27.79ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:26:55.103 JupyterHub log:181] 302 GET / -> /hub/ (@172.18.207.29) 1.56ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:26:55.108 JupyterHub log:181] 302 GET /hub/ -> /hub/login (@172.18.207.29) 0.86ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:26:55.141 JupyterHub log:181] 200 GET /hub/login (@172.18.207.29) 27.69ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:27:10.103 JupyterHub log:181] 302 GET / -> /hub/ (@172.18.207.29) 1.84ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:27:10.111 JupyterHub log:181] 302 GET /hub/ -> /hub/login (@172.18.207.29) 0.74ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:27:10.145 JupyterHub log:181] 200 GET /hub/login (@172.18.207.29) 30.30ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:27:25.129 JupyterHub log:181] 302 GET / -> /hub/ (@172.18.207.29) 1.23ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:27:25.133 JupyterHub log:181] 302 GET /hub/ -> /hub/login (@172.18.207.29) 0.77ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:27:25.253 JupyterHub log:181] 200 GET /hub/login (@172.18.207.29) 27.17ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:27:40.101 JupyterHub log:181] 302 GET / -> /hub/ (@172.18.207.29) 1.09ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:27:40.106 JupyterHub log:181] 302 GET /hub/ -> /hub/login (@172.18.207.29) 0.69ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:27:40.143 JupyterHub log:181] 200 GET /hub/login (@172.18.207.29) 32.72ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:27:55.101 JupyterHub log:181] 302 GET / -> /hub/ (@172.18.207.29) 1.08ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:27:55.105 JupyterHub log:181] 302 GET /hub/ -> /hub/login (@172.18.207.29) 0.81ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:27:55.136 JupyterHub log:181] 200 GET /hub/login (@172.18.207.29) 27.47ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:28:10.228 JupyterHub log:181] 302 GET / -> /hub/ (@172.18.207.29) 1.71ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:28:10.234 JupyterHub log:181] 302 GET /hub/ -> /hub/login (@172.18.207.29) 1.13ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:28:10.267 JupyterHub log:181] 200 GET /hub/login (@172.18.207.29) 28.57ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:28:25.101 JupyterHub log:181] 302 GET / -> /hub/ (@172.18.207.29) 1.28ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:28:25.106 JupyterHub log:181] 302 GET /hub/ -> /hub/login (@172.18.207.29) 0.97ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:28:25.140 JupyterHub log:181] 200 GET /hub/login (@172.18.207.29) 30.08ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:28:40.101 JupyterHub log:181] 302 GET / -> /hub/ (@172.18.207.29) 1.76ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:28:40.108 JupyterHub log:181] 302 GET /hub/ -> /hub/login (@172.18.207.29) 1.94ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:28:40.145 JupyterHub log:181] 200 GET /hub/login (@172.18.207.29) 32.96ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:28:55.100 JupyterHub log:181] 302 GET / -> /hub/ (@172.18.207.29) 0.88ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:28:55.109 JupyterHub log:181] 302 GET /hub/ -> /hub/login (@172.18.207.29) 0.92ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:28:55.150 JupyterHub log:181] 200 GET /hub/login (@172.18.207.29) 36.64ms

After a few hours jupyterhub stops working and has the following error in its logs:

[E 2020-08-19 11:03:55.159 JupyterHub log:181] 500 GET /hub/login (@172.18.207.29) 50.32ms
jupyter_hub.1.ry6cazaavyrc@swarm_manager    | [I 2020-08-19 11:04:10.102 JupyterHub log:181] 302 GET / -> /hub/ (@172.18.207.29) 1.46ms
jupyter_hub.1.ry6cazaavyrc@swarm_manager    | [I 2020-08-19 11:04:10.127 JupyterHub log:181] 302 GET /hub/ -> /hub/login (@172.18.207.29) 1.47ms
jupyter_hub.1.ry6cazaavyrc@swarm_manager    | [E 2020-08-19 11:04:10.136 JupyterHub web:1792] Uncaught exception GET /hub/login (172.18.207.29)
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |     HTTPServerRequest(protocol='http', host='jupyter.ifora.hse.ru', method='GET', uri='/hub/login', version='HTTP/1.1', remote_ip='172.18.207.29')
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |     Traceback (most recent call last):
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/tornado/web.py", line 1703, in _execute
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         result = await result
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/login.py", line 135, in get
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         self.finish(self._render(username=username))
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/nativeauthenticator/handlers.py", line 151, in _render
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         {'next': self.get_argument('next', '')},
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py", line 1160, in render_template
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         template = self.get_template(name)
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py", line 1154, in get_template
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         return self.settings['jinja2_env'].get_template(name)
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/environment.py", line 883, in get_template
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         return self._load_template(name, self.make_globals(globals))
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/environment.py", line 857, in _load_template
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         template = self.loader.load(self, name, globals)
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/loaders.py", line 426, in load
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         return loader.load(environment, name, globals)
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/loaders.py", line 426, in load
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         return loader.load(environment, name, globals)
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/loaders.py", line 426, in load
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         return loader.load(environment, name, globals)
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       [Previous line repeated 958 more times]
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/loaders.py", line 127, in load
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         code = environment.compile(source, name, filename)
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/environment.py", line 628, in compile
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         source = self._parse(source, name, filename)
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/environment.py", line 539, in _parse
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         return Parser(self, source, name, encode_filename(filename)).parse()
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/parser.py", line 937, in parse
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         result = nodes.Template(self.subparse(), lineno=1)
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/parser.py", line 919, in subparse
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         rv = self.parse_statement()
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/parser.py", line 141, in parse_statement
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         return getattr(self, "parse_" + self.stream.current.value)()
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/parser.py", line 279, in parse_extends
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         node.template = self.parse_expression()
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/parser.py", line 448, in parse_expression
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         return self.parse_condexpr()
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/parser.py", line 453, in parse_condexpr
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         expr1 = self.parse_or()
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/parser.py", line 466, in parse_or
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         left = self.parse_and()
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/parser.py", line 475, in parse_and
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         left = self.parse_not()
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/parser.py", line 486, in parse_not
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         return self.parse_compare()
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/parser.py", line 490, in parse_compare
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         expr = self.parse_math1()
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/parser.py", line 513, in parse_math1
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         left = self.parse_concat()
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/parser.py", line 524, in parse_concat
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         args = [self.parse_math2()]
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/parser.py", line 534, in parse_math2
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         left = self.parse_pow()
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/parser.py", line 545, in parse_pow
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         left = self.parse_unary()
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/parser.py", line 563, in parse_unary
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         node = self.parse_primary()
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/parser.py", line 580, in parse_primary
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         next(self.stream)
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/lexer.py", line 387, in __next__
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         self.current = next(self._iter)
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/lexer.py", line 660, in wrap
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         yield Token(lineno, token, value)
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |     RecursionError: maximum recursion depth exceeded
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |
jupyter_hub.1.ry6cazaavyrc@swarm_manager    | [E 2020-08-19 11:04:10.175 JupyterHub web:1220] Uncaught exception in write_error
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |     Traceback (most recent call last):
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/tornado/web.py", line 1218, in send_error
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         self.write_error(status_code, **kwargs)
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py", line 1244, in write_error
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         html = self.render_template('error.html', **ns)
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py", line 1160, in render_template
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         template = self.get_template(name)
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py", line 1154, in get_template
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         return self.settings['jinja2_env'].get_template(name)
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/environment.py", line 883, in get_template
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         return self._load_template(name, self.make_globals(globals))
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/environment.py", line 857, in _load_template
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         template = self.loader.load(self, name, globals)
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/loaders.py", line 426, in load
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         return loader.load(environment, name, globals)
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/loaders.py", line 426, in load
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         return loader.load(environment, name, globals)
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/loaders.py", line 426, in load
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         return loader.load(environment, name, globals)
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       [Previous line repeated 959 more times]
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/loaders.py", line 127, in load
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         code = environment.compile(source, name, filename)
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/environment.py", line 628, in compile
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         source = self._parse(source, name, filename)
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/environment.py", line 539, in _parse
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         return Parser(self, source, name, encode_filename(filename)).parse()
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/parser.py", line 45, in __init__
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         self.stream = environment._tokenize(source, name, filename, state)
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/environment.py", line 573, in _tokenize
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         stream = self.lexer.tokenize(source, name, filename, state)
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/lexer.py", line 616, in tokenize
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         return TokenStream(self.wrap(stream, name, filename), name, filename)
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/lexer.py", line 334, in __init__
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         next(self)
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/lexer.py", line 387, in __next__
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         self.current = next(self._iter)
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       File "/usr/local/lib/python3.6/dist-packages/jinja2/lexer.py", line 660, in wrap
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |         yield Token(lineno, token, value)
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |     RecursionError: maximum recursion depth exceeded
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |
jupyter_hub.1.ry6cazaavyrc@swarm_manager    | [E 2020-08-19 11:04:10.193 JupyterHub log:173] {
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       "X-Forwarded-Host": "jupyter.ifora.hse.ru",
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       "X-Forwarded-Proto": "http",
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       "X-Forwarded-Port": "80",
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       "Referer": "http://172.18.207.37/hub/",
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       "User-Agent": "Go-http-client/1.1",
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       "Connection": "close",
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       "X-Forwarded-For": "172.18.207.29,::ffff:10.255.0.2",
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       "Host": "jupyter.ifora.hse.ru",
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |       "X-Real-Ip": "172.18.207.29"
jupyter_hub.1.ry6cazaavyrc@swarm_manager    |     }
jupyter_hub.1.ry6cazaavyrc@swarm_manager    | [E 2020-08-19 11:04:10.193 JupyterHub log:181] 500 GET /hub/login (@172.18.207.29) 61.07ms

Please help me solve the problem.

Dockerfile jupyterhub:

FROM jupyterhub/jupyterhub:1.2

USER root



RUN pip install notebook
RUN pip install --no-cache dockerspawner
RUN pip3 install psycopg2-binary
RUN pip install --no-cache jupyterhub-nativeauthenticator

jupyterhub_config.py:

import os

#Определение переменной JPY_DB_PASSWORD для подключения к базе данных jupyter на сервере  pg_cluster_01
#f = open("/var/run/secrets/jupyterhub_postgres_pass", "r")
#os.environ['JPY_DB_PASSWORD'] = (f.read())

#Изменение рабочей директории пользователей в контейнере
notebook_dir = '/home/jovyan'
c.SwarmSpawner.notebook_dir = notebook_dir


#Монтирование папки с хоста в папку home пользователей в контейнера
def create_dir_hook(spawner):
    username = spawner.user.name  # get the username
    volume_path = os.path.join('/home/data/jupyterhub_data/', username)
    if not os.path.exists(volume_path):
        os.makedirs(volume_path)
        os.chmod(volume_path, 0o777)
    mounts_user = [
                   {'type': 'bind',
                    'source': '/home/data/jupyterhub_data/' + username,
                    'target': '/home/jovyan', }
                   ]
    spawner.extra_container_spec = {
        'mounts': mounts_user
    }


c.Spawner.pre_spawn_hook = create_dir_hook

#Определение пользователя admin
c.Authenticator.admin_users = {'admin'}

#Автоматически создавать пользователя на сервере после добавления пользователя через панель управления jupyterhub
c.LocalAuthenticator.create_system_users = True




#Подключение к базе данных  postgresql

c.JupyterHub.db_url = "postgresql://{}@{}:{}/{}".format(
    os.environ['JPY_DB_USER'],
    os.environ['POSTGRES_PORT_5432_TCP_ADDR'],
    os.environ['POSTGRES_PORT_5432_TCP_PORT'],
    os.environ['JPY_DB_NAME']
)


#Использовать собственную аутентификацию jupiterhub
c.JupyterHub.authenticator_class = 'nativeauthenticator.NativeAuthenticator'

#Сделать интерфейс jupyterlab по умолчанию
c.SwarmSpawner.default_url = '/lab'

#Использовать свой docker образ для серверов пользователей
c.SwarmSpawner.image = 'abashkin/jupyterhub_singleuser:1.2'


c.ConfigurableHTTPProxy.should_start = True
#c.Spawner.default_url = '/lab'
c.JupyterHub.spawner_class = 'dockerspawner.SwarmSpawner'
c.SwarmSpawner.network_name = "jupyter_jupyterhub-net"
c.SwarmSpawner.extra_host_config = { 'network_mode': "jupyter_jupyterhub-net" }
#c.SwarmSpawner.extra_create_kwargs.update({ 'volume_driver': 'local' })
c.SwarmSpawner.remove_containers = True
c.SwarmSpawner.debug = True
c.JupyterHub.hub_ip = '0.0.0.0'
c.JupyterHub.hub_port = 8081
c.SwarmSpawner.host_ip = "0.0.0.0"
c.SwarmSpawner.extra_placement_spec = {
 'constraints': ['node.hostname == cruncher_01'],
}

c.JupyterHub.port = 8000
#c.JupyterHub.ssl_key = os.environ['SSL_KEY']
#c.JupyterHub.ssl_cert = os.environ['SSL_CERT']

c.SwarmSpawner.http_timeout = 300
c.SwarmSpawner.start_timeout = 300

config docker swarm service:

version: '3.7'

networks:
  jupyterhub-net:
    driver: overlay

configs:
  jupyterhub_config:
    file: ./config/jupyterhub_config.py

services:
  hub:
    image: registry.ifora.hse.ru/jupyterhub:1.2
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      JPY_DB_USER: jupyter
      POSTGRES_PORT_5432_TCP_ADDR: 172.18.207.27
      POSTGRES_PORT_5432_TCP_PORT: 5432
      JPY_DB_NAME: jupyter
    configs:
      - source:  jupyterhub_config
        target: /srv/jupyterhub/jupyterhub_config.py
    deploy:
      replicas: 1
      restart_policy:
        condition: any
      placement:
        constraints: [node.hostname == swarm_manager]
    networks:
      - jupyterhub-net
    ports:
      - 8000:8000
    logging:
      driver: gelf
      options:
        gelf-address: "udp://172.18.207.46:12201"
        tag: "jupyter_hub"

Expected behaviour

Correct work with nginx reverse proxy

Actual behaviour

Jupyterhub stops working

How to reproduce

Go to '...'
Click on '....'
Scroll down to '....'
See error

Your personal set up

OS:

Version: jupyterhub:1.2

Configuration:

The text was updated successfully, but these errors were encountered:

welcome · 2020-08-19T13:16:19Z

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

1kastner · 2020-08-23T10:09:10Z

Please encapsulate your code into backticks (`) so that they are formatted appropriately. You can just edit your old entry.

As we can see in the logs, the error message stems from Jinja and it seems pretty awkward to me. I am not deeply rooted in the projects though. Have you tried running a debugger as described in https://code.visualstudio.com/docs/python/debugging to see what the template engine tries to do before it dies and creates problems for you?

manniche · 2021-05-25T13:02:03Z

This doesn't so much seem to be a problem with JupyterHub or the dockerspawner, but more with your nginx configuration. I would suggest that you direct these questions to https://discourse.jupyter.org which is a most welcoming forum.

As for your issue: JupyterHub will do a redirect to /hub if you hit it at the base path /, as indicated by the logs:

jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:25:55.101 JupyterHub log:181] 302 GET / -> /hub/ (@172.18.207.29) 0.97ms
jupyter_hub.1.vkcroo6bu1zt@swarm_manager    | [I 2020-08-18 12:25:55.106 JupyterHub log:181] 302 GET /hub/ -> /hub/login (@172.18.207.29) 0.66ms

And the recursion limit being reached:

jupyter_hub.1.ry6cazaavyrc@swarm_manager    |     RecursionError: maximum recursion depth exceeded

Maybe try to stop nginx from also trying to do the redirection by adding to your location configuration:

    location / {
      [... the other configuration ...]
      proxy_redirect  off;
      [... the other configuration ...]
    }

Energy23 · 2023-02-21T11:52:32Z

I've managed to run several JupyterHubs inside Docker container.
I've set my nginx as followed:

    location / {
            proxy_pass https://127.0.0.1:8000;

            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

            #proxy_set_header X-NginX-Proxy true;
        # websocket headers
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
            proxy_set_header X-Scheme $scheme;

            proxy_buffering off;
        }
        location ~* /(user/[^/]*)/(api/kernels/[^/]+/channels|terminals/websocket)/? {
            proxy_pass https://127.0.0.1:8000;

            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

            #proxy_set_header X-NginX-Proxy true;

            # WebSocket support
            proxy_http_version 1.1;
            proxy_set_header Upgrade "websocket";
            #proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_read_timeout 86400;
            proxy_set_header X-Scheme $scheme;
            proxy_buffering off;

        }

abashkinas added the bug Something isn't working label Aug 19, 2020

minrk added question and removed bug Something isn't working labels Feb 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jupyterhub in docker container not working with nginx reverse proxy #390

Jupyterhub in docker container not working with nginx reverse proxy #390

abashkinas commented Aug 19, 2020 •

edited by minrk

Loading

welcome bot commented Aug 19, 2020

1kastner commented Aug 23, 2020

manniche commented May 25, 2021

Energy23 commented Feb 21, 2023 •

edited

Loading

Jupyterhub in docker container not working with nginx reverse proxy #390

Jupyterhub in docker container not working with nginx reverse proxy #390

Comments

abashkinas commented Aug 19, 2020 • edited by minrk Loading

Bug description

Expected behaviour

Actual behaviour

How to reproduce

Your personal set up

welcome bot commented Aug 19, 2020

1kastner commented Aug 23, 2020

manniche commented May 25, 2021

Energy23 commented Feb 21, 2023 • edited Loading

abashkinas commented Aug 19, 2020 •

edited by minrk

Loading

Energy23 commented Feb 21, 2023 •

edited

Loading