Reloading a worker from the web

[nggit]

You can reload the worker from the web with sys.exit(3), whereas shutdown with sys.exit(0).

import sys

# http://localhost:8000/reload
@app.route('/reload')
async def reload_handler(**server):
    sys.exit(3)

It also works if using Tremolo as ASGI server. You can do sys.exit(3) from the handler in the app.

In app frameworks that catch BaseException rather than Exception, it won't work. But app frameworks shouldn't do that.

It appears that this feature is exclusive to Tremolo, because the other ASGI servers I tried gave unexpected results.

[gnat]

This is an extremely awesome, flexible feature but...

I've noticed when sys.exit(3) is used to reload, this happens:

• @app.on_worker_start runs as expected. Cool.
• @app.on_worker_stop does NOT run! InvalidStateError is thrown first, interrupting the event loop and exiting early. See below.

Tremolo 0.2.9 on Python 3.12

import asyncio
from tremolo import Tremolo

app = Tremolo()

import sys
@app.route('/reload')
async def reload(**server):
	sys.exit(3)

@app.on_worker_start
async def on_start(**_):
	print("Starting.. I always print")

@app.on_worker_stop
async def on_stop(**_):
	print("Stopping.. I never print when /reload happens")

if __name__ == '__main__':
	app.run('0.0.0.0', 9999, worker_num=2, reload=True, debug=True)

After visiting http://127.0.0.1:9999/reload

gnat@gnat:~/Projects/tremolo_test$ python3 ./test.py
Starting Tremolo (tremolo 0.2.9, cpython 3.12.7, linux)
------------------------------------------------------------------------
Routes:
  400 -> error_400()
  404 -> error_404(request=None, globals=None, status=(404, b'Not Found'), stream=False)
  500 -> error_500()
  b'^/+(?:\\?.*)?$' -> index(status=(503, b'Service Unavailable'))
  b'^/+reload(?:/+)?(?:\\?.*)?$' -> reload()

Options:
  run(host=0.0.0.0, port=9999, worker_num=2, reload=True, debug=True, log_level=DEBUG, reuse_port=True, shutdown_timeout=30, app=None, app_dir=/home/gnat/Projects/tremolo_test, module_name=test)
Starting.. I always print
------------------------------------------------------------------------
Tremolo main (pid 841329) is running 
Starting.. I always print
[2025-04-05 04:01:39] Tremolo worker (pid 841330) is started at 0.0.0.0 port 9999
[2025-04-05 04:01:39] Tremolo worker (pid 841332) is started at 0.0.0.0 port 9999
Process Process-1:
Traceback (most recent call last):
  File "/home/gnat/Projects/tremolo_test/tremolo/tremolo.py", line 439, in _worker
    loop.run_forever()  # until loop.stop() is called
    ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/base_events.py", line 641, in run_forever
    self._run_once()
  File "/usr/lib/python3.12/asyncio/base_events.py", line 1986, in _run_once
    handle._run()
  File "/usr/lib/python3.12/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/home/gnat/Projects/tremolo_test/tremolo/lib/http_protocol.py", line 243, in _handle_request
    await self.request_received(self.request, response)
  File "/home/gnat/Projects/tremolo_test/tremolo/http_server.py", line 297, in request_received
    await self._handle_response(func, kwargs)
  File "/home/gnat/Projects/tremolo_test/tremolo/http_server.py", line 149, in _handle_response
    data = await agen
           ^^^^^^^^^^
  File "/home/gnat/Projects/tremolo_test/./test.py", line 9, in reload
    sys.exit(3)
SystemExit: 3

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/gnat/Projects/tremolo_test/tremolo/managers.py", line 44, in _target
    func(*args, **kwargs)
  File "/home/gnat/Projects/tremolo_test/tremolo/tremolo.py", line 444, in _worker
    exc = task.exception()
          ^^^^^^^^^^^^^^^^
asyncio.exceptions.InvalidStateError: Exception is not set.
A worker process died (1). Restarting...
Starting.. I always print
[2025-04-05 04:01:45] Tremolo worker (pid 841360) is started at 0.0.0.0 port 9999

[gnat]

Edited. More details, with example. @nggit

[nggit]

Thank you for testing this feature!

1. The current clean up requirements are:
   await self._serve_forever() must be completed either with/without errors. See: finally block there.

2. The issue you found is because SystemExit (a BaseException-level) instead appears in:
   loop.run_forever(). So no clean up occurs.

3. I tried to fix it in the past, but since it caused the response to unintuitively show "request timeout", I postponed it until I forgot about it.

4. This ensure on_worker_stop on reload #251 patch should create the behavior you want. While this is correct, it still has the flaws I mentioned in point 3.

[nggit]

yield-ing before exit can be done as a workaround for point 3:

@app.route('/reload')
async def reload(chunked=False, **server):
    yield b'Reloading...'
    sys.exit(3)

If you find other issues worth fixing please raise them @gnat !