Getting error while trying to log "WatchtowerWarning: Received message after logging system shutdown"

[dhiyan-nirva-ai]

"watchtower_init_.py:381: WatchtowerWarning: Received message after logging system shutdown
warnings.warn("Received message after logging system shutdown", WatchtowerWarning)"

Getting the above error while logging ,

[EddyIonescu]

Ran into this too, supplying use_queues=False to the CloudWatchLogHandler constructor function seemed to fix it. Might not be closing properly whenever you stop/start the server (like Django or Flask) you're using: #38

[ymguerra]

Hi, I am also having this issue, in my case I use Django + gunicorn (gevent), if I set use_queues=False it works. I think is somehow related to Gunicorn + gevent because we share the same configuration with others servers (celery and websocket) and all the logs are send from those servers but not from our Web server, any idea?

[o-nikolas]

I think is somehow related to Gunicorn + gevent

Linking #89 for context

[Insighttful]

Hitting this issue using Uvicorn 0.24 and Fastapi 0.105.

[medley56]

I think I've fixed this issue by flushing all CloudWatchLogHandlers before exiting any process. I'm guessing the issue is that the logging system shuts down without waiting for the cloudwatch logs to finish transmitting.

The following code doesn't work if you have handlers attached to child loggers. This only flushes the handlers attached to the root logger.

def flush_cloudwatch_logs():
    """Force flush of all cloudwatch logging handlers. For example at the end of a process just before it is killed.

    Returns
    -------
    None
    """
    root_logger = logging.getLogger()
    for handler in root_logger.handlers:
        if isinstance(handler, watchtower.CloudWatchLogHandler):
            handler.flush()

[pascencio]

I have the same issue with uvicorn=0.30.5, fastapi[standard]=0.112.0 and watchtower=3.2.0. Disabling the queue option works fine but is too slow.

Please help!!

I will tray to fork and create a pull request if catch the issue.

[LazyBusyYang]

Same issue as #171 (comment) with fastapi. Found a solution:
To be brief, start uvicorn with uvicorn.run(..., log_config=None, ...), specify the log_config value, CloudWatchLogHandler will work normally.
What's the side effect: log format of uvicorn becomes different.

How to locate the problem? Manually raise an exception at CloudWatchLogHandler.close() like this:

    def close(self):
        """
        Send any queued messages to CloudWatch and prevent further processing of messages.
        This method does nothing if ``use_queues`` is set to False.
        """
        # FIXME: don't add filter if it's already installed
        self.addFilter(_boto_filter)
        # Avoid waiting on the queue again when the close called twice.
        # Otherwise the second call, as no thread is running, it will hang
        # forever
        if self.shutting_down:
            return
        self.shutting_down = True
        raise Exception('test')
        ...

Run your program again, you will find why close() is called. My traceback stack is like this:

Traceback (most recent call last):
  File "/pvc/home/Code/.../main.py", line 37, in <module>
    ret_val = main(args)
  File "/pvc/home/Code/.../main.py", line 19, in main
    server.run()
  File "/pvc/home/Code/...", line 237, in run
    uvicorn.run(self.app, host=self.host, port=self.port)
  File "/pvc/home/.local/lib/python3.10/site-packages/uvicorn/main.py", line 517, in run
    config = Config(
  File "/pvc/home/.local/lib/python3.10/site-packages/uvicorn/config.py", line 274, in __init__
    self.configure_logging()
  File "/pvc/home/.local/lib/python3.10/site-packages/uvicorn/config.py", line 366, in configure_logging
    logging.config.dictConfig(self.log_config)
  File "/pvc/home/.conda/envs/.../lib/python3.10/logging/config.py", line 811, in dictConfig
    dictConfigClass(config).configure()
  File "/pvc/home/.conda/envs/.../lib/python3.10/logging/config.py", line 538, in configure
    _clearExistingHandlers()
  File "/pvc/home/.conda/envs/.../lib/python3.10/logging/config.py", line 275, in _clearExistingHandlers
    logging.shutdown(logging._handlerList[:])
  File "/pvc/home/.conda/envs/.../lib/python3.10/logging/__init__.py", line 2183, in shutdown
    h.close()
  File "/pvc/home/.conda/envs/.../lib/python3.10/site-packages/watchtower/__init__.py", line 533, in close
    raise Exception('test')
Exception: test

Reading uvicorn/config.py, you'll know the fix.

[jacobggman]

@LazyBusyYang Thank you so much! You are the GOAT