Add "reinstall()" method to make it easier in spawn multiprocessing

[monchin]

Make it easier when using loguru in spawn multiprocessing environment

[monchin]

It seems black does not check docstring's length.

[Delgan]

Thanks for the PR @monchin!

However, I'm going to wait a little before merging it, because there are quite a few special cases I'd like to think about and a lot of unit tests that need to be implemented. I'll probably make some commits after yours to complete the development, if you're OK with that.

It seems black does not check docstring's length.

Indeed, docstrings are not automatically formatted by black, we have to do it manually.

[monchin]

I'm not familiar with ruff, seems it failed because I created a new tests/test_reinstall.py file

[Delgan]

Thanks for the update.

I'm not familiar with ruff, seems it failed because I created a new tests/test_reinstall.py file

Similarly to black, the ruff tool notably replaces isort and is in charge of re-ordering the import statements.

You can try to install the pre-commit hooks to automatically fix your files before git commit:

$ pre-commit install --install-hooks

Or you can manually fix the file with:

$ ruff tests/test_reinstall.py --fix

[indigoviolet]

@Delgan What is the status of this? this would be a really nice addition to have.

[Delgan]

Thanks for your contribution and sorry for the delay.

The logger.reinstall() method is a great addition. I'll merge the PR as-is and see if further refinements are needed (thinking about some edge cases).

Refs: #818 #912 #1064

[wynemo]

hello, guys, does it actually work?

run it on linux, it fails:

import multiprocessing
from loguru import logger

def subworker(_logger):
    _logger.reinstall()
    _logger.info("Child")


def test_process_spawn():
    spawn_context = multiprocessing.get_context("spawn")
    logger.add("file.log", context=spawn_context, enqueue=True, catch=False)

    process = spawn_context.Process(target=subworker, args=(logger,))
    process.start()
    process.join()
    assert process.exitcode == 0
    _logger.info("Main")
    _logger.remove()

if __name__ == "__main__":
    test_process_spawn()

ubuntu@oracle:/tmp$ ./venv/bin/python test.py
Traceback (most recent call last):
  File "/tmp/test.py", line 21, in <module>
    test_process_spawn()
  File "/tmp/test.py", line 14, in test_process_spawn
    process.start()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/lib/python3.10/multiprocessing/context.py", line 288, in _Popen
    return Popen(process_obj)
  File "/usr/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/lib/python3.10/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_io.TextIOWrapper' object

am i missing something?

edit: never mind, just call logger.remove before test, because "sys.stderr" sink is not picklable

[monchin]

Thanks for your contribution and sorry for the delay.

Thank you for merging and it's my pleasure.

I wanted to make loguru name rotation files the same way as std logging when rotating every midnight, and found that actually I can use logger.add(f"file.log.{time:YYYY-MM-DD}). Now I can replace std logging with loguru from next release！

[Delgan]

@wynemo The new reinstall() method only helps with updating the global logger after it's inherited by a child process. However, this does not change the fact that the logger must be inheritable to begin with.

In your case, you're facing an error because the default sink, which is sys.stderr, can't be inherited (because it contains non-picklable attributes). This is a common problem. You need first to remove the default sink, then re-add it with enqueue=True (which ensures the sink can be inherited):

spawn_context = multiprocessing.get_context("spawn")
logger.remove()
logger.add(sys.stderr, context=spawn_context, enqueue=True)
logger.add("file.log", context=spawn_context, enqueue=True, catch=False)