Possibility of multiprocess-safe logging without inheriting the logger?

[julian-goettingen]

I have had problems with colorized output on multiprocessing with windows and I did find the solution here:
#108

Unfortunately in my case the creation of the multiprocessing-pool is within a library.
Now I can solve the problem by patching the library (I probably will) but I wonder if a solution without touching initargs is possible, for the following reason:

In 2019 when the issue linked above was raised the newest stable python-version was 3.7 and this problem only affected windows.
But with python3.8, macOS switched the default to spawn and in python3.14, even linux will switch the default to spawn, thus eventually making default multiprocessing consistent across platforms. https://docs.python.org/3.12/library/multiprocessing.html

Here is an idea, no idea if it is feasible with the library-internals of loguru:
multiprocessing-processes know their parent-process and can obtain its pid (even if they were spawned instead of forked) This unnecessarily wordy article shows it: https://superfastpython.com/multiprocessing-parent/
(This is a python>=3.8 feature, but since 3.7 is EOL I think that's ok)
So if the main process expects more processes to create their own global loguru-loggers, then it would be theoretically possible for it to set up a synchronization-scheme in advance that the other processes will discover and hook into when those processes initialize their global loguru-logger.

[Delgan]

Hi @julian-goettingen.

I agree that combining loguru and multiprocessing with "spawn" start method is not very convenient as of today. I was planning some kind of new logger.connect() method to permit inter-process communication without inheriting the logger. The idea would be to communicate using a TCP socket, message sent by child process would be received and logged by the main one.

That would require the method to be called by both parent and child processes (with separate arguments to differentiate sender from receiver), which is not as straightforward as when the "fork" start method is used.

However, it's important to note that inheriting the logger should be preferred whenever possible, as it provides better performance than any other solution. I also plan to add a new logger.reinstall() method so that one could just start a process with initializer=logger.reinstall to make sure the inherited logger is globally available.

Regarding your proposal of develop a protocol allowing child process to discover the parent logger, I have not a clear idea about how to implement this in a robust way. I'm open to technical suggestions if you have any.

[julian-goettingen]

About the discovery-mechanism:
All three big operating systems have named pipes, so I thought something should be possible with that. The __main__-process could spawn a process that will be the sole true writer doing disk-IO. This process reads on a named pipe called <__main__-proc_id>-new . If any process initialiizes their loguru-logger, they can find this name of this <__main__-proc_id>-new -pipe and send a a signal to the writer process notifying it of their existence. Then, the two processes could create a new pipe, <__main__-proc_id>-<child_proc_id> or something, where the child could write its messages.

Not sure how this would play out with multithreading, I don't use that at all and I don't know how loguru currently handles it, I am sure you have some complexity in place for that.

About performance:
Why would inheriting the logger yield a better performance than a TCP protocol? My understanding is that loguru uses multiprocessing-queues and a multiprocessing-queue is just a pipe where we send pickled data. This article seems to suggest TCP can keep up with pipes: https://www.baeldung.com/linux/ipc-performance-comparison
Is there something on the python side of things that I am not seeing?

[Delgan]

Thanks for your input.

---

All three big operating systems have named pipes, so I thought something should be possible with that. The __main__-process could spawn a process that will be the sole true writer doing disk-IO. This process reads on a named pipe called <__main__-proc_id>-new . If any process initialiizes their loguru-logger, they can find this name of this <__main__-proc_id>-new -pipe and send a a signal to the writer process notifying it of their existence. Then, the two processes could create a new pipe, <__main__-proc_id>-<child_proc_id> or something, where the child could write its messages.

I think named pipes support in Python is very OS-dependent, for example os.mkfifo() only works on Linux. The mechanism you describe might work but it seems to assume the child process always acts as a "client". There may be use case for starting the logger "server" from a child process. This is the kind of edge cases I find difficult to address without requiring the user to explicitly states the intent.

If inter-process communication without logger inheritance is needed by the user, I imagine something like that in main.py:

if multiprocessing.parent_process() is None:
    logger.connect(is_server=True)  # Main process: will receive and log all messages
else:
    logger.connect(is_server=False)  # Child process: all logs are sent to the server

We can imagine doing this automatically if is_sever is None, but that would still require to explicitly call logger.connect() in both child and parent.

---

Not sure how this would play out with multithreading, I don't use that at all and I don't know how loguru currently handles it, I am sure you have some complexity in place for that.

Yeah, that's likely not a big problem.

---

About performance:
Why would inheriting the logger yield a better performance than a TCP protocol? My understanding is that loguru uses multiprocessing-queues and a multiprocessing-queue is just a pipe where we send pickled data. This article seems to suggest TCP can keep up with pipes: https://www.baeldung.com/linux/ipc-performance-comparison
Is there something on the python side of things that I am not seeing?

Yes you're right, Loguru uses a pipe internally when enqueue=True and the logger is inherited.

The Python documentation seems to recommend pipe over socket for performance:

If you need fast IPC between two processes on one machine, you should look into pipes or shared memory. If you do decide to use AF_INET sockets, bind the “server” socket to localhost. On most platforms, this will take a shortcut around a couple of layers of network code and be quite a bit faster.

There are also benchmarks that suggest that pipe are faster than sockets, although I agree it's not very significant.

I also suspect I might need to implement "digest authentication" for security reasons, as done by Listener and Client, which will add overhead.

[julian-goettingen]

I see your concerns and why you want the explicit connect-method. I don't see a better way either.
I do believe that a lot of people use loguru because it is so good out-of-the-box and configuration is optional. It really replaces print in a way most logging-libraries in most languages don't.

But then again, everyone who does parallel programming is aware that print will not really work either, so I guess this is fine.

[wohali]

@Delgan First, thanks for your hard work on this library. It's so much better than using just logger!

As @julian-goettingen notes, the multiprocessing default start method is changing in Python 3.14, but from fork to forkserver on POSIX platforms. I expect the same limitations on loguru will exist under forkserver as they will under spawn.

Also, the current documentation heavily implies that spawn is only for Windows, but this is not the case. Multiprocessing programs that require GPU access (e.g. CUDA with CuPy or PyTorch), require the use of spawn instead of fork. I recommend changing this in a future release.

For now, I am passing the logger instance through my multiprocessing.Pool initializer, and setting a global logger there, but I look forward to improved future options!

[Delgan]

@wohali Thank you for your interest and technical insights regarding the future Loguru.

I agree that "fork as default on Linux" is outdated, and that "spawn" is now preferred in most situations.

The reinstall() method discussed in this thread has been implemented in #1069 and is therefore available on the Loguru master branch. It offers a simple and elegant solution for propagating the logger configuration to newly spawn processes, without having to manually change the global logger.

The second function mentioned in this thread, logger.connect(), will be mainly useful for interoperability with FastAPI. It remains one of my priorities once I have finished clearing up my backlog.

After that, I definitely plan to refresh the documentation with the new best practices.