broker: preserve log levels in system instance

[garlick]

Problem: all Flux logs go to the systemd journal (via stderr) at LOG_INFO level, which means they cannot be usefully filtered using the journalctl(1) -p option.

When log-stderr-mode=local is selected, use a prefix that can be interpreted by systemd instead of embedding the level name in the log message. For example:

$ flux logger --severity=err help

before:

logger.err[0]: help

after:

<3>logger[0]: help

Edit: as a bonus, journalctl output for flux is now properly colorized by severity.

[chu11]

LGTM!

[grondo]

Is there a reason to replace the log level string with the prefix, instead of just adding the prefix for systemd's benefit?
AFAIK, the <N> prefix is dropped when log messages go from the systemd journal to /var/log/messages or other log message destinations. When debugging, I have found it useful to grep these files for things like broker.err to find error messages when I don't know what I'm looking for. Unless I've missed something, that will no longer be possible after this change?

[garlick]

Oh, you are right, for some reason I was assuming the syslog severity levels landed in our log files but they do not. (I guess I also assumed that if systemd knows the log level, it would preserve it when forwarding to syslog, although I didn't check that).

Sure, I will add that back in.

[garlick]

Thanks, I'll set MWP

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83.91%. Comparing base (d4929cf) to head (1824d14).
Report is 3 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6918      +/-   ##
==========================================
+ Coverage   83.90%   83.91%   +0.01%     
==========================================
  Files         540      540              
  Lines       90525    90526       +1     
==========================================
+ Hits        75957    75967      +10     
+ Misses      14568    14559       -9     

Files with missing lines	Coverage Δ
src/broker/log.c	76.02% <100.00%> (+0.07%)	⬆️

... and 6 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.