Time-based log rotation

[bmoylan]

Some log archival tools only operate on rotated/compressed logfiles. For services which do not log very often, it can take weeks/months to reach the 1GB limit to trigger a rotation.

FR: Allow configuring a maximum logfile age before rotation. This likely requires changing Lumberjack. See some previous discussion on natefinch/lumberjack#17 and natefinch/lumberjack#54.

If we do not think it will be straighforward to change Lumberjack, natefinch/lumberjack#17 proposes the following goroutine to force rotations on a periodic schedule:

log := lumberjack.Logger{ /* some config */ }
go func() {
    for {
        <-time.After(time.Hour*24)
        log.Rotate()
    }
}()

This will not preclude size-based rotations, and if the timer fires soon after a size-based rotation, you may end up with weirdly small files.

[bmoylan]

I've pushed bmoylan/lumberjack#1 which proposes a change to lumberjack for rotating after a max time

[bmoylan]

Just found natefinch/lumberjack#76 which sounds like he probably would not accept the contribution

[nmiyake]

Thanks for raising this issue!

Took a look at the PR, and one thing to note is that, as currently implemented, I don't think that it makes the guarantee that log files are rotated at least once per time interval since the rotation check happens on Write. To make this guarantee, I do think that we'd need some goroutine that is running and would initiate rotation after the time period has elapsed. Although in general a running program will probably log to things like service logs at an interval where this isn't an issue, for logs that depend more on user actions (like event logs), this may be more of a possibility.

If at all possible, I would also like to avoid forking... Even though it's possible to fork or convert to an internal library here (since I don't think we expose the API externally at all), being on mainline would be nice.

What do you think of an implementation that would start a goroutine that does the file age comparison itself? I would imagine the high-level logic to be something like:

• When creating/initializing logger, look for output file and record creation time
• Start a timer that will fire at creation time + configured max log file age
• When the timer fires, check creation time of output file
  • If it is the same as before, then perform rotation and restart timer
  • If it is different, then update timer to fire at new creation time + configured max log file age

There is a small edge case here (we determine that file is old, decide to call "rotate", but a write that triggers size-based rotation happens right before the "rotate" call and thus we double-rotate and have a log file with just one line), but I think it's small enough to be acceptable (and even when it's hit, you don't lost data -- just have a file that is strangely small). If we really cared about this, then we could make a wrapper struct that implements the Logger interface (and does this goroutine) and then perform the locking at this higher level.