Vector keeps reading from a log file that has been deleted when no space left on device

[duanchao2035]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

When there is no remaining space on the device, I found that vector continues to read those deleted files. By checking the logs, I suspect it might be due to a failure in writing checkpoints. The issue persists even after the device frees up space until Vector is restarted.

2024-10-22T03:20:48.690801Z INFO source{component_kind="source" component_id=multiline-symlinks component_type=file}:file_server: vector::internal_events::file::source: Stopped watching file. file=/sysroot/var/log/zbs/netbouncer/[email protected] reached_eof="true" 2024-10-22T04:08:18.904447Z ERROR vector::internal_events::file::source: Failed writing checkpoints. error=No space left on device (os error 28) error_code="writing_checkpoints" error_type="writer_failed" stage="receiving" internal_log_rate_limit=true 2024-10-22T04:08:18.905785Z ERROR vector::internal_events::file::source: Internal log [Failed writing checkpoints.] is being suppressed to avoid flooding. 2024-10-22T04:08:28.955636Z ERROR vector::internal_events::file::source: Internal log [Failed writing checkpoints.] has been suppressed 46 times. 2024-10-22T04:08:28.955648Z ERROR vector::internal_events::file::source: Failed writing checkpoints. error=No space left on device (os error 28) error_code="writing_checkpoints" error_type="writer_failed" stage="receiving" internal_log_rate_limit=true 2024-10-22T04:08:29.032913Z ERROR vector::internal_events::file::source: Internal log [Failed writing checkpoints.] is being suppressed to avoid flooding. 2024-10-22T04:08:39.106843Z ERROR vector::internal_events::file::source: Internal log [Failed writing checkpoints.] has

Configuration

Version

0.38.0-alpine

Debug Output

Example Data

No response

Additional Context

No response

References

No response

[jszwedko]

Hi @duanchao2035 !

So long as Vector is running, I don't believe the failure to write the checkpoints will cause Vector to fail to continue reading from files. Just if Vector stops, it will start from further back in the file than it should (from the stale checkpoint). Are you finding that Vector is ceasing reading from the input files when the disk is full? That is: nothing is flowing to the sinks?

Otherwise, I think my recent response to another issue, #22273 (comment), may be relevant here as well. Vector will hold open deleted files until it has finished reading from them. You can see that comment for some knobs to tweak to control that behavior or improve file processing speed.