When Vector's file source stops watching a file (due to deletion, rotation, or rotate_wait timeout), there's currently no visibility into whether all data was successfully read. Operators have no way to detect or measure data loss scenarios such as:
- Files deleted before Vector finished reading them (e.g., kubelet cleaning up old container logs)
- Aggressive log rotation where old files are removed faster than Vector can read
- Vector falling behind on high-volume log files
There is an existing metric:files_unwatched_total(https://vector.dev/docs/reference/configuration/sources/internal_metrics/#files_unwatched_total) metric only counts number of times vector stopped watching a file before hitting EOF, but doesn't provide you with how many "bytes" were lost. Without knowing bytes loss, it could be the entire file or just a small amount of bytes at the end of the file.
Proposed Solution:
Add a new counter metric files_unwatched_bytes_unread_total that tracks the number of bytes that were not read from a file.