Skip to content

Addfiles_unwatched_bytes_unread_total metric to track unread bytes when files are unwatched #24675

@akashvbabu91

Description

@akashvbabu91

When Vector's file source stops watching a file (due to deletion, rotation, or rotate_wait timeout), there's currently no visibility into whether all data was successfully read. Operators have no way to detect or measure data loss scenarios such as:

  • Files deleted before Vector finished reading them (e.g., kubelet cleaning up old container logs)
  • Aggressive log rotation where old files are removed faster than Vector can read
  • Vector falling behind on high-volume log files

There is an existing metric:files_unwatched_total(https://vector.dev/docs/reference/configuration/sources/internal_metrics/#files_unwatched_total) metric only counts number of times vector stopped watching a file before hitting EOF, but doesn't provide you with how many "bytes" were lost. Without knowing bytes loss, it could be the entire file or just a small amount of bytes at the end of the file.

Proposed Solution:
Add a new counter metric files_unwatched_bytes_unread_total that tracks the number of bytes that were not read from a file.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions