Collector stops reading new file after copytruncate rotation

[paulojmdias]

Component(s)

pkg/stanza/fileconsumer

What happened?

Description

When using a copy-truncate log rotation method, the filelog receiver supposedly stops reading new log lines from the active file after a rotation.

This happens intermittently and is not always reproducible. It depends on the precise timing between the rotation operation, file truncation, and the collector’s polling interval.

After rotation, debug logs show entries such as:

Reading lost file path=/var/log/app/app.log

But no new log entries from /var/log/app/app.log appear afterward.

The collector continues polling, but does not seem to reattach to the new file created by logrotate.

Steps to Reproduce

1. Configure the receiver to monitor only the base log file:

receivers:
  otlpjsonfile:
    include: [/var/log/app/app.log]
    start_at: beginning
    poll_interval: 500ms

2. Continuously write log lines to /var/log/app/app.log

3. Use a logrotate configuration file with the copytruncate option, for example (/etc/logrotate.d/app):

/var/log/app/app.log {
    rotate 5
    size 1M
    copytruncate
    missingok
    notifempty
    compress
    delaycompress
}

4. Trigger a rotation manually:

logrotate -f /etc/logrotate.d/app

5. Continue writing logs to /var/log/app/app.log after the rotation.
6. Observe that in some runs, the collector stops collecting new lines from the active log file. In other runs, the collector resumes correctly, depending on timing and poll overlap.

Expected Result

The collector should detect the rotation, complete reading any remaining lines from the old file, and then immediately resume reading from the new /var/log/app/app.log.
New log lines written after the rotation should be collected seamlessly.

Actual Result

Occasionally, after a rotation with copytruncate, the collector supposedly stops collecting new data.
The debug logs show:

Reading lost file path=/var/log/app/app.log

But the collector does not emit new log entries from the file until it is restarted or reconfigured.
This behavior is not consistent and appears to depend on the timing between rotation and polling.

Collector version

v0.137.0

Environment information

Environment

OS: Ubuntu 22.04

OpenTelemetry Collector configuration

receivers:
  otlpjsonfile:
    include: [/var/log/app/app.log]

exporters:
  debug:
    verbosity: detailed

extensions: {}

service:
  extensions: []
  pipelines:
    logs:
      receivers: [otlpjsonfile]
      processors: []
      exporters: [debug]

Log output

{"level":"debug","ts":"2025-10-19T17:59:57.814Z","caller":"fileconsumer/file.go:157","msg":"Consuming files","resource":{"service.instance.id":"1f208ef0-43ad-40b1-843b-0450afa19318","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"otlpjsonfile/app","otelcol.component.kind":"receiver","otelcol.signal":"logs","component":"fileconsumer","paths":["/var/log/app/app.log"]}
{"level":"debug","ts":"2025-10-19T18:00:02.814Z","caller":"fileconsumer/file_other.go:57","msg":"Reading lost file","resource":{"service.instance.id":"1f208ef0-43ad-40b1-843b-0450afa19318","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"otlpjsonfile/app","otelcol.component.kind":"receiver","otelcol.signal":"logs","component":"fileconsumer","path":"/var/log/app/app.log"}
{"level":"debug","ts":"2025-10-19T18:00:04.614Z","caller":"fileconsumer/file.go:157","msg":"Consuming files","resource":{"service.instance.id":"1f208ef0-43ad-40b1-843b-0450afa19318","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"otlpjsonfile/app","otelcol.component.kind":"receiver","otelcol.signal":"logs","component":"fileconsumer","paths":["/var/log/app/app.log"]}
{"level":"debug","ts":"2025-10-19T18:00:06.014Z","caller":"fileconsumer/file.go:125","msg":"matched files","resource":{"service.instance.id":"1f208ef0-43ad-40b1-843b-0450afa19318","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"otlpjsonfile/app","otelcol.component.kind":"receiver","otelcol.signal":"logs","component":"fileconsumer","paths":["/var/log/app/app.log"]}
{"level":"debug","ts":"2025-10-19T18:00:06.215Z","caller":"fileconsumer/file.go:125","msg":"matched files","resource":{"service.instance.id":"1f208ef0-43ad-40b1-843b-0450afa19318","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"otlpjsonfile/app","otelcol.component.kind":"receiver","otelcol.signal":"logs","component":"fileconsumer","paths":["/var/log/app/app.log"]}
{"level":"debug","ts":"2025-10-19T18:00:06.414Z","caller":"fileconsumer/file.go:125","msg":"matched files","resource":{"service.instance.id":"1f208ef0-43ad-40b1-843b-0450afa19318","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"otlpjsonfile/app","otelcol.component.kind":"receiver","otelcol.signal":"logs","component":"fileconsumer","paths":["/var/log/app/app.log"]}
{"level":"debug","ts":"2025-10-19T18:00:07.414Z","caller":"fileconsumer/file.go:157","msg":"Consuming files","resource":{"service.instance.id":"1f208ef0-43ad-40b1-843b-0450afa19318","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"otlpjsonfile/app","otelcol.component.kind":"receiver","otelcol.signal":"logs","component":"fileconsumer","paths":["/var/log/app/app.log"]}
{"level":"debug","ts":"2025-10-19T18:00:07.614Z","caller":"fileconsumer/file.go:157","msg":"Consuming files","resource":{"service.instance.id":"1f208ef0-43ad-40b1-843b-0450afa19318","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"otlpjsonfile/app","otelcol.component.kind":"receiver","otelcol.signal":"logs","component":"fileconsumer","paths":["/var/log/app/app.log"]}
{"level":"debug","ts":"2025-10-19T18:00:07.814Z","caller":"fileconsumer/file.go:157","msg":"Consuming files","resource":{"service.instance.id":"1f208ef0-43ad-40b1-843b-0450afa19318","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"otlpjsonfile/app","otelcol.component.kind":"receiver","otelcol.signal":"logs","component":"fileconsumer","paths":["/var/log/app/app.log"]}
{"level":"debug","ts":"2025-10-19T18:00:16.013Z","caller":"fileconsumer/file.go:125","msg":"matched files","resource":{"service.instance.id":"1f208ef0-43ad-40b1-843b-0450afa19318","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"otlpjsonfile/app","otelcol.component.kind":"receiver","otelcol.signal":"logs","component":"fileconsumer","paths":["/var/log/app/app.log"]}
{"level":"debug","ts":"2025-10-19T18:00:16.213Z","caller":"fileconsumer/file.go:125","msg":"matched files","resource":{"service.instance.id":"1f208ef0-43ad-40b1-843b-0450afa19318","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"otlpjsonfile/app","otelcol.component.kind":"receiver","otelcol.signal":"logs","component":"fileconsumer","paths":["/var/log/app/app.log"]}
{"level":"debug","ts":"2025-10-19T18:00:16.413Z","caller":"fileconsumer/file.go:125","msg":"matched files","resource":{"service.instance.id":"1f208ef0-43ad-40b1-843b-0450afa19318","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"otlpjsonfile/app","otelcol.component.kind":"receiver","otelcol.signal":"logs","component":"fileconsumer","paths":["/var/log/app/app.log"]}

Additional context

Based on analysis and local testing, this issue, IMO, can occur because of how the collector manages fingerprints and file tracking during rotation events where both the pre-rotation and post-rotation files share the same fingerprint or when the new file is empty during the poll.

1. Duplicate fingerprint handling in makeReaders()
   

   opentelemetry-collector-contrib/pkg/stanza/fileconsumer/file.go

   Lines 215 to 223 in 4e7fcea

   	if r := m.tracker.GetCurrentFile(fp); r != nil {
   	m.set.Logger.Debug("Skipping duplicate file", zap.String("path", file.Name()))
   	// re-add the reader as Match() removes duplicates
   	m.tracker.Add(r)
   	if err := file.Close(); err != nil {
   	m.set.Logger.Debug("problem closing file", zap.Error(err))
   	}
   	continue
   	}

I think here if both files have the same fingerprint (as temporarily happens with copy-truncate), the new file can be skipped for that poll, leaving the collector attached to the old inode.

2. File matching order in readLostFiles()
   

   opentelemetry-collector-contrib/pkg/stanza/fileconsumer/file_other.go

   Lines 31 to 33 in 4e7fcea

   	if newReader.Fingerprint.StartsWith(oldReader.Fingerprint) {
   	continue OUTER
   	}

The fingerprint check happens before verifying the filename, which may prevent proper detection of the rotated file.

I think those conditions together can cause the collector to remain attached to the old (truncated) file and fail to reattach to the new active file.

Link to possible related previous issues/PRs:

• [pkg/stanza/fileconsumer] Compare the oldReader.offset and newReader.file.size when creating newReader #19428
• [fix] [pkg/stanza/fileconsumer] do not update fingerprint size when less data has been read #22937
• Fix Rotation file data lost in specified state #19766
• make new reader during the rotation and exclude the same file each cycle #19767

I'm ok with contributing to fix this, but I want first thoughts to ensure I'm thinking in the right direction 🙏

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners:

• pkg/stanza/fileconsumer: @andrzej-stencel

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[braydonk]

I'm trying to understand the exact issue, but I am getting mixed up by some terminology in the issue. Here's what I'm understanding:

old file = the copy that was made
active file = the file that was truncated

I would expect it not to connect to the old file, or the copy that was made of what was originally in the file, because copytruncate creates an entirely new file. Unless you configure the matcher to also match for the new copied file's name, there is no way for the collector to discover the new file. (I explain this in my talk at KubeCon Observability Day last year, it's in reference to Fluent Bit but explains the same principal).

I would expect collection to continue from the active file, i.e. the file that was truncated to zero. I would also expect it to have detected the rotation on account of the fingerprint of its file changing, but I would not have expected it to then skip that as a duplicate file even if the fingerprint ends up the same as before when it's re-evaluated. This does seem like a bug if that's what you're observing.

Is the latter the root of the bug? If it's the former then I think it's working as intended, if it's the latter then that's a bug we can investigate.

[paulojmdias]

I think you are understanding it correctly. The steps are like:

1. Logrotate copies the app.log to the app.log.1
2. Truncates the app.log

However, the collector, after step 2, stops reading any data from the app.log, but we continue seeing collector logs like the file exists and supposedly is being consumed.

{"level":"debug","ts":"2025-10-19T17:59:57.814Z","caller":"fileconsumer/file.go:157","msg":"Consuming files","resource":{"service.instance.id":"1f208ef0-43ad-40b1-843b-0450afa19318","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"otlpjsonfile/app","otelcol.component.kind":"receiver","otelcol.signal":"logs","component":"fileconsumer","paths":["/var/log/app/app.log"]}
{"level":"debug","ts":"2025-10-19T18:00:02.814Z","caller":"fileconsumer/file_other.go:57","msg":"Reading lost file","resource":{"service.instance.id":"1f208ef0-43ad-40b1-843b-0450afa19318","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"otlpjsonfile/app","otelcol.component.kind":"receiver","otelcol.signal":"logs","component":"fileconsumer","path":"/var/log/app/app.log"}
{"level":"debug","ts":"2025-10-19T18:00:04.614Z","caller":"fileconsumer/file.go:157","msg":"Consuming files","resource":{"service.instance.id":"1f208ef0-43ad-40b1-843b-0450afa19318","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"otlpjsonfile/app","otelcol.component.kind":"receiver","otelcol.signal":"logs","component":"fileconsumer","paths":["/var/log/app/app.log"]}
{"level":"debug","ts":"2025-10-19T18:00:06.014Z","caller":"fileconsumer/file.go:125","msg":"matched files","resource":{"service.instance.id":"1f208ef0-43ad-40b1-843b-0450afa19318","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"otlpjsonfile/app","otelcol.component.kind":"receiver","otelcol.signal":"logs","component":"fileconsumer","paths":["/var/log/app/app.log"]}
{"level":"debug","ts":"2025-10-19T18:00:06.215Z","caller":"fileconsumer/file.go:125","msg":"matched files","resource":{"service.instance.id":"1f208ef0-43ad-40b1-843b-0450afa19318","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"otlpjsonfile/app","otelcol.component.kind":"receiver","otelcol.signal":"logs","component":"fileconsumer","paths":["/var/log/app/app.log"]}
{"level":"debug","ts":"2025-10-19T18:00:06.414Z","caller":"fileconsumer/file.go:125","msg":"matched files","resource":{"service.instance.id":"1f208ef0-43ad-40b1-843b-0450afa19318","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"otlpjsonfile/app","otelcol.component.kind":"receiver","otelcol.signal":"logs","component":"fileconsumer","paths":["/var/log/app/app.log"]}
{"level":"debug","ts":"2025-10-19T18:00:07.414Z","caller":"fileconsumer/file.go:157","msg":"Consuming files","resource":{"service.instance.id":"1f208ef0-43ad-40b1-843b-0450afa19318","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"otlpjsonfile/app","otelcol.component.kind":"receiver","otelcol.signal":"logs","component":"fileconsumer","paths":["/var/log/app/app.log"]}
{"level":"debug","ts":"2025-10-19T18:00:07.614Z","caller":"fileconsumer/file.go:157","msg":"Consuming files","resource":{"service.instance.id":"1f208ef0-43ad-40b1-843b-0450afa19318","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"otlpjsonfile/app","otelcol.component.kind":"receiver","otelcol.signal":"logs","component":"fileconsumer","paths":["/var/log/app/app.log"]}
{"level":"debug","ts":"2025-10-19T18:00:07.814Z","caller":"fileconsumer/file.go:157","msg":"Consuming files","resource":{"service.instance.id":"1f208ef0-43ad-40b1-843b-0450afa19318","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"otlpjsonfile/app","otelcol.component.kind":"receiver","otelcol.signal":"logs","component":"fileconsumer","paths":["/var/log/app/app.log"]}
{"level":"debug","ts":"2025-10-19T18:00:16.013Z","caller":"fileconsumer/file.go:125","msg":"matched files","resource":{"service.instance.id":"1f208ef0-43ad-40b1-843b-0450afa19318","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"otlpjsonfile/app","otelcol.component.kind":"receiver","otelcol.signal":"logs","component":"fileconsumer","paths":["/var/log/app/app.log"]}
{"level":"debug","ts":"2025-10-19T18:00:16.213Z","caller":"fileconsumer/file.go:125","msg":"matched files","resource":{"service.instance.id":"1f208ef0-43ad-40b1-843b-0450afa19318","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"otlpjsonfile/app","otelcol.component.kind":"receiver","otelcol.signal":"logs","component":"fileconsumer","paths":["/var/log/app/app.log"]}
{"level":"debug","ts":"2025-10-19T18:00:16.413Z","caller":"fileconsumer/file.go:125","msg":"matched files","resource":{"service.instance.id":"1f208ef0-43ad-40b1-843b-0450afa19318","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"otlpjsonfile/app","otelcol.component.kind":"receiver","otelcol.signal":"logs","component":"fileconsumer","paths":["/var/log/app/app.log"]}

[braydonk]

Thanks for clarifying. I will try to look deeper later today or tomorrow and see if I can find anything. @andrzej-stencel let me know if you might have any ideas in advance of me taking a look.

[braydonk]

@paulojmdias can you expand a bit on what your testing setup looks like?

I.E by this:

Continuously write log lines to /var/log/app/app.log

Do you mean you have an app that is generating OTLP JSON log lines to this filepath? Is it a dummy generator that's writing the same lines over again or is it an actual app generating logs in a non-uniform way?

[paulojmdias]

Thank you @braydonk 🙏

I'm still trying to find a pattern to properly reproduce it to find a root cause. If I find something, I'll also let here for help.

@paulojmdias can you expand a bit on what your testing setup looks like?

I.E by this:

Continuously write log lines to /var/log/app/app.log

Do you mean you have an app that is generating OTLP JSON log lines to this filepath? Is it a dummy generator that's writing the same lines over again, or is it an actual app generating logs in a non-uniform way?

It is a Python app generating logs similar to what we have here. Gather from this snippet.

[paulojmdias]

Quick update about this issue debug. Yesterday, I enabled the file_storage extension on this input in multiple deployments, to try to get more useful data to identify the root cause.

The following config was used:

receivers:
  otlpjsonfile:
    include: [/var/log/app/app.log]

exporters:
  debug:
    verbosity: detailed

extensions:
  file_storage:
    directory: /var/lib/opentelemetry-collector

service:
  extensions: [file_storage]
  pipelines:
    logs:
      receivers: [otlpjsonfile]
      processors: []
      exporters: [debug]

So, when the rotation occurs, the collector stops exporting data, and nothing is logged (we have disabled the Debug log level on the collector).

cat /srv/opentelemetry-collector/receiver_otlpjsonfile

��
  ���������
           �{M���T��
�����3�             ���������
]knownFiles1
{"Fingerprint":{"first_bytes":"eyJyZXNvdXJjZV9sb2dzIjogW3sicmVzb3VyY2UiOiB7ImF0dHJpYnV0ZXMiOiBbeyJrZXkiOiAidGVsZW1ldHJ5LnNkay5sYW5ndWFnZSIsICJ2YWx1ZSI6IHsic3RyaW5nX3ZhbHVlIjogInB5dGhvbiJ9fSwgeyJrZXkiOiAidGVsZW1ldHJ5LnNkay5uYW1lIiwgInZhbHVlIjogeyJzdHJpbmdfdmFsdWUiOiAib3BlbnRlbGVtZXRyeSJ9fSwgeyJrZXkiOiAidGVsZW1ldHJ5LnNkay52ZXJzaW9uIiwgInZhbHVlIjogeyJzdHJpbmdfdmFsdWUiOiAiMS4zNy4wIn19LCB7ImtleSI6ICJzZXJ2aWNlLm5hbWUiLCAidmFsdWUiOiB7InN0cmluZ192YWx1ZSI6ICJhcHAifX0sIHsia2V5IjogInNlcnZpY2UudmVyc2lvbiIsICJ2YWx1ZSI6IHsic3RyaW5nX3ZhbHVlIjogIjguMzEuMCJ9fSwgeyJrZXkiOiAic3RhY2siLCAidmFsdWUiOiB7InN0cmluZ192YWx1ZSI6ICJvYnNlcnZhYmlsaXR5In19LCB7ImtleSI6ICJwcm92aWRlciIsICJ2YWx1ZSI6IHsic3RyaW5nX3ZhbHVlIjogInByb3ZpZGVyIn19LCB7ImtleSI6ICJqdXJpc2RpY3Rpb24iLCAidmFsdWUiOiB7InN0cmluZ192YWx1ZSI6ICJqdXIifX0sIHsia2V5IjogInJlZ2lvbiIsICJ2YWx1ZSI6IHsic3RyaW5nX3ZhbHVlIjogInJlZyJ9fSwgeyJrZXkiOiAic2VydmljZSIsICJ2YWx1ZSI6IHsic3RyaW5nX3ZhbHVlIjogImFwcCJ9fSwgeyJrZXkiOiAicm9sZSIsICJ2YWx1ZSI6IHsic3RyaW5nX3ZhbHVlIjogImdlbmVyaWMifX0sIHsia2V5IjogImluc3RhbmNlIiwgInZhbHVlIjogeyJzdHJpbmdfdmFsdWUiOiAiaW5zdGFuY2UtMDAwMDAxIn19LCB7ImtleSI6ICJzZXJ2aWNlLmluc3RhbmNlLmlkIiwgInZhbHVlIjogeyJzdHJpbmdfdmFsdWUiOiAiNjc1ZjRjZTM0NzYyNTRjM2IyYTI1ODNlOWVhYzY0ODIifX0sIHsia2V5IjogImFwX29ic2VydmFiaWxpdHlfc2RrLnZlcnNpb24iLCAidmFsdWUiOiB7InN0cmluZ192YWx1ZSI6ICI3LjUxLjAifX1dfSwgInNjb3BlX2xvZ3MiOiBbeyJzY29wZSI6IAo=="},"Offset":128867793,"RecordNum":2641,"FileAttributes":{"log.file.path":"/var/log/app/app.log"},"HeaderFinalized":false,"FlushState":{"LastDataChange":"2025-10-21T18:00:01.834609593Z","LastDataLength":0},"TokenLenState":{"MinimumLength":0},"FileType":""}
default
]knownFiles1
{"Fingerprint":{"first_bytes":"eyJyZXNvdXJjZV9sb2dzIjogW3sicmVzb3VyY2UiOiB7ImF0dHJpYnV0ZXMiOiBbeyJrZXkiOiAidGVsZW1ldHJ5LnNkay5sYW5ndWFnZSIsICJ2YWx1ZSI6IHsic3RyaW5nX3ZhbHVlIjogInB5dGhvbiJ9fSwgeyJrZXkiOiAidGVsZW1ldHJ5LnNkay5uYW1lIiwgInZhbHVlIjogeyJzdHJpbmdfdmFsdWUiOiAib3BlbnRlbGVtZXRyeSJ9fSwgeyJrZXkiOiAidGVsZW1ldHJ5LnNkay52ZXJzaW9uIiwgInZhbHVlIjogeyJzdHJpbmdfdmFsdWUiOiAiMS4zNy4wIn19LCB7ImtleSI6ICJzZXJ2aWNlLm5hbWUiLCAidmFsdWUiOiB7InN0cmluZ192YWx1ZSI6ICJhcHAifX0sIHsia2V5IjogInNlcnZpY2UudmVyc2lvbiIsICJ2YWx1ZSI6IHsic3RyaW5nX3ZhbHVlIjogIjguMzEuMCJ9fSwgeyJrZXkiOiAic3RhY2siLCAidmFsdWUiOiB7InN0cmluZ192YWx1ZSI6ICJvYnNlcnZhYmlsaXR5In19LCB7ImtleSI6ICJwcm92aWRlciIsICJ2YWx1ZSI6IHsic3RyaW5nX3ZhbHVlIjogInByb3ZpZGVyIn19LCB7ImtleSI6ICJqdXJpc2RpY3Rpb24iLCAidmFsdWUiOiB7InN0cmluZ192YWx1ZSI6ICJqdXIifX0sIHsia2V5IjogInJlZ2lvbiIsICJ2YWx1ZSI6IHsic3RyaW5nX3ZhbHVlIjogInJlZyJ9fSwgeyJrZXkiOiAic2VydmljZSIsICJ2YWx1ZSI6IHsic3RyaW5nX3ZhbHVlIjogImFwcCJ9fSwgeyJrZXkiOiAicm9sZSIsICJ2YWx1ZSI6IHsic3RyaW5nX3ZhbHVlIjogImdlbmVyaWMifX0sIHsia2V5IjogImluc3RhbmNlIiwgInZhbHVlIjogeyJzdHJpbmdfdmFsdWUiOiAiaW5zdGFuY2UtMDAwMDAxIn19LCB7ImtleSI6ICJzZXJ2aWNlLmluc3RhbmNlLmlkIiwgInZhbHVlIjogeyJzdHJpbmdfdmFsdWUiOiAiNjc1ZjRjZTM0NzYyNTRjM2IyYTI1ODNlOWVhYzY0ODIifX0sIHsia2V5IjogImFwX29ic2VydmFiaWxpdHlfc2RrLnZlcnNpb24iLCAidmFsdWUiOiB7InN0cmluZ192YWx1ZSI6ICI3LjUxLjAifX1dfSwgInNjb3BlX2xvZ3MiOiBbeyJzY29wZSI6IAo=="},"Offset":128867793,"RecordNum":2641,"FileAttributes":{"log.file.path":"/var/log/app/app.log"},"HeaderFinalized":false,"FlushState":{"LastDataChange":"2025-10-21T18:00:01.834609593Z","LastDataLength":0},"TokenLenState":{"MinimumLength":0},"FileType":""}
default

For trying to fix this, we tried the following options:

• Restart the collector
  • Did not take any effect as the collector will try to resume from previously known offset(s)

{"level":"info","ts":"2025-10-22T10:00:21.350Z","caller":"fileconsumer/file.go:62","msg":"Resuming from previously known offset(s). 'start_at' setting is not applicable.","resource":{"service.instance.id":"86650e1d-6d16-4ef7-8f5e-08eaccbc4b79","service.name":"otelcol-contrib","service.version":"0.137.0"},"otelcol.component.id":"otlpjsonfile","otelcol.component.kind":"receiver","otelcol.signal":"logs","component":"fileconsumer"}

• Delete the storage file rm -f /var/lib/opentelemetry-collector/receiver_otlpjsonfile and restart the collector
  • As expected, the collector starts to read the file
• Force logrotate to run again with logrotate -f
  • As expected, the collector starts to read the file

I'm not sure if it can be related to the first_bytes, which have the maximum value of 1000 bytes, and in this case, is there a possibility of the first 1000 bytes being always equal due to the log line structure, and can cause this issue?

[andrzej-stencel]

Thanks @paulojmdias for the report. The code is pretty complex and it's not impossible that a bug has sneaked in, even if the code was originally written to handle the copy-truncate scenario correctly.

A couple questions:

• Did this start to occur after a Collector upgrade? Was there a previous version that didn't exhibit this behavior?
• Are the logs in "Log output" sampled or filtered in some other way? The timings on those logs look suspicious. Can you make sure log sampling is disabled with service::telemetry::logs::sampling::enabled: false? Also make sure to include both INFO and DEBUG logs.

With regards to the finding in "Additional context":

1. Duplicate fingerprint handling in makeReaders()

   This part of code only applies when more than one file is matched. In your scenario there is only one file to track: /var/log/app/app.log, so I don't believe this code has anything to do with the behavior.

2. File matching order in readLostFiles()

   I'm not seeing a whole in this logic.

In summary:

To be able to pin down the issue, we need a minimal reproducible scenario, even if the reproduction rate is not 100%. The scenario must consist of the Collector config plus a detailed description of the process writing to the file watched by the Collector. Ideally a shell script, or a Python program, something that can be reliably run on anyone's machine that produces the problematic behavior of the Collector.

[paulojmdias]

• Did this start to occur after a Collector upgrade? Was there a previous version that didn't exhibit this behavior?

I cannot give an exact answer to that. I'm pretty sure this occurs at least from v0.128.0, but possibly it occurs on previous versions. We have started using the otlpjsonfile receiver around that version.

• Are the logs in "Log output" sampled or filtered in some other way? The timings on those logs look suspicious. Can you make sure log sampling is disabled with service::telemetry::logs::sampling::enabled: false? Also, make sure to include both INFO and DEBUG logs.

The logs are not sampled or filtered by the collector itself, I just try to share here only what can be useful for debugging. 🙏

To be able to pin down the issue, we need a minimal reproducible scenario, even if the reproduction rate is not 100%. The scenario must consist of the Collector config plus a detailed description of the process writing to the file watched by the Collector. Ideally a shell script, or a Python program, something that can be reliably run on anyone's machine that produces the problematic behavior of the Collector.

Sure, I created a scenario using a single container with a Python application that generates logs using the same class we want to use to reproduce the issue, and added dummy resource attributes to ensure the first 1000 bytes on each log line have the same fingerprint.

This is trying to simulate the same scenario as I'm facing the issue, which is in the same VM (here we will use containers) we have all the processes running together.

The code is on this repo https://github.com/paulojmdias/otlpjson-log-generator

Please let me know if this will help 🙏

[paulojmdias]

@andrzej-stencel @braydonk I reproduced the issue with the setup shared above.

I also think this issue seems to happen since the collector identifies files fingerprint by the first 1000 bytes of their content. When a log file is rotated using copytruncate, the new file can sometimes start with the same bytes as the previous one.

In this case, the fingerprint matches and the stored offset is higher than the new file size, the collector wrongly assumes it is the same file and stops reading.

I tested locally by increasing the fingerprint size like to 4096, the issue is not happening with, but will only makes this less likely and does not solve it completely.

IMO, a safer fix would be check the identifiers like inode, device and/or even mtime to confirm the file identity for compute the fingerprint.

I will open a PR tomorrow for this discussion.