Files marked .bad after "Too many open files" read error

[rdingwall]

The current diskqueue implementation does not discriminate between different types of errors returned by os.OpenFile(). We experienced a problem in production where an nsqd host ran out of file descriptors and os.OpenFile() returned a "Too many open files" error, so a number of NSQ diskqueue files were incorrectly marked as .bad, requiring manual repair (by renaming them and updating the cursor position in the metadata files). It would be good if NSQ was able to discriminate between transient "try again" read errors vs other more serious errors.

I don't have a PR to fix unfortunately, I just thought I would flag up.

[mreiferson]

Hi, thanks for the feedback.

I'm not sure what you would expect the behavior to be in that case. The only possible behavior I can think of that would avoid needing to manually repair files would be if nsqd refused to proceed until it was restarted — is that desirable vs. current behavior?

[rdingwall]

If nsqd encountered EMFILE or ENFILE on a diskqueue file, I would expect the behaviour to be something along the lines of:

• Client publishes a message: return an error to the client
• New client subscribing: return an error to the client
• Existing client subscription: either graceful recovery (subscription goes quiet until the file is readable again) or terminate the subscription and let the client reconnect

I think?

[mreiferson]

Unfortunately, given nsqd's implementation (memory and disk) this is a little more complicated:

Client publishes a message: return an error to the client

Not all messages reach disk and so it would only make sense to error when a message "overflowed" to disk — which is basically how nsqd currently behaves.

New client subscribing: return an error to the client

Existing client subscription: either graceful recovery (subscription goes quiet until the file is readable again) or terminate the subscription and let the client reconnect

As per above, since there are commonly cases where most messages don't reach disk, it doesn't make sense to penalize all clients of the topic.

If all messages went to disk, e.g. if nsqd moved to a "log" style architecture, it would be far easier to make decisions about what appropriate behavior is for situations like this.