Allow advertisement or entries chain truncation #132
Conversation
When traversing an advertisement chain or entries chain, handle a NotFound error as an indication that the chain was intentionally truncated, and ingest the portion of the chain that was found.
| // found. Consider this to mean the chain was truncated at this point, | ||
| // and return what was found so far. | ||
| if errors.Is(err, ipld.ErrNotExists{}) { | ||
| log.Warnw("stopping ipld traversal due to content not found") |
There was a problem hiding this comment.
it would be great to record these and be able to occasionally re-try?
it's easy to imagine mis-configured provider web servers that could get into states where a 404 is returned when the data is only temporarily unavailable.
There was a problem hiding this comment.
Right, this is certainly a desirable.
I imagine the bulk of the complexity in facilitating this is the semantics of retry: persisting the state of retriable CIDs while not blocking the ad ingestion. That would need to have some upper limit per provider per retry.
But first, a quick sanity check: @gammazero do we surface this error to providers at /providers endpoint?
There was a problem hiding this comment.
This is still an error if the first CID asked for is not found. It is only considered intentional if, at some point in chain traversal, a NotFound is returned. So, if a web server is misconfigured, then the first requested CID should result in error. So, I think it is safe to assume that a NotFound, if it occurs at a later point in traversal, is intentional.
|
Related to this PR: ipni/specs#25 |
|
Changing this to draft until there is a decision on ipni/specs#25 |
When traversing an advertisement chain or entries chain, handle a NotFound error as an indication that the chain was intentionally truncated, and ingest the portion of the chain that was found.