Joining Fetch

[englishm]

This PR adds the desired "Join" functionality as discussed at IETF 121 in Dublin¹

Adding this functionality to Fetch unblocks further simplifications of Subscribe, allowing us to more clearly delineate between "past" (Fetch) and "future" (Subscribe). (see #598, #510 (review), etc.)

There are several ways we could allow a publisher to "atomically" join a Fetch and a Subscribe.
At IETF 121, @wilaw presented slides showing API options:

1. a new macro-like singular JOIN message that is decomposed at the publisher into both a SUBSCRIBE and a FETCH
2. a modified form of FETCH that can be joined together with an existing Subscribe

The consensus of the discussion seemed to heavily favor the latter design, so, as was requested, this PR is written as a modified form of Fetch: Joining Fetch.

This PR:

• Adds "Fetch Type" field to Fetch messages
• Redefines pre-existing behavior as a Fetch Type: "Standalone Fetch"
• Defines new Fetch Type: "Joining Fetch"
• Defines new error code: "Invalid Subscribe ID"

Footnotes

1. 21:18 in the recording. Note: the Meetecho page includes the chat, synced to the video. ↩

[englishm]

(Fixed some capitalization inconsistencies)

[afrind]

Individual Review:

Thanks for this. I think it's an improvement over "Current Group" in subscribe.

[englishm]

Individual Review:

Thanks for this. I think it's an improvement over "Current Group" in subscribe.

Thanks for the quick feedback @afrind!

I'll fix up the definitions for how to resolve the range relative to the Subscribe so the terminology matches the current draft text and the details are more clear.

[wilaw]

Looking good Mike. I added a few points for clarification.

[wilaw]

Are there limits on this value? Is zero allowed? Is 1000 allowed? Is -1 allowed?

[englishm]

Consolidating discussion of this point.

For simplicity, we should disallow PGC = 0. It essentially overlaps the FETCH with the SUBSCRIBE. Forcing PGC >= 1 seems cleaner.

I'm in favor of allowing "Preceding Group Offset" = 0 so that we can have the option of using Fetch to retrieve the first portion of a group when Subscribe(LatestObject) is used to retrieve the rest. The updated text hopefully makes it clear that the Fetch and Subscribe do not overlap.

Is 1000 allowed?

Yes. Depending on how group ID are set for a given track this may make sense.

That said, it is not necessarily advisable to use a Joining Fetch to Fetch a large number of groups. In that case it may be better to wait for Subscribe OK and then make a series of smaller Fetch requests to fill the desired range.

I'm not opposed to putting some kind of limit on this value, but it's hard to say what it ought to be for all circumstances. I think it may be better to just allow publishers to reject joining fetch requests with PGOs larger than they wish to fulfill by responding with a Fetch Error. (I just created a separate issue about defining more error codes: #644)

Is -1 allowed?

No. This field is defined as [Preceding Group Offset (i),] and this Variable-Length Integer Encoding does not permit negative values.

[afrind]

Individual Review

[fluffy]

I think we need to fix up the "what happens" when no content case but I like the overall PR here and seems to meet the use cases.

[englishm]

@ianswett let me know if there are any other editorial changes you'd like to see here.

[afrind]

Individual Review

[afrind]

I think we need to resolve what's going on with Subscribe(LatestGroup). Options:

1. Remove it (it splits past and future)
2. Subscribe(LatestGroup) is identical to Subscribe(LatestObject) + Joining Fetch(PGO=0) -- it sends a Subscribe Ok, a FetchOK, a Fetch stream and any future subgroups.
3. Special rules for publishers/relays that definitively have the entire current group cached

I guess it can be in another PR.

[englishm]

I agree. I've tried to write this PR in a way where 1 is a viable option, but I think that actually modifying Subscribe should be done in a separate PR.

[afrind]

I agree. I'll wait for this to land since I don't think it will impact the decision and then open a separate issue.

[englishm]

@ianswett if we pursue 1 here, does Joining Fetch seem less awkward?

[afrind]

Individual Comments

[afrind]

I agree. I'll wait for this to land since I don't think it will impact the decision and then open a separate issue.

[ianswett]

Thanks for the update to make it only apply to Latest Object subscriptions, this is MUCH simpler IMO.

[ianswett]

Nit: I don't think you need a MUST here. This is just explaining how the mechanism works and why.

Suggested change

	The Resolved Subscribe Start values for a Joining Fetch MUST correspond to the
	The Resolved Subscribe Start values for a Joining Fetch correspond to the

[englishm]

This is actually a requirement, but I see how it's easy to read it as just descriptive. The important part is the "within the same session" which precludes responding to Subscribe at relay, but forwarding the Joining Fetch and having it resolved against a different Subscribe at an upstream relay. We don't necessarily want to entirely preclude forwarding upstream, but it's important that the Joining Fetch be resolved within the same session as the associated Subscribe so that this join point is consistent.

This text originated after the 2024-12-18 weekly interim where the main issue raised (mainly by Cullen and Alan, if I recall correctly) was a desire to clarify requirements relating to forwarding behavior. My initial stab at text for that was in this commit: 9dfceaf followed by some discussion tightening it up here: #638 (comment)

If this is too subtle, we could expand it a bit again or rephrase it to make the requirement more clear, but the MUST here is intentional.

[ianswett]

I think everything in this document is in the context of a single session, except for some advice on relays, so I thought that aspect was a bit out of place. For example, SUBSCRIBE_UPDATE and SUBSCRIBE_CANCEL don't clarify that you're altering a subscription within the session because it's implied.

[fluffy]

The more I think about the joining case where the Fetch and associated Subscribe share the same subscribe ID seems like it might cause problems for us later. Say we added an extension that did something that worked on a Fetch or Subscribe, there would be know way to know what it referred to.

I think it would be better to have joining fetch have a new ID and in the request parameters cary the ID of the associated subscribe.

[ianswett]

I do think it's less confusing to have a separate ID for the Fetch, since it makes some of the questions I previously had (ie: If I do SUBSCRIBE_CANCEL, does it also cancel the Fetch with the same ID?) obvious.

[englishm]

I think we do still need to handle this zero case separately.

e.g. Subscribe Largest Group = 3, Subscribe Largest Object = 0, Preceding Group Offset = 2

The new text here would have us create a Fetch with:

• StartGroup = 1
• StartObject = 0
• EndGroup = 3
• EndObject = 0

This would overlap an entire group since an EndObject of 0 in a Fetch means "all objects in the group"

I think the rest of the text consolidation here is fine, but we do need to keep this - 1 for the Fetch EndGroup when Subscribe Largest Object is 0.

[ianswett]

Thanks Mike, I added a paragraph to address this and try to briefly explain why.