Clarify Latest Object Filter #853

martinduke · 2025-03-24T17:48:45Z

Fixes #758

Since the current Local Object definition is ambiguous, I eliminated it and replaced it with two unambiguous definition.

I am open-minded about naming.

draft-ietf-moq-transport.md

afrind · 2025-03-24T18:14:04Z

draft-ietf-moq-transport.md

+SUBSCRIBE_OK, whether or not the ID is higher. This is most useful for tracks
+where the Group ID does not imply a temporal relationship.


Same comment on utility - though another key difference is that this filter might be useful when there is re-ordering (eg: if the relay has seen 2-2 because 2-1 was temporarily delayed, you will still get 2-1)

See above. In the additional case you suggest, using this filter is very janky compared to Joining Fetch -- I wouldn't recommend it in this case.

I also changed it to make it clear that stuff that arrives by Upstream FETCH is not included in this filter.

afrind · 2025-03-24T18:14:44Z

You may need to scan the document for places where Latest Object is referenced. I think at least Joining Fetch mentions it.

wilaw · 2025-03-24T19:41:47Z

This still doesn't address the case where the relay has received a fraction of an Object (i.e at least the first byte but not the last byte). What does it return as Largest Object in the SUBSCRIBE_OK? Does it return the fractional Object, or the last complete Object it received? I feel we should make this clear for relay implementers.

afrind · 2025-03-24T19:43:19Z

This still doesn't address the case where the relay has received a fraction of an Object (i.e at least the first byte but not the last byte). What does it return as Largest Object in the SUBSCRIBE_OK? Does it return the fractional Object, or the last complete Object it received? I feel we should make this clear for relay implementers.

This is tracked in your issue #592 and we'll make a separate PR soon.

martinduke · 2025-03-24T21:08:10Z

This still doesn't address the case where the relay has received a fraction of an Object (i.e at least the first byte but not the last byte). What does it return as Largest Object in the SUBSCRIBE_OK? Does it return the fractional Object, or the last complete Object it received? I feel we should make this clear for relay implementers.

This is tracked in your issue #592 and we'll make a separate PR soon.

Indeed, this PR references the Largest Group/Object fields in SUBSCRIBE_OK, so I prefer that the in-progress issue be addressed there.

martinduke · 2025-03-24T21:56:31Z

You may need to scan the document for places where Latest Object is referenced. I think at least Joining Fetch mentions it.

Done

afrind · 2025-03-24T22:19:29Z

draft-ietf-moq-transport.md

+All Objects (0x6): All objects that arrive via upstream SUBSCRIBE at, or are
+created by, the publisher after the object indicated by the Largest Group ID and
+Largest Object ID fields in the SUBSCRIBE_OK, whether or not the Location is
+higher.  This is most useful for tracks where the Group ID does not imply a


Maybe change it to:

All objects that arrive via upstream SUBSCRIBE at, or are created by, the publisher after processing the SUBSCRIBE, whether or not the Location is higher.

And I still assert All Objects has some utility in applications even with temporal groups.

the Location is higher than what? I added another clause.

I don't see the use case. But you're the editor, feel free to add more text.

fluffy

This is not at all a clarification. This is a fundamental chance to core part of the draft and I would think the chairs are failing to acknowledge we had consensus on that.

I think you should redo this PR as a new filter time based on higher object but also keep the latest object and make it clear that is is object that arive after this point in time are forwarded on the subscribe.

martinduke · 2025-03-25T01:15:34Z

This is not at all a clarification. This is a fundamental chance to core part of the draft and I would think the chairs are failing to acknowledge we had consensus on that.

I think you should redo this PR as a new filter time based on higher object but also keep the latest object and make it clear that is is object that arive after this point in time are forwarded on the subscribe.

As we discussed in Denver, you and I read the same text and drew radically different conclusions as to what it meant. To me, that requires clarification.

Is the "AllObjects" filter not describing the behavior you want? Or is just that you want to retain the LatestObject name?

afrind · 2025-03-25T01:54:55Z

This is not at all a clarification. This is a fundamental chance to core part of the draft and I would think the chairs are failing to acknowledge we had consensus on that.

Different people in the working group were looking at the same text (for LatestObject) and interpreting it differently. If anything, the current draft supports the "Higher" interpretation of Latest Object.

Latest Object (0x2): Specifies an open-ended subscription beginning from the current object of the current group.

Each new Object belonging to the track within the subscription range is forwarded to each active subscriber

If a subscriber wants to subscribe to Objects both before and after the Latest Object, it can send a SUBSCRIBE for the Latest Object followed by a FETCH

I don't see anything in this PR that is a fundamental change. The goal is to end up with two different filter types that each have an unambiguous meaning. I'm not sure why the name or code point matters as long as applications have a mechanism to get the behavior they want.

suhasHere · 2025-03-25T12:21:17Z

draft-ietf-moq-transport.md

+created by, the publisher after processing the SUBSCRIBE, whether or not the
+Location is higher than the highest observed when the SUBSCRIBE is processed.
+This is most useful for tracks where the Group ID does not imply a temporal
+relationship.


Subscription filters should be agnostic of group structure. This seems to be adding application semantics and feels fragile to me.

I am bit unsure on how this filter is addressing the original linked issue

The original issue is that people read the LatestObject definition and had two different interpretations. Both interpretations serve valid use cases. So now there are two precise filter definitions, one for each use case.

draft-ietf-moq-transport.md

suhasHere · 2025-03-25T12:24:50Z

draft-ietf-moq-transport.md

+Higher Objects (0x5): Specifies an open-ended subscription. All objects with
+a Location greater than the Largest Group ID and Latest Object ID fields in the
+SUBSCRIBE_OK will be delivered by the publisher. This is most useful where the
+Group ID implies a temporal relationship.


I am not comfortable of adding group relationship in these filters.

I think it's useful to provide a hint to the reader on why you'd want to use each of these, but I won't lie down in the road over that sentence.

suhasHere

I see the intent for resolving Will's original issue on clarifying the interpretation. But I think this PR is doing more than just that. Wondering if we can split into 2 PRs , if needed.

I don't see a reason to express group structures in the filters though.

martinduke · 2025-03-25T16:29:51Z

I see the intent for resolving Will's original issue on clarifying the interpretation. But I think this PR is doing more than just that. Wondering if we can split into 2 PRs , if needed.

I don't see a reason to express group structures in the filters though.

Which use case should I delete from the PR? I think Higher Objects is critical to keep.

afrind · 2025-03-25T18:12:06Z

@suhasHere

I am not comfortable of adding group relationship in these filters.

I don't understand the discomfort. LatestObject already has a group relationship:

Latest Object (0x2): Specifies an open-ended subscription beginning from the current object of the current group.

It sounds like you want a filter that just returns any object that arrives or is created after subscribe. That is the purpose of the "All Objects" filter. Or do you want something else?

I see the intent for resolving Will's original issue on clarifying the interpretation. But I think this PR is doing more than just that. Wondering if we can split into 2 PRs , if needed.

This isn't intended to resolve Will's issue at all. This is addressing #758 (LatestObject subscribe filter is ambiguous).

I don't see a reason to express group structures in the filters though.

Have a read through the linked issue and let us know if you have a concrete suggestion to move forward. I think this PR is in the right direction, since it offers both behaviors that have been requested by the working group.

suhasHere · 2025-03-26T12:29:01Z

I don't understand the discomfort. LatestObject already has a group relationship:

I was referring to the sentence that was indicating if a group implies a temporal relationship or not. I don't think that matters for filters.

suhasHere · 2025-03-26T12:43:31Z

Have a read through the linked issue and let us know if you have a concrete suggestion to move forward. I think this PR is in the right direction, since it offers both behaviors that have been requested by the working group.

I did comment on the original issue where out of order groups can still happen even when groups are temporal and sequential. The AllObjects filter will not address the flow IIUC.

I think this fix needs to address that flow and perhaps clarify:

what does it really mean to include Group/Object info in Subscribe Ok
Does those are just hints and receivers should handle if things on wire doesn't exactly match (esp when Dsc order was set by the original pub or upstream relay)

martinduke · 2025-03-26T14:55:52Z

Have a read through the linked issue and let us know if you have a concrete suggestion to move forward. I think this PR is in the right direction, since it offers both behaviors that have been requested by the working group.

I did comment on the original issue where out of order groups can still happen even when groups are temporal and sequential. The AllObjects filter will not address the flow IIUC.

I think this fix needs to address that flow and perhaps clarify:

what does it really mean to include Group/Object info in Subscribe Ok

Does those are just hints and receivers should handle if things on wire doesn't exactly match (esp when Dsc order was set by the original pub or upstream relay)

The fact that objects can arrive out of order does not, in any way, imply that receivers should expect senders to forward objects that are outside the subscription range defined by the SUBSCRIBE_OK.

Neither does Descending Order.

Maybe you want that use case, and that's fine. That's what the AnyObjects filter is for.

afrind · 2025-03-26T16:09:07Z

I was referring to the sentence that was indicating if a group implies a temporal relationship or not. I don't think that matters for filters.

+1 to that. We can just describe the properties of how the filters work.

what does it really mean to include Group/Object info in Subscribe Ok

I do think this clarification is useful.

For Highest - this is the exclusive start of your range: you will only receive objects with Location > than the value in OK
For all other filter types - this is informational about the state of the relay and has no bearing on what you will receive

suhasHere · 2025-03-26T16:32:50Z

The fact that objects can arrive out of order does not, in any way, imply that receivers should expect senders to forward objects that are outside the subscription range defined by the SUBSCRIBE_OK.

Neither does Descending Order.

The problem here is group is arriving out of order. If we can get rid of descending order for Subscriber, almost all the confusion here is gone IIUC.

suhasHere · 2025-03-26T16:36:23Z

For Highest - this is the exclusive start of your range: you will only receive objects with Location > than the value in OK

I am still not sure how would this satisfy descending group order issue . How would a end subscriber even know to use Highest vs AllObjects when the groups are all sequential but publisher is sending it in dsc order. Are we saying that , if one asks for Highest and if the groups follow descending order, then the end subscribers should expect not to receive any objects from groups that are behind the one listed in the SubscribeOk. If so, then probably we need to add a note to that extent for dsc group order delivery.

martinduke · 2025-03-26T16:38:18Z

The fact that objects can arrive out of order does not, in any way, imply that receivers should expect senders to forward objects that are outside the subscription range defined by the SUBSCRIBE_OK.
Neither does Descending Order.

The problem here is group is arriving out of order. If we can get rid of descending order for Subscriber, almost all the confusion here is gone IIUC.

Descending order is irrelevant to this discussion. The issue is what filter the sender applies to group/object IDs for a subscription.

We all agree, I hope, that for Absolute Range the sender does not send objects < Start, whether the order is Ascending or Descending, and whether or not objects arrive out of order.

The HigherObjects filter is one interpretation of the current LatestObject text. A HigherObject filter is equivalent to an AbsoluteStart filter with a start value of (largest group, largest_id + 1) and behaves in exactly the same way.

An AllObjects filter conforms to the other interpretation. There is no limit on forwarded IDs.

martinduke · 2025-03-26T16:51:21Z

For Highest - this is the exclusive start of your range: you will only receive objects with Location > than the value in OK

I am still not sure how would this satisfy descending group order issue . How would a end subscriber even know to use Highest vs AllObjects when the groups are all sequential but publisher is sending it in dsc order. Are we saying that , if one asks for Highest and if the groups follow descending order, then the end subscribers should expect not to receive any objects from groups that are behind the one listed in the SubscribeOk. If so, then probably we need to add a note to that extent for dsc group order delivery.

It was well-understood by many people who read the LatestObject text that the group in the Subscribe OK was the left limit of what you would receive, whether descending or ascending. The point of descending is not to get stuff deep in the past, it is to get the highest Group IDs first. For descending, old groups are especially irrelevant, so I must not understand the nature of this objection.

Moreover, you are asking for a clarifying note. That is the entire purpose of this PR!

For Highest - this is the exclusive start of your range: you will only receive objects with Location > than the value in OK

I am still not sure how would this satisfy descending group order issue . How would a end subscriber even know to use Highest vs AllObjects when the groups are all sequential but publisher is sending it in dsc order. Are we saying that , if one asks for Highest and if the groups follow descending order, then the end subscribers should expect not to receive any objects from groups that are behind the one listed in the SubscribeOk. If so, then probably we need to add a note to that extent for dsc group order delivery.

Suhas -- can you please explain exactly what breaks with Descending order? To me it's quite straightforward that you dont't deliver things before the start of the filter. Descending is meant to prioritize the latest groups.

afrind · 2025-03-28T13:53:22Z

Is All Objects the same as AbsoluteStart={0,0} ?

martinduke · 2025-03-28T14:23:16Z

Is All Objects the same as AbsoluteStart={0,0} ?

Sort of? Except that everything except AllObjects uses Location is a proxy for time. AllObjects uses actual time of arrival. So as long as a (0,0) filter is only applied to new arriving objects, then yes.

fluffy

This makes breaking changes to Latest Object. I disagree that was ever ambiguous and certainly don't see a reason to make breaking changes to it. Can you separate this into two PR. One that proposes the new higher objects filter and another that clarifies the latest object.

If a publisher is publishing object 1,2,3,4,5,6, and a relay has already received objects 1,2,3, and then the subscribe what whatever the name of latest object/all objects is, I want the subscribe to get 4,5,6 and not get 1,2,3 which is why I think all objects is a bad name as that would imply it would get 1,2,3 as well as 4,5,6. SO Like the name latest object more than higher objet.

fluffy · 2025-04-03T22:25:25Z

draft-ietf-moq-transport.md

+the 'Higher Objects' filter.
+
+Higher Objects (0x5): Specifies an open-ended subscription. All objects with
+a Location greater than the Largest Group ID and Latest Object ID fields in the


I think this is the wrong definition and will result in really hard to resolve race conditions. I think you should take it off the highest group / object the relay has received, not what was in the Subscribe OK. I'm speaking from implementation experience here. Think hard about the case of a non cacheing relay that got the object refereed to the the subscribe OK multiple object before it got the OK

Would that actually work, given that joining FETCH relies on the SUBSCRIBE_OK-provided values?

afrind · 2025-04-14T14:30:44Z

Is All Objects the same as AbsoluteStart={0,0} ?

I created #927 as an alternate to this PR, along these lines.

#927) This is an alternate to #853, attempting to resolve #758. Fixes: #758 Closes #726 --------- Co-authored-by: ianswett <[email protected]> Co-authored-by: Suhas Nandakumar <[email protected]>

afrind · 2025-04-19T15:11:12Z

Addressed by #927

martinduke added 2 commits March 24, 2025 17:44

Clarify Latest Object

a15ceff

typos

2fea33f

martinduke requested review from kixelated, suhasHere, vasilvv and ianswett as code owners March 24, 2025 17:48

martinduke changed the title ~~Sub filter~~ Clarify Latest Object Filter Mar 24, 2025

martinduke requested review from fluffy and afrind and removed request for vasilvv, kixelated and suhasHere March 24, 2025 17:49

afrind reviewed Mar 24, 2025

View reviewed changes

afrind mentioned this pull request Mar 24, 2025

Clarification on what current object means in SUBSCRIBE, SUBSCRIBE_OK and FETCH #592

Open

martinduke added 2 commits March 24, 2025 21:12

alan comments

93b1ecc

typo

c31f2f6

afrind reviewed Mar 24, 2025

View reviewed changes

Alan comments

bfd0911

afrind added the Design Issues or PRs that change how MoQ works including the wire format. label Mar 24, 2025

fluffy suggested changes Mar 25, 2025

View reviewed changes

suhasHere reviewed Mar 25, 2025

View reviewed changes

draft-ietf-moq-transport.md Show resolved Hide resolved

suhasHere reviewed Mar 25, 2025

View reviewed changes

martinduke and others added 2 commits March 27, 2025 14:35

Eliminated hints on use

46f777a

Merge branch 'main' into sub-filter

d626e11

afrind mentioned this pull request Mar 28, 2025

Make SUBSCRIBE filters into optional parameters #879

Open

afrind added the Needs Discussion Tags for issues which need discussion, ideally at an interim or IETF meeting. label Mar 31, 2025

fluffy suggested changes Apr 3, 2025

View reviewed changes

fluffy reviewed Apr 3, 2025

View reviewed changes

afrind mentioned this pull request Apr 14, 2025

Clarify how SUBSCRIBE filters work, allow Absolute start before Latest #927

Merged

afrind closed this Apr 19, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clarify Latest Object Filter #853

Clarify Latest Object Filter #853

martinduke commented Mar 24, 2025

afrind Mar 24, 2025

martinduke Mar 24, 2025

afrind commented Mar 24, 2025

wilaw commented Mar 24, 2025

afrind commented Mar 24, 2025

martinduke commented Mar 24, 2025

martinduke commented Mar 24, 2025

afrind Mar 24, 2025

martinduke Mar 24, 2025

fluffy left a comment

martinduke commented Mar 25, 2025

afrind commented Mar 25, 2025

suhasHere Mar 25, 2025

martinduke Mar 25, 2025

suhasHere Mar 25, 2025

martinduke Mar 25, 2025

suhasHere left a comment

martinduke commented Mar 25, 2025

afrind commented Mar 25, 2025

suhasHere commented Mar 26, 2025

suhasHere commented Mar 26, 2025

martinduke commented Mar 26, 2025

afrind commented Mar 26, 2025

suhasHere commented Mar 26, 2025

suhasHere commented Mar 26, 2025

martinduke commented Mar 26, 2025

martinduke commented Mar 26, 2025

afrind commented Mar 28, 2025

martinduke commented Mar 28, 2025

fluffy left a comment

fluffy Apr 3, 2025

vasilvv Apr 7, 2025

afrind commented Apr 14, 2025

afrind commented Apr 19, 2025

		SUBSCRIBE_OK, whether or not the ID is higher. This is most useful for tracks
		where the Group ID does not imply a temporal relationship.

Clarify Latest Object Filter #853

Clarify Latest Object Filter #853

Conversation

martinduke commented Mar 24, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

afrind commented Mar 24, 2025

wilaw commented Mar 24, 2025

afrind commented Mar 24, 2025

martinduke commented Mar 24, 2025

martinduke commented Mar 24, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fluffy left a comment

Choose a reason for hiding this comment

martinduke commented Mar 25, 2025

afrind commented Mar 25, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

suhasHere left a comment

Choose a reason for hiding this comment

martinduke commented Mar 25, 2025

afrind commented Mar 25, 2025

suhasHere commented Mar 26, 2025

suhasHere commented Mar 26, 2025

martinduke commented Mar 26, 2025

afrind commented Mar 26, 2025

suhasHere commented Mar 26, 2025

suhasHere commented Mar 26, 2025

martinduke commented Mar 26, 2025

martinduke commented Mar 26, 2025

afrind commented Mar 28, 2025

martinduke commented Mar 28, 2025

fluffy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

afrind commented Apr 14, 2025

afrind commented Apr 19, 2025