RFC: Generic Filters for Subscriptions and FETCH #1164

afrind · 2025-08-20T17:53:53Z

Based on proposal in #1068

Fixes: #441

One thing that is strange is allowing Group ID filters here, since these overlap with the built-in filters in Subscribe and Fetch. This may be what Victor was suggesting in Stockholm.

The pathological case of this encoding is for every other object, which would double the length of the Values array. We could fix it by shifting the start value left by one and using the LSB to indicate single vs range, but it seemed like overkill.

Based on proposal in #1068 One thing that is strange is allowing Group ID filters here, since these overlap with the built-in filters in Subscribe and Fetch. This may be what Victor was suggesting in Stockholm. The pathological case of this encoding is for every other object, which would double the length of the Values array. We could fix it by shifting the start value left by one and using the LSB to indicate single vs range, but it seemed like overkill.

draft-ietf-moq-transport.md

suhasHere · 2025-08-20T23:27:28Z

draft-ietf-moq-transport.md

+#### OBJECT FILTER Parameter
+
+The OBJECT_FILTER parameter(Parameter Type 0x05) MAY appear in SUBSCRIBE,
+SUBSCRIBE_UPDATE, PUBLISH_OK, TRACK_STATUS or FETCH message.  It is a structure


how about adding this to SubscribeNamespace as well ?

It's unclear what adding a filter to SUBSCRIBE_NAMESPACE would do without #1047? Or you have different use cases in mind? Let's keep this focused on subscriptions.

adding it to namespace will enable object filtering across the tracks, where the base case is a single subscription . I was suggesting it would be nice to think the filter more generically

@suhasHere I am amenable to using the same parameter to filter Tracks in response to SUBSCRIBE_NAMESPACE rather than Objects in SUBSCRIBE, but it requires more semantics (for example, most operands defined here don't make any sense). Can you file an issue explaining the requirements for Track filtering first?

suhasHere · 2025-08-20T23:28:43Z

draft-ietf-moq-transport.md

+Object Filter {
+  Type (0x5),
+  Length (i),
+  Operand and Flag (i),


Suggested change

Operand and Flag (i),

Operand and Operator (i),

Where Operator can be one !-, =, >, < . I think this will cover a good set of use-cases.

All of those can be express via range filters + negation:

= -> Equal != -> Negate + Equal > -> Range with no end < -> Negate + Range with no end

It's also unclear what < > mean with multiple ranges.

I realized < X is also trivially implemented as a range from [0,X]

fluffy

I can see how this might make sense for fetch but it seems to really complicate subscribe and I don't see the use cases it solves. To be specific, I can not imagine a subscriber where I want groups 100 to 200, then skip group 201 to 299, then give me groups 300 to 400.

This is a fairly significant breaking change and I think we should talk more about what we are trying to fix before getting into this. I do see some value in writing down a more generic and normalize filter that provides more capabilities as long as it does not complicate things so we may end up at exactly this but I'd like to consider the goals more fist.

draft-ietf-moq-transport.md

fluffy · 2025-08-21T13:38:53Z

draft-ietf-moq-transport.md

+Operand No Filter.
+
+This parameter MAY appear more than once, and filters are cumulative.  When
+filters are applied, delivery rules regarding Subgroups and FETCH responses are


I don't want to deal with an unlimited number of filters operations. How about a max of 4 unless someone can come up with a very good use case for more. Keep in mind I would like to be able to implement this on something like a FPGA or silicon for wire speed filtering.

A max seems very sensible.

fluffy · 2025-08-21T13:43:06Z

draft-ietf-moq-transport.md

+filters are applied, delivery rules regarding Subgroups and FETCH responses are
+modified.  Within a Subgroup, the Publisher is allowed to send an Object on a
+Subgroup stream even when it is not the next Object if the expected Objects
+did not pass the filter.  In a FETCH response, the Subscriber can only infer


I might be reading what you are saying here wrong but ...

If you mean you can send stuff that does not pass the filter, then no, I don't think this should be allowed. I think the stream should be ended if this happens on reliable stream and the things should just be filtered if on datagram.

If you mean that if the filter remove object N but object N+1 is the subgroup passes the filter, you can send N+1 even though N is missing ... then I need to think about what this does to caches downstream.

I am not saying you should send stuff that doesn't pass the filter. I was thinking the latter. For example f you have a subgroup with objects 0-9, and a filter that says only even objects, then you can put [ 0, 2, 4, 5, 8 ] in a single stream, even though you know you are omitting objects.

That said, I think maybe that is a really bad idea, and we should just use the normal logic and the publisher has to reset the stream and open a new one wherever it skips.

draft-ietf-moq-transport.md

afrind

I can see how this might make sense for fetch but it seems to really complicate subscribe and I don't see the use cases it solves. To be specific, I can not imagine a subscriber where I want groups 100 to 200, then skip group 201 to 299, then give me groups 300 to 400.

See the linked issue (and also #441) where @wilaw lists out some use cases. You may be right about having a list of allowed ranges though.

This is a fairly significant breaking change and I think we should talk more about what we are trying to fix before getting into this. I do see some value in writing down a more generic and normalize filter that provides more capabilities as long as it does not complicate things so we may end up at exactly this but I'd like to consider the goals more fist.

Completely agree we need to spend time discussing this before moving forward (note "RFC" in PR title). I wrote this to help kickstart the discussion, and because it's among the biggest "API level" changes remaining, which the chairs have asked us to prioritize.

draft-ietf-moq-transport.md

afrind · 2025-08-21T15:25:28Z

draft-ietf-moq-transport.md

+Object Filter {
+  Type (0x5),
+  Length (i),
+  Operand and Flag (i),


I realized < X is also trivially implemented as a range from [0,X]

afrind · 2025-08-21T15:26:16Z

draft-ietf-moq-transport.md

+Operand No Filter.
+
+This parameter MAY appear more than once, and filters are cumulative.  When
+filters are applied, delivery rules regarding Subgroups and FETCH responses are


A max seems very sensible.

afrind · 2025-08-21T15:29:46Z

draft-ietf-moq-transport.md

+filters are applied, delivery rules regarding Subgroups and FETCH responses are
+modified.  Within a Subgroup, the Publisher is allowed to send an Object on a
+Subgroup stream even when it is not the next Object if the expected Objects
+did not pass the filter.  In a FETCH response, the Subscriber can only infer


I am not saying you should send stuff that doesn't pass the filter. I was thinking the latter. For example f you have a subgroup with objects 0-9, and a filter that says only even objects, then you can put [ 0, 2, 4, 5, 8 ] in a single stream, even though you know you are omitting objects.

That said, I think maybe that is a really bad idea, and we should just use the normal logic and the publisher has to reset the stream and open a new one wherever it skips.

draft-ietf-moq-transport.md

wilaw · 2025-08-22T09:09:13Z

I can see how this might make sense for fetch but it seems to really complicate subscribe and I don't see the use cases it solves. To be specific, I can not imagine a subscriber where I want groups 100 to 200, then skip group 201 to 299, then give me groups 300 to 400.

Sparse non-contiguous ranges are really a filter use-case for FETCH. Since we want a common filter parameter between FETCH and SUBSCRIBE, these just won't be used much in SUBSCRIBE, but will be used with FETCH.

For SUBSCRIBE, the use-cases for filters outside of Sparse non-contiguous ranges are quite real and some examples are described below. Additionally, when combined with extensions, we get some very nice behaviors.

Sensors send temperature summary data in the first Object of each Group, then updates every minute as a new object. I want to monitor only the summary data by receiving only the first object in every group
FILTER ( =, object ID, 0 )
A live video feed codes its groups using unix epoch time. I want to subscribe to the feed starting tonight at 7pm
FILTER ( >, group ID, 1752138462)
Only forward me live security camera groups where the activity level is greater than or equal to 25%. The activity level is added to each object as the custom extension with a type of 400.
FILTER (>=, extension 400, 25)
Only forward me live temperature data from zip code 94925. The zip code is stored as the custom extension header 1444.
FILTER (=, extension 1444, 94925)
Only forward me log data from the media player with subscriber ID 847362511. The subscriber ID is sent in extension header 768 by the media player.
FILTER (=, extension 768, 847362511)

wilaw

Thanks for implementing this Alan!

My main feedback is about making the length of the match inclusive rather than exclusive. When dealing with GroupIDs etc it doesn't matter much, but when we get into use-cases where the values represent real-world discrete things (like ZIP codes, device IDs, temperatures, serial numbers etc), then specifying the inclusive end of the range seems better than one more than that.

draft-ietf-moq-transport.md

wilaw · 2025-08-22T09:23:49Z

draft-ietf-moq-transport.md

+Values is an array of integers which encode the values of interest.  The
+remainder of the array is a sequence of pairs indicating the start and length of
+the matching range.  The Start is encoded as a delta from the previous End, or


The remainder of the array is a sequence of pairs indicating the start and length of
the matching range

Saying that the second value indicates the length of the matching range is contradictory to the examples, as it suggests the length within which samples will match. For example a start of 10 and a length of 3 indicates that the matching range is from 10..13 , whereas it is intended to be 10..12.

I think explicitly defining the matching range is more intuitive for users than specifying a non-inclusive final value. For example, if I am filtering on zip code data, and I want 94552, i have to set the filter length to point at the imaginary 94553, which is not a valid zip code.

So suggestion is modify the definition so that the length is an inclusive match. This changes the example given later and also relaxes the requirement around which values can be zero. To specify matches for 10, 11, 12, 14 and 34-max we would code [10, 2, 2, 0, 20]

API and wire format don't need to be the same. I figured most APIs would take in a list of absolute ranges and run through an encoder which produce the values on the wire -- especially since subsequent range offsets are delta encoded. It's also a little strange to say it's a sequence of [Delta, Length-1] pairs, though it does buy one more value for one byte encodings by giving a meaning to 0 length.

draft-ietf-moq-transport.md

yekuiwang · 2025-09-18T20:12:49Z

draft-ietf-moq-transport.md

+Values is an array of integers which encode the values of interest.  The
+remainder of the array is a sequence of pairs indicating the start and length of


What does "the remainder" here mean?

I think a previous version had the first element optionally be the extension Id, but it was pulled out

yekuiwang · 2025-09-18T20:16:50Z

draft-ietf-moq-transport.md

+from 0 for the first element.  The length indicates the number of elements
+including Start to match.  An odd number of elements indicates the final range


If we specify the length field to indicate the number of elements minus 1, then the same number of bits can represent more elements, i.e., the capability or bit efficiency is slightly improved.

It only saves one value though at the expense of some readability?

Yes. Just a suggestion for consideration. In some other standards this is always done.

draft-ietf-moq-transport.md

Co-authored-by: Ye-Kui Wang <[email protected]>

draft-ietf-moq-transport.md

afrind · 2025-09-30T16:44:15Z

Parking pending Design Team input

mzanaty · 2025-10-07T14:18:00Z

In the Toronto interim, it was decided to setup a design team for object/track filters (PR#1164, I#441, 1068, 1206). Mo, Martin, Suhas, Victor, Will, YK volunteered. Please let me know if you want to join these meetings. Next meeting is Friday October 10 at 9:00am PDT.
Immediately after the interim, we had an impromptu initial meeting with the folks still lingering in Toronto: Mo, Victor, Cullen, Mike, Magnus. We converged on a simplified design.

The 'Operands' were removed in favor of separate parameter types for each filter type.
The 'Values' array was removed in favor of a single range, Start/End inclusive with no delta coding. End=0 means no end unless Start=0. For multiple ranges, use multiple parameters.
The 'Negation' flag was removed. We discussed allowing Start>End to mean negation. Maybe unnecessary, since any desired ranges can be expressed without negation using an extra range. For example, not 3-7 can be expressed as 0-2 and 8-0.
The track filter only specifies NumberOfTopTracks. The selection metric is a new Selector extension. Some folks preferred this opaque Selector to avoid leaking private info, so the relay network is not aware these are voice activity, temperature, etc. metrics. However, this means only a single metric can be the Selector for all subscribers. An alternative is to specify the extension ID as well, which can be the Selector or another extension. List feedback preferred this alternative.
A setup parameter will negotiate the number of filter ranges supported, including 0 to disable support.

mzanaty · 2025-10-11T04:10:19Z

Oct 10, 2025 design meeting with Mo, Suhas, Victor, Will, YK.

Parameters in Subscribe, Subscribe_Update, Fetch, Publish_Ok:
No Filter { Type=TBD+0, Length=0 }
Group ID Filter { Type=TBD+2, Length, Start, [End] }
Subgroup ID Filter { Type=TBD+4, Length, Start, [End] }
Object ID Filter { Type=TBD+6, Length, Start, [End] }
Pub Priority Filter { Type=TBD+8, Length, Start, [End] }
Extension ID Filter { Type=TBD+a, Length, Extension ID, Start, [End] }

Parameter in Subscribe_Namespace:
Track Selection Filter { Type=TBD+c, Length, Extension ID, MaxTracksSelected }

Setup Parameters:
MAX_OBJECT_FILTER_RANGES (=0 to disable)
MAX_TRACKS_SELECTED (=0 to disable)
Applies to each type of filter per subscription

Track Selection Details

Multiple Filters
- Object filters evaluate first, then track selection.
Tie Breaker
- Same selection metric value, keep old track.
- If app wants new track it can embed time in metric.
Suggest to change metric infrequently? Start of group? Avoid erratic selection churn.
Evict silent tracks? Avoid rogue publishers that send a max metric to evict others then send nothing to remain on top.
Track Alias and Name needed upon new track selection
- Send PUBLISH to deliver this
- Parameters in PUBLISH may differ upon each new track selection
  - Delivery Timeout
  - Max Cache Duration
  - Publisher Priority

Consider in later PR: Remove SUBSCRIPTION_FILTER parameter? Need another way to say Next Group.

afrind · 2025-10-11T14:07:31Z

@mzanaty Thanks for the update.

Send PUBLISH to deliver this

Does that mean evicted tracks also get PUBLISH_DONE to keep the total number of subscriptions under this SUBSCRIBE_NAMESPACE=MaxTracksSelected? I'd prefer that because it keeps the subscription state machine the same. If you don't do this, you will leak subscriptions causing excessive state, and, with current mechanisms, likely run out of request IDs.

Consider in later PR: Remove SUBSCRIPTION_FILTER parameter? Need another way to say Next Group.

I'd prefer that the draft have only one group filter mechanism at a time, so if you want to replace SUBSCRIPTION_FILTER, wait to add Group ID Filter until that follow up PR.

suhasHere · 2025-10-14T04:45:35Z

Does that mean evicted tracks also get PUBLISH_DONE to keep the total number of subscriptions under this SUBSCRIBE_NAMESPACE=MaxTracksSelected? I'd prefer that because it keeps the subscription state machine the same. If you don't do this, you will leak subscriptions causing excessive state, and, with current mechanisms, likely run out of request IDs.

This will be challenging. since this a dynamic selection and track can be selected at different points in time. Those tracks are all active , sending publish_done is incorrect here.

afrind · 2025-10-14T13:55:30Z

Those tracks are all active

What if MaxSelected is 5 and there are 1000 tracks, but every track has a moment where it's selector is in the top 5. Then I end up with 1000 subscriptions. Is that the intent?

mzanaty · 2025-10-20T05:01:58Z

Oct 17, 2025 design meeting with Martin, Mo, Suhas, Victor, YK.

Decided to defer details for Fetch and removing Subscription Filter until core design is done.

Decided to put all Start/End ranges of a single filter type in a single parameter. Discussed but rejected putting all filter types in a single super parameter. Subscribe_Update always replaces all ranges of a given filter type, or removes the filter if no ranges (Length=0), which eliminated the "No Filter" type.

Parameters in Subscribe, Subscribe_Update, Publish_Ok: (defer Fetch details)
Group Filter { Type=TBD1, Length, [Start, [End]]… }
Subgroup Filter { Type=TBD2, Length, [Start, [End]]… }
Object Filter { Type=TBD3, Length, [Start, [End]]… }
Priority Filter { Type=TBD4, Length, [Start, [End]]… }
Extension Filter { Type=TBD5, Length, [Extension ID, Start, [End]]… }

End is optional in the last range, meaning no End if omitted. For the Extension Filter, also allow End=0 to mean no End when Start>0, to allow different Extension IDs to have no End.

Need to clarify how filters (specifically Object or Extension Filters) can cause object gaps in subgroup streams, and how FIN is handled.

No changes to the Track Selection Filter. It remains as below.

Parameter in Subscribe_Namespace:
Track Selection Filter { Type=TBD6, Length, Extension ID, MaxTracksSelected }

Discussed whether deselected tracks get signaled in control messages like Publish Done. That results in extra control messages and subscription state churn if deselcted tracks later get reselected. However, many deselected tracks can pollute subscription state, so it may be good to allow publishers to purge state for many old deselected tracks. Need more discussion if "many old" can be left to implementations or we need to specify guidance.

afrind · 2025-10-20T15:19:02Z

Group Filter { Type=TBD1, Length, [Start, [End]]… }

For your next meeting can you discuss the interaction with SUBSCRIPTION_FILTER, or if you are replacing it entirely?

Need to clarify how filters (specifically Object or Extension Filters) can cause object gaps in subgroup streams, and how FIN is handled.

See #1295. I think the answer is along those lines -- if you omit an object from a subgroup, you MUST reset the stream.

Need more discussion if "many old" can be left to implementations or we need to specify guidance.

We definitely need some guidance, even if the algorithm to close deselected subscriptions is implementation specific. Eventually you can run out of request IDs.

mzanaty · 2025-10-20T15:31:28Z

The final design intends to replace SUBSCRIPTION_FILTER entirely. But there are details (such as Next Group) that the team wants to defer until the core design aspects are finalized first.

afrind requested review from ianswett, suhasHere and vasilvv as code owners August 20, 2025 17:53

afrind commented Aug 20, 2025

View reviewed changes

draft-ietf-moq-transport.md Show resolved Hide resolved

Add priority to the filter.

beb7072

suhasHere reviewed Aug 20, 2025

View reviewed changes

fluffy requested changes Aug 21, 2025

View reviewed changes

afrind commented Aug 21, 2025

View reviewed changes

draft-ietf-moq-transport.md Outdated Show resolved Hide resolved

draft-ietf-moq-transport.md Show resolved Hide resolved

draft-ietf-moq-transport.md Outdated Show resolved Hide resolved

draft-ietf-moq-transport.md Outdated Show resolved Hide resolved

Apply suggestions from code review

455b640

afrind commented Aug 21, 2025

View reviewed changes

draft-ietf-moq-transport.md Outdated Show resolved Hide resolved

wrap and newline

5705af4

afrind commented Aug 21, 2025

View reviewed changes

draft-ietf-moq-transport.md Outdated Show resolved Hide resolved

draft-ietf-moq-transport.md Show resolved Hide resolved

More cleanup

bd66412

wilaw suggested changes Aug 22, 2025

View reviewed changes

afrind added the Design Issues or PRs that change how MoQ works including the wire format. label Aug 31, 2025

afrind mentioned this pull request Sep 4, 2025

Clarify Objects from different subgroups can be sent on the same FETCH stream #1193

Closed

yekuiwang reviewed Sep 18, 2025

View reviewed changes

draft-ietf-moq-transport.md Outdated Show resolved Hide resolved

yekuiwang reviewed Sep 18, 2025

View reviewed changes

afrind commented Sep 18, 2025

View reviewed changes

draft-ietf-moq-transport.md Outdated Show resolved Hide resolved

Apply suggestions from code review

d5c2340

Co-authored-by: Ye-Kui Wang <[email protected]>

afrind commented Sep 18, 2025

View reviewed changes

draft-ietf-moq-transport.md Outdated Show resolved Hide resolved

Update draft-ietf-moq-transport.md

19e15cd

afrind commented Sep 18, 2025

View reviewed changes

draft-ietf-moq-transport.md Outdated Show resolved Hide resolved

Update draft-ietf-moq-transport.md

65a7e24

afrind mentioned this pull request Sep 22, 2025

Semantic FIN and Filters / Forward #1233

Closed

This was referenced Sep 25, 2025

Filter / select across multiple tracks in a namespace #1206

Open

Make START LOCATION and END LOCATION params #1059

Closed

afrind assigned mzanaty Sep 26, 2025

afrind mentioned this pull request Sep 26, 2025

Ad Insertion #527

Open

afrind added the Parked Issue we may discuss later or close as OBE label Sep 30, 2025

yekuiwang mentioned this pull request Oct 17, 2025

Possible improvements on MoQ support of VOD applications #1316

Open

wilaw mentioned this pull request Nov 4, 2025

SUBSCRIBE does not need a forward parameter if we have filters #1352

Open

		Values is an array of integers which encode the values of interest. The
		remainder of the array is a sequence of pairs indicating the start and length of

		from 0 for the first element. The length indicates the number of elements
		including Start to match. An odd number of elements indicates the final range

RFC: Generic Filters for Subscriptions and FETCH #1164

Are you sure you want to change the base?

RFC: Generic Filters for Subscriptions and FETCH #1164

Uh oh!

Conversation

afrind commented Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fluffy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

afrind left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wilaw commented Aug 22, 2025

Uh oh!

wilaw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

afrind commented Aug 20, 2025 •

edited

Loading