Nested filters, filter syntax, and other filter problems

[MartyGentillon]

Currently, it is difficult to use filters to find values for the arguments to filter operations. It would be nice to change that. As an example use case, consider a Pokemon twiki, I want to list all pokemon with a particular type. The type tiddly is called Grass Type, and Pokemon keep their type in a type field containing, for example, Grass. To make this list I might want to use a filter like:

[tags[pokemon]field:type[<currentTiddler>split[" "]first[]]]

But, currently this can only be done with a bunch of <$set> widgets and variables in the filter.

I think it would be useful to have a filter notation which supported nested filter calls like the one above. Doing it in a backward compatible manner may present a problem, but I believe that it is mostly possible, barring edge cases (There is still design work to be done here).

This leads to a few questions:

1. What do people think of that kind of syntax extension?
2. What are the current problems that people have with filters? (example filters would be useful.)
3. What do the devs consider to be problems with the current filter implementation?
4. What other plans do people have / changes are people working on regarding filter syntax and parsing?

[pmario]

You can not use the type field, because TW uses it for the tiddler-mime-type. MIME-types are specified.

In your case the filter would be [tag[pokemon]] +[ptype[Grass Type]] ... I also don't see the need to add a field-value "Grass Type" .. IMO "Grass" would be good enough, since the field-name already contains the info that it is a type.

I also don't know, why you would want to use <currentTiddler> in this context.

[MartyGentillon]

The problem comes when linking from the Grass Type page to the pokemon that have a ptype of Grass from within a template which gets displayed on all Pokemon Type tiddlies. Because this code belongs in the template, it has <currentTiddler> to grab Grass Type. Right now, the workaround I have is to add a name to the tiddlers that are linked.

I should also note, this is a single example of the kind of thing that it would be nice to be able to do trivially without having to jump through hacks. I severely doubt that this is the only time I might want nested filters. The purpose of the example was to demonstrate an example of a time when nested filters would be useful.

Overall, this is an ease of use concern; when someone guesses something should work, it would be ideal if it did work. (Though, of course, backward compatibility is a serious concern to implementing something like this).

[pmario]

OK, I do understand the use-case now.

TW filters are optimized for speed and they should do 1 thing well. So the different elements can be combined to more complex filters, which are still fast.

• One of the main goals if we build a filter is. Go from many input titles to a view input titles as fast as you can. eg:

[tag[pokemon]] is a shortcut for [all[tiddlers]tag[pokemon]] ... where all[tiddlers] represents the "starting set of tiddlers". All of them, even those that are not tagged. ... tag[pokemon] will reduce the set of tiddler drastically. So that's good.

• A second goal is: Don't execute "code" in a loop, if it doesn't change.

Your suggested filter would exactly do that: [tags[pokemon]field:type[<currentTiddler>split[" "]first[]]] ... [<currentTiddler>split[" "]first[]] will not change in the loop, but it will need to be executed for every tiddler that is tagged "pocemon"

As you found out: A fast solution is:

<$set name=type filter="[<currentTiddler>split[ ]first[]]">
  <$list filter="[tag[pocemon]ptype<type>]">
    <$link /><br>
  </$list>
</$set>

The set-widget is only calculated once and the result is written to a variable named "type". Accessing a variable in a filter is much faster as recalculating the value over and over again.

At the beginning slow filters don't matter much, but once you go up to 1000+ tiddlers you can feel it. ... There are wikis out there with 60 000 tiddlers. ... So speed matters.

[MartyGentillon]

Memoization is a thing, so I don't think that the other solution would necessarily break the "don't execute code in a loop that doesn't change" rule. But that is a good point, any implementation like that would need to include memoization.

[MartyGentillon]

Also, the idea that the run order would be from outside to inside doesn't really hold. I mean, like any functional programming language, the filter would basically operate by reference substitution. Which implies you actually want to calculate from the most nested element out.

[joshuafontany]

Hi! You want the filter[] or subfilter[] operators, but you will want to set up the "subfilter" as a Macro Definition (to avoid the nested [] syntax you have discovered does not work).

https://tiddlywiki.com/#subfilter%20Operator:%5B%5Bsubfilter%20Operator%5D%5D%20%5B%5Bfilter%20Operator%5D%5D

To adapt the example to your use-case:

\define typeFilter() [get[ptype]addsuffix[ Type]match:caseinsensitive[Grass Type]]

{{{ [tag[pokemon]filter<typeFilter>] }}}

There is a bug where a javacript error happens if you use Variable syntax in filters defined as macros. So, you will have to redefine the
\define typeFilter() [get[ptype]addsuffix[ Type]match:caseinsensitive[Grass Type] filter in each "PType" tiddler, but then its as simple as {{{ [tag[pokemon]filter<typeFilter>] }}} to generate the list.

Best,
Joshua Fontany

[joshuafontany]

I think there is a way to avoid "hard-coding" the type to check against, but
\define typeFilter() [get[ptype]addsuffix[ Type]match:caseinsensitive{!!title}]
does not work. I'll file a bug report around these, as they are very new operators.

So, you could shorten that to \define typeFilter() [get[ptype]match:caseinsensitive[Grass]] for now.

[joshuafontany]

#5879

[AnthonyMuscio]

Is the test ptype[Grass] or ptyle<currentType> or a simple get[ptype] which returns nothing if no value so get[ptype]match<currentType>

[MartyGentillon]

Keep in mind, the point isn't the example, but the question of can we make gotchas go away by finding a more extensive grammar for filters that both 1) supports all current filters and 2) allows for more expressive queries without things becoming over-complicated. Also, as a survey for what problems people encounter with filters.

This isn't a problem to be solved, it's a design question to be answered.

[AnthonyMuscio]

Marty,
I think it important to ask such questions.

Unfortunately I don't understand your example, because in part it is not optimal anyway. Perhaps you can think of a more robust real world example?

For me. only now. is my own understanding and the features of tiddlywiki bringing us to a point where we can use a range of methods to simplify filters, which is amazing given they ultimately let us do lots of complex things. 5.1.23 and the next release 5.1.24/5.2.0 continues this trend.

• It will be be possible to build macros that construct filters such as <<myfilter param param>> and use them inside filters [<myfilter param param>].
• Further I do not think we have yet explored all the possibilities recently added such as the reduce and
  map operators and the new run prefixes which do a lot (have an accumulator), as well not to mention long standing methods like fieldname[value] eg: display-detail[yes] which help create language like statements.
• The use of filtered transclusions {{{ filter }}} to generate values for attributed to html or widgets is also very powerful.
• Filter and sub filter operators
• Designing macros that rather than passing a tiddler name you actually pass a filter as a parameter, which can be just a single tiddler name, many title or a filter.

I suppose the point I am trying to make is we do have opportunities to handle many cases of nested filters and making them easy to read by the use of macros, variables and fieldnames. But of course simply nesting list widgets can address perhaps any possible case. I know it is tempting to load filters up with more and more syntax, especially since they always feel like they are already "nested" but their is no harm actually nesting widgets.

I have actually written macros to which you pass some parameters which themselves contain nested list widgets, with recursive or self calling macros which effectively allow you to request in a single macro the output that comes from deeply nested algorithm's. That is it resembles what you are asking for in this thread.

Personally where possible I write all list Widgets in macros to operate on the currentTiddler, so the keys are always (tiddler) titles and from that I can extract data from the tiddler, this elegant design method will only be complicated if we do too much nesting inside filters.

Regards
Tony

[MartyGentillon]

The goal isn't so much to add a bunch of syntax, as it is to support the affordances already there. The basic affordance that brackets provide in programming languages is that they can easily be nested. So, we are already using a syntax which, by the symbols used to build it, implies nesting. (In English I can comment on something with a parenthetical (parentheticals can be nested in written language too)).

One of the big rules I have for such things is to try and be consistent. As much as possible, this would be about taking the current syntax and making it so that it can be used consistently throughout the filter. Any string in a filter followed by a set of brackets is treated the same way, regardless of whether it is in the position of an parameter or not.

[MartyGentillon]

Also, using macros as a workaround does present some problems, at least if you use text substitution. After all, text substitution is non-syntactic and therefore could result in situations where a filter run can have commands injected into it and therefore modify things. That would be... non-ideal. (Though, it is possible that the newer features you discuss could resolve that, I am, after all, basing much of my thinking on the current version.)

[AnthonyMuscio]

Marty,

Please understand I am in problem solving mode for you. This means my language will be "egoless" in nature and please do not assume any personal criticism between my lines.

Because I still cant understand the case you describe (perhaps my problem) I feel you understanding of filters may be part of the issue. Perhaps you could indicate your filter skills level. If advanced then I expect you can find a work around, or to state it more clearly, if less mature I expect I can give you a solution. My filter skills are now quite advanced.

A few tips that may lead you where you want

• In a macro you can include "dynamic filters" using $param$ and $(varname)$ within a filter (these are substitutions)
• Use filter and subfilter operators to include whole filter definitions filter[$param$] subfilter[$varname$]
• Use operator[$param$] or operator[$(varname)$]
• See getvariable to programaticaly name a variable from which to retrieve values

When people ask about nested filters, as I have done in the past, even proposing something similar to you in this thread. I have come to learn that given filters resemble nesting already (ie can contain a list of lists) people willfully ignore the ability to nest list widgets, or use a filtered transclusion within a list. The value of nested list widgets is with each nesting you can set the variable=variablename and this variable is available for display, computation or conditional code within the nested code. ie you could have four nested lists and all four variables would be available.

I even wrote a macro whose parameter was filters:"filter1 filter2 filter3" which generated a three level nested list programaticaly.

Post script: I am not familiar with Poekmon however I am currently building a solution (containing tiddler object-types and lots of nesting) for my self, much more advanced that I speculate is in your question. So I am confident I can help.

[AnthonyMuscio]

Post script I think the doco at https://groktiddlywiki.com/read/#Text%20Substitution could be improved.

[Jermolene]

Hi @MartyGentillon with the upcoming v5.2.0 release, @joshuafontany's code can be simplified to a single filter:

{{{ [tag[pokemon]] :filter[get[ptype]addsuffix[ Type]match:caseinsensitive[Grass Type]]}}}

See https://tiddlywiki.com/prerelease/#Filter%20Expression for more details.

[MartyGentillon]

Which is indeed an improvement over the current state. But, it's worth keeping in mind that, as I mentioned in the Discord when asking why this wasn't possible, from my perspective, this is largely about both consistency of design and also removing the need for workarounds given how workarounds require the user to sacrifice brain space remembering to and dealing with them.

The fact of the matter is that, in the current syntax, when I initially type [ to open a filter, I enter a context wherein I can use a filter operator, but then when I type [ to provide that filter with its argument, I enter a different context wherein I can only provide a bareword argument value. This is an inconsistency in syntax that it would be nice if could be made to go away. There is this affordance provided by bracket based syntax for nesting which the current parser doesn't deliver. And it seems possible to create a grammar which doesn't suffer this flaw.

Presuming that this can be done efficiently and while maintaining reasonable backwards compatibility, it seems to be the kind of rough edge that we should be seeking to file off*. As mentioned before, both memoization and executions strategies can be used to meet the performance needs. And, since any filter which would trigger the new behavior currently triggers a parse error, backwards compatibility seems like it shouldn't be a large problem. So, the feature seems manageable.

The bigger problem when considering such a work is developing an understanding of both current design issues related to the filter parser (what are they broadly considered to be?) and also current use cases that need to be supported.

• Another example of this kind of edge case is the one discussed in Grok TiddlyWiki related to nested transclusion and the necessity of macros.