ListWidget join parameter

[andrewgoz]

I'm currently using the list widget to generate dynamic JSON data. JSON doesn't like dangling commas, which makes working with the list widget tricky. My current code "snippet" looks like:

<$list counter="is">
<$text text={{{ [<is-first>match[no]then[,]] }}}/>content
</$list>

but what I'd really like is:

<$list join=",">
content
</$list>

Comments?

[linonetwo]

This is a solved problem in echarts plugin, see this code, I parse content as js, so tailing comma won't be a problem

tiddly-gittly/tw-echarts@254c56a

[linonetwo]

and in echarts examples, we used to support comma problem by using list widget's count field, but is deprecate, and example only exist in old commits.

---

Update: the counter usage example in echarts config, it still works, but not necessarily. @andrewgoz

The logic to add tailing comma is <$list filter="[match[no]]">`,`</$list>

https://github.com/tiddly-gittly/tw-echarts/blob/e254da05f8887dd3d2351c7ddbff32586f75a737/src/echarts/addons/Gk0Wk/PluginsSize.tid

title: $:/plugins/Gk0Wk/echarts/addons/PluginsSize
type: text/vnd.tiddlywiki

`{
  "tooltip": {
    "trigger": "item"
  },
  "series": [
    {
      "type": "treemap",
      "visibleMin": 300,
      "name": "Plugin Size",
      "leafDepth": 1,
      "data": [
`
<$list filter="[!plugin-type[]!title[$:/core]!title[$:/temp/info-plugin]]" variable="pluginName" counter="counter1">
`        {
          "name": "`<$list filter="[<pluginName>has[name]]"><$text text={{{ [<pluginName>get[name]] }}}/></$list><$list filter="[<pluginName>!has[name]]"><$text text={{{ [<pluginName>has[name]else<pluginName>split[/]nth[4]] }}}/></$list>`",
          "value": `<$text text={{{ [<pluginName>get[text]length[]else[0]] }}}/>`,
          "children": [
`
<$list filter="[<pluginName>plugintiddlers[]]" variable="pluginShadow" counter="counter2">
`          {
            "name": "`<$text text={{{ [<pluginShadow>removeprefix<pluginName>removeprefix[/]else<pluginShadow>] }}}/>`",
            "value": `<$text text={{{ [<pluginShadow>get[text]length[]else[0]] }}}/>`
          }`<$list filter="[<counter2-last>match[no]]">`,`</$list>`
`
</$list>
`          ]
        }`<$list filter="[<counter1-last>match[no]]">`,`</$list>`
`
</$list>
`
      ],
      "levels": [
        {
          "itemStyle": {
            "borderColor": "#555",
            "borderWidth": 4,
            "gapWidth": 4
          }
        },
        {
          "colorSaturation": [0.3, 0.6],
          "itemStyle": {
            "borderColorSaturation": 0.7,
            "gapWidth": 2,
            "borderWidth": 2
          }
        }
      ],
      "upperLabel": {
        "show": true,
        "height": 30
      }
    }
  ]
}`

I'm not sure what those ` means in the code, it is written by @Gk0Wk .

[Jermolene]

Hi @linonetwo the "counter" attribute of the list widget is not deprecated; it has many valid use cases. We do warn that it has a performance impact in interactive situations.

Could you point to the missing examples?

[linonetwo]

Sorry, I mean using counter in echarts config is deprecated, because it is too hard for human brain to think about iterations...

[linonetwo]

@Jermolene Added the example. Do you know what those ` means? Seems @Gk0Wk try to wrap the widget call with ` ` pairs.

[Jermolene]

Hi @linonetwo the purpose of the backticks is to wrap text that should not be wikified, to ensure that it is not inadvertently corrupted by the wikification process.

[Gk0Wk]

@Jermolene Added the example. Do you know what those ` means? Seems @Gk0Wk try to wrap the widget call with ` ` pairs.

Prevnt the text surrounding from parsing by TW5 parser, maybe. I read something like this before.

[andrewgoz]

I started with JavaScript parsing, but switched to JSON to address concerns about unwanted code execution. I will see about making this change myself when I have some time...

[linonetwo]

Such a code execution does not consume much CPU time, but the time you spend on it consumes lots of Your time!

[Jermolene]

Adding a "join" attribute to the list widget would be a more complex undertaking than it appears because of the impact on the code that optimises interactive changes to the list widget.

[andrewgoz]

On face value I wouldn't have thought it more complicated than automating what's in my example snippet? As in, adjusting the generated item template. However, the list widget code is a more complicated than I was expecting...

[Jermolene]

The list widget code is one of the most complex parts of TiddlyWiki's internals due to the optimisations that are made.

Besides that complexity, there are a couple of other factors:

• The "join" attribute relies on the ability to place wikitext constructions within an attribute value. This is also the case with the "emptyMessage" attribute. It's a clumsy pattern because authors have to take care to make sure that the quoting is correct, and it is hard for editors to provide syntax colouring etc for nested wikitext
• The widget layer of the design of wikitext is designed to be the smallest possible, primitive units of functionality needed to make TW work. The widget layer is not where we seek to optimise for brevity or ease of typing; for those situations we use specific parsing rules. So for example, the \define pragma is a shortcut for the set widget that is significantly easier to type for many scenarios

[rmunn]

I've started working on adding a join attribute, re-using some of the code that handles counters but trying to optimize it for the join attribute case (where the only thing list items will care about is whether or not they're last in the list). In the process, I discovered that the existing list-widget code actually has a bug: when items are added to a list but nothing else is changed, it will fail to update the <<counter-last>> variable correctly, resulting in list renderings like "abc" that should have been "a, b, c". #7693 contains that bug report.

Once I get that bug fixed, I should be able to submit a PR adding a join attribute.

[pmario]

I think this should not be part of the list-widget. This feature seems to be very usecase related and it should be done within the list-template or the code in the list-widget

I'm against this feature

[rmunn]

A "join" is part of nearly every list-handling feature in every programming language I'm aware of. That TW requires you to write <$text text={{{ [<is-last>match[no]then[,]] }}}/> instead of join="," is the opposite of user-friendly. It's one of the most common things you need when writing lists that aren't HTML <li> sequences.

[pmario]

If that's the case, then there should be a core template that does exactly that. So users write:

title: test-list

<$list filter="[tag[HelloThere]]" template="$:/cnt/join-by-comma" counter="cnt"/>
<$list filter="[tag[HelloThere]]" template="$:/cnt/join-by-pipe" counter="cnt"/>

<$list filter="[tag[HelloThere]]" template="$:/cnt/join-by-you-name-it" counter="cnt"/>

title: $:/cnt/join-by-comma

<$link /><$text text={{{ [<cnt-last>match[no]then[,]] }}}/>

title: $:/cnt/join-by-pipe

<$link /><$text text={{{ [<cnt-last>match[no]then[|]] }}}/>

and so on.
No new functionality needed. Can be done with TW v5.3.1 out of the box.
The template has countless possibilities with no limits. It's the documentation that is missing.

In my examples, the template title is built like this: $:/<counter-value>/<template-name>.
It could be different, but that's just how we do define the convention.

Instead of cnt we could use list or is or something similar that makes sense.

[pmario]

If templates are not your business we could create core macros like list-join similar to list-links

\procedure list-join(filter, sep:",")

<$list filter=<<filter> template=<<template>> counter="cnt" >
	<$link /><$text text={{{ [<cnt-last>match[no]then<sep>] }}}/>
</$list>
\end

<<list-join "[tag[HelloThere]]">>

[rmunn]

That list-join macro looks good, though I'd also add another parameter for the template rather than using <$link />, as some use cases (like @andrewgoz's JSON output need that started this discussion) would want <$text text={{!!title}} /> (or just <<currentTiddler>> as that's probably way faster to process).

However, see #7694 (comment) for some dramatic performance measurements. The counter-based approach took, on average over four runs, 1,117.5 ms (1.1 seconds) to update an item near the start of a 5,000-item list. And that's only measuring the time it took to run handleListChanges; the actual update to the page were visibly slower than that time, but I couldn't measure them with any precision. The code from PR #7694, though, took 14.75 ms on average over four runs to make the same change.

There's just no comparison. The counter-based code has to assume that you might be using the counter in your list-item template. So if you add or remove an item near the start of the list, it has to update every item after the one that changed because the counter value will be different for the items after the change (they're now at different indices). But the join=", " code knows that you don't care about the position in the list (the index), only about whether the item is last or not. So it is able to simply update the changed item and up to two items at the end of the list (the one that used to be last but maybe isn't last after the change, and the one that is now last but maybe wasn't last before the change).

[rmunn]

PR #7694, still in draft mode for now until I write some unit tests (running out of time to work on that today), adds a join attribute to the list widget. It also doesn't contain any documentation yet, but basically there's a join="whatever" attribute now. It handles arbitrary wikitext so you could write join="<br/>" if you want (though putting HTML inside attributes is tricky, you'd actually have to use variables and so on). That incurs a relatively costly parsing step, so I also created joinText="whatever" for the use cases where you just want a comma. The joinText value is treated as a plain text string, and no wiki parsing occurs. (You can still do joinText=<<variable>> or joinText={{{ [<myvar>addsuffix[ ]] }}}, what I mean is that after the value is retrieved from the variable or the filter expression or whatever, it's treated as plain text and inserted as-is into the list in between list items. So joinText="<br/>" would insert the literal string <br/> rather than an HTML line break, whereas join="<br/>" would insert an HTML line break element.)

P.S. PR #7694 also fixes bug #7693.

[andrewgoz]

I have nothing to add, other than to thank @rmunn for your current and ongoing efforts on this. From the sounds of it, progress is good and I'm hoping the changes (when they're done) will be accepted.

[pmario]

According to the OP @andrewgoz wants a plain-text output that is joined with a comma. IMO here is one possible solution.
Maybe no list-widget needed at all.

\function list-join(filter, sep:", ") [subfilter<filter>join<sep>]

<<list-join "[tag[HelloThere]]">>