feat: serialize AST node back to wikitext string #8258

linonetwo · 2024-06-12T17:28:01Z

Currently just a demo, I will gradually move code from https://github.com/tiddly-gittly/wikiast/tree/master/packages/wikiast-util-to-wikitext to here.

Try it with tiddler (Well, if official website have share plugin installed, I can put a link here):

AAA test

---

\```js
var match = reEnd.exec(this.parser.source)
\```

end

and run this in console:

$tw.utils.serializeParseTree($tw.wiki.parseTiddler('New Tiddler').tree)

vercel · 2024-06-12T17:28:06Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
tiddlywiki5	✅ Ready (Inspect)	Visit Preview	Jun 12, 2024 5:39pm

… text" This reverts commit 8a12498.

pmario · 2024-06-13T08:24:26Z

The problem here is, that the output of you test does not produce the same rendered HTML output as the original source. See the screenshot.

linonetwo · 2024-06-13T09:27:44Z

First line is wrapped by a p element, but I haven't find the rule that produce the p element. And there was no p html in the source, so I may need to add some attribute to ast to show this p element is "fake".

Anyway, this is just a demo for API, do you like the API name and their usage? The further task is just adding exports.serailize = xxx to each rule.

pmario · 2024-06-13T09:33:59Z

First line is wrapped by a p element, but I haven't find the rule that produce the p element. And there was no p html in the source, so I may need to add some attribute to ast to show this p element is "fake".

IMO the current parser has a "systemic" problem with redundant P-tags, that we probably may need to solve first.

Working around this problem with "fake" elements will just increase your code complexity, instead of solving the underlaying problem. It will also create an other dependency, that will make solving the problem, in the core, in backwards compatible way even harder as it is now.

pmario · 2024-06-13T09:40:11Z

core/modules/wiki.js

+};
+
+exports.getParser = function(type,options) {
+	options = $tw.utils.extend({},options);


IMO your code will throw an RSOD if options is undefined
So options = options || {} should be used instead. We use this pattern all over the places.

I copy $tw.utils.extend({},options) from somewhere nearby, I thought this will work.

linonetwo · 2024-06-13T14:04:01Z

redundant P-tags

Can't remove then, many of my plugin style sheet is already depending on this structure. And p > text for block is following HTML5 semantic.

Another option is handle it like text node, regard it as a special case.

pmario · 2024-06-13T14:05:53Z

And p > text for block is following HTML5 semantic.

That's true, but we do have a lot of p > div which should not happen at all.

linonetwo · 2024-07-29T16:47:05Z

Handle the block rule by adding a rule: 'parseBlock' to WikiParser.prototype.parseBlock.

I've added some test for it.

…n of it

Jermolene · 2025-05-29T07:40:29Z

It kepts receiving conflict from other PRs, when will we merge this? @Jermolene

Apologies @linonetwo I appreciate your patience. Now that this PR is planned for v5.4.0 it will be merged once we release v5.3.7, when we'll start v5.4.0. (While we could start v5.4.0 in another branch alongside the development of v5.3.7, the problem is that we would just be shifting the friction you are experiencing with merge conflicts to a different branch).

github-actions · 2025-05-29T07:41:36Z

📊 Build Size Comparison: `empty.html`

Branch	Size
Base (master)	2528.4 KB
PR	2532.1 KB

Diff: ⬆️ Increase: +3.7 KB

github-actions · 2025-05-29T07:46:16Z

📊 Build Size Comparison: `empty.html`

Branch	Size
Base (master)	2528.4 KB
PR	2532.1 KB

Diff: ⬆️ Increase: +3.7 KB

linonetwo · 2025-05-29T07:47:03Z

Okay, I just help other PR fixing its test. Hope AI could help me do that in the future.

Looking forward to v5.4.0. I check this PR regularly because when someone in QQ group ask me about WYSIWYG editor's bug I will talk about this PR once and check it...

linonetwo · 2025-07-12T07:52:18Z

Our serializer is based on rule field. And type field is for widgets. This is because many rules are producing type: "html", and block (p) don't even have a type.

This could cause some bug hard to debug when using it. So I write it here for future me (or AI that helps me).
P.S. won't be a problem if we have type-check. This is a tipical bug that typescript is created for.
gpt4.1 is too silly to fix this, but claude4 could find it in a second, so, no need for typescript!

linonetwo · 2025-09-12T12:08:03Z

Any chance merging this? Do you think this new core plugin is feature completed?

Jermolene · 2025-09-15T09:08:21Z

Hi @linonetwo I'm hoping we can merge #9135 in the next day or two, and then I believe we can start merging v5.4.0 PRs.

Leilei332 · 2025-10-10T12:39:34Z

@Jermolene IMO this PR is ready to be merged now.

github-actions · 2025-10-11T08:24:05Z

📊 Build Size Comparison: `empty.html`

Branch	Size
Base (master)	2421.1 KB
PR	2424.8 KB

Diff: ⬆️ Increase: +3.7 KB

Copilot

Pull Request Overview

This PR introduces a wikitext serialization feature that can convert parsed AST (Abstract Syntax Tree) nodes back to wikitext format. It implements a new plugin with serializers for various wikitext rules and utilities for reverse-engineering wikitext from parse trees.

Key changes:

Adds comprehensive serialization support for most wikitext syntax elements
Introduces a new void widget type for handling non-rendering nodes
Updates parser rules to include metadata needed for serialization

Reviewed Changes

Copilot reviewed 115 out of 117 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
plugins/tiddlywiki/wikitext-serialize/utils/parsetree.js	Core serialization utilities with main `serializeWikitextParseTree` function
plugins/tiddlywiki/wikitext-serialize/rules/*.js	Individual serializers for each wikitext rule (tables, headings, emphasis, etc.)
core/modules/widgets/void.js	New void widget for handling non-rendering parse tree nodes
core/modules/parsers/wikiparser/rules/*.js	Updates to parser rules to add serialization metadata
editions/test/tiddlers/tests/*.js	Test files and test data for serialization functionality

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

plugins/tiddlywiki/wikitext-serialize/rules/styleblock.js

plugins/tiddlywiki/wikitext-serialize/rules/list.js

core/modules/widgets/void.js

plugins/tiddlywiki/wikitext-serialize/rules/filteredtranscludeblock.js

github-actions · 2025-10-11T08:33:21Z

📊 Build Size Comparison: `empty.html`

Branch	Size
Base (master)	2421.1 KB
PR	2424.8 KB

Diff: ⬆️ Increase: +3.7 KB

…block.js Co-authored-by: Copilot <[email protected]>

Co-authored-by: Copilot <[email protected]>

github-actions · 2025-10-11T08:34:01Z

📊 Build Size Comparison: `empty.html`

Branch	Size
Base (master)	2421.1 KB
PR	2424.8 KB

Diff: ⬆️ Increase: +3.7 KB

Co-authored-by: Copilot <[email protected]>

github-actions · 2025-10-11T08:34:06Z

📊 Build Size Comparison: `empty.html`

Branch	Size
Base (master)	2421.1 KB
PR	2424.8 KB

Diff: ⬆️ Increase: +3.7 KB

github-actions · 2025-10-11T08:34:12Z

📊 Build Size Comparison: `empty.html`

Branch	Size
Base (master)	2421.1 KB
PR	2424.8 KB

Diff: ⬆️ Increase: +3.7 KB

github-actions · 2025-10-11T08:34:24Z

📊 Build Size Comparison: `empty.html`

Branch	Size
Base (master)	2421.1 KB
PR	2424.8 KB

Diff: ⬆️ Increase: +3.7 KB

github-actions · 2025-10-11T08:34:31Z

📊 Build Size Comparison: `empty.html`

Branch	Size
Base (master)	2421.1 KB
PR	2424.8 KB

Diff: ⬆️ Increase: +3.7 KB

linonetwo added 3 commits June 13, 2024 01:23

refactor: extract a new $tw.wiki.getParser

37338b1

feat: allow $tw.utils.getParseTreeText to render other rules' text

8a12498

feat: two example getText handler

09c2e71

linonetwo added 2 commits June 13, 2024 01:29

Revert "feat: allow $tw.utils.getParseTreeText to render other rules'…

a0708a1

… text" This reverts commit 8a12498.

refactor: keep original getParseTreeText not touched

f882cbd

vercel bot deployed to Preview June 12, 2024 17:31 View deployment

refactor: use serialize in rules

be3f037

vercel bot deployed to Preview June 12, 2024 17:39 View deployment

linonetwo changed the title ~~Feat/to string~~ feat: serialize AST node back to wikitext string Jun 12, 2024

pmario reviewed Jun 13, 2024

View reviewed changes

linonetwo added 5 commits July 29, 2024 21:43

Merge remote-tracking branch 'upstream/master' into feat/to-string

4ddcbc6

refactor: $tw.utils.extend({},options) -> options || {}

4e896d1

Update codeinline.js

179651a

Create test-wikitext-serialize.js

8cbf855

DEBUG: only run my tests for development, remove before PR merge

a075303

linonetwo mentioned this pull request Jul 29, 2024

Add start and end properties to WikiText AST nodes for all elements. #7866

Closed

linonetwo added 3 commits July 30, 2024 00:43

lint: if

05dec72

feat: add rule: 'parseBlock' metadata

1a6a071

feat: handle tailing \n that may be missing

84e27c0

linonetwo added 4 commits July 30, 2024 00:56

feat: allow recursive

078967c

feat: generate more rule and tests

2aff9fe

feat: generate more rule and tests

bfd116a

fix: remove pragma:true, otherwise following text will become childre…

d067871

…n of it

Update macrodef.js

3da7564

Update test-wikitext-parser.js

0b40531

linonetwo mentioned this pull request Jun 3, 2025

TEST: Use biome instead of eslint dprint #9058

Draft

saqimtiaz moved this from In progress to Needs review in Planning for v5.4.0 Jun 17, 2025

saqimtiaz moved this from Needs feedback to Ready in Planning for v5.4.0 Jun 17, 2025

linonetwo mentioned this pull request Sep 14, 2025

DEV Edition - Code Formatting Best Practice Documentation #9176

Open

Merge remote-tracking branch 'upstream/master' into feat/to-string

a483074

Copilot AI review requested due to automatic review settings October 11, 2025 08:23

Copilot AI reviewed Oct 11, 2025

View reviewed changes

lint: fix

79822f2

linonetwo and others added 4 commits October 11, 2025 16:33

Update plugins/tiddlywiki/wikitext-serialize/rules/filteredtransclude…

a91fa06

…block.js Co-authored-by: Copilot <[email protected]>

Update core/modules/widgets/void.js

82c2ad3

Co-authored-by: Copilot <[email protected]>

Update plugins/tiddlywiki/wikitext-serialize/rules/list.js

821e961

Co-authored-by: Copilot <[email protected]>

Update plugins/tiddlywiki/wikitext-serialize/rules/list.js

c26564c

Co-authored-by: Copilot <[email protected]>

Update plugins/tiddlywiki/wikitext-serialize/rules/styleblock.js

163b8cf

Co-authored-by: Copilot <[email protected]>

feat: serialize AST node back to wikitext string #8258

Are you sure you want to change the base?

feat: serialize AST node back to wikitext string #8258

Uh oh!

Conversation

linonetwo commented Jun 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vercel bot commented Jun 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pmario commented Jun 13, 2024

Uh oh!

linonetwo commented Jun 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pmario commented Jun 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pmario Jun 13, 2024

Choose a reason for hiding this comment

Uh oh!

linonetwo Jun 13, 2024

Choose a reason for hiding this comment

Uh oh!

linonetwo commented Jun 13, 2024

Uh oh!

pmario commented Jun 13, 2024

Uh oh!

linonetwo commented Jul 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Jermolene commented May 29, 2025

Uh oh!

github-actions bot commented May 29, 2025

📊 Build Size Comparison: empty.html

Uh oh!

github-actions bot commented May 29, 2025

📊 Build Size Comparison: empty.html

Uh oh!

linonetwo commented May 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

linonetwo commented Jul 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

linonetwo commented Sep 12, 2025

Uh oh!

Jermolene commented Sep 15, 2025

Uh oh!

Leilei332 commented Oct 10, 2025

Uh oh!

github-actions bot commented Oct 11, 2025

📊 Build Size Comparison: empty.html

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 11, 2025

📊 Build Size Comparison: empty.html

Uh oh!

github-actions bot commented Oct 11, 2025

📊 Build Size Comparison: empty.html

Uh oh!

github-actions bot commented Oct 11, 2025

📊 Build Size Comparison: empty.html

Uh oh!

github-actions bot commented Oct 11, 2025

📊 Build Size Comparison: empty.html

Uh oh!

github-actions bot commented Oct 11, 2025

📊 Build Size Comparison: empty.html

Uh oh!

linonetwo commented Jun 12, 2024 •

edited

Loading

vercel bot commented Jun 12, 2024 •

edited

Loading

linonetwo commented Jun 13, 2024 •

edited

Loading

pmario commented Jun 13, 2024 •

edited

Loading

linonetwo commented Jul 29, 2024 •

edited

Loading

📊 Build Size Comparison: `empty.html`

📊 Build Size Comparison: `empty.html`

linonetwo commented May 29, 2025 •

edited

Loading

linonetwo commented Jul 12, 2025 •

edited

Loading

📊 Build Size Comparison: `empty.html`

📊 Build Size Comparison: `empty.html`

📊 Build Size Comparison: `empty.html`

📊 Build Size Comparison: `empty.html`

📊 Build Size Comparison: `empty.html`

📊 Build Size Comparison: `empty.html`

📊 Build Size Comparison: `empty.html`