Move to CommonMark

[minchinweb]

I've been thinking for a while that I would like to move to a CommonMark Markdown parser. The goal would be to have consistence performance, backed by a spec that has worked through many of the edge cases. As well, the hope is to make it easier to move Markdown content between applications (like to/from Obsidian).

Also, Pelican has taken to installing a CommonMark parser (markdown-it-py) as a sub-dependency of rich, which Pelican has taken to using as a CLI output library. So we already have the parser installed!

One complication is that the original Python commonmark.py parser (available on PyPI as commonmark) has been deprecated in favour of markdown-it-py (available as markdown-it-py[plugins]).

Markdown-it-py does support plugins. This is another complication in that I use several Markdown plugins currently, and not entirely standard Markdown as is.

Another issue is how to deal with front matter; I think Pelican is using it's own standard.

Available, existing Pelican Plugins:

• Markdown-IT reader -- actually supports Markdown-it. Available on PyPI. Last updated Dec 2021. Bundles and auto-activates a few plugins
• Pelican FrontMark -- relies on the commonmark package. Available on PyPI. Last updated in Dec 2022.
  • has an open issue about dealing with dates, as expected by Pelican\
  • has an (unsubmitted) PR to allow using the MARKDOWN setting. c.f. Issue #3
• pelican_commonmark plugin -- relies on the commonmark package. Intended to be available as part of the pelican-plugins repo, but never went beyond PR from 2015.
• CommonMark for Pelican -- relies on the commonmark package. Available on PyPI. Last updated Jan 2022. I actually have an open issue (from 2016!) here due to the commonmark package changing their API, and this never got updated to match.
• pelican-gfm plugin -- actually implementing GitHub Flavoured Markdown. Never progressed past a PR for 2019.

There doesn't appear to be anything in the pelican-plugins organization.

c.f. getpelican/pelican#2984

[minchinweb]

• release Markdown-IT reader to support arbitrary plugins
  • see https://github.com/MinchinWeb/minchin.pelican.readers.commonmark
• [-] MD-it plugin to support Pelican-style frontmatter
  • this does not seem to be needed; Pelican the existing code is there to pull off and read the front matter before passing it to the Reader
• Pelican (or MD-it) plugin to pull title from (first) H1 tag (to support Obsidian vault, as used)
• MD-it plugin to support checklists, as used by Obsidian theme
  • https://github.com/MinchinWeb/minchin.md-it.fancy-tasklists
• [-] MD-it plugin for code highlighting
  • the base code for the Reader seems to do this
• MD-it plugin for markdown.extensions.extra
• [-] MD-it plugin for markdown.extensions.meta
• MD-it plugin for superfences
• MD-it plugin for caret
  • use single caret for superscript
  • use double caret for insertion (underline?)
• MD-it plugin for tilde
  • use single tildes for subscript -- done with mdit-py-plugins v0.5.0
  • use double tildes for deletion -- built-in (but needs enabling) in markdown-it-py
• MD-it plugin for smarty
• MD-it plugin for arithmatex -- so MathJax equations (always) make it through
• MD-it plugin for smart symbols
• MD-it plugin for def_list
  • mdit_py_plugins.deflist.deflist_plugin
• ?? MD-it plugin for progress bar
• ?? MD-it plugin for admonition
• ?? MD-it plugin for HTML5 pictures
• ?? MD-it plugin for emoji
• MD-it plugin needed?? for markdown=1 to work?
• ?? MD-it plugin for Heading Anchors (already in mdit-py-plugins) (c.f. Funnel's fix)

[minchinweb]

Update todo list.

Also, a footnote plugin: mdit_py_plugins.footnote.footnote_plugin

[minchinweb]

Can now pull H1 tag if no title, but it has to be built in to the reader (no plugin provides both the metadata and the content).

[minchinweb]

Can now use Markdown-IT's front matter plugin to separate front matter from the body of the post. May need to do a lot of metadata wrangling now to get it to work as expected....

There is still an issue where if it pulls out the title, it strings it along with spaces between each letter.

[minchinweb]

Frontmatter is now processed as YAML; this is built in to the (WIP) Pelican reader.

Wikilinks are now handled by a dedicated plugin --> https://github.com/MinchinWeb/minchin.pelican.plugins.wikilinks

Backlinks can be handled by an existing plugin (by someone else!) --> https://github.com/renyuneyun/pelican-interrefs

[minchinweb]

minchinweb/minchin.pelican.plugins.wikilinks#1

Re Wikilinks: they could be handled as a Markdown-IT plugin, that relies on Pelican to actually render the link targets (as a follow-on processing step). This would likely be better, as it would provide less false positives. To investigate.

[minchinweb]

Additional icons for checkboxes:

• https://github.com/r-u-s-h-i-k-e-s-h/Obsidian-CSS-Snippets/blob/Collection/Snippets/Checkboxes%20-%20Minimal%20theme.md
• https://github.com/r-u-s-h-i-k-e-s-h/Obsidian-CSS-Snippets/blob/Collection/Snippets/Checkboxes%20-%20Origami%20theme.md
• https://github.com/r-u-s-h-i-k-e-s-h/Obsidian-CSS-Snippets/blob/Collection/Snippets/Checkboxes%20-%20SlRvb's%20checkboxes%20(ITS%20theme).md
• https://publish.obsidian.md/slrvb-docs/ITS+Theme/Alternate+Checkboxes

[egberts]

WOW! I was about to embark down on the Markdown-IT integration path for Pelican.

But I am spending time studying Pelican internals and writing up a flowchart on this.

[egberts]

Related?

getpelican/pelican#3344

[minchinweb]

WOW! I was about to embark down on the Markdown-IT integration path for Pelican.

I wrote a (Pelican) plugin to do this -->

https://github.com/MinchinWeb/minchin.pelican.readers.commonmark

I've published this to PyPI; you should be able to use it as is. Please try it out, and let me know if you run into any issues or shortcomings. I use it regularly on my side.

One disappointment I discovered with using Markdown-IT-Py is that very few of the Markdown-IT plugins originally written in JS have been ported to Python yet.

[egberts]

And it WORKS! Nice job.

This is an evaluation mode, not an implementation. But nice job.

[minchinweb]

Superscript plugin -- https://github.com/hasgeek/funnel/blob/main/funnel/utils/markdown/mdit_plugins/sup_tag.py

Subscript plugin -- https://github.com/hasgeek/funnel/blob/main/funnel/utils/markdown/mdit_plugins/sub_tag.py

(Although this second one may now ship with the "official" plugins now)

Funnel had several other plugins in the same directory which may be of interest.

[minchinweb]

• plugin for inline tag cross references; e.g. markdown text with a #tag in the middle