Colour handling improvements #8702

Jermolene · 2024-10-25T09:02:10Z

Introduction

This PR brings several new features for end users:

Automatically switching between a dark and light palette as the operating system setting changes (and to do so without making the wiki dirty)
Customisation options for palettes. For example, users might choose a base hue, with the colours of the palette automatically adapting to it
A generalisation of the dark vs. light mechanism to allow an arbitrary number of distinct schemes that are dynamically selected. For example, a palette that has a different scheme for night, morning, day and evening that automatically change with the time of day

There are also new capabilities for palette authors:

Inheritance for palettes, making it easy to create chains of variants of a base palette
Self contained palettes that can contain both dark and light variants (or variants for any other custom scheme)

To make all of these new features possible, this PR also includes some useful new general purpose mechanisms and features:

Background actions that are triggered whenever there is a change to the results of a specified filter
Several new filter operators for manipulating colour values. The underlying functionality comes from the color.js library
New media query tracking mechanism that can track the results of any CSS media query (not just dark mode), storing the results in a shadow $:/info/... tiddler
New changecount filter operator
New :apply filter run prefix

Try the Demo

A preview build is available at https://deploy-preview-8702--tiddlywiki-previews.netlify.app/

Changes to Palettes

Palette entries are now defined as filters that must be evaluated, rather than wikitext that must be wikified.

This makes it possible to create palettes that reference and modify other colours. For example:

tiddler-controls-foreground-selected: [tf.interpolate-colours[background],[foreground],[0.9]]

Retrospectively re-interpreting palettes entries as filters would normally mean that <<colour background>> would no longer work. Instead these entries are automatically converted to the updated form [function[colour],[background]].

Palette Compilation

The key idea underpinning these changes is a fundamental change to the way that TiddlyWiki handles palettes. At the moment, palette entries are named items that can contain either a CSS colour string, a CSS colour token like "inherit", or can use the <<colour>> macro to reference another colour palette entry. Thus, palette entries have to be wikified before they can be used. This has turned out to be very limiting and doesn't provide a viable path to the complex colour manipulations shown above. Switching to filters might make things worse, by encouraging authors to use complex expressions within palettes.

The fix is compiled palettes: at the point of switching to a new palette, the colours within it are "compiled" to raw CSS colour values (typically but not necessarily in #rrggbbaa format). This allows palette entries to be used directly, without the requirement to wikify them.

The static palette is created in a new system tiddler $:/temp/palette-colours by an action procedure that is invoked at startup and when switching to a new palette.

There should not be any backwards compatibility issues because the use of background actions means that any code that changes $:/palette will automatically trigger the recompilation of the palette.

This change also allows us to change the <<colour>> procedure to be a function, which allows it to be used as the value for a style attribute:

<div style.background=<<colour tiddler-background>>>

Automatic Palette Readability Tests

Palettes can opt to include readability tests as special palette entries. The results of these tests are shown at the bottom of the palette chooser. For example:

?base-background-ink: [tf.check-colour-contrast[base-background],[base-ink],[45]]

The test framework looks for palette entries starting with a question mark and runs the associated filters. The filter should return nothing if there are no errors, or a textual error message if the conditions are violated. Sample output:

alert-contrast: 42.357: alert-background/foreground contrast is too low
background-foreground-contrast: 41.915: background/foreground contrast is too low
base-ink-secondary: 42.357: base-ink/base-secondary contrast is too low
base-paper-ink: 41.915: base-paper/base-ink contrast is too low
base-paper-primary: 32.682: base-paper/base-primary contrast is too low
base-paper-tertiary: 30.502: base-paper/base-tertiary contrast is too low
code-contrast: 28.565: code-background/code-foreground contrast is too low

Colour Manipulation

We need a colour manipulation library that can calculate variants of colours. Only color.js met the requirements of being able to work with P3 colours and the OKLCH colour space. It also includes a CSS colour string parser which can replace the simple one that TiddlyWiki has always incorporated.

Media Query Tracker

The CSS media query tracker allows a media query to be bound to an info tiddler so that the current state of the query is reflected in the value of the tiddler. The value is updated dynamically.

The use of the info mechanism for the CSS media query tracker means that these tiddlers are dynamically created as shadow tiddlers within the $:/temp/info plugin, and so do not appear in tiddler lists.

The mechanism is used to implement the existing dark mode tiddler with the following configuration:

title: $:/core/wiki/config/MediaQueryTrackers/DarkLightSwitch
tags: $:/tags/MediaQueryTracker
media-query: (prefers-color-scheme: dark)
info-tiddler: $:/info/browser/darkmode
info-tiddler-alt: $:/info/darkmode

Note the use of info-tiddler-alt to specify a redundant secondary info tiddler. This is used by the dark mode tracker to maintain compatibility while changing the info tiddler title for consistency.

Background Actions

The new background actions mechanism allows action strings to be invoked automatically in the background whenever the results of a filter change.

The preview includes a demonstration background action that displays an alert with a list of tiddlers in the story river whenever the story river changes:

title: SampleBackgroundAction: Story Change
tags: $:/tags/BackgroundAction
track-filter: [list[$:/StoryList]]

<$action-sendmessage $message="tm-notify" $param="SampleBackgroundAction: Story Change" list={{$:/StoryList!!list}}/>

Story List:

<ol>
<$list filter="[enlist<list>]">
<li>
<$text text=<<currentTiddler>>/>
</li>
</$list>
</ol>

`apply` Filter Run Prefix

An experimental new filter run prefix that makes it possible to use computed values as variables within a filter run. For example:

\function tf.interpolate-colours(paletteEntryA,paletteEntryB,weight)
[function[colour],<paletteEntryA>] [function[colour],<paletteEntryB>] :apply[<weight>colour-interpolate:oklch<$1>,<$2>]
\end tf.interpolate-colours

Backwards Compatibility

The current content of $:/PaletteManager is moved into $:/PaletteEditor, and $:/PaletteManager repurposed as the control panel palette switcher
$:/config/DefaultColourMappings/ now only supports CSS colours, and not indirections via <<colour X>> or [function[colour],[X]]

References

The media query tracker and background actions mechanisms were originally developed in a separate pull request
[IDEA] Tools for manipulating colours and palettes #5236 discusses the general idea of dynamically computed colours, and includes a very neat demo from @cdruan
A thread discussing using the colour macro as an attribute value - https://talk.tiddlywiki.org/t/how-to-use-the-colour-macro-as-an-attribute-value/10051
A thread discussing improvements to colour handling, including a similar idea for palette previews - https://talk.tiddlywiki.org/t/idea-approach-to-palettes-and-themes/9431

Progress

Documentation
- Palettes and colour schemes
- Media Query Tracker
- Background actions
- changecount operator
- colour-interpolate operator
- colour-lighten operator
- colour-darken operator
- colour-get-oklch operator
- colour-set-oklch operator
- colour-contrast operator
- colour-best-contrast operator
- Adopt palettes from https://yatagarasu.tiddlyhost.com/

@linonetwo

The replacement library from https://colorjs.io/ is much, much larger but I think we can develop a custom build that uses treeshaking to whittle the code down to the bits that we need. @linonetwo does that sound feasible? I intend the explore further improvements but I wanted to start by establishing a library that can do modern P3 and OKLCH colour calculations.

Really just syntactic sugar for the wikify widget

Using the new wikify operator. Currently has a bug whereby redirected colours (like "tiddler-background") do not work. Direct colours like "background" do work. Note the hacks needed to makeFakeWidgetWithVariables work

github-actions · 2024-10-25T09:02:22Z

Confirmed: Jermolene has already signed the Contributor License Agreement (see contributing.md)

themes/tiddlywiki/vanilla/base.tid

core/wiki/paletteswitcher.tid

core/modules/utils/dom/color.js/tiddlywiki.files

kookma · 2024-11-04T03:38:02Z

Is there a demo to test colour handling improvement?

Jermolene · 2024-11-04T18:18:08Z

Is there a demo to test colour handling improvement?

Hi @kookma I've posted a preview to tiddlyhost

See #8702 (comment)

These are mostly RGB entries that were previously missing, filled in with values from Vanilla. The goal is still not to have any direct RGB colours in the palette, just computed colours derived from the base colours

Makes debugging easier, and works in CSS and as a style.prop assignment

pmario · 2025-03-07T16:30:32Z

core/palettes/TwentyTwenties.tid

-button-border:
+button-background: 
+button-border: 
+button-foreground: 


Do we need spaces here?

I intend to fill in those entries later.

pmario · 2025-03-07T16:35:39Z

core/palettes/TwentyTwenties.tid

@@ -76,7 +78,7 @@ dropdown-background: [tf.colour[background]]
 dropdown-border: [tf.colour[muted-foreground]]
 dropdown-tab-background-selected: [tf.colour[background]]
 dropdown-tab-background: [tf.interpolate-colours[base-paper],[base-ink],[0.9]]
-dropzone-background: rgba(0,200,0,0.7)
+dropzone-background: [tf.colour[base-secondary]colour-set-alpha[0.7]]


I do like this possibility: [tf.colour[base-secondary]colour-set-alpha[0.7]] but once the "alpha" values are settled, would it be possible to have eg: colour-set-alpha<lighter>, colour-set-alpha<lighter> colour-set-alpha<lightest> and eg: dark, darker, darkest -- What do you think?

In the same realm. I know colour-set-alpha is the right technical term, but our users would probably prefer darken-colour<percent>

Question Does "alpha" set transparency or does it calculate a lighter or darker colour?

Hi @pmario to darken a colour one would use colour-set-oklch:l[0.5] to set the lightness.

We don't yet have operators to extract the components of a colour in a particular colour space, but once we have those then it would be possible to make macros for setting the alpha or brightness in relative steps.

Question Does "alpha" set transparency or does it calculate a lighter or darker colour?

Apologies, to be clear it sets transparency

github-actions · 2025-06-19T16:14:28Z

📊 Build Size Comparison: `empty.html`

Branch	Size
Base (master)	2529.0 KB
PR	2820.5 KB

Diff: ⬆️ Increase: +291.5 KB

⚠️ Warning: Size increased significantly.

Leilei332 · 2025-06-20T09:30:59Z

core/modules/background-actions.js

+(function(){
+
+/*jslint node: true, browser: true */
+/*global $tw: false */
+"use strict";


Function wrappers should be removed now.

Thanks @Leilei332 I've removed them in a4c84d7 – I think they still need to be removed from a number of my other open PRs.

Leilei332 · 2025-06-20T09:37:39Z

core/modules/background-actions.js

+	return id;
+};
+
+BackgroundActionDispatcher.prototype.untrackFilter = function(enterValue) {


Since ES2017 is the baseline now, I think we are ready to migrate to class (since refactoring to class is easy in VSCode). We should also use function argument default value and use const in some occasions.

Good point, which again applies to my other PRs.

Jermolene · 2025-06-20T14:21:04Z

We need to discuss the considerable increase in size of the core incurred by this PR. Nearly 300KB is a considerable impact; I think it's 11% of the current size of empty.html.

📊 Build Size Comparison: empty.html

Branch Size
Base (master) 2529.5 KB
PR 2822.2 KB
Diff: ⬆️ Increase: +292.6 KB

My original thinking was that we could conceal the size impact by adopting compression for the core, but it would be reasonable to expect that the percentage increase in size would be roughly similar.

I wondered if it might be possible to reduce the size of the library through tree shaking. I don't have experience of the tooling but perhaps others might be able to comment. However, we are using some substantial subsystems from the library (such as parsing CSS colours, which I would imagine might be a hefty chunk of code), and so the impact might not be enormous.

Leilei332 · 2025-06-23T10:32:42Z

Some tags' foreground colors in this PR looks weird to me.

Jermolene added 11 commits October 21, 2024 09:03

Include colour.js license and mark version number

93d1c05

Introduce wikify operator

467a1a4

Really just syntactic sugar for the wikify widget

Rewrite colour macro as a function

a23ee16

Using the new wikify operator. Currently has a bug whereby redirected colours (like "tiddler-background") do not work. Direct colours like "background" do work. Note the hacks needed to makeFakeWidgetWithVariables work

Merge branch 'master' into colour-improvements

e1e73d2

Fix typo

1e5c69e

Fix nested colour definitions

4af573a

Fix up the tests

96b85ed

Add colour-lighten and colour-darken operators

d372729

Remove obsolete comment

b5a22e3

Improve colour palette switcher with previews

fc36941

Palette chooser styling tweaks

151f61a

pmario reviewed Oct 25, 2024

View reviewed changes

themes/tiddlywiki/vanilla/base.tid Show resolved Hide resolved

Jermolene added 5 commits October 30, 2024 20:38

Merge branch 'master' into colour-improvements

4445111

Merge branch 'master' into colour-improvements

4f2754d

Fix palette chooser when displayed in "Tools" dropdown

bc0fde6

Merge branch 'master' into colour-improvements

85fa913

Add expertimental colour-oklch operator

a8fb071

pmario reviewed Nov 3, 2024

View reviewed changes

core/wiki/paletteswitcher.tid Show resolved Hide resolved

pmario reviewed Nov 3, 2024

View reviewed changes

core/modules/utils/dom/color.js/tiddlywiki.files Show resolved Hide resolved

Jermolene added 2 commits November 4, 2024 10:39

Merge branch 'master' into colour-improvements

81b7bb4

Indentation for palette switcher

ff5c846

Jermolene added 5 commits November 4, 2024 19:26

Merge branch 'master' into colour-improvements

23eccd1

Restore default styling for chosen chooser item

c607440

See #8702 (comment)

Remove logging

348f717

Testing Netlify CI

c3ce9ca

Testing CI

a4d9303

Merge branch 'master' into colour-improvements

75ba085

saqimtiaz mentioned this pull request Feb 28, 2025

[IDEA] SetTimeout / SetInterval / SetAlarmClock widget #8967

Closed

Jermolene added 9 commits March 6, 2025 21:38

Merge branch 'master' into colour-improvements

8467fa3

Fix from mismerging from master

624cf95

TwentyTwenties: copy over missing palette entries

250ee90

These are mostly RGB entries that were previously missing, filled in with values from Vanilla. The goal is still not to have any direct RGB colours in the palette, just computed colours derived from the base colours

Preview: Fix sidebar link colour

3001696

Include palette entry name in palette entries

ee977de

Makes debugging easier, and works in CSS and as a style.prop assignment

Add colour-set-alpha operator

d42e3d3

Fix colour-interpolate with missing colours

4c95ae5

TwentyTwenties improvements

5e27462

Fix tests

5897b82

pmario reviewed Mar 7, 2025

View reviewed changes

Jermolene mentioned this pull request Mar 7, 2025

Introduce multi-valued variables and let filter run prefix #8972

Draft

Merge branch 'master' into colour-improvements

e86eb28

Jermolene mentioned this pull request Apr 6, 2025

[BUG] testcase colours don't exist for anything other than vanilla #9004

Closed

Merge branch 'master' into colour-improvements

54382d6

saqimtiaz added this to Planning for v5.4.0 May 20, 2025

saqimtiaz moved this to In progress in Planning for v5.4.0 May 20, 2025

This was referenced Jun 3, 2025

Introduce background actions and media query tracker mechanisms #8555

Closed

[IDEA] Introduce a vanilla dark palette #9083

Open

Leilei332 mentioned this pull request Jun 18, 2025

Fix appearance of disabled element #9091

Closed

Merge branch 'master' into colour-improvements

e9d87fb

Leilei332 reviewed Jun 20, 2025

View reviewed changes

Jermolene added 4 commits June 20, 2025 17:23

Merge branch 'master' into colour-improvements

5335ebf

Remove function wrappers

a4c84d7

Adopt class syntax

dcf0f44

Adopt more ES2017 features

7f2a47f

Colour handling improvements #8702

Are you sure you want to change the base?

Colour handling improvements #8702

Uh oh!

Conversation

Jermolene commented Oct 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Introduction

Try the Demo

Changes to Palettes

Palette Compilation

Automatic Palette Readability Tests

Colour Manipulation

Media Query Tracker

Background Actions

apply Filter Run Prefix

Backwards Compatibility

References

Progress

Uh oh!

github-actions bot commented Oct 25, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kookma commented Nov 4, 2024

Uh oh!

Jermolene commented Nov 4, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 Build Size Comparison: empty.html

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jermolene commented Jun 20, 2025

Uh oh!

Leilei332 commented Jun 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Jermolene commented Oct 25, 2024 •

edited

Loading

`apply` Filter Run Prefix

github-actions bot commented Jun 19, 2025 •

edited

Loading

📊 Build Size Comparison: `empty.html`

Leilei332 commented Jun 23, 2025 •

edited

Loading