Using New UUID v7 as a Tiddler ID for MWS Related Data Storages

[pmario]

It is proposed to use UUID v7 as a tiddler ID for MWS related data storages.

• The UUID v6, v7 and v8 specs are described at RFC 9562 from May 2024.
• The old spec RFC 4122 (v1 to v5) was published in July 2005. A lot has happened since then.

The Update Motivation describes the technical details and what's special about the new UUID versions.

Also have a closer look at: DBMS and Database Considerations

I will summarize what I think is important for TW in my own thoughts here:

Database related

1. UUID v7 rises monotonically since it uses an 48bit Unix epoch timestamp in the most significant bits
2. Together with the random elements of the UUID, collisions should be avoided
3. Since v7 UUIDs are monotonic, they make perfect primary keys for databases
4. Since they are universally unique, they also work with distributed DBs without collision
5. Once a UUID is created it must not be changed anymore
   5.1 They can be stored as text, where they need more storage space
   5.1.2 For easier debugging at the beginning it should be stored as text
   5.3 They can be stored as 16byte binary data for better storage (See Database Considerations link above)

For TW

6. Theoretically, we would not need to store the created field anymore, since it can be replicated from the tiddler ID. See "Backwards Compatibility Considerations"
7. The modified field may have the same properties as created. But UUID v8 may be an option too
8. TWs created and modified fields can not be used for database IDs anymore, since we do have $:/config/TimestampDisable

For Users

9. From time to time we have requests for tiddler UUIDs that are guaranteed to not change
10. With the proposed addition it is easy to provide this field

Core Internals

20. Currently there are 5 fields which use special treatment by the core js code.
    20.1 created, modified, color, tags and list
    20.2 See source code at boot.js

21. created and modified implement 2 functions
    21.1 parse ... which converts TW data format to Unix date timestamp
    21.2 stringify ... which converts a Unix timestamp into a TW date, which is almost human readable 20250213171512000

Backwards Compatibility Considerations

30. There should be a new tiddler field UUID7. I think that name should be not used by any user yet, since the spec is very new.
31. The core provides UUID7 read only functions, but there are no user facing functions to modify it.
32. As seen at 21) there are already function calls if created and modified are accessed or written, there should be no performance difference to the new UUID7 fields
33. The modified field stays as it is and can be influence by $:/config/TimestampDisable
34. The created field will be calculated from UUID7 instead of a JS Date() variable

Possible Upgrade Path(s)

40. The TW browser upgrader and the browser $:/Import process call an upgrade function, that will allow us to generate new IDs from existing created and or modified fields, if needed

41. If there is no created field like for shadow tiddlers, TODO !!!!!!!!!!!!!!!

42. UUID7 replaces created field so $:/config/TimestampDisable will not effect created anymore

43. modified-7 field is added //may be UUID v8// TODO !!!!!!!!!!!!

JSON export

50. It should be straight forward to export the new and or old format

Source of Truth

60. Once implemented the new format is the source of truth
61. Upgrade and Export will act accordingly

In the Future

70. In database storage backends, it should be possible to have the same title for different tiddlers.

Server Side

80. Tiddler UUID7 will be the source of truth for the primary key
81. created and modified are just a data fields
82. modified-7 may be UUID v8 with a SHA256 hash in custom_b and custom_c structure -- TODO !!!!!!!!!!!!!! more thoughts needed

Implementation Details (MVP prototype)

UUID code would look as follows. TODO: proper testing.

It implements uuid
.v7 ... which should return a string eg: 0195002c-02ac-743d-a29f-cb3f978015c0
.validate ... validates an input if created from a 3rd party
.parse ... convert UUID string into byte array
.stringify ... converts byte array to UUID string

const hex = [];
for (let i = 0; i < 256; ++i) {
    hex.push((i + 0x100).toString(16).substr(1));
}

function rng() {
    if (typeof crypto !== 'undefined' && crypto.getRandomValues) {
        let rnds = new Uint8Array(16);
        crypto.getRandomValues(rnds);
        return rnds;
    }
    throw new Error('No secure random number generator available.');
}

function v7() {
    const now = Date.now();
    const rnds = rng();

    rnds[0] = (now >>> 8) & 0xff;
    rnds[1] = now & 0xff;
    rnds[2] = ((now >>> 24) & 0x0f) | 0x70;
    rnds[3] = (now >>> 16) & 0xff;
    rnds[4] = (now >>> 32) & 0xff;

    rnds[8] = (rnds[8] & 0x3f) | 0x80;  // Set the version to v7

    return stringify(rnds);
}

function validate(uuid) {
    const regex = /^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i;
    return regex.test(uuid);
}

function parse(uuid) {
    if (!validate(uuid)) {
        throw new Error('Invalid UUID format');
    }
    const bytes = new Uint8Array(16);
    uuid.toLowerCase().replace(/[0-9a-f]{2}/g, (hex, i) => {
        bytes[i >> 1] = parseInt(hex, 16);
    });
    return bytes;
}

function stringify(arr) {
    return (hex[arr[0]] + hex[arr[1]] + hex[arr[2]] + hex[arr[3]] + '-' +
            hex[arr[4]] + hex[arr[5]] + '-' +
            hex[arr[6]] + hex[arr[7]] + '-' +
            hex[arr[8]] + hex[arr[9]] + '-' +
            hex[arr[10]] + hex[arr[11]] +
            hex[arr[12]] + hex[arr[13]] +
            hex[arr[14]] + hex[arr[15]]).toLowerCase();
}

module.exports = {
    v7,
    validate,
    parse,
    stringify
};

@Arlen22, @Jermolene, @saqimtiaz -- what do you think?

@sukima This may also be interesting for your TiddlyMap replacement as tiddler ID.

[Arlen22]

I would love that.

From a database perspective, everything gets a lot more complicated if users are allowed to change the primary identifier, whether that's the ID or title, so having a fixed ID would be wonderful.

Currently MWS is using the insert id for a revision number. It's ingenious, but it's not a sufficient solution for revision tracking and a globally unique identifier would be a lot more useful.

Having a primary key like this is basically required for revision tracking so I probably would have proposed it whenever we got around to that, but you know way more about the tiddler side of things than I do.

I'm pretty sure the title would still have to be unique, and we could easily maintain a unique index on the server side.

Should the recipe system overlay based on UUID or title or both?

If the same identifier has two different titles on two different layers they would both be visible unless we Make sure we take the top identifier and top title.

[pmario]

From a database perspective everything gets a lot more complicated if users are allowed to change the primary key.

My suggestion should not indicate, that we should allow to change the primary key. Once the primary key is generated, it should never ever be changed again.

I'm pretty sure the title would still have to be unique, and we could easily maintain a unique index on the server side.

Yes. That's why I did mention point 70: In the future it may be possible to have the same title. But I think with the current client we can not handle that - yet.

Should the recipe system overlay based on UUID or title or both?

I am not sure, what you mean here.

If the same identifier has two different titles on two different layers they would both be visible unless we Make sure we take the top identifier and top title.

The current system with unique tiddler titles has a big advantage at the moment. The title in the "latest" bag wins. This mechanism for example is essential for the Anna Freud Manuals.

So that's one of the biggest USPs that TW has at the moment.

[Arlen22]

So what happens if a tiddler in a lower bag gets edited? I'm assuming it would have the same ID and just be copied to the top bag, essentially forking it. Then if that forked tiddler gets renamed, what do we do then? Do we reveal the previous title? Or do we keep it hidden because the same ID is being used in a higher layer?

I guess the clue is in the word fork. If it gets renamed you don't want the old version to suddenly appear. But we could make that configurable in MWS depending how much revision tracking is happening.

I mean some people aren't going to want to bother with the idea of forking and they just want the title to show up again when they rename a tiddler, while other people would be seriously bothered by having the old tiddler show up again.

I'm not objecting at all, I think it's a great idea. I'm just trying to think through scenarios we'll probably have to address.

MWS is going to make a list of all titles available to the client, so people can build UI around checking whether the same tiddler is in multiple layers.

[pmario]

Tiddlers from lower level bags work like shadow tiddlers. They are cloned into the top layer bag. When deleted they behave exactly like shadow tiddlers. The shadow tiddler will take over again.

That's a mechanism that our users already know from plugins. So there will be no problems. We did have all of this already with TiddlySpace. The only thing that is needed is a good visual indicator.

With TS we used a blue circle for local / private / editable tiddlers and a pink circle for global shadow tiddlers. Users that are not editors will only see pink tiddlers.

[Jermolene]

Thanks @pmario. UUIDs have come up several times before over the years. After a good deal of discussion and though, I have come to the conclusion that there is no benefit in rearchitecting TiddlyWiki to use UUIDs as the primary key for tiddler storage, and that it could only be done with significant redundancy, and poor performance. I understand that it may take some further discussion to persuade others, and so let's do so as much as needed.

To start with, you note:

From time to time we have requests for tiddler UUIDs that are guaranteed to not change

The simplest way to handle that request is to use a field to store the UUID. Some tweaking is necessary to assign a new UUID each time a tiddler is created. One of the characteristics of UUIDs is that no matter where and when they are created they are guaranteed to be unique, which means that there shouldn't be multiple tiddlers sharing the same UUID. With the UUID in an ordinary field, the standard machinery of TiddlyWiki ensures that the UUID is preserved across edits.

Most of the rest of the OP is really about a separate idea: to rearchitect TiddlyWiki/MWS to use UUIDs as the primary key. The fact that the post is so long hints at the problem: this is a very complex proposal that adds a good deal of complexity to TiddlyWiki.

There is immediately a fundamental incompatibility because using UUIDs as the primary key breaks the usual semantics of a new tiddler overwriting a previously existing tiddler with the same title. Additional mechanisms can be overlayed to add the title field semantics, but the result is much, much more complicated than what we have now.

One of the key characteristics of TiddlyWiki is that it establishes a very, very simple and easy to understand data model that is rich enough to support all the higher level structures that a conventionally encountered in programming. For that reason, making the basic semantics more complicated is not acceptable.

With all that said, I think there may be a role for UUIDs in MWS: as identifiers for revisions of a tiddler. This approach does not cause any problems with TiddlyWiki's existing semantics, and it is useful for external tooling to be able to refer unambiguously to a snapshot of a tiddler.

[pmario]

After a good deal of discussion and though, I have come to the conclusion that there is no benefit in rearchitecting TiddlyWiki to use UUIDs as the primary key for tiddler storage, and that it could only be done with significant redundancy, and poor performance.

With the OP I did not intend to change the internal data structure for the client side object store. I do not intend to get rid of the TW title field.

The intention is to enhance the possibilities using database backends.

I wanted to suggest, that there should be a UUID in the core, that we can use if we need one. The side effect of the new UUID v7 spec is time based sorting built into a tiddler field, that will never be touched, once it exists.

As I pointed out at 8) in the OP -- Since $:/config/TimestampDisable exists in the core the created field can not be used anymore for a unique database key / index, since users are allowed to modify it. The new UUID7 field must not change after it is created.

Most of the rest of the OP is really about a separate idea: to rearchitect TiddlyWiki/MWS to use UUIDs as the primary key. The fact that the post is so long hints at the problem: this is a very complex proposal that adds a good deal of complexity to TiddlyWiki.

The OP is long, since I needed to go into detail to clearly explain my thoughts and some extended links to existing code.

Yes. I did think about it, as I did visualise the MWS-original internal database structure, where the tiddler_id is an integer. IMO a counter makes a poor database key. It's OK for a prototype but not for production.

I made the graphical schemas to better understand the MWS database design and connections.

MWS Original Database Structure (click to show image)

I know that not every aspect of the OP is thought through 100%.

The new MWS-schema used by @Arlen22 is slightly improved. It separates session and ACL data from tiddler data. But it also uses an integer for the tiddler primary key, which I think can be improved using an UUID7.

MWS Arlen Database Structure (click to show image)

[Arlen22]

@pmario I'm very glad you brought this up, as I've been considering exactly the same thing. I looked at UUIDv7 a bit at the beginning but I needed more discussion on it, and considering where I'm at now, I think this is perfect timing.

My thought isn't as much about syncing changes to the client, since that's simple enough with ascending integers. Rather I'm looking at the requirements of scale, and ascending integers are simply not practical at scale. I should probably change the tiddler_id to a UUID sooner rather than later. It is a revision ID, not a tiddler ID, serving as a modified field, not a created field. I should probably rename the field as well for clarity.

At the moment MWS has no need for a static tiddler id. However, if a revision history is implemented, suddenly a static UUID becomes critical, because a tiddler being renamed is most likely renamed because of its content, and the history of that content needs to persist across the title change. The history of all fields including the title must be preserved, and so a static tiddler ID that can never be changed becomes critical.

Almost any collaborative tooling is going to need a static UUID to overcome renames. Document edit sessions, revision history, and conflict resolution, to name a few.

This also means we need to make it harder to overwrite tiddlers. Rather than allowing tiddlers to be overwritten, we should in most cases require the other tiddler to be deleted manually by the user, to prevent accidental loss of an illustrious edit history. For that matter, some will opt to never allow a tiddler to actually be deleted, but rather banished forever to the shadow realm of deleted tiddlers. It will also allow a recycle-bin like system to efficiently handle multiple deletions of the same title (this would be keyed on the revision ID rather than tiddler ID.

I'm sure we could look to plenty of sources for inspiration when it comes to edit history. How has edit history been implemented in the various tiddlywiki spheres, like tiddlyspace?

[Jermolene]

With the OP I did not intend to change the internal data structure for the client side object store. I do not intend to get rid of the TW title field.

But if the semantics remain the same on the client, then they must also remain the same on the server. Any use of UUIDs or other special indexing schemes needs to be invisible from the point of view of the client.

The intention is to enhance the possibilities using database backends.

Do you mean that you are suggesting internal enhancements to the server that are invisible from the clients perspective?

I wanted to suggest, that there should be a UUID in the core, that we can use if we need one. The side effect of the new UUID v7 spec is time based sorting built into a tiddler field, that will never be touched, once it exists.

You haven't addressed my argument as to why UUID should not be in the core. If we expose the UUID in a separate field then we will have created an incompatibility with standalone TiddlyWiki.

I don't understand why time based sorting is important? Why not just a timestamp field on the server that is invisible to the client?

As I pointed out at 8) in the OP -- Since $:/config/TimestampDisable exists in the core the created field can not be used anymore for a unique database key / index, since users are allowed to modify it. The new UUID7 field must not change after it is created.

Why would anyone want to use the created field as a database key? It is obviously not unique.

Yes. I did think about it, as I did visualise the MWS-original internal database structure, where the tiddler_id is an integer. IMO a counter makes a poor database key. It's OK for a prototype but not for production.

Anything like a tiddler_id is an internal concern of the server and must not be exposed to the client within the ordinary tiddler structure.

[Leilei332]

IMO the purpose of using UUID shouldn't be storing timestamps. It may be better to identify unique tiddler by its UUID rather than the title.

[pmario]

It does not matter if the UUID also contains a timestamp -- It is a UUID, that can be used as such. If you read the OP carefully, you will see that UUID v7 has some unique properties that make it very useful for database primary keys / indexes.

As Jeremy pointed out client side internal TW store has to be title based for performance reasons. See: #6116 (comment) section 6 Object store for more details.

Since databases can have secondary indices, which are updated at "write time" we do have more possibilities.

[Arlen22]

The UUID containing a timestamp helps to reduce the already miniscule risk of collisions (although I'm sure some could argue the risk is increased, and I don't know the mathematical answer to that). But more importantly it makes it very easy to sort by created time. If we have a revision UUID as well it could be sorted by modified time.

If we want to use UUIDs for revision tracking, all of these features will come in handy, I think.

[pmario]

If we want to use UUIDs for revision tracking, all of these features will come in handy, I think.

I did think about tiddler revisions using UUID-v8, which allows for a custom format. But I did not think about that one in detail. See Possible Upgrade Paths section 43 in the OP.

Revisions can be dealt with at a later date, without breaking compatibility.