IPIP-499: UnixFS CID Profiles #499
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,6 +1,6 @@ | ||
| --- | ||
| title: 'IPIP-0499: UnixFS CID Profiles' | ||
| date: 2026-01-14 | ||
| ipip: proposal | ||
| editors: | ||
| - name: Michelle Lee | ||
|
|
@@ -54,110 +54,125 @@ This proposal introduces **configuration profiles** for CIDs that represent file | |
|
|
||
| ## Motivation | ||
|
|
||
| While CIDs and UnixFS DAGs are cryptographically verifiable, the same file or directory can produce different CIDs across UnixFS implementations, because DAG construction parameters like chunk size, DAG width, and layout vary between tools. Often, these parameters are not even configurable by users. | ||
|
|
||
| This creates two problems: | ||
|
|
||
| - **Broken hash semantics:** Unlike standard hash functions where identical input produces identical output, UnixFS CIDs depend on DAG construction parameters. Simple CID comparison leads to false-negatives. | ||
| - **Verification overhead:** Without knowing the original parameters, users must retrieve and compare entire DAGs to verify content, adding storage, bandwidth, and complexity. | ||
|
|
||
| A potential solution is to define configuration profiles: well-known parameter presets that implementations can adopt when common conventions for DAG creation are desired. | ||
|
|
||
| See related discussion at <https://discuss.ipfs.tech/t/should-we-profile-cids/18507> | ||
|
|
||
| ### UnixFS parameters | ||
|
|
||
| The following [UnixFS](https://specs.ipfs.tech/unixfs/) parameters were identified as factors that affect the resulting CID: | ||
|
|
||
| 1. CID version, e.g. CIDv0 or CIDv1 | ||
| 1. Multibase encoding for the CID, e.g. `base32` | ||
| 1. Hash function used for all nodes in the DAG, e.g. `sha2-256` | ||
| 1. UnixFS file chunking algorithm and chunk size (e.g., fixed-size chunks of 256KiB) | ||
| 1. UnixFS DAG layout: | ||
| - `balanced`: builds a balanced tree where all leaf nodes are at the same depth. Optimized for random access, seeking, and range requests within files (e.g., video). | ||
| - `trickle`: builds a tree optimized for streaming, where data can be consumed before the entire file is available. Useful for logs and other append-only data structures where random access is not important. | ||
| 1. UnixFS DAG width (max number of links per `File` node) | ||
| 1. [HAMTDirectory](https://specs.ipfs.tech/unixfs/#dag-pb-hamtdirectory) fanout: the branching factor at each level of the HAMT tree (e.g., 256 leaves). | ||
| 1. [HAMTDirectory threshold](https://specs.ipfs.tech/unixfs/#when-to-use-hamt-sharding): max `Directory` size before converting to `HAMTDirectory`, based on `PBNode.Links` count or estimated serialized [dag-pb](https://ipld.io/specs/codecs/dag-pb/spec/) size: | ||
| - `links-count`: `PBNode.Links` length (child count). Simple but ignores varying entry sizes. | ||
| - `links-bytes`: sum of `PBNode.Links[].Name` and `PBNode.Links[].Hash` byte lengths. Underestimates actual size by ignoring UnixFS Data, Tsize, and protobuf overhead. | ||
| - `block-bytes`: full serialized dag-pb node size. Most accurate, accounts for varint `Tsize` and optional metadata such as `mode` or `mtime`. | ||
| 1. Leaves: either [dag-pb wrapped](https://specs.ipfs.tech/unixfs/#dag-pb-node) or [raw](https://specs.ipfs.tech/unixfs/#raw-node) | ||
| 1. Whether empty directories are included in the DAG. Some implementations may apply filtering. | ||
| 1. Whether hidden entities (including dot files) are included in the DAG. Some implementations may apply filtering. | ||
| 1. Directory wrapping for single files: in order to retain the name of a single file, some implementations have the option to wrap the file in a `Directory` with link to the file. | ||
| 1. Presence and accurate setting of `Tsize` (correct UnixFS has `Tsize` of child sub-DAGs). | ||
| 1. [Symlink](https://specs.ipfs.tech/unixfs/#dag-pb-symlink) handling: preserved as UnixFS Type=4 nodes, or followed (dereferenced to target). | ||
| 1. [Mode](https://specs.ipfs.tech/unixfs/#mode-field): optional POSIX file permissions. | ||
| 1. [Mtime](https://specs.ipfs.tech/unixfs/#mtime-field): optional modification timestamp. | ||
|
|
||
| ### Divergence in current implementations | ||
|
||
|
|
||
| We analyzed the default settings across the most popular UnixFS implementations in the ecosystem. The table below documents the divergence that prevents deterministic CID generation today: | ||
|
|
||
| | Parameter | kubo (CIDv0) | helia | storacha | kubo (CIDv1) | dasl | | ||
| | ----------------------------- | ------------------------ | -------------------- | ------------------ | ----------------------------- | ------------ | | ||
| | Based on | v0.39 (`unixfs-v0-2015`) | @helia/unixfs 6.0.4 | w3cli 7.12.0 | v0.39 (`test-cid-v1` profile) | spec 2025-12 | | ||
| | CID version | CIDv0 | CIDv1 | CIDv1 | CIDv1 | CIDv1 | | ||
| | Hash function | sha2-256 | sha2-256 | sha2-256 | sha2-256 | sha2-256 | | ||
| | Chunking algorithm | fixed-size | fixed-size | fixed-size | fixed-size | N/A | | ||
| | Max chunk size | 256KiB | 1MiB | 1MiB | 1MiB | N/A | | ||
| | DAG layout | balanced | balanced | balanced | balanced | N/A | | ||
| | DAG width (children per node) | 174 | 1024 | 1024 | 174 | N/A | | ||
| | HAMTDirectory fanout | 256 blocks | 256 blocks | 256 blocks | 256 blocks | N/A | | ||
| | HAMTDirectory threshold | 256KiB (links-bytes) | 256KiB (links-bytes) | 1000 (links-count) | 256KiB (links-bytes) | N/A | | ||
| | Leaves | dag-pb | raw | raw | raw | N/A | | ||
| | Empty directories | included | included | excluded | included | N/A | | ||
| | Hidden entities | excluded (opt-in) | excluded (opt-in) | excluded (opt-in) | excluded (opt-in) | N/A | | ||
| | Symlinks | preserved | followed | followed | preserved | N/A | | ||
| | Mode (permissions) | excluded (opt-in) | excluded (opt-in) | not supported | excluded (opt-in) | N/A | | ||
| | Mtime (modification time) | excluded (opt-in) | excluded (opt-in) | not supported | excluded (opt-in) | N/A | | ||
|
|
||
| **Terminology:** | ||
|
|
||
| - `included`: Always included in the DAG (no option to exclude) | ||
| - `excluded`: Always excluded from the DAG (no option to include) | ||
| - `opt-in`: Excluded by default; implementations provide a flag to include (e.g., `--hidden` in Kubo/Storacha, `hidden: true` in Helia) | ||
| - `opt-out`: Included by default; implementations provide a flag to exclude | ||
| - `preserved`: Symlinks stored as UnixFS Type=4 nodes with target path (per [UnixFS spec](https://specs.ipfs.tech/unixfs/)). Note: Kubo (v0.39) `--dereference-args` only follows symlinks passed as CLI arguments; symlinks found during recursive traversal are always preserved. | ||
| - `followed`: Symlinks dereferenced and treated as target files/directories | ||
|
|
||
mishmosh marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| ## Detailed design | ||
|
|
||
| We introduce a set of **named configuration profiles**, each specifying the complete set of parameters for generating UnixFS CIDs. When implementations use these profiles, they guarantee that the same input, processed with the same profile, will yield the same CID across different tools and implementations. | ||
|
|
||
| ### The `unixfs-v1-2025` modern profile | ||
|
|
||
| Based on the research above, we define **`unixfs-v1-2025`** as an opinionated profile for implementations that want to adopt deterministic CID generation for UnixFS DAGs with CIDv1. | ||
|
|
||
| | Parameter | `unixfs-v1-2025` | | ||
| | ----------------------------- | -------------------- | | ||
| | CID version | CIDv1 | | ||
| | Hash function | sha2-256 | | ||
| | Chunking algorithm | fixed-size | | ||
| | Max chunk size | 1MiB | | ||
| | DAG layout | balanced | | ||
| | DAG width (children per node) | 1024 | | ||
| | HAMTDirectory fanout | 256 blocks | | ||
| | HAMTDirectory threshold | 256KiB (block-bytes) | | ||
|
Member
There was a problem hiding this comment. I don't think Helia can currently do block-bytes. How much of a problem is this for making this the "modern" profile? Or does that just mean there's work to do.
Member
There was a problem hiding this comment. Oh, I see from your table that links-bytes is the most common across the implementations, so why not just use it? It's not precise, but it means that you don't need to change implementations as much to conform.
Member
There was a problem hiding this comment. @rvagg just mean there's work to do to implement I hear you - standardizing on
Both implementations need code changes to align on any method. There's no free lunch here. Given that we're already introducing The goal of IPIP-499 is to create sensible modern profile that is implementable without poking your eyes out. I feel it needs (This is my "strong belief weakly held", I'm happy to be convinced otherwise :))
Member
There was a problem hiding this comment. reasonable, as long as there is people-budget to do all the work on this 👍 |
||
| | Leaves | raw | | ||
| | Empty directories | included (opt-out) | | ||
| | Hidden entities | excluded (opt-in) | | ||
| | Symlinks | preserved | | ||
| | Mode (permissions) | excluded (opt-in) | | ||
| | Mtime (modification time) | excluded (opt-in) | | ||
|
|
||
| ### The `unixfs-v0-2015` legacy profile | ||
|
|
||
| This profile documents the default UnixFS DAG construction parameters used by Kubo through version 0.39 when producing CIDv0. It is provided for users who depend on CIDv0 identifiers generated by Kubo and need to reproduce them with other implementations, or verify content against existing CIDv0 references. The year 2015 in the name indicates that the majority of these parameters were picked a decade ago, when the initial go-ipfs alpha software was implemented, and these defaults were never contested since then. | ||
|
|
||
| | Parameter | `unixfs-v0-2015` | | ||
| | ----------------------------- | -------------------- | | ||
| | CID version | CIDv0 | | ||
| | Hash function | sha2-256 | | ||
| | Chunking algorithm | fixed-size | | ||
| | Max chunk size | 256KiB | | ||
| | DAG layout | balanced | | ||
| | DAG width (children per node) | 174 | | ||
| | HAMTDirectory fanout | 256 blocks | | ||
| | HAMTDirectory threshold | 256KiB (links-bytes) | | ||
| | Leaves | dag-pb | | ||
| | Empty directories | included | | ||
| | Hidden entities | excluded (opt-in) | | ||
| | Symlinks | preserved | | ||
| | Mode (permissions) | excluded (opt-in) | | ||
| | Mtime (modification time) | excluded (opt-in) | | ||
|
|
||
| ### User benefit | ||
|
|
||
| Profiles provide 3 key advantages for working with content-addressed data: | ||
|
|
||
| 1. **Predictable, deterministic behavior:** Profiles restore intuitive hash-like behavior: identical input data always produces identical CIDs, regardless of which implementation generates them. | ||
|
|
||
| 2. **Lightweight verification:** Users can verify content without needing to rely on additional merkle proofs or CAR files. | ||
|
|
||
|
|
@@ -167,7 +182,7 @@ Profiles provide 3 key advantages for working with content-addressed data: | |
|
|
||
| UnixFS data encoded with the CID profiles defined in this IPIP remains fully compatible with existing implementations, since it conforms to the [UnixFS specification](https://specs.ipfs.tech/unixfs/). | ||
|
|
||
| To generate CIDs in compliance with this IPIP, implementations MUST support the `unixfs-v1-2025` profile. The `unixfs-v0-2015` profile is provided for backward compatibility and MAY be supported by implementations that need to produce CIDs matching historical Kubo output. | ||
|
|
||
| Implementations SHOULD allow users to inspect default values and adjust configuration options related to CID generation. | ||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@obo20 any chance someone from Pinata yeam can provide what defaults you use currently, so we can list your "CID profile" next to Storacha one? Understanding divergence would be very useful.