scaling to indefinite and changing groups of files

[okdistribute]

Hey all,

I've been in conversations with Internet Archive to integrate Dat into their dweb portal. One of their requirements is to serve data from Dat over the peer-to-peer network without having to scan all of the files ahead of time (upwards of 50 petabytes of archives...). Opening this on behalf of @mitra42 who is project lead on the Archive's dweb portal.

The way they have approached this with other protocols such as IPFS, Gun, and Webtorrent is to have a peer that pretends that it contains everything, and then once an archive is requested, to go fetch it from Internet Archive storage and serve it on the network.

For example, they want to be able to handle requests for the string /foo/bar/baz.mp3 over the network. Quoting @mitra42:

Looking at that doc [How dat works by vtduncan], I'm struggling a bit with the route from a path.
dat://12334/foo/bar/baz.mp3 to the data.

It seems, if I read it correctly, that this requires all the files to
have previously been put into the metadata "file", so doesn't scale to
indefinite and changing groups of files, i.e. there doesn't appear to be
point (unlike in GUN for example) where the string /foo/bar/baz.mp3 is
sent from one peer to the other, for that peer to then figure out what
to send back.

Wondering if this is a use case that this WG think is interested to support directly in the protocol (e.g., file-aware transfers vs. only block transfers)?

Thanks,

[pfrazee]

Well let's see...

The wire protocol exchanges hypercore blocks, not files. As a result, peers don't explain what files they're trying to fetch when they communicate with each other. They only explain which data-chunks they want. That doesn't help.

One option might be to update the hyperdrive data structure so that it can have "placeholder" entries. These would be empty files with a "is_placeholder" flag set. A wire-protocol extension message could ask the owning peer to "hydrate" the file, in which case it would overwrite the placeholder entry with a real entry. Then replication could happen as usual.

Another option might be to leverage one-way mounts, once they're implemented. In that scenario, the owning peer would be sharding their file-set into a tree of dats. They would optimistically create and share the dats, but not populate them -- when a peer first connects, they would then populate the dats.

Let me step through the one-way mount solution. Let's say we have the following file tree:

/alice/fun/stuff
/alice/work/emails
/alice/work/backups
/bob/fun/games
/bob/fun/pics
/bob/work/
/carla/

The top level has three folders:

/alice
/bob
/carla

At initialization, a top-level dat would be created, and then 3 more dats: one for each folder. They would remain empty. The top-level dat would mount each of them.

# toplevel dat:
/alice -> dat://alice
/bob -> dat://bob
/carla -> dat://carla

# dat://alice
(empty)

# dat://bob
(empty)

# dat://carla
(empty)

All 4 dats are swarmed. Let's suppose a reading client wants to pull down the 'bob' folder. It would request a block from the 'bob' hyperdrive. When this happens, the owning peer would immediately mint 2 more dats, the 'bob-fun' dat and the 'bob-work' dat, and mount them to the bob dat:

# dat://bob
/fun -> dat://bob-fun
/work -> dat://bob-work

# dat://bob-fun
(empty)

# dat://bob-work
(empty)

These 2 new dats would then also be swarmed. This continues recursively. Basically, we're using entire dats as placeholders, and using the request of any block as a trigger to "hydrate" it.

The good news is that one-way mounts will not require additional hits to the DHT. The replication connection for the top-level dat can multiplex all of the child mounts, making this design relatively efficient over the network. It also would not require any unplanned changes to the protocol.

[martinheidegger]

This is now theory land: Another way to go about this would be more "lowlevel". Every DAT consists of 2 hypercores, metadata & content. What if it were to contain X hypercores? One per folder and one per file. The special hypercore dir/ will contain a list of operations.

For a file, it might contain an entry

{ "op": "add", "file": "Readme.md", "time":12312313, "hash": "YI8Qx5b/Tpbu5Hiw72RoSduT3qL3ZgYWZEMx5QwTPZ4=", "type": "file" }

... which means that for the latest version of the Readme.md you would need to ask the server for the hypercore file/YI8Qx5b/Tpbu5Hiw72RoSduT3qL3ZgYWZEMx5QwTPZ4= to receive the files.

For a folder, it might contain en entry:

{ "op": "add", "file": "css", "time":12312313, "hash": "85FydCC60+APHJsgWHJBOISXnITmtx9dwRUC4wi+pak=", "type": "dir" }

... which means that folder the history of this folder you would need to ask the server for the hypercore: dir/85FydCC60+APHJsgWHJBOISXnITmtx9dwRUC4wi+pak= which would contain the version history for this folder.

The server would need to lookup all files in a folder and just generate the hashes and upon request of a hypercore would start creating those hypercores (like a stream placeholder).

The crux of the issue, and the reason why the webarchive can not really do what they mean to is the problem that you can't quite know how big the entire archive is (something that is prominently displayed in DAT right now).

[pfrazee]

@martinheidegger Correct me if I'm wrong, but I think, for the most part, that's how my one-way-mounts solution works.

[martinheidegger]

@pfrazee It is similar, but on a lower level. Hyperdrive → Hypercore. Also it uses hashes instead of direct file lookups for deletions.

[pfrazee]

@martinheidegger What's the upside of introducing a different hypercore datastructure than hyperdrive? If you stick with hyperdrive, you avoid having to create a niche solution.

[martinheidegger]

Codename for this thing: hyperfs. If you build hyperfs, the way to lookup files will be vastly different from how it currently works in hyperdrive (pre multi-writer). You need to add some structure on top of hyperdrive to do that. It seems to me that having the hyperdrive structure (and overhead) doesn't give any benefit to this: What use is they metadata core if its always the same structure? (one node in metadata declares if its a file or directory, with the content stream only in use when its a file). As the linked DAT's mean that they could be theoretically by different owners, we need to create separate replication streams for all those DATs, If they are but "subcores" we can assume (guarantee?) that they have been generated with the same write key.

[mitra42]

Thanks for the thinking .. I'm hoping we can use this exercise as an example of how DAT might work with a large and changing site like the Archive. IMHO (and from prior experience with how the Web built on legacy Gopher and FTP data), how well a protocol gets adopted depends in part on how well it integrates with legacy data systems, and its great to be able ot use the Archive to test out some of these ideas.

I could be wrong, but I don't think either of those scenarios work for large data sets with an unknown (and changeable) set of items. For example, the obvious structure to me would be dat://// , in @martinheidegger 's suggestion this has two issues
a) one DAT at the top level, pointing at the others and one dat for each directory (what we call an ) - will fail because i) the files are added to all the time (approx once per second) and ii) There would be roughly 50million item ids to keep track of. and iii) practically we can't run any process ahead of time over all the ids.

Of course - we could do this in a different way - similar to how we work around IPFS's inherent weaknesses - where we created a DAT for each item the first time a user requested it, and then shared the address of the DAT out of band (via GUN or HTTP), but with that solution, Wort, Gun or WebTorrent are always going to work better than DAT.

[pfrazee]

@martinheidegger You're free to build any data structure you want on hypercore, but you're going to have to get that structure adopted widely if you want it solve the problem.

@mitra42 Sharding the dataset into multiple mounted dats is going to give you a couple benefits:

1. You can do just-in-time construction of each folder's dat, and avoid constructing dats which have no demand
2. You can retire & swap out a folder's dat if the folder's history ever gets too large (effectively garbage collecting the history)

Are you certain that sharding the dataset to one-dat-per-folder, and only updating folder-dats that have peers, is not enough for you to scale?

[okdistribute]

I wonder how we might use something like kappa-db/multifeed for this 'hyperfs' implementation.. any thoughts @noffle?

[martinheidegger]

@pfrazee That issue persists with both a hyperdrive and a hypercore solution. Pulling it one abstraction level higher doesn't make it quicker to be used.

@mitra42 I am seconding @pfrazee here: but I have some issues comprehending all you wrote (some formatting issues here). The only bottle-neck that I see with my example is that / can not be entirely cleared quickly (which is something worth considering to add). Since each directory/core only keeps track of is children any change operation would be influencing a disconnected part of the structure. Like IPFS I used a "hash" to specify the content hypercore, but this could also "just" be a random key to behave more like DAT does today (more direction of hyperdrive approach). The protocol of dat is built to be quicker and more efficiently than HTTP for larger files (lower overhead).

[okdistribute]

@pfrazee i disagree that it has to be adopted widely for the archive's case, because most of the peers would be served from the client side webrtc (watching the same video at the same time, for example)

[pfrazee]

@Karissa you're just losing wider ecosystem compatibility. If the situation really is niche, then you can do that, but I think you're losing some of the benefits of dat. In the case of cabal the custom data structure makes total sense, but in this case hyperdrive + the upcoming one-way-mounts feature has a good chance to solve the issue.

[okdistribute]

Yeah, this is why I was sort of hoping it could be something at the hypercore level -- being able to expose the filename (if known, optionally) in the request

[okdistribute]

or rather, any payload for extra request content needed

[pfrazee]

@Karissa You could probably do that via a wire-protocol extension message. It's got some extra overhead but if it was only turned on for dats doing this style of on-demand hydration, it's not super crazy. You'll just need the peers to support the extension.

[martinheidegger]

@pfrazee I don't think we can come up with a solution that doesn't require the entire ecosystem to update. I believe a case could be made that the next update multiwritercough* could be that thing. But also once more these solutions are not backwards compatible! One important aspect of DAT is that we know the entire size of it at a versions time. No matter if it's mounts or different cores: we wouldn't be able to tell how big the data space is.

[pfrazee]

@martinheidegger The one-way mounts are a planned update for the entire ecosystem, though.

[martinheidegger]

Planned doesn't mean done ;)

[mitra42]

@pfrazee asked "Are you certain that sharding the dataset to one-dat-per-folder, and only updating folder-dats that have peers, is not enough for you to scale?”

Sure - this can be done, but then you have the issue that you can’t do this on 50m directories ahead of time, so you have to do this creation of one dat-per-folder between the time the user navigates to the page and the time the metadata is returned, since you might have to rebuild a DAT with a few GB of data (the raw version of the file), so it’s very slow on first access to an item, and in fact slows it down for all users, not just those who will use DAT to retrieve the files, so in practice it won’t happen. (For IPFS we only put the actual files requested into IPFS because this step is so slow for IPFS).
The Archive might be a good place to try an extension, and then role it out to the ecosystem if, and only if, it works well - we have the advantage (during the experiment phase) of controlling both the client, and the peer at the Archive.

@martinheidegger: You are definitely going to lose the “size of the entire DAT” for any large changing data set, accumulating size back up the chain isn’t practical,

Updating the top-level “/“ DAT is possible but much harder since a) I can’t crawl 50m items and b) even if I could, doing that update at item creation time requires hacking a large code-base with substantial (and reasonable) institutional resistance to change, while anything I can do on top of the existing data doesn’t face that barrier and is more open to experimentation.

I truly believe that name lookup on large (number of items) changing data sets is a crucial bottleneck, its a massive weakness in IPFS, and a big advantage in GUN and Wort. Both AT an Wort allow for “hijacking” during that name process, i.e. mapping a name from their ecosystem to a HTTP request. I think it would be great for DAT if it could think this through., for the Archive we can use GUN/Wort/HTTP to discover the DAT archive id, but that wouldn’t be was useful for testing DAT on its own scaling.

[pfrazee]

@mitra42 If that's your read then I'd agree that you should try the wire-protocol extension.

[martinheidegger]

can’t do this on 50m directories ahead of time

That is not what we are suggesting here. In pfrazee's approach you'd only serve a DAT once you have it but link it ahead of time. My approach would do the similar thing on hyperprotocol level (if the protocol asks for a key, a hook will be called to create this item - or return 0 if its not replicated/existing).

definitely going to lose the “size of the entire DAT”

This sounds to me like a more-important issue than you might think. The size allows prediction on the client how much needs to be downloaded and gives an indication of how much would need to be shared. (also you gain some optimization options by preserving this).

@mitra42 Asking a different question here: Why, one data set and not split it up and serve a lot of smaller DATs? A DAT - particularly in the next version - can easily hold few gigabytes of small files and perform well. Without breaking any infrastructure: If a central indexing DAT contains a list of DATs for sections of the web, then clicking on a link (in beaker browser) the service could index the content of that section.

@martinheidegger The one-way mounts are a planned update for the entire ecosystem, though.

I havn't seen the DEP of this yet. What was the consensus on the "loose size of entire DAT" issue (which persists with mounts, from a user perspective).

---

Thinking about a protocol-level approach here:

In the protocol there is no distinction between files and other data. The mirror-folder system indexes the file system and adds one-by one files to the DAT. There is currently no metadata that tells the client if a mirror-folder process is running on a folder (same like there is no information about file streaming). If this information was given, a reader could let the writer know "please prioritize this folder". And the writer could simply tell the mirror-folder process to prioritize that folder first. This seems like a hack of a system though - very fragile.

Note: I have other tasks to look after from now until tomorrow. Will answer/give ideas with delay.

[mitra42]

@martinheidegger - there are two possible approaches here - one DAT for the whole of the IA, or one DAT per item with a root-level DAT as an inde, both of them appear to have (solvable or unsolvable) scaling problems.

For the one DAT per item approach maybe I'm missing something - where do I find the link for a specific DAT, given that I can't run a process on 50m items to create the links for 50m (not yet created) DATs ahead of time in order to put these into the index , the scaling problem has just been pushed to a different (maybe better) point. If we could calculate the pointers for the DATs without processing the content (e.g. by a hash function on the name) then it would be easier, and wouldnt create a slowdown at the point we are requesting Archive (not DAT) metadata, though its still non-trivial to generate a list of all item ids, and even more non-trivial to map those DAT ids back to not-yet-created DAT archives.

If its one DAT per item, then certainly we can get sizes on each DAT, but obviously not on the index which will change size approximately once per second as items are created .... how in DAT are you now handling data sets which get appended to and so don't know the size ahead of time?

[RangerMauve]

I second the idea of using a protocol extension, but I don't think you need to use mounts.

Think of it like this:

• You have the entire internet archive on an FS of some sort
• You start with an empty Dat representing it
• Peers replicate the archive (sparse) and attempt to load the file they want
• If they 404, send a message down the replication protocol using an extension to say "hey, add this file"
• The archive peer will then add the file to the archive, and it'll become part of the swarm

This way you don't need to index ahead of time, and content is lazy-loaded into the archive. And once the file is loaded, it'll be spread across the P2P network as needed.

Some concerns:

• You need to hope that you're connected to the archive peer when finding a new file or this extension stuff won't help (Maybe we could work around that by broadcasting?)
• This method assumes the files are static and changes are created in new files rather than extending existing ones. If the files change you'll need some sort of process to watch those changes
• The internet archive will slowly be duplicated within the content feed. Might need to do some spooky magic at the hypercore level to have it load those sections of the feed from the actual FS on the internet archive peer.

I believe this method would be similar to what the internet archvie is doing for other protocols and requires a minimal amount of change to the existing hyperdrive code.

[mitra42]

I think this is the right kind of approach,

A couple of detail points/questions.

Changes: wouldn't this work the same as DAT does now, i.e. if a change happens after a file is added then presumably the same mechanism (and I don't know how DAT does this) to handle an update would be used.

Connectivity: Could some variant of the mechanism (again I don't know how DAT does this) that is used to find which peers have the metadata/file now be used to forward the query (with the file name extension) with the peer connected to the file system eventually getting the request and loading it.

Http: Instead of connected to a File system, the peer should map the DAT to a URL request e.g. DAT/1234// maps to http://dweb.me/arc/archive.org/download//. (for a filesystem this would just be a URL file:/something/something)

Even better: Note that this could (in some cases) also work at multiple peers not physically at the Archive, where any peer could fetch a missing file via HTTP, this is how WebTorrent does it now - there is a HTTP field in the magnet link that allows any peer to get missing files direct from the Archive and start seeding them itself. (This is wonderfully fast in WebTorrent, which is why it is currently our preferred decentralized video player)

[mitra42]

Note that for the anti-censorship case, you wouldnt want to rely on the HTTP fallback direct from each peer, as ideally if you are behind a censorwall you want to be connected to some peer that is able to connect via Http.

[RangerMauve]

Changes: It depends on how the raw data is stored. If possible you could hook into FS events and check if the file is in the metadata feed and add the change

Connectivity: Kind of. Basically, you discover peers for the archive and open up a hypercore-protocol stream with them to start replicating append-only logs called hypercores. The protocol is set up in such a way that you could pass custom messages in addition to the ones used for hypercores. That's where you could send the request out.

HTTP: Not sure what you mean, is the data not going to be stored directly in the archive?

Even Better: Yeah, I get what you mean. However, peers are unable to add data into a Dat archive, only the internet archive could add files to the archive due to the way security works in Dat. Now that I think of it, I don't think it'd be very easy because Dat doesn't use content addressing the way WebTorrent and IPFS do.

[mitra42]

Changes: Might have to do that later, probably from the other direction, having something periodically crawl the archive comparing sha's against that in the current metadata.
Http: Sure the data is directly in the Archive, but for most really large sites I've seen (including the Archive's 50 petabytes), that isn't a file system, its servers responding to HTTP.
Even Better: yes - understood about the security.

[martinheidegger]

i.e. if a change happens after a file is added then presumably the same mechanism (and I don't know how DAT does this) to handle an update would be used.

In the hypercore protocol you need to specify ranges of interest. If the range doesn't have a clearly defined end the client will receive always the newest available data. (add-only-protocol)

where do I find the link for a specific DAT

Well, one thing that has been part of this discussion is "mounts":

Mounts merge dats together by causing one dat to act as a folder within another dat.

(source: datproject/planning)

This means that for IA you would have one DAT that contains the links to all other DAT's and you would add one for every folder you have. As soon as someone becomes a peer to a particular DAT (and asking for additional data) you could start/resume the mirroring of that particular DAT.

[mitra42]

Lets be precise .... when you say "contains the links to all other DAT's and you would add one for every folder you have." Do you mean all the folders you have, or all the ones you make. This is important, because as said above you simply can't create one file with links to all the possible (50 million and changing all the time) folders/DATs.

[martinheidegger]

@mitra it is stack-able. You could a root-set of 512 keys, each containing 512 more, each with 512 in them, which would mean 3 more look-ups for getting to the actual folder.

[mitra42]

@martinheidegger the size isn't the point, its the impracticality of walking a tree this size. Its extending the same requirement that applies for files to the items (directories) that there is a need to request them, so that the superpeer can only (on first request) add them to DAT

[martinheidegger]

@mitra42 Lets assume we have 256 characters x 2: aa, ab, 'ac` ... etc. The root DAT contains DAT ids for each of the first 2 chars that exist. Each DAT contains again 65535 possible DAT's. But those sub-dats are only created "as stub" meaning that none contain any content. Upon the first peer for each of the DAT's. a crawler starts adding a crawler for those blocks.

1. IA creates root-DAT: 1f0b532e
2. IA looks in root-folder for every entry, and creates a DAT for each two-letter variable, for example: aa=54a31a6a
3. IA registers a peer for 54a31a6a
4. time passed - A client connects to 54a31a6a
5. IA notices this and looks up which files starting with aa exist and adds a DAT for the next two characters, i.e. it finds aasu=d0dc0b9c and adds that.
6. time passed - A client is updated with DAT id for aasu
7. A client connects to d0dc0b9c
8. IA looks up which files are starting with aasum. exists. At this point we are three levels in, that may be enough or we continue with further levels down.

[mitra42]

Let me run through this ...

First ... item ids are not case sensitive, and include a-z0-9 and some punctuation, so lets assume approx 64 chars not 256, so 64^2 = 4000 pairs per level.

• Step 1 fine :-)
• Step 2 - "looks in root-folder for every entry" doesn't mean anything, we have to assume that all possible two letter pairs exist (~50m / 4k ~ 11k items per two letter pair so we'll probably want to go one more level as you suggest.
• Step 3 - create 4k peers. (I'm going to assume DAT can handle connecting to 4k peers ?)
• Step 4 - but what does this next step mean - why would a client connect to 54a31a6a, is that because its looking for an item starting with aa
• Step 5 - I guess it could then create another 4k DATS (one for each of the next pair of characters) and another 4000 peers. Steps 6,7,8 then lead to of the order of 3 items per peer (and will be maybe as many as 30 given some letter combinations are more popular).

This means for each item requested, its going to create ~4000 + 3 peers. Which means this isn't just a hack, its got a really nasty overhead, when what we really want to do is indicate from the client to IA precisely which item its looking for and ONLY create a dat for that item.

[mitra42]

Would a better hack be something like (with the protocol extension proposed somewhere near the top of this thread).
1: Create top level DAT
2: Client connect to that DAT
3: Client requests the actual item "commute" (maybe as a mount, maybe as a file)
4: That request is communicated via the protocol extension
5: IA sees that request, creates an empty DAT for, and adds to the top level DAT either the item as a mount point, OR a file with only as content the id of the DAT
6: Client connects to the new DAT
7: Client requests file in that new DAT via protocol extension
8: IA adds the files to that DAT, and syncs them to the client.

This would be far less hacky, and also not have the massive overhead that I think (cant be sure) is present in the proposal you made.

[martinheidegger]

The problem/good-thing about having an IA DAT link is that you can replicate the entire IA. In order for this to not kill your hardware, you need to limit/prioritize what part of the IA archive is added in what order. On another hand: The whole file-tree of IA itself is also probably too big to be kept on your average hard drive; which means that each client connecting to IA-DAT would only connect/request data from a DAT that it wants to access: Which can be implemented with 1 connection at a time, or limit to (say) 5 connections - depends on the client software (Beaker?) to implement this.

[mitra42]

Sorry, I don't see how that comment relates to either the problem, or potential solution.

The whole point is to limit what is added BASED ON WHAT THE CLIENTS REQUEST, and then only replicate to the client the parts it wants.

Then on connections - its not the per-client connections, its that your proposal would - if I understood it - mean creating all these 4000 peers even when there is only 1 connection.

[martinheidegger]

@mitra42 1 client wants to get the DAT for aasum.com/index.html (the first domain with aa i found) - it would open a connection to the root DAT, get the DAT for aa, lose the root DAT, open the aa DAT; wait until aasu is there then closes the aa and opens aasu etc. ... on and on it goes until a dat contains the entire file for a DAT. This is one connection that the client needs. On the server side: You need one server that "serves" all those DATs. One "server" can easily announce to be serving 4000 DAT's a the same time, when the request for one particular comes in. No need to open 4000 ports or keep 4000 DAT's in memory. So if one client connection comes in it needs to serve the DAT's.

This doesn't mean that a DAT client wouldn't be able to simply open 4000 connections to the peer for each DAT. It also doesn't mean that the same server would need to provide all those DAT's; it could easily be split up into several servers to index the data.

[martinheidegger]

This would be far less hacky, and also not have the massive overhead that I think (cant be sure) is present in the proposal you made.

The problem with this model is that there can (and should be in future) proxies for DAT's. Those proxies will serve a DAT as a peer to other peers. They will never be connected to the IA server! As such the proxies would need to aggregate and relay the messages of the clients to IA server. Effectively making it a lot harder to distribute the data.

[mitra42]

Right - but if I understood the conversation above those messages can also be relayed. Note that again, if I understood it correctly, the messages would only get relayed to IA (as they are in GUN) when that data is not already in the DAT, which is not really any different in terms of working through proxies than your solution. i.e. data is served from a Proxy if that Proxy has previously seen the data, otherwise the request ends up back at the IA server.

[okdistribute]

@mitra42 I like your proposal which is sort of a combination of using the protocol extension and the dat mount points as @pfrazee described above.

[martinheidegger]

@mitra42 Implementing a message-array is possible - in a form that all connected peers relay all the messages in the whole network but it is a unprecedented step.

. i.e. data is served from a Proxy if that Proxy has previously seen the data

For that to work the proxies would need to become a lot more "intelligent" than they are now. Proxies do not currently need to have a concept for "file-system", which means they can be used for any kind of data in future.

For a message-relay-protocol to be added a-top of that I am worried about unpredictable amount of data to be sent through the system. How would the messages need to be constructed in a way to prevent system overload? Limit per peer / time? Limit of size ? Should messages may have a age limit? Message consume-ability? Will all messages just be sent? Should peers have a way to tell others that they received a message? Namespace for message-types?

[mitra42]

@Karissa can you jump in here, I though I understood that the extended messages would essentially pass through the exact same routing as a request for a block, but they'd be asking for a file so the load would be the same - am I missign something.

[okdistribute]

@martinheidegger those are valid concerns but perhaps would come up in the development process, and aren't necessarily blockers for this work.

@mitra42 I will discuss with @mafintosh this week about the proposed plan you and @pfrazee had, and get back to you about viability

[martinheidegger]

A few more questions/complexity: How do we prevent spamming the system? How is order ensured (very hard in a distributed system)? Are the senders of messages verified? (This would make proxying harder)

It is my worry that messaging opens the gate towards tracking and user profiling. Something that I would very much like to avoid. See #49

Imo. If the IA case could be done without messaging, that would be awesome. Maybe @aral could add his POV?

[aral]

@martinheidegger I’ve mostly skimmed this (rather long thread) so apologies if I am not 100% up to date but some thoughts/concerns:

If by messaging you mean an ephemeral messaging channel, this is something that I am exploring also for Hypha based on @pfrazee’s DEP-0000 for peer authorisation. You can see it in action (without encryption, right now, going to look into adding it today/tomorrow) at https://source.ind.ie/hypha/spikes/multiwriter-2

So an emphemeral messaging channel is essential for usable peer/device authentication in multiwriter so if one is being implemented at core, I’d like to see this use case covered.

My concern regarding optimising Dat for IA is this: it’s like optimising Mastodon for Stephen Fry. We all love Stephen Fry but he has very different needs (tens of millions of followers, etc., than do regular folks on Mastodon.) Can Mastodon be both a great solution for someone talking one-way to millions of people and for people who want to chat among themselves in small groups? More precisely, and I’ve seen this multiple times now with various technologies, starting with (I’m ashamed to say), Flash: there is a chasm between the requirements of people/individuals and enterprises. A single tech is rarely great at making both of those groups happy. And, usually, given that the money comes from enterprises, tech evolves to meet their needs at the cost of not exploring optimisations that better meet the needs of individuals.

What’s special for me about the Dat ecosystem is that it is focussed on the needs of individuals. I hope we continue to keep the focus there and aren’t sidetracked by optimising for enterprise, especially at the cost of optimising for individuals. (Not that I’m saying that’s what’s happening here but that is what it made me think of and I’d be remiss to not share my concerns on a project that has given me so much hope and that I care so deeply about.)

These are just general concerns. Now that corporate funding is also starting to enter the picture, I feel it might be timely to consider them. Please use/expand upon them/or keep them at the backs of your heads if you feel they’re useful or don’t afford them another thought if you feel they are irrelevant :)

PS. The one thing I would not like to see is the emergence of any privileged, centralised nodes. If anything, any centralised nodes that are absolutely unavoidable should be less privileged. (In Hypha, for example, there is an always-on node but it doesn’t hold any secret keys.)

[mitra42]

Totally valid concerns @aral - I see this as an exploration, can DAT be used for bigger systems and is it appropriate, I watched how the early days of gopher were driven by being able to integrate data from the legacy ftp, and then how the web's early growth was driven by being able to integrate data from gopher and ftp, and I think an ability to integrate with larger, sparse, file systems is going to be crucial to the various Dweb protocols (DAT, IPFS, etc) being able to grow and become the norm that eventually obsoletes the legacy centralized systems.

I think the proposal we are exploring, is to figure out the general case of how to bring a file into DAT that isn't already there, preferably without having to go to some other protocol to request it. That is generically useful in all kinds of situations (including just mounting existing file systems onto DAT without duplicating all the data).

The Archive, in this case is a good place to experiment, especially since we can explore, and if it doesn't work we can continue to use IPFS, GUN, WebTorrent or even HTTP.

The archive is a classic large system in that it has a massively long tail - I used to have the number handy, but it was something like 10 items that had been downloaded over a million times, and 10 million items downloaded only once, don't quote me on that, I could be an order of magnitude out, but I think you get the idea.)

Such a long tail creates a potential to replicate - on demand - just the parts of IA that are requested, this then becomes useful for example in the rural areas of developing countries, or behind censor-walls, where local decentralized replication of material pulled from the legacy web is a useful application. Sure we can do it with HTTP, but its also a great potential use-case for DAT (or IPFS, GUN & WebTorrent).

[martinheidegger]

So an emphemeral messaging channel is essential for usable peer/device authentication in multiwriter so if one is being implemented at core, I’d like to see this use case covered.

@aral I am very hesitant to support (even ephemeral) messages - as they might be affecting the quality of the DAT network negatively. See my questions above two comments for reference. It is my worry that the mere possibility of a two-way channel has terrible effects on the trust-worthyness of a DAT source. It is to me a fundamental property of DAT that everything is rather fixed. I am looking at this from the perspective of someone who wants to transfer very-private-data. If that person connects to the DAT network, that person wants to be sure that all the data transferred contains as little private information as necessary. With an extendable messaging protocol attached, the clients (and servers) could exchange - over time - more and more information with each other. Enabling easier tracking of the person (eventually to the point that cookie policies might be in order?).

You are having a usability issue here. You want to make the authentication process smoother. Which is a admirable goal, but I wonder if QR codes and Camera's (as used by Google Authenticator, Signal, WhatsApp, ...) might not be a better solution for the authentication?

@mitra42 I think the proposal we are exploring ...

Even though you might be hesitant to look at my proposal: I ask you to take another look. I believe it should easily possible to implement your use-case with the current state of DAT.

[aral]

@mitra42 Thanks, Mitra. I can definitely see the value in on-demand replication (and perhaps rather naïvely thought sparse replication was able to handle this). It’s also a feature I need for Hypha as huge upfront replication (HUR?) is a usability nightmare that can kill an alternative system dead at hello. (Can you image where the web would be if we had to download a whole site before we could get started… and yet some p2p implementations feel that this is an acceptable onboarding experience.) Anyway, I don’t want to digress or fork this thread. Will continue to monitor it silently + best of luck with your efforts :)

[aral]

@martinheidegger I understand your concerns; they’re definitely valid. Re: the use of ephemeral messages in Hypha: my plan is currently to use them only between nodes that the same person owns and only using symmetric encryption. I do not have any use cases beyond authentication at the moment. And I’m happy with it as currently implemented (as a protocol extension). If we want truly ephemeral messaging between people as a feature later (this is a valid social use case), we can use asymmetrically-encrypted ephemeral messages without any changes to the current protocol extension.

My goal is to match (if not exceed) the usability of centralised systems, otherwise (for Hypha), we’re dead in the water from the word go. QR-Codes are a good possible secondary means of distributing the uncensorable read key especially in physical environments. But the addition of any extra complexity at onboarding is a showstopper for what I want to try and achieve (which I realise is not the same as everyone else’s use cases/success criteria.)