BranchStore/KeyStore/Permissions

[DyrellC]

Currently we track Cursors, Psk's and Exchange Keys in a KeyStore for a channel-centric management of access control, but in v2.0 we intend to bring more granularity of access management to the data tree topology, making it more branch-centric.

While managing branches, there are 4 pieces of information that we will need to keep track of in v2.0 going forward:

1. Cursors - The state of each publisher within a branch
2. Permissions - The permissions a user has within the branch (Administrative, Read Only, Read/Write)
   a. Links connecting the message chains to the permission granting keyload message
   b. Individual user permissions as seen in a keyload message (with a default as either Read Only or Read/Write)
3. Pre Shared Keys (PSK's) - Keys disseminated outside of the streams protocol between users to give them read access to a branch
4. Exchange Keys - The public exchange key disseminated from a user to allow for encryption logic

PSK's and Exchange Keys make sense to be separated out into their own storage imo, but the cursors and permissions should likely stay together as they're so closely related. In the current state, permission storing requires a mapping of links to the permission granting keyload, which requires that we either keep track of the latest link in the chain (as is currently proposed, but carries limitations in utility) or keep track of all links above the keyload (bloats the user instance and decreases performance). Additionally, permissions need to be stored in the branch for access during each read/write operation.

Questions:

• Should Permissions be stored with Cursors in one store?
• Should PSK's and Exchange Keys be in their own storage location separate from Permissions and Cursors?
• Should Permissions store single links or multiple mapped to the permission granting keyload?
• Is there a way to circumvent the need for this link mapping for permissions?

[DyrellC]

Is there a way to circumvent the need for this link mapping for permissions?

If we have a general branch store logic as follows:

pub struct BranchStore {
    branches: HashMap<String, CursorStore>
}

pub struct CursorStore {
    cursors: HashMap<Id, Cursor>,
    permissions: Vec<Permission>
}

Would we even need to track the keyloads/links specifically? Or would it not suffice that the act of separating out cursors and permissions into individual branches would permit that users would always be checking/updating permissions only for a specific subsection of the channel at any given time anyways?

For example:

A channel with an Author and a Subscriber. The Author wants to set up a branch to send messages to the Subscriber, but after a few messages, the Author chooses to give the Subscriber write access to the branch. This would look like this: 

Author starts channel: 
BranchStore {
    branches: [
        {
            "branch0": { 
                cursors: [(AuthId, Cursor)], 
                permissions: [Admin(AuthId)],
            } 
        }
    ]
}

Author creates new branch, setting Subscriber as Read Only: 
BranchStore {
    branches: [
        {
            "branch0": { 
                cursors: [(AuthId, Cursor)], 
                permissions: [Admin(AuthId), Read(SubId)] 
            },
            "branch1": { 
                cursors: [(AuthId, Cursor)], 
                permissions: [Admin(AuthId), Read(SubId)] 
            } 
        }
    ]
}

Author changes permissions for Subscriber on "branch1" to ReadWrite:
BranchStore {
    branches: [
        {
            "branch0": { 
                cursors: [(AuthId, Cursor)], 
                permissions: [Admin(AuthId), Read(SubId)] 
            },
            "branch1": { 
                cursors: [(AuthId, Cursor), (SubId, Cursor)], 
                permissions: [Admin(AuthId), ReadWrite(SubId)] 
            } 
        }
    ]
}

Any read or write action would first need to fetch (or create) a branch in the BranchStore, and the cursors and permissions would be reflected there. Are the links necessary to enforce this at all?

[arnauorriols]

ah, I see now that I wasn't missing anything after all 😌 . This is what I have in mind as well. From a conceptual level at least

[DyrellC]

Nah I think we both are on the same page about how to tackle this one.

[arnauorriols]

I think it would be useful to have a more abstract listing of what we need in terms of the state of an Streams client. In this comment I want to first recap on the capabilities we envision users will have in v2.0, and then translate those into abstract requirements for the "State".

User Capabilities

From a very high-level perspective, all a Streams user can do is:

• Instantiate a Streams client with a ed25519, PSK or DID identity
• Create a stream
• Subscribe to other's streams
• Accept subscribers to own's stream
• Publish public and private messages with a topic
• Read public and private messages by topic and/or publisher
• Create topic branches
• Control access to branches of own's stream (read, read/write, admin)
• Recover an instance statelessly by rereading the stream
• Recover an instance from a backup

State requirements

The previous capabilities can be distilled into the following requirements from a State point of view:

• (1) Know own's identity
  • this one is kinda obvious
• (2) Know Stream owner's identity
  • the user must know from who can admin signals be expected and trusted
• (3) Be able to retrieve the Spongos state corresponding to the previous message to which the current message is linked
  • This is required both when wrapping a message as a publisher, and when unwrapping a message as a consumer, as both these Spongos procedures start from the state of the previous message. The unavailability of the Spongos state of the previous message implies the incapability to wrap/unwrap the message and any following one linked to it.
• (4) Be able to retrieve the address of the "last" message of a branch by topic
  • The application will publish messages to a "topic". Under the hood, the user client will retrieve the address that corresponds to the "last" known message that was published to that topic, to which the new message will be linked
• (5) Know the set of branches/topics
  • In order to fetch messages, subscribers poll the transport layer for new messages in all the possible addresses where a new message should be expected. The set of possible addresses is comprised by the combination of branches, the subscribers that have write permissions over the branch, and the sequence number that the new message would have within the branch.
• (6) Be able to retrieve the set of subscribers with write access (aka publishers) by topic
  • In order to fetch messages, subscribers poll the transport layer for new messages in all the possible addresses where a new message should be expected. The set of possible addresses is comprised by the combination of branches, the subscribers that have write permissions over the branch, and the sequence number that the new message would have within the branch.
• (7) Know the number of messages sent in each topic
  • In order to fetch messages, subscribers poll the transport layer for new messages in all the possible addresses where a new message should be expected. The set of possible addresses is comprised by the combination of branches, the subscribers that have write permissions over the branch, and the sequence number that the new message would have within the branch.
• (8) Be able to retrieve the set of branches where a subscriber has write access
  • In order to fetch messages, subscribers poll the transport layer for new messages in all the possible addresses where a new message should be expected. The set of possible addresses is comprised by the combination of branches, the subscribers that have write permissions over the branch, and the sequence number that the new message would have within the branch.
• (9) Be able to retrieve the set of subscribers included in a particular branch where the user is admin and their permissions over it
  • The admins of a branch must keep track of the subscribers and their their permissions in order to be able to to do "stateful" modifications of these, instead of forcing the application to provide the full list of subscribers any time the permissions are to be changed
• (10) Be able to retrieve a PSK by its corresponding hash
  • To refer to a PSK when setting permissions, the admin use its hash (aka PSKid), in order to avoid having to store the Psk at application level.

Questionable requirements

• Know all subscribers of own's stream
  • Must the owner of a Stream really keep the identifier of all the subscribers of the stream? If you think about it, when changing the access permissions of the subscribers, the application will have to provide the identifier of the subscribers anyway, so storing the subscriber identifiers does not add any value from this point of view. Arguably, it could be used to check if the user is setting permissions of an unsubscribed user. But is this really an error? In which sense does this harm the consistency of the state? Is it a matter of UX?
  • I would argue, actually, that the whole subscription flow does not add any value and can be removed. if the subscription address must be exchanged OOB anyway, what's the point of sending the subscription message? The subscriber can exchange the identifier OOB, end of story. Note that unsubscription would still be useful, in order for a publisher to signal the other subscribers that they can safely remove her from their state without having to rely on the admin changing the branches permissions.

[DyrellC]

The subscription flow argument makes sense. Since you need to manually adjust permissions anyways, and the links for subscription would need to be provided OOB, the subscription message may not serve as much of a purpose now as it was originally intended to. With that said however, I think we need to discuss the relevance of them with respect to Alias use (which I feel we haven't really dived into enough).

[arnauorriols]

It is true that the more information we add into the Subscribe message, the more convenient it seems to share it via the subscription message. On the other hand, provided that we cannot avoid an OOB exchange anyway, we have to weight in just how much worth it is.

[arnauorriols]

Comments on op given the abstract requirements described in the previous comment:

1. Cursors - The state of each publisher within a branch

This covers requirement 5, 6 and 7, although we should consider if its really necessary to keep a cursor per publisher. I would explore further the possibility to keep a single cursor per branch, as the tuple (publisher, topic, branch-seq-num) is already unique within a stream

2. Permissions - The permissions a user has within the branch (Administrative, Read Only, Read/Write)
   a. Links connecting the message chains to the permission granting keyload message
   b. Individual user permissions as seen in a keyload message (with a default as either Read Only or Read/Write)

why is 2.a necessary?

3. Pre Shared Keys (PSK's) - Keys disseminated outside of the streams protocol between users to give them read access to a branch

This covers requirement 10

4. Exchange Keys - The public exchange key disseminated from a user to allow for encryption logic

Besides the PSK, the other identity types will have the exchange keys inherent to their identifiers. In the case of ed25519 the x25519 can be directly derived, while in the case of DID, the exchange key will be referred to as a verification method. We still need to keep in state the set of subscribers and their permissions for each branch, as per requirement 9, but not another one specific for the exchange keys.

PSK's and Exchange Keys make sense to be separated out into their own storage imo, but the cursors and permissions should likely stay together as they're so closely related.

Not putting my foot down, but I feel it might have a chance of being more comprehensible and semantically expressive if all the state is kept in a single State struct, instead of scattered in overlapping stores.

In the current state, permission storing requires a mapping of links to the permission granting keyload, which requires that we either keep track of the latest link in the chain (as is currently proposed, but carries limitations in utility) or keep track of all links above the keyload (bloats the user instance and decreases performance). Additionally, permissions need to be stored in the branch for access during each read/write operation.

Branches will be identified by topics, therefore the mapping will be topic/set{permissions}. Why do we need to track any link?

• Should Permissions be stored with Cursors in one store?
• Should PSK's and Exchange Keys be in their own storage location separate from Permissions and Cursors?

I feel having everything together enhances understanding the state management as a whole, which is a good thing imo

• Should Permissions store single links or multiple mapped to the permission granting keyload?

neither, although I hope I'm not missing anything

• Is there a way to circumvent the need for this link mapping for permissions?

yes! map permissions to topic. although I hope I'm not missing anything...

[DyrellC]

why is 2.a necessary?

To be honest I don't think it is, I prefer the removal of that as a requirement but wanted to use it as a staging ground for conversation regarding Brords current implementation

but not another one specific for the exchange keys.

Agreed, with DID logic encapsulating the exchange logic, the only other instance to be concerned about is the ed25519/x25519 which, as you pointed out, is easily derivable and shouldn't need to be stored.

Not putting my foot down, but I feel it might have a chance of being more comprehensible and semantically expressive if all the state is kept in a single State struct, instead of scattered in overlapping stores.

I'm definitely open to keeping the logic together, but @kwek20 expressed concern about there being too many things in KeyStore simultaneously. However, if we don't need to map links the way that they are in the current implementation (which we hopefully can work around to avoid all the bloat that comes along with it), then there shouldn't be an issue with keeping it all together and not becoming too full in itself.

Branches will be identified by topics, therefore the mapping will be topic/set{permissions}. Why do we need to track any link?
neither, although I hope I'm not missing anything
yes! map permissions to topic. although I hope I'm not missing anything...

This was my general thought as well, the elimination of the links alleviates most concerns I see coming from the KeyStore. I just wanted to make sure I was addressing all the relevant concerns in the discussion here. Generally it seems you and I are aligned on the approach we should take, but I'd like to see what @kwek20 has to say about all this before we look to implementation (although I have a pseudo implementation done from a couple months back that adheres to this general approach with the topic mapping).

[arnauorriols]

This has evolved a lot since the last comment. Are we content with the current implementation?

[DyrellC]

Yes I think we're good