Skip to content

Latest commit

 

History

History
269 lines (229 loc) · 13.2 KB

02-24jan2018.md

File metadata and controls

269 lines (229 loc) · 13.2 KB
Raw

dat protocol WG (#2)

24 Jan 2017 via IRC (#datprotocol)

Agenda discussion

People

  • Mafintosh
  • Joe
  • Paul
  • Tara
  • Bryan
  • Karissa
  • Danielle

Agenda

  • JOE Summary/Review of action items from previous meeting (#2)
  • BRYAN Discussion and approval of DEP 1 (dat-ecosystem-archive/DEPs#2)
  • BRYAN structure (how many DEPs) and division of labor for expressing existing protocol in DEP form
  • BRYAN multi-writer API ergonomics (eg, making single-writer case simple)1
  • TARA post-mortem of Dat discovery failing on 1/20, future for DHT2

Summary/Review of meeting 1

  • Most follow-ups done
  • main ones we didn't get done are on discussion later in the agenda (DEP proposals)
  • we have a new datprotocol.com site https://deploy-preview-6--datprotocol.netlify.com/ waiting for PR
  • Need to wait for merge on DEP repo to do auto-generation for website
  • Whitepaper moved, need to remove from docs.datproject.org site still (and move references)

DEP-0001: DEP Process

Rendered: https://github.com/bnewbold/dat-deps/blob/meta-dep/proposals/0001-dep-process.md

Specific points ("burried ledes") to review:

  • DEP text licensing: standardize on CC-BY, with explicit authorship
    • +1 (pfrazee, mafintosh, dcrobins, taravancil) (DECISION)
  • "Working Group" has ultimate decision making

    • "how formalized should the working group be? voting? nominations for membership? should that formalism/rules live elsewhere?"

    • Concern of tyranny of structureless system

    • We need structure

      • +1 karissa, mafintosh, taravincil

    • Start with current group and have single consensus system internally

    • Use consensus

    • New member happen in DEP or elsewhere?

      • +1 in issues in working-group repo (ACTION: setup template issue)

    • WG membership, and changes to it, belong in a separate repo

    • formalizing voting rules sounds like a P-DEP

    • Define Consensus (consensus = no objections)

      • A: lack of veto/block within a period
      • B: a single individual or group veto can be overridden by everybody else agreeing
        • +1 mafintosh/tara
      • A "default-accept consensus", and term B "voting consensus"
      • Consent is all WG members, minus 1 (DECISION)

    • Is vote by org or by member?

      • if we define "voting consensus" as "unanimous minus one" then it shouldnt matter
      • we'd all be individuals (eg, you could block something even if tara liked it), and only consider orgs when it comes down to an override
      • WG membership vs DEP decisions
      • unless/until we have a voting process that's more complex than unanimous minus one, that we use member voting and not org voting
        • +1 mafintosh, bryan, tara, joe, dcr
        • (DECISION)

  1. how is active WG membership tracked, and what is the process for changing it?

    • listing members in the WG README. To change it open an issue and wait for consensus from current WG members (until DEP formalized process)
      • +1 tara
    • Use "voting" consensus for WG membership, do not rush process
      • +1 mafintosh, bryan, tara
    • Use datprotocol/working-group to:
      • record active members, and this process, in its readme
      • (ACTION, DECISION)
      • +1 karissa, tara

  2. how does the WG come to consensus on protocols?

    • loose consensus (faster decision) to accept a draft, then more careful ("voting consensus"?) for draft->action
      • +1 pfrazee, maf, tara (DECISION)
    • Loose consesnsus
      • 3 weeks
        • +1 maf, pfrazee (DECISION)
        • at minimum, whoever merges the PR should review. So if 3 weeks go by and there's been no activity, the person to do the acceptance can review and merge or veto
          • +1 bryan
          • "If no review takes place in the fixed time window, the default is to close (reject) until a member is willing to commit to review."
            • +1 pfrazee (DECISION)
      • No veto/block within time period
        • +1 tara
      • before invoking the consensus process, some WG member ("editor", any person, not a specific role) checks it over for completeness
      • Expectation is entire WG doesn't have to review every DEP before it becomes draft
      • loose-consensus veto override by voting consensus?
        • clarify: if someone objects, to override that we have to reach voting consensus?
        • +1 bryan, tara
    • What does "Draft" imply for implementation?
      • Note: bt BEPs are only drafts but all implemented (example)
      • Should there be one process for accepting draft + bringing draft to completion or two seprate ones?
      • Similar to BT, where ship features that are still in draft
      • OR draft is more "draft", not implementation
      • I feel like we (Dat/hyper*) are more like Python/Rust in that we want to put proposed stuff out on the table sooner, to help with implementation
      • does "Active" signify anything about whether the DEP must be adopted to be conformant with Dat?
        • for now we should not mandate too much, and should focus on makign the DEPs readable and useful
          • +1 maf
      • drafts don't imply institutional mandate or even support
        • +1 (DECISION)
    • specific timelines (2 or 3 weeks) for review process
      • (see above)
    • SUMMARY:
      • 3 week period of loose consensus with a voting-consensus override against vetoes
        • +1 drc, bryan, maf (DECISION)
  • joint authorship (by working group) for DEP-0001
    • clarify: authorship is good for individual things and all future DEPs, but for this first one it felt like a unified front might be good?
    • +1 tara, jh, pfrazee, maf, drc (DECISION)
  • action item for DEP-0001 will be that Bryan re-edit and submit (ACTION)
    • clarify: that means we'll be starting the draft-acceptance process for DEP-0001, correct
      • YES
    • which is three weeks. do we want to short-circuit that as a special case and say we'll give a merge/no-merge decision in 2 weeks (at the next meeting)?
      • +1 pfrazee, joe, maf (DECISION)

Upcoming DEPs

Order of priority?

hyperdb - standard - noffle, mafintosh, bnewbold multi-writer - standard - mafintosh, bnewbold wire-protocol - standard (PFRAZEE, mafintosh review) hypercore (or hyperregister?) - standard (PFRAZEE, mafintosh review) discovery - informative (PFRAZEE, TARA) nomenclature - informative (JOE)

  • SLEEP should become a DEP, but existing whitepaper is pretty good, so less rush?

    • We should DEP it tho
      • +1 bryan

  • mafintosh prefer not to sit and write these =), happy to review

  • (ACTION) summarize DEP needs, owners of each DEP

  • to clarify this discussion, i'd say the priorities are to get some of the work-in-progress stuff (hyperdb) documented, because it isnt' anywhere yet and get stuff that isn't well covered in the whitepaper into DEPs, like hyperdrive

  • the wire protocol is sort of a priority because encryption isn't in the whitepaper

  • pfrazee: I actually have a pretty good idea of how I could get initial drafts done of the wire protocol, hypercore, hyperdrive, and discovery

    • take those and rest leave as-is
      • +1 bryan (DECISION)

Multi-Writer Concerns

(ACTION): PUNTED TO NEXT MEETING

  • mafintosh: tldr in agreement on this

Similar to git merge strategies: https://git-scm.com/docs/merge-strategies

https://github.com/mafintosh/hyperdb#dbgetkey-callback

db.get(key, callback) -> [feed1_chunk, feed2_chunk, ...]

Only get multiple chunks if there is ambiguity.

Concern is that app developers will just take the first element, which will result in weird behavior. Alternatives:

  • db.get_stream(key, feed_id, callback) -> feed_chunk
  • db.get(key, callback_single, callback_multiple)
  • db.get(key, callback) -> single or error
  • ship with default merge/resolution algorithm; also selection via config, or callback to implement custom. more like git.
  • others? brainstorm?

post-mortem of Dat discovery failing on 1/20, future for DHT

  • make everyone was aware the Dat network effectively had an outage on 1/20
    • discovery server went down (now fixed)
    • bug in code made fallback from being used (now fixed)
    • mafintosh; its a bit intertwined:
      1. we had a bug we discovered by writing the integration tests we decided last meeting where a retry/timeout loop didn't run as expected meaning that a timeout from dns servers was never discovered. the discovery code was relying on that in case of dns failure (it would never hit secondary discovery). We fixed that around 12 days ago, but beaker (and prob lots of dat clients) were still using an older version.
      2. which meant that when one of the dns servers had an outage (they do sometimes, various reasons) the fallbacks didn't run
      3. the bittorrent DHT was still active but just totally failed to work
    • would it be helpful to have a third DNS server?
      • Yes
      • Archive could host one with poor uptime (ACTION, bryan)
      • mafintosh has dingie ops setup
    • tl;dr classic bad luck
    • Integration tests working!
  • maf: what do you think is the most realistic/pessimistic timeline for hyperdht?
    • code is all there and working for hyperdht. the missing part is integration (close to nothing done there)
  • is the bittorrent DHT totally broken today? or just unreliable?
    • we are just hitting some limitations with how we use it
    • maf idea: we write a new dns server, backed by hyperdht
      • so it's a dns server network that people can easily help host. that should give us much much much better uptime for our current stack
      • with close to no changes needed to the clients
      • would the idea be that hyperdht would help the dns servers share their table?
        • yes
      • for now just DNS -> hyperdht
      • good way to stress test hyperdht
      • then when we have bandwidth we integrate the dht into the clients
      • maf: estimate is that we could prototype that in a day or two

  • technical question about that: are DNS registrations sent to both/all servers? and do queries go to both/all? or round-robin?

Decisions Made

  • Working Group Meta Info, Process
    • Meetings
      • Keep doing IRC meetings =)
    • WG Membership:
      • Add members via Voting Consensus
      • Voting Consensus is all WG members, minus 1
      • Unless/until we have a voting process that’s more complex than unanimous minus one, we use member voting and not org voting
      • Use datprotocol/working-group to record active members, and this process, in its readme
  • DEPs
    • DEP text licensing: standardize on CC-BY, with explicit authorship
    • DEP-0001
      • joint authorship
      • starting draft-acceptance process
      • use 2 week review (so we can accept at next meeting)
    • DEP Review & Acceptance
      • Before invoking the consensus process, any WG member must check it over for completeness
      • Consensus Process
        • Loose Consensus to accept a draft
        • Voting Consensus for draft->action
        • Loose Consensus veto override requires Voting Consensus
      • 3 week review period
        • If no review takes place in the review period, the default is to close (reject) until WG member is willing to commit to review.
    • DEP Draft vs Active
      • Drafts don’t imply institutional mandate or even support
      • Does "Active" signify DEP must be supported
        • Will fully clarify as more examples come up
    • Upcoming DEPs
      • Prioritize DEPs not in paper

ACTION ITEMS

  • Working Group Meta Info
    • datprotocol/working-group Repo Updates
      • Record active members (@joehand)
      • Document WG member add/remove process (@joehand)
      • Issue template for adding new WG members (@joehand)
      • Add link/info about integration tests (@joehand)
    • Datprotocol.com
  • DEPs
    • DEP-0001
      • Edit and submit based on meeting decisions (@bnewbold)
      • Review PR before next meeting (WG)
    • Upcoming DEPs
      • High Priority - * Prepare Drafts*
        • wire-protocol - standard (@pfrazee, @mafintosh review)
        • hyperdb - standard (@bnewbold, @mafintosh review)
        • multi-writer - standard (@bnewbold, @mafintosh review)
        • nomenclature - informative (@joehand, WG review)
      • Lower Priority (already in paper)
        • discovery - informative (@taravancil/@pfrazee, @mafintosh review)
        • hypercore|drive - standard (@pfrazee, @mafintosh review)
        • SLEEP - standard
  • Other
    • Network Uptime Improvements
      • IA host DNS server (@bnewbold ping maf about it)
      • Prototype DNS -> Hyperdht
  • Next Meeting (Feb 7, 9:30am PST)
    • Setup issue for agenda (@joehand)
      • Add multi-writer concerns we punted (@joehand)
    • Make template for notes (@joehand)
    • Make recurring calendar invites (@joehand)