refactor: ⎆ Migration to pull-streams (from Node.js Streams)

[dignifiedquire]

As noted in #362 I did the due diligence of confirming that using pull-streams all the way through solves our issues with secio. The result of this can be seen in libp2p/js-libp2p-switch#67. (Warning lots of module linking required to get this running)

The next step is to move from the prototype into a fully tested version for all modules.
Below is a list of the modules that will need work.

ipfs

• js-ipfs - PR [WIP] Move to pull-streams #440
• interface-ipfs-core - PR fix(files): don't expect a specific sorting of files ipfs-inactive/interface-js-ipfs-core#70
• js-ipfs-api - delayed - Issue Migrate to pull-streams ipfs-inactive/js-ipfs-http-client#367
• js-ipfs-unixfs-engine - PR [WIP] Pull streams ipfs-inactive/js-ipfs-unixfs-engine#64
• js-libp2p-ipfs (Only testing) - PR pull ipfs-inactive/js-libp2p-ipfs-nodejs#17
• js-libp2p-ipfs-browser (Only testing) - PR refactor: use pull-streams ipfs-inactive/js-libp2p-ipfs-browser#84
• js-ipfs-merkle-dag - PR [WIP] refactor: move to pull-streams js-ipfs-merkle-dag#44
• js-ipfs-block-service - PR refactor: Migrate to pull-streams js-ipfs-block-service#23
• js-ipfs-ipld - PR refactor: migrate to pullstreams ipld/js-ipld#54
• js-ipfs-repo - PR [WIP] Move to pull-streams js-ipfs-repo#76
• New interface-pull-blob-store (replaces abstract-blob-store) https://github.com/ipfs/interface-pull-blob-store
• New idb-pull-blob-store https://github.com/ipfs/js-idb-pull-blob-store
• New fs-pull-blob-store https://github.com/ipfs/js-fs-pull-blob-store
• js-ipfs-bitswap - PR [WIP] Pull streams js-ipfs-bitswap#25
• js-multistream - PR [WIP] pull-streams multiformats/js-multistream-select#19

libp2p

• interface-connection - PR Migrate to pull-streams libp2p/interface-connection#10
• js-libp2p-identify - PR [WIP] move to pull-streams libp2p/js-libp2p-identify#2
• js-libp2p-secio - PR [WIP] Refactor to use pull streams libp2p/js-libp2p-secio#8
• js-libp2p-spdy - PR [WIP] Move to pull-streams libp2p/js-libp2p-spdy#16
• js-libp2p-websockets - PR [WIP] Move to pull-streams libp2p/js-libp2p-websockets#18
• js-libp2p-tcp - PR Migrate to pull-streams libp2p/js-libp2p-tcp#20
• js-libp2p-webrtc-star PR Migrate to pull-streams libp2p/js-libp2p-webrtc-star#12
• js-libp2p-ping - PR migrate to pull-streams libp2p/js-libp2p-ping#11
• js-libp2p-swarm - PR [WIP] Add secio & move to pull-streams libp2p/js-libp2p-switch#67
• interface-stream-muxer - PR [WIP] Pull-streams libp2p/interface-stream-muxer#7
• interface-transport - PR https://github.com/libp2p/interface-transport#8

Documentation

• Add a note about pull-streams to every readme
• Create a FAQ document collecting all important information on how to use pull-streams, how to convert them, why we use them etc.

[daviddias]

Thank you for captaining this @dignifiedquire :) :) Let's make sure not to forget documentation :)

[dignifiedquire]

Draft for the readme note.

This module uses pull-streams for all stream based interfaces. You can find details of why we use them in this document. If you need to convert them to node-streams, you can do so easily with pull-stream-to-stream like this:

const toStream = require('pull-stream-to-stream')
const pullStream = getPullStream()
const myNodeStream = toStream(pullStream)

[jbenet]

before doing the work of chaning every module, i highly recommend you pick a small set and work with them-- see how they go. You wont understand how an abstraction fits a project until you play with it for a while. and def better to find that out with a small set than after changing everything.

[dignifiedquire]

@jbenet we already did that as outlined at the beginning of this issue. I ported the minimal needed set to get secio in swarm working and see how things are working out, and they worked out great. Afterwards I talked with @diasdavid and we decided to move forward. The result of this is this endeveaour of moving all things into high quality ports.

[RichardLitt]

Sub. Let me know if I can help with READMEs.

[dignifiedquire]

Thanks @RichardLitt ❤️ will ping you as soon as I have a consistent message ready.

[daviddias]

Missing from the list:

• js-multistream

[dignifiedquire]

Missing from the list:

js-multistream

added

[daviddias]

One more: libp2p/interface-transport#8