Bucket Pinning API for IPFS and Filecoin

[andrewxhill]

In December of last year, we began implementing the IPFS Pinning API Spec directly inside buckets. This work is exciting because it would allow any bucket to instantly become a pinning endpoint for your app and include all the permissions and synchronization tools within buckets.

Buckets are themselves dynamic, synchronizable, and permission-based. So imagine being able to subscribe or pull changes to your pin sets from other bucket nodes (e.g., hosting providers like the Hub). Easily set up mirrors and ensure replication of your entire pin set. Imagine having a pinning API that you could share with your collaborators, revoke permissions, or enable multiple applications to add pins simultaneously.

We are excited by the potential of a pinning API on buckets. Below, we'll detail a few additional areas we think will excite people. Please let us know how you would use the new API or any design feedback you think will help us get it right.

Pinning and DIDs

• Our internal MVP of a bucket pinning API highlighted weaknesses in our existing identity models and auth methods. We want to ensure that we support flexible identity tooling and integrate with other networks easily. The same goes for permissions and authentication. With that in mind, we've started our journey toward broad DID support in all our tools (Hub, Buckets, and Threads). We've written up the start of our progress toward a more flexible system that supports DIDs here. The bucket pinning API would be an immediate beneficiary of this work.

Filecoin Pinning API

• Multiple projects have asked us for a simple "pinning API" for Filecoin. The idea would be to send the API a CID and get it stored as a deal on Filecoin. By supporting a set of additional tags that would allow data to bypass IPFS and store on Filecoin, or store on both, the Bucket Pinning API could offer a Filecoin pinning API.

Multi-protocol

Buckets are really "buckets of protocols" as much as they are "buckets of files." Today, there is various support for IPFS, IPNS, HTTP, and gRPC. There is a clear path to make the pinning API on buckets available over more than HTTP (as in the spec). We'll share some further discussions on making these services available over Libp2p directly, but please let us know if your app or project is interested in this topic.

A pinning API wherever you run buckets

We want to enable a pinning API on buckets such that anyone who runs the bucket daemon gets a fully-featured pinning API out of the box. Fastest Pinning API deployment ever?

Feedback

Are you interested in a new kind of Pinning API, where pin sets are dynamic documents that themselves can be synchronized over the decentralized web? If so, let us know your thoughts.

• Would you mirror your pin sets with buckets?
• Are you waiting for a Filecoin Pinning API?
• What is the ideal way to pay? Native crypto with per CID transactions? USD invoices? Something else?
• How else would you want to use the API?

[alanshaw]

There is a clear path to make the pinning API on buckets available over more than HTTP (as in the spec). We'll share some further discussions on making these services available over Libp2p directly, but please let us know if your app or project is interested in this topic.

It would be interesting if the pinning API was available over libp2p and was able to use the Peer ID as the auth token instead of having to create a JWT out-of-band.

[carsonfarmer]

This is a great suggestion. Our goal is to have pretty much all these APIs we're exposing available over libp2p. In fact, with the peer id bit, there's a really interesting path here. Because the peer id is derived from a key, it is possible to represent it as a did, and use that for the auth token. I like this idea so much, and its something we've been playing around with internally.
Anyway, long story short, we should be able to expose a peer id as did:p2p:, and then decode it to extract the relevant doc (which is really just basically a did:key.

Anyway, long story short: yes, thanks for the suggestion... I think we'll be able to support this soon!

[andrewxhill]

Quoting @sanderpick from [slack engineering](https://slack.textile.io/

Making some progress on exposing a bucket as an IPFS pinning service:

> ipfs pin remote service ls
mybucket http://127.0.0.1:8000/thread/bafk6chlb2ez6ee7iwjq7fhjsv7wffzea57sa/buckets/bafzbeidkhyny26jte426cd72pwuttyivtge5ijh36vlj45vczzpizwtx7y

Interact with the bucket from IPFS:

> ipfs pin remote add --service=mybucket --name=mypin --background bafybeig24a3nubcbescvhwb5rlab7v4r676rpqa3w2uulwjd472qt3rtda
CID:    bafybeig24a3nubcbescvhwb5rlab7v4r676rpqa3w2uulwjd472qt3rtda
Name:   mypin
Status: queued

The pin is tracked in the bucket’s metadata:

> ipfs pin remote service ls
mybucket http://127.0.0.1:8000/thread/bafk6chlb2ez6ee7iwjq7fhjsv7wffzea57sa/buckets/bafzbeidkhyny26jte426cd72pwuttyivtge5ijh36vlj45vczzpizwtx7y

Making some progress on exposing a bucket as an IPFS pinning service:

> buck get
{
  "thread": "bafk6chlb2ez6ee7iwjq7fhjsv7wffzea57sa",
  "_id": "bafzbeidkhyny26jte426cd72pwuttyivtge5ijh36vlj45vczzpizwtx7y",
  "owner": "did:key:12D3KooWRQkPKVnGLxBhZbbi8zi8QECVZwbSjfYV7mak5vk99T9p",
  "name": "unnamed",
  "version": 1,
  "path": "/ipfs/bafybeibbva2q5u5huwu2l5n25jzgvo2xxxbavi2ex5jujyu6lf7a7iu3ni",
  "metadata": {
    "": {
      "roles": {
        "*": 1,
        "did:key:12D3KooWRQkPKVnGLxBhZbbi8zi8QECVZwbSjfYV7mak5vk99T9p": 3
      },
      "updated_at": 1615792297065949700
    },
    ".textileseed": {
      "roles": {},
      "updated_at": 1615792297065949700
    },
    "bqpesoi5aegy776lugbf7dpadr7h4ztoo": {
      "roles": {},
      "info": {
        "pin": {
          "created": "2021-03-15T07:44:53.074051506Z",
          "pin": {
            "cid": "bafybeig24a3nubcbescvhwb5rlab7v4r676rpqa3w2uulwjd472qt3rtda",
            "name": "mypin",
            "origins": [
              "/ip4/192.168.64.1/tcp/4001/p2p/12D3KooWFSFtCvJCuimAnSF2n2zW67NroPv8Y29w77JN1q9WpJ4L",
              "/ip4/127.0.0.1/tcp/4001/p2p/12D3KooWFSFtCvJCuimAnSF2n2zW67NroPv8Y29w77JN1q9WpJ4L",
              "/ip6/::1/tcp/4001/p2p/12D3KooWFSFtCvJCuimAnSF2n2zW67NroPv8Y29w77JN1q9WpJ4L",
              "/ip4/192.168.64.1/udp/4001/quic/p2p/12D3KooWFSFtCvJCuimAnSF2n2zW67NroPv8Y29w77JN1q9WpJ4L",
              "/ip4/127.0.0.1/udp/4001/quic/p2p/12D3KooWFSFtCvJCuimAnSF2n2zW67NroPv8Y29w77JN1q9WpJ4L",
              "/ip6/::1/udp/4001/quic/p2p/12D3KooWFSFtCvJCuimAnSF2n2zW67NroPv8Y29w77JN1q9WpJ4L",
              "/ip4/75.142.226.110/tcp/11340/p2p/12D3KooWFSFtCvJCuimAnSF2n2zW67NroPv8Y29w77JN1q9WpJ4L",
              "/ip4/75.142.226.110/udp/11340/quic/p2p/12D3KooWFSFtCvJCuimAnSF2n2zW67NroPv8Y29w77JN1q9WpJ4L"
            ]
          },
          "requestid": "bqpesoi5aegy776lugbf7dpadr7h4ztoo",
          "status": "queued"
        }
      },
      "updated_at": 1615794293141921800
    }
  },
  "created_at": 1615792297065949700,
  "updated_at": 1615794676375010600
}

[Z0rlord]

awesome, thanks, just saw this. Working through the textile tutorial on gr9. Your talk was helpful. Best Regards, Zorie

-- Zorie Barber Main: [+1 213-538-2972](tel:+1202135382972) [Meeting Request](https://calendly.com/zorie-1/30min)

This e-mail is the property of WisperLuxe, llc, Shibumi Hotels NA, llc, and Shibumi Crypto Capital Digital Asset Fund, lp. It is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential, or otherwise protected from disclosure. Distribution or copying of this e-mail, or the information contained herein, to anyone other than the intended recipient is prohibited.

Sent with [ProtonMail](https://protonmail.com) Secure Email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

…

On Monday, March 15, 2021 4:46 PM, Andrew W. Hill ***@***.***> wrote: Quoting ***@***.***(https://github.com/sanderpick) from [slack engineering](https://slack.textile.io/ > Making some progress on exposing a bucket as an IPFS pinning service: > ipfs pin remote service ls mybucket http://127.0.0.1:8000/thread/bafk6chlb2ez6ee7iwjq7fhjsv7wffzea57sa/buckets/bafzbeidkhyny26jte426cd72pwuttyivtge5ijh36vlj45vczzpizwtx7y > Interact with the bucket from IPFS: > ipfs pin remote add --service=mybucket --name=mypin --background bafybeig24a3nubcbescvhwb5rlab7v4r676rpqa3w2uulwjd472qt3rtda CID: bafybeig24a3nubcbescvhwb5rlab7v4r676rpqa3w2uulwjd472qt3rtda Name: mypin Status: queued > The pin is tracked in the bucket’s metadata: > ipfs pin remote service ls mybucket http://127.0.0.1:8000/thread/bafk6chlb2ez6ee7iwjq7fhjsv7wffzea57sa/buckets/bafzbeidkhyny26jte426cd72pwuttyivtge5ijh36vlj45vczzpizwtx7y > Making some progress on exposing a bucket as an IPFS pinning service: > buck get { " thread " : " bafk6chlb2ez6ee7iwjq7fhjsv7wffzea57sa " , " _id " : " bafzbeidkhyny26jte426cd72pwuttyivtge5ijh36vlj45vczzpizwtx7y " , " owner " : " did:key:12D3KooWRQkPKVnGLxBhZbbi8zi8QECVZwbSjfYV7mak5vk99T9p " , " name " : " unnamed " , " version " : 1, " path " : " /ipfs/bafybeibbva2q5u5huwu2l5n25jzgvo2xxxbavi2ex5jujyu6lf7a7iu3ni " , " metadata " : { " " : { " roles " : { " * " : 1, " did:key:12D3KooWRQkPKVnGLxBhZbbi8zi8QECVZwbSjfYV7mak5vk99T9p " : 3 }, " updated_at " : 1615792297065949700 }, " .textileseed " : { " roles " : {}, " updated_at " : 1615792297065949700 }, " bqpesoi5aegy776lugbf7dpadr7h4ztoo " : { " roles " : {}, " info " : { " pin " : { " created " : " 2021-03-15T07:44:53.074051506Z " , " pin " : { " cid " : " bafybeig24a3nubcbescvhwb5rlab7v4r676rpqa3w2uulwjd472qt3rtda " , " name " : " mypin " , " origins " : [ " /ip4/192.168.64.1/tcp/4001/p2p/12D3KooWFSFtCvJCuimAnSF2n2zW67NroPv8Y29w77JN1q9WpJ4L " , " /ip4/127.0.0.1/tcp/4001/p2p/12D3KooWFSFtCvJCuimAnSF2n2zW67NroPv8Y29w77JN1q9WpJ4L " , " /ip6/::1/tcp/4001/p2p/12D3KooWFSFtCvJCuimAnSF2n2zW67NroPv8Y29w77JN1q9WpJ4L " , " /ip4/192.168.64.1/udp/4001/quic/p2p/12D3KooWFSFtCvJCuimAnSF2n2zW67NroPv8Y29w77JN1q9WpJ4L " , " /ip4/127.0.0.1/udp/4001/quic/p2p/12D3KooWFSFtCvJCuimAnSF2n2zW67NroPv8Y29w77JN1q9WpJ4L " , " /ip6/::1/udp/4001/quic/p2p/12D3KooWFSFtCvJCuimAnSF2n2zW67NroPv8Y29w77JN1q9WpJ4L " , " /ip4/75.142.226.110/tcp/11340/p2p/12D3KooWFSFtCvJCuimAnSF2n2zW67NroPv8Y29w77JN1q9WpJ4L " , " /ip4/75.142.226.110/udp/11340/quic/p2p/12D3KooWFSFtCvJCuimAnSF2n2zW67NroPv8Y29w77JN1q9WpJ4L " ] }, " requestid " : " bqpesoi5aegy776lugbf7dpadr7h4ztoo " , " status " : " queued " } }, " updated_at " : 1615794293141921800 } }, " created_at " : 1615792297065949700, " updated_at " : 1615794676375010600 } — You are receiving this because you are subscribed to this thread. Reply to this email directly, [view it on GitHub](https://github.com/textileio/textile/discussions/499#discussioncomment-485875), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AHYR2M44C7IGRSOTD6AJEKDTDZW2TANCNFSM4XZGAKYQ).

[codynhat]

This would be very cool.

We just finished a simple integration with the IPFS Pinning Service API to anchor AR/3D content on the Geo Web and pin the data using a pinning service.

We are really interested in creating a default solution for users who do not want to manage their own pinning service powered by Buckets.

I need to get caught up on the latest with DIDs and Threads and check out @sanderpick's PR. Hoping to explore this soon!