Skip to content

Meeting some of the dragons of IPFS

Jump to bottom
Gorka Ludlow edited this page Oct 16, 2017 · 10 revisions

We decided to build a decentralized app where users could shoot GIFs and share them. A decentralized app means we wanted to avoid running servers that had ownership and authority over the generated content. We found there are communities, platforms, tools and other resources for such a mission.

We chose the IPFS path:

IPFS (the InterPlanetary File System) is a new hypermedia distribution protocol, addressed by content and identities. IPFS enables the creation of completely distributed applications. It aims to make the web faster, safer, and more open.

IPFS is a distributed file system that seeks to connect all computing devices with the same system of files. In some ways, this is similar to the original aims of the Web, but IPFS is actually more similar to a single bittorrent swarm exchanging git objects.

IPFS Overview @ Github

They offer a set of tools (in a variety of flavors: Go, Node & browser JS) that can be used to access and interact with their decentralized network.

Our plan: build a static single page application. Our stack included create-react-app which under the hood implements:

  • React
  • Webpack
  • Babel

The webapp

The idea behind a decentralized app is that the content is not uploaded to a server but rather stored and served directly from the client machine. What this means is that we had to store our files in the user's device and then find a way to make this content accessible to others.

ipfs-js offers an abstraction layer to interact with the IPFS network in order to add/retrive files to their network. After the files have been added a unique id is returned which can be used to retrieve the files's content from any IPFS gateway in a regular http fashion: https://ipfs.io/ipfs/UNIQUE_HASH.

We coded our process into adding our files into the IPFS network, received the unique ids and started rendering the images into our webapp. And we felt very happy. Our code was working.

We thought we were ready to continue building new features, but then we discovered a problem:

Build process failure

If you are not familiar with create-react-app the command yarn build executes a process that creates/updates an optimized bundle of HTML files (html, js and css) for your code. This was the output we received:

$ yarn build
yarn run v1.2.0
$ react-scripts build
Creating an optimized production build...
Failed to compile.

Failed to minify the code from this file:

        ./node_modules/cids/src/index.js:23

Read more here: http://bit.ly/2tRViJ9

error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

The shortened url redirects to: npm-run-build-fails-to-minify with the following:

You may occasionally find a package you depend on needs compiled or ships code for a non-browser environment. This is considered poor practice in the ecosystem and does not have an escape hatch in Create React App.

To resolve this:

Open an issue on the dependency's issue tracker and ask that the package be published pre-compiled (retaining ES6 Modules). Fork the package and publish a corrected version yourself. If the dependency is small enough, copy it to your src/ folder and treat it as application code.

Hmmm. So somebody was shipping a package that wasn't browser environment friendly. Some research led to this issue with a comment that quotes the exact same output we received.

This error comes from the uglify step in create-react-app's build process. It does not know how to work with code written in es6 syntax. There where several ways we could go at this:

  • use a tool to minify that can work with code written in es6 (open issue here)
  • transpile the dependencies's source code ourselves
  • remove the minify step in the build process

Transpiling source code

We decided to research some more. We found that the dependency cids had its code base written in es6 syntax. We also found out their npm package had its main entry pointing to their src/index.js file rather that to their dist/index.js file.

We dug a little deeper and found that they were using a tool called AEgir to build their project and that this tool did not include a transpilation step in their build process. We thought to ourselves: "let`s contribute by adding such a step once we find this is the solution to our problem" this way we could help avoid this problem in the future.

Rather than submitting an issue (and the accompanying PR) and having to wait for an answer we directly started transpiling the source code of the dependency. This way we could be sure our solution worked. And it did, but another problem popped up: another dependency caused the same error. No biggy. We saw they were using AEgir as well, and out solution would benefit them as well.

Using babel-cli it is easy to transpile code in es6 syntax:

$ babel-cli SOURCE_FILE -o DESTINATION_FILE --presets=es2015

The build process was working again.

Errors in production

We decided to take our new shiny build for a test drive. We did not get far. Two new errors appeared - and these where not happening in our dev environments:

  • Uncaught Error: second argument must be a CID
  • Mixed Content: The page at 'PUBLIC_URL' was loaded over HTTPS, but attempted to connect to the insecure WebSocket endpoint 'ws://star-signal.cloud.ipfs.team/socket.io/?EIO=3&transport=websocket'. This request has been blocked; this endpoint must be available over WSS.

Oh my. At least we had seen this before, here. We had hit a known wall, worked our way around it to find another known wall.

We tried some quick hacking: lets force wss on that request to see if it works. No. And surprisingly, not even an error to let us know what was happening. Later we realized the second error had nothing to do with this issue.

We had one last idea to make our build process work: taking out the minify step.

Minification is needed

Our regular build goes for 3.7M while the minified one for 1.5M. We do need to minify our deployed code but right now we had to take this step out of the build process to make it work.

After writing this journal entry it was easier to pinpoint what had happened, to sum up:

  • create-react-app uses uglify 2 and some of our dependencies (mostly from IPFS) where throwing an error when minifying (their codebase is built with es6 syntax and uglify 2 is not able to understand it).
  • By transpiling their code base before minifying we hoped the process would succeed, and it did, but another error popped up. This error is referenced here. Seems their library checks against class names. minify libraries overwrite class names thus the check fails.
  • Tried using uglify 3 since it understands es6 syntax but the name checking error still renders and the app does not work.

The network lagging and newest changes

At this point that we started seeing slow responses from the IPFS network. Our GIFs (we were rendering them via an IPFS gateway) started rendering some time after the html had loaded and sometimes not loading at all. This could've been the reason why we say no errors nor warnings at all when we tried to force wss on the connection - still a maybe.

18 days ago there were some changes that disrupted the way the pubsub features works (discussed here). Even though this changes do not implement breaking compatibility features they caused a critical high severity issue.

4 days ago there was a surge in nodes in the IPFS network (discussed here). The network wasn't ready for this kind of problem, a nice problem to have, but still a big problem.

Clone this wiki locally