diff --git a/Makefile b/Makefile index 4f4f34c0..72b356d6 100644 --- a/Makefile +++ b/Makefile @@ -10,6 +10,9 @@ common: models rpc indexer: common yarn indexer build +dag-indexer: common + yarn dag-indexer build + file-retriever: common yarn file-retriever build diff --git a/README.md b/README.md index de1c1127..54e4196e 100644 --- a/README.md +++ b/README.md @@ -35,6 +35,10 @@ sequenceDiagram Gateway->>Node: Retrieve object metadata from the node ``` +## Getting Started + +- [Local Development Guide](docs/development.md) - Run the gateway locally for development + For a more detailed over on each service check out: - [File retriever](docs/file-retriever.md) diff --git a/docker/object-mapping-indexer/docker-compose.yml b/docker/object-mapping-indexer/docker-compose.yml index df34967d..f1d231cd 100644 --- a/docker/object-mapping-indexer/docker-compose.yml +++ b/docker/object-mapping-indexer/docker-compose.yml @@ -5,6 +5,9 @@ volumes: services: subspace-node: image: ${NODE_DOCKER_TAG} + profiles: + - node + - production volumes: - subspace-node-data:/var/subspace:rw ports: @@ -39,6 +42,8 @@ services: image: ${OBJECT_MAPPING_INDEXER_DOCKER_TAG} container_name: object-mapping-app working_dir: /app + profiles: + - production ports: - '${OBJECT_MAPPING_INDEXER_PORT:-3000}:3000' environment: diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 00000000..9eecacbd --- /dev/null +++ b/docs/development.md @@ -0,0 +1,83 @@ +# How to launch files gateway? + +As is detailed on [docs](https://github.com/autonomys/auto-files-gateway/), files gateway is a three component system and should be launched in this order: + +## 1. Subspace Node + +Not essential you can optionally connect a remote RPC though you will have to wait for your published IPLD nodes to be archived. + +## 2. Database + +Both DAG indexer and object mapping indexer needs a database for storing some data. There you can launch a postgres at: `psql://postgres:postgres@localhost:5432/postgres` by executing the command `docker compose -f docker/object-mapping-indexer/docker-compose.yml --profile development up -d db` + +## 3. DAG indexer + +This will store the IPLD nodes metadata on a database which enables features like partial file retrieval and speeds up retrieval by not retrieving `Inlink` nodes. + +You will have to setup the following `.env` at the root of the repo + +```bash +# subquery params +RPC_ENDPOINTS=wss://rpc-0.mainnet.autonomys.xyz/ws # or your local +CHAIN_ID=0x66455a580aabff303720aa83adbe6c44502922251c03ba73686d5245da9e21bd +START_BLOCK=0 # this might fail if node is not in archival mode + +# database +DB_USER=postgres +DB_PASSWORD=postgres +DB_DATABASE=postgres +DB_HOST=127.0.0.1 +DB_PORT=5432 +``` + +And launch it with: + +```bash +make dag-indexer +yarn dag-indexer start +``` + +## 4. Object Mapping Indexer + +For setting up the object mapping indexer, you will need to provide the following config at `services/object-mapping-indexer/.env` + +```bash +NODE_RPC_URL=wss://rpc-0.mainnet.autonomys.xyz/ws # or your local +DATABASE_URL=postgres://postgres:postgres@localhost:5432/postgres +``` + +And, for launch it: + +```bash +make indexer +yarn indexer start +``` + +## 5. File Retriever + +Finally, the last piece of the files gateway is the file retriever which has the following config at `services/file-retriever/.env` + +```bash +SUBSPACE_GATEWAY_URLS=http://localhost:9955 +OBJECT_MAPPING_INDEXER_URL=http://localhost:3000 +VICTORIA_ACTIVE=false +VICTORIA_ENDPOINT= +VICTORIA_USERNAME= +VICTORIA_PASSWORD= +API_SECRET=random-secret # the apikey needed for file retrieval +CORS_ORIGIN=* +MAX_SIMULTANEOUS_FETCHES=10 # optional +CACHE_DIR=./.cache # optional +CACHE_MAX_SIZE=1024000000 # optional +CACHE_TTL=86400000 # optional + +``` + +And, for launch it: + +```bash +make file-retriever +yarn file-retriever start +``` + +And with that files gateway would be fully deployed locally