ADR - Custom mirror node/JSON-RPC-Relay services

[mamorales1]

Context

In the accelerator, we currently use Hedera mirror node to get information about accounts, tokens or transactions, and we use the Hashio service that provides a JSON-RPC API interface, based on the open-source Hedera JSON-RPC-Relay, not only to use MetaMask in order to operate through the UI, but also to invoke any smart contract view function for getting cost-free queries.

Both ways, which are publicly available, are highly recommended by Hedera and, for sure, are the most commonly used by the Hedera community. But there are some other options that allow us to compare response times and reliability and, having alternative options to configure both services can help users to guarantee the operation of the applications and overcome moments where main services could be, for example, saturated or don’t respond properly.

Therefore, we want users to have the opportunity to configure other options in order to use both mirror node and Hedera JSON-RPC-Relay, so they could configure as many services as they want, in any moment and change the service they are using whenever they want.

Proposal

Although using these two services, mirror node and Hedera JSON-RPC-Relay, is done in a very similar way in the accelerator, since both uses a base URL which is not currently configurable so these URLs are static, we are going to explain our proposal separately: first the mirror node service and secondly the Hedera JSON-RPC-Relay.

Mirror node service

As we have just said, mirror node service uses a base URL, so we have a MirrorNodeAdapter class which is the only class that performs mirror node REST API calls.

The following diagram represents the accelerator architecture when calling the mirror node in order to get an account information or the list of stable coins an account can work with, through both the CLI and the UI.

The structure of the REST API is described into this swagger: Swagger UI

The REST API base URL currently depends only on the Hedera network, so this base URL is obtained whenever the user changes the account is working with:

private getMirrorNodeURL(environment: Environment): string {
        switch (environment) {
            case mainnet:
                return 'https://mainnet-public.mirrornode.hedera.com';
            case previewnet:
                return 'https://previewnet.mirrornode.hedera.com';
            case testnet:
                return 'https://testnet.mirrornode.hedera.com';
            case local:
                return 'http://127.0.0.1:5551';
            default:
                return '';
        }

So this is the current situation about the use of mirror node service, and now we are going to explain how we will allow users to configure other mirror node services and how theses users could use them.
    }

Manage mirror node services

The functionality that will be added to the accelerator in order to users can use other mirror node services would be different in CLI and the UI. Basically, through the CLI users would be able to manage mirror node services, which allows:

• To set up a new mirror node service.

• To remove an existing mirror node service.

• To view all existing mirror node services.

• To select an existing mirror node service to use in the application.

On the contrary, in the case of the UI, users would be able to use different services through configuration, but they would not be able to manage these services through the UI.

Next, we will explain how these functionalities would be implemented in each module.

CLI

A new option ‘Manage mirror node’ would be present into the main menu ‘Configuration’ option. Once the user has selected this option, it would select the Hedera network before choosing one of the previous managment options. So the path would be as follows:

The information about mirror node services would be stored in the CLI configuration file, so this services would be treated as in the case of the accounts. They would have a name, it belongs to a Hedera network and it would also have a base URL and a mechanism of authentication/identification as an option. For now, we would only introduce the possibility to pass an API key through a HTTP header as we will see later.

So a mirror node service in the CLI configuration file would be as follows:

mirrors:
  - name: Arkhia
    network: testnet
    baseUrl: http://pool.arkhia.io/hedera/testnet/api/v1/
    apiKey: 123456
    header: x-api-key
    selected: true

The selected property would indicate that this service is currently be used, so if the user changes the service to use the property value, for both the previously service and the new one, should be also changed, this property allows that the selected service would continue to be used next time the CLI is started by the user. A selected service would always exist for each Hedera network.

Although mirror services information would be stored in the CLI configuration file, it wouldn’t be possible to add new services during the configuration file creation process as in the case of the accounts, so when the file is created only the service provided by Hedera would be present for each network. So the only way to add new mirror node services to this file would be using the option described above.

mirrors:
  - name: Hedera
    network: testnet
    baseUrl: https://testnet.mirrornode.hedera.com/api/v1/
    selected: true
  - name: Hedera
    network: previewnet
    baseUrl: https://previewnet.mirrornode.hedera.com/api/v1/
    selected: true
  - name: Hedera
    network: mainnet
    baseUrl: https://mainnet-public.mirrornode.hedera.com/api/v1/
    selected: true

When the user selects an account to work with, the account id is always shown in the screen along with every question asked to the user. And the same occurs with the token id when the user selects the stable coin to work with.

We would also show the mirror node service name to be always in the screen.

Currently, and as explained before, mirror node service is statically configured in SDK MirrorNodeAdapter class, so there is no option for the CLI to indicate the SDK the mirror node service to use, although when the user selects an account to work with, the chooseAccount function of the CLI WizardService class sets the SDK network service, including mirror node service which is not used subsequently:

await Network.setNetwork(
      new SetNetworkRequest({
        environment: currentNetwork.name,
        consensusNodes:
          currentNetwork.consensusNodes.length > 0
            ? currentNetwork.name
            : undefined,
        mirrorNode:
          currentNetwork.mirrorNodeUrl.length > 0
            ? currentNetwork.mirrorNodeUrl
            : undefined,
      }),
    );

So the first thing to do before managing new mirror node services would be, given an initial configuration with a mirror node service in the CLI configuration file, to set the SDK network service with this selected mirror node service in order to be used in the SDK MirrorNodeAdapter class instead of using the getMirrorNodeURL function shown above. But the mirrorNode property of the SDK network service couldn't be just a string, but an object containing the name, the base URL and the API key and http header (these last two properties would be, of course, optional). This way, the MirrorNodeAdapter would use the network service to know which mirror node service to use.

Once the mirror node service depends on the CLI configuration file, let’s see above listed management options.

Set up mirror node
When selecting this option, the wizard would ask the user to introduce required data:

• Introduce the name

• Introduce the base url

• Do you need an API key? (Yes/No)

• If so:

• Introduce your http header name

• Introduce your api key

Note that the Hedera network is not requested to the user since this environment was previously selected before choosing this management option.

It couldn’t exist a mirror node service with the same name or base url as another existing service, so this should be checked when introducing this data.

Once the user has introduced all required data, a new mirror node service data structure, as shown above, is added to the configuration file and the user would be requested whether to add another service. If it wasn’t the case, the mirror node service management options would be shown to the user.

This options would be entirely managed by the CLI, in the SetConfigurationService class, so the SDK wouldn’t be involved.

Remove mirror node
When the user selects this option, a list with all mirror node services belonging to the previously selected Hedera network would be shown.

Before removing the selected service, the user will be asked to confirm the decision. Next, the list without the just removed service is shown again so the user could remove another service, or go back to the mirror node services management options.

The user wouldn’t be able to remove the currently used mirror node service.

As in the case of the set up, this option to remove mirror node services would be managed in the CLI SetConfigurationService class.

Display configured nodes
Through this option the user could see a list with all currently configured mirror node services for the previously selected Hedera network, showing all properties for each service: name, base url, the api key (showing only some characters, like we already do with account's private key), and the header used for sending this API key (these last two properties would only appear if the service has them).

This option would be also managed in the CLI SetConfigurationService class.

Change current node
This option allows the user to change the mirror node service is being currently used. Once the user has changed the service, this would start to work instead of the previous one, and the mirror node service name shown through the prompt we previously refer would also change.

Changing the mirror node service to use would cause:

• The ‘selected’ property of the service in the CLI configuration file has to be change from true to false for the service that is no longer used, and from false to true for the new selected service.

• The mirrorNode property of the SDK network service, which is the property that reports the MirrorNodeAdapter class what mirror node service to use, has to be updated, so the SetConfigurationService that manage this option should update this property.

UI

Unlike the CLI, the UI does not allow the user to manage different mirror node services, so the service the user wants to use in the UI would be allways configured in the UI .env file.

This .env file is currently used to store certain configuration needed by the UI in order to work properly. Below is the content of a UI .env file:

REACT_APP_LOG_LEVEL=TRACE
REACT_APP_FACTORIES='[{"Environment":"testnet","STABLE_COIN_FACTORY_ADDRESS":"0.0.4041403"}]'
GENERATE_SOURCEMAP=true

The target of this document is not to describe a UI .env file, but it is important to understand that this is an environment file, in which the user sets some environment variables, which is unmodifiable through the UI, so if the user wants to start the UI with a different value of any environment variable currently set in the file, the user should change manually the value and, later or, starts the UI again.

The file should not contain more than a mirror node service for each Hedera network, and this is the service which would be used when the UI starts.

This is an example of how these services would be configured in the .env file:

MIRROR_NODE_SERVICES='[{"Environment":"testnet","BASE_URL":"http://pool.arkhia.io/hedera/testnet/api/v1/", "API_KEY": "123456", "HEADER": "x-api-key"}]'

SDK

In addition to change the mirrorNode property of the network service so that it doesn't continue to be a string, but an object containing the name, the base URL and the API key and http header, so the SetNetworkRequest should also be changed, the MirrorNodeAdapter class would also change for this to use the mirror node service whose data are stored into the mirrorNode property of the network service, and also to invoke the service in a different way if needed. Regarding this last, we would only manage, for the moment, services which need to pass an API key through a http header, in addition to services which don’t need any authentication/identification method.

This is an example of how Arkhia’s mirror node service could be consumed:

const axios = require('axios');

const baseUrl = <ARKHIA_REST_API_URL_HERE>;
const headerKey = <ARKHIA_API_KEY_HERE>;
const config = {
  method: 'get',
  url: baseUrl,
  headers: {
    'x-api-key': headerKey
  }
};

const standardApiRequests = {
  transactions: transactions,
  contracts: contracts,
  tokens: tokens,
  accounts: accounts,
  balances: balances,
  blocks: blocks,
  schedules: schedules,
  tokens: tokens,
};
const getMirrorNodeRestData = async (apiRequest) => {
  config.url = ${baseUrl}/${apiRequest};
  await axios(config)
    .then(function (response) {
      if (!response.data) {
        console.error(Something went wrong);
        return;
      }
      console.info(response.data);
      console.info(Retrieved ${response.data[apiRequest].length} results);
    })
    .catch(function (error) {
      console.log(error);
  });
}
getMirrorNodeRestData(standardApiRequests.tokens);
process.on('uncaughtException', (error) => {
  console.log(error)
});
process.on('unhandledRejection', (error) => {
  console.log(error)
});

Hedera JSON-RPC-Relay

As in the case of mirror node service, the Hashio service based on the open-source Hedera JSON-RPC-Relay is used in the accelerator, in this case, in two classes: RPCQueryAdapter and RPCTransactionAdapter. The first class uses the service to call any stable coin smart contract view function regardless the wallet used by the user, while the second class uses the service to call any stable coin smart contract function, which involves a change of state, when using MetaMask.

The following diagram illustrates the accelerator architecture when calling any StableCoin input port operation, so all commands are executed through the RPCTransactionAdapter when using MetaMask while all queries are executed through the RPCQueryAdapter regardless the wallet.

Both classes includes a init function where ethers library is used to build a JsonRpcProvider depending on the URL which depends, in turn, on the Hedera network

This is the init function of the RPCTransactionAdapter:

async init(debug = false): Promise<string> {
        this.provider = new ethers.providers.JsonRpcProvider(
             `https://${this.networkService.environment.toString()}.hashio.io/api`,
        );
        !debug && (await this.connectMetamask(false));
        const eventData = {
            initData: {
                account: this.account,
                pairing: '',
                topic: '',
            },
            wallet: SupportedWallets.METAMASK,
        };
        this.eventService.emit(WalletEvents.walletInit, eventData);
        return this.networkService.environment;
    }

This is the init function of the RPCQueryAdapter:

async init(customUrl?: string): Promise<string> {
        this.provider = new ethers.providers.JsonRpcProvider(
             `https://${this.networkService.environment.toString()}.hashio.io/api`,
        );
        return this.networkService.environment;

**Manage Hedera JSON-RPC-Relay**
As in the case of the mirror node service, management of Hedera JSON-RPC-Relay services would be different in the CLI and the UI, allowing CLI users:

- To set up a new JSON-RPC-Relay service.

- To remove an existing JSON-RPC-Relay service.

- To view all existing JSON-RPC-Relay services.

- To select an existing JSON-RPC-Relay service to use in the application.

While UI users would only be able to set the JSON-RPC-Relay service to use in the environment configuration file.

    }

CLI
Very similar to what happens with the mirror node services, a new option ‘Manage JSON-RPC-Relay’ would be present into the main menu ‘Configuration’ option so, when selected, available Hedera networks would be shown for the user to select one before choosing a management option. So the path would be as follows:

The information about JSON-RPC-Relay services would be also stored in the CLI configuration file, and each service would also have the same properties as mirror node services: a name, the Hedera network, a URL and a mechanism of authentication/identification as an option, and, in the same way, we would only introduce the possibility to pass an API key through a HTTP header right now.

So a JSON-RPC-Relay service in the CLI configuration file would be as follows:

mirrors:
  - name: Hashio
    network: testnet
    url: https://testnet.hashio.io/api
    selected: true

The selected property would have the same behavior as previously explained in the case of mirror node services and, in the same way, it wouldn’t be possible to add new JSON-RPC-Relay services during the configuration file creation process, that would have the following initial configuration regarding JSON-RPC-Relay services:

json-rpc-relay:
  - name: Hashio
    network: testnet
    baseUrl: https://testnet.hashio.io/api
    selected: true
  - name: Hashio
    network: previewnet
    baseUrl: https://previewnet.hashio.io/api
    selected: true
  - name: Hashio
    network: mainnet
    baseUrl: https://mainnet.hashio.io/api
    selected: true

The currently used service would be shown in the CLI prompt for the user to know in every moment the JSON-RPC-Relay service that is being used.

JSON-RPC-Relay service is currently used in SDK RPCQueryAdapter and RPCTransactionAdapter classes statically, so we have to get the selected JSON-RPC-Relay service from the CLI configuration file to these two SDK classes, and we would do it as we explained in the case of mirror node services: initializing the network service using the SetNetworkRequest object where rpcNode property will be change to be an object containing the name, the URL and, optionally, the API key and http header, so the SDK network service would always contain the JSON-RPC-Relay service to be used in above classes.

Set up JSON-RPC-Relay
When selecting this option, the wizard would ask the user to introduce required data:

• Introduce the name

• Introduce the url

• Do you need an API key? (Yes/No)

• If so:

• Introduce your http header name

• Introduce your api key

The Hedera network is not requested to the user since this environment was previously selected before choosing this management option.

It couldn’t exist a JSON-RPC-Relay service with the same name or url as another existing service.

Once the user has introduced all required data, a new JSON-RPC-Relay service data structure, as shown above, is added to the configuration file and the user would be requested whether to add another service. If it wasn’t the case, the JSON-RPC-Relay service management options would be shown to the user.

This options would be entirely managed by the CLI, in the SetConfigurationService class, so the SDK wouldn’t be involved.

Remove JSON-RPC-Relay
When the user selects this option, a list with all JSON-RPC-Relay services belonging to the previously selected Hedera network would be shown.

Before removing the selected service, the user will be asked to confirm the decision. Next, the list without the just removed service is shown again so the user could remove another service, or go back to the JSON-RPC-Relay services management options.

The user wouldn’t be able to remove the currently used JSON-RPC-Relay service.

As in the case of the set up, this option to remove JSON-RPC-Relay services would be managed in the CLI SetConfigurationService class.

Display JSON-RPC-Relays
Through this option the user could see a list with all currently configured JSON-RPC-Relay services for the previously selected Hedera network, showing all properties for each service: name, url, the api key, and the header used for sending this API key (these last two properties would only appear if the service has them).

This option would be also managed in the CLI SetConfigurationService class.

Change current JSON-RPC-Relay
This option allows the user to change the JSON-RPC-Relay service is being currently used. Once the user has changed the service, this would start to work instead of the previous one, and the JSON-RPC-Relay service name shown through the prompt we previously refer would also change.

Changing the JSON-RPC-Relay service to use would cause:

• The ‘selected’ property of the service in the CLI configuration file has to be change from true to false for the service that is no longer used, and from false to true for the new selected service.

• The rpcNode property of the SDK network service, which is the property that reports both the RPCQueryAdapter and RPCTransactionAdapter classes what JSON-RPC-Relay service to use, has to be updated, so the SetConfigurationService that manage this option should update this property.

UI
In the same way as in the case of mirror node services, JSON-RPC-Relay services are managed through the UI .env file. So the user should set the JSON-RPC-Relay service to be used, being able to do it for each Hedera network. Below is an example:

JSON-RPC-RELAY_SERVICES='[{"Environment":"testnet","URL":"https://testnet.hashio.io/api"}]'

In this case, we have used the default service, Hashio, which does not require any API key. If the user wants to use another service, the .env file should be edited and the UI should also be restarted. There will be no way to change this .env file through the UI in order to use another service.

SDK
As it happened with mirrorNode property of SetNetworkRequest in mirror node services, in the case of JSON-RPC-Relay, rpcNode property has to be also changed to an object containing the name, the URL and the API key and http header, instead of a string and, in the same way, RPCQueryAdapter and RPCTransactionAdapter classes should use the information stored by the rpcNode property to use a certain JSON-RPC-Relay service, previously selected by the user, instead of having the URL of the service statically written in the method that uses it. Moreover, the way we currently consume the only one existing service should be extended in order to be able to consume services which require any type of authentication/identification method, although, for now, we would only implement a mechanism to consume services which don’t need any method of authentication/identification or need an API key provided by a http header.

Conclusion

After the above study we have reached the following conclusions:

• When a user firstly starts the CLI, the Hedera mirror node service will be configured in the CLI configuration file to be used.

• A user can add as many mirror node services as desired through the CLI.

• Only mirror node services that don’t need any authentication/identification method or need an API key provided by a http header can be used both through the CLI and the UI.

• A user can select, through the CLI, any mirror node service to be used between those previously configured.

• Once the user has selected a mirror node service through the CLI, this would be used until the user selects another one, although stops the CLI and starts again.

• A user can remove any mirror node service previously configured but the Hedera mirror service through the CLI.

• A user can see all configured mirror node services in the CLI.

• When a user starts the UI, the mirror node service selected by the user will be configured in the UI .env file.

• There is no way to manage mirror node services through the UI.

• When a user firstly starts the CLI, the JSON-RPC-Relay mirror node service will be configured in the CLI configuration file to be used.

• A user can add as many JSON-RPC-Relay services as desired through the CLI.

• Only JSON-RPC-Relay services that don’t need any authentication/identification method or need an API key provided by a http header can be used both through the CLI and the UI.

• A user can select, through the CLI, any JSON-RPC-Relay service to be used between those previously configured.

• Once the user has selected a JSON-RPC-Relay service through the CLI, this would be used until the user selects another one, although stops the CLI and starts again.

• A user can remove any JSON-RPC-Relay service previously configured but the Hedera mirror service through the CLI.

• A user can see all configured JSON-RPC-Relay services in the CLI.

• When a user starts the UI, the JSON-RPC-Relay service selected by the user will be configured in the UI .env file.

• There is no way to manage JSON-RPC-Relay services through the UI.