External App Tutorial

This tutorial will focus on how to create an app on top of OneLedger blockchain protocol. This app will be referred as "external app" in this tutorial.

Introduction

Blockchain is a Distributed Ledger Technology (DLT), that abstracts away trust among 2 or more parties by introducing a technology that is unhackable, immutable and trusted. Any use case that involves trust between 2 or more parties, is where blockchain can be applied, and so it’s thought of regularly in Supply Chain and Logistics, Finance, Insurance, Litigation, Health Care and many others.

Unlike Bitcoin with Proof of Work (PoW), OneLedger blockchain network is based on Proof of Stake (PoS), which will increase transactions per second (TPS) and create sidechains for higher throughput.

An external app will contains 4 layers in total to utilize the OneLedger blockchain network.

• Data Layer: store varies data that will be used in the app
• Transaction Layer: any actions in the blockchain system will be executed as a transaction, and this transaction will be processed multiple times in the blockchain network to let all the blockchain node reach a consensus state.
• RPC Layer: in order to interact with blockchain network, for example, trigger the transactions in transaction layer, or query data from data layer, we need RPC layer to be able to make RPC request to block chain network.
• Block Function Layer: block function provides the ability to run transactions without manually triggering it by RPC request, it will allow automatic or routine based transaction to be executed.

💡Simply put, an external app will use different transactions to perform corresponding actions that deal with data in store(s) to achieve certain functionalities. We will also use block function to run transactions automatically in the block ender and provide rpc functions to support querying result from outside of blockchain.

0. Pre-Start

System Requirements

Ensure your system meets the following requirements:

Operating System must be one of the following:

• macOS
  • Ensure Xcode Developer Tools is installed(only command line version is needed)
• A Debian-based Linux distribution
  • Ubuntu is recommended for the smoothest install, otherwise you will have to set up your distribution for installing PPAs
• Go version 1.11 or higher
• git

Configurations

Setup Environment Variables

Use which go to check were Golang is installed

In your .bashrc or .zshrc file, set up environment variables as below:

export GOPATH="folder for all your Golang project"
export PATH=$PATH:$GOPATH/bin
export GO111MODULE=auto
export GOROOT="folder where Golang is installed"
export OLROOT="$GOPATH/src/github.com/Oneledger"
export OLTEST="$OLROOT/protocol/node/scripts"
export OLSCRIPT="$OLROOT/protocol/node/scripts"
export OLSETUP="$OLROOT/protocol/node/setup"
export OLDATA="$GOPATH/test"

and source it(choose .bashrc or .zshrc that you are using)

source ~/.bashrc

Install ClevelDB

For Ubuntu:

sudo apt-get update
sudo apt install build-essential

sudo apt-get install libsnappy-dev

wget https://github.com/google/leveldb/archive/v1.20.tar.gz && \
  tar -zxvf v1.20.tar.gz && \
  cd leveldb-1.20/ && \
  make && \
  sudo cp -r out-static/lib* out-shared/lib* /usr/local/lib/ && \
  cd include/ && \
  sudo cp -r leveldb /usr/local/include/ && \
  sudo ldconfig && \
  rm -f v1.20.tar.gz

For MacOS:

brew install leveldb

Clone the repo

cd $OLROOT
git clone github.com/Oneledger/protocol

Install the required dependencies

cd ./protocol
git checkout develop
git checkout -b name-of-your-branch
make update
make install_c

If no error occurs, you are good to go.

Choose your IDE

You can use Goland or VScode. For Goland, it comes with a 30-day trial.

Steps to create an external app

It takes 8 major steps to create an external app:

• Create your external app folder
• Create error codes package
• Create data package
• Create action(transaction) package
• Create block function package
• Create RPC package
• Register the app into OneLedger Blockchain
• Create integration test using python or any script you like

This tutorial will use an example app called "bid" to show the steps you need to take to build an external app.

This example app will provide the functionality of bidding on OneLedger Naming Service.

OneLedger Naming Service: The Domains created on the OneLedger Network can be tied directly to an Account (OneLedger Address) and can be associated to a website URLs, so that Businesses can connect their web properties to a OneLedger Account and send crypto-payments to any online business.

1. Create your external app folder

Inside protocol folder, there is a folder named external_apps, everything need to be done will be in this folder.

The structure inside external_apps is shown in the below, which includes a common utility folder, an example project folder and an initialization file.

.
├── bid(example project folder)  
├── common(common utility folder)
└── init.go

The first step for external app is to create your own external app folder, just like bid as the example.

2. Create error codes package

Create error package and error codes file to define all the error codes used in your external app, you will be provided a range of error codes that are pre-allocated to this external app to avoid conflict.

All external app error codes will be a six digit number starts with 99, and for each external app, there are 100 error codes available. For example, 990100 to 990199.

All packages inside your app folder should follow the naming convention of app name + underscore + package name, such as bid_error.

.
├── bid(example project folder)  
│   └── bid_error
│       └── codes.go
├── common(common utility folder)
└── init.go

Later on you will be adding error codes into this file as below. (with your dedicated error codes)

const (
	BidErrInvalidBidConvId = 990001
	BidErrInvalidAsset = 990002
	...
)

3. Create data package

Data package takes care of the functionality to store, set, get and iterate data related to your external app. There will be data structs to represent single entry of data object, and there will be data stores to hold the data. You can use multiple data structs or stores if needed.

Data in all stores is saved in a non-relational key-value database universally, with different prefix in the key we can differentiate data entries in different stores.

Create data folder as below

.
├── bid(example project folder)  
│   ├── bid_data
│   └── bid_error
├── common(common utility folder)
└── init.go

Define basic data types that will be used in the external app

Create a new file called types.go in your data package.

.
├── bid(example project folder)  
│   ├── bid_data
│   │   └── types.go
│   └── bid_error
├── common(common utility folder)
└── init.go

This file will store all the basic types you will need in the app. This way they can be easily maintained in the future.

type (
	BidConvId            string
	BidConvState         int
	BidAssetType         int
	BidConvStatus        bool
	BidOfferStatus       int
	BidOfferType         int
	BidOfferAmountStatus int
	BidDecision          bool
)

Define data layer errors

Create a new file called errors.go

.
├── bid(example project folder)  
│   ├── bid_data
│   │   ├── errors.go
│   │   └── types.go
│   └── bid_error
├── common(common utility folder)
└── init.go

Inside this file, we will define errors that potentially can be triggered in data layer.

var (
	ErrInvalidBidConvId                 = codes.ProtocolError{bid_error.BidErrInvalidBidConvId, "invalid bid conversation id"}
	ErrInvalidAsset                     = codes.ProtocolError{bid_error.BidErrInvalidAsset, "invalid bid asset"}
	...
)

Here we define errors using the error code from bid_error package in last step, and combine it with an error message.

Later on we can add more errors to this file.

Define constants and other components in init.go file

.
├── bid(example project folder)  
│   ├── bid_data
│   │   ├── errors.go
│   │   ├── init.go
│   │   └── types.go
│   └── bid_error
├── common(common utility folder)
└── init.go

We will define some constants as below

const (
	//Bid States
	BidStateInvalid   BidConvState = 0xEE
	BidStateActive    BidConvState = 0x01
	BidStateSucceed   BidConvState = 0x02
	...
)

Since in the bid example app, we have two stores, and we define a master stores to hold both of them.

type BidMasterStore struct {
	BidConv  *BidConvStore
	BidOffer *BidOfferStore
}

var _ data.ExtStore = &BidMasterStore{}

func (bm *BidMasterStore) WithState(state *storage.State) data.ExtStore {
	bm.BidConv.WithState(state)
	bm.BidOffer.WithState(state)
	return bm
}

We need to implement WithState method as above to impelent data.ExtStore interface for our use. And we can use var _ data.ExtStore = &BidMasterStore{} to check if the implementation is successful.

todo put the paragraph link to ## Define data stores that will be used in the external app

This part can be done in ## Define data stores that will be used in the external app if one store is enough for your app. If it's done in there, remember to implement WithState method as above.

As default, init function is not needed in this part, but since the bid asset is a generic concept and allow any asset implement the BidAsset interface, here we need to register the concrete type of the asset in the init function. And later we will use different serializer to perform (de)serialization.

Define structs that will be used in the external app

This is the structs that you want to utilize to represent single entry of data object in the external app.

For example, this is a note if your app is a notebook, a product if your app is a product management system.

.
├── bid(example project folder)  
│   ├── bid_data
│   │   ├── errors.go
│   │   ├── init.go
│   │   ├── bid_conversation.go
│   │   └── types.go
│   └── bid_error
├── common(common utility folder)
└── init.go

In bid example app, we have two different data structs, bid conversation and bid offer. Let's use bid conversation as an example.

This bid_conversation.go will at least contains a struct definition and a constructor as below.

⚠️ Remember to use proper json tag so that it can be successfully serialized and deserialized. The naming convention we use for json tag is camel style.

type BidConv struct {
	BidConvId   BidConvId    `json:"bidId"`
	AssetOwner  keys.Address `json:"assetOwner"`
	AssetName   string       `json:"assetName"`
	AssetType   BidAssetType `json:"assetType"`
	Bidder      keys.Address `json:"bidder"`
	DeadlineUTC int64        `json:"deadlineUtc"`
}

func NewBidConv(owner keys.Address, assetName string, assetType BidAssetType, bidder keys.Address, deadline int64, height int64) *BidConv {
	return &BidConv{
		BidConvId:   generateBidConvID(owner.String()+assetName+bidder.String(), height),
		AssetOwner:  owner,
		AssetName:   assetName,
		AssetType:   assetType,
		Bidder:      bidder,
		DeadlineUTC: deadline,
	}
}

In the bid conversation, we have the bid conversation id to make each conversation unique.

Since this app is to let people bidding on an asset that belongs to an owner, we have owner and bidder here. The data type is OneLedger address.

🛠If your data object is an entity that designed to be owned by or traded/exchanged among users, you can use keys.Address as data type for that field. This represents an address on the OneLedger blockchain network. The String() method for address will return the string value for that address.

We also have bid asset name and type that will be used in the validation and exchange process. For example, this asset needs to be under owner's address, and it needs to be valid in the period of bidding.

The "height" here represents the block height, which is used as another factor to make this id unique.

Define data stores that will be used in the external app

This the storage struct that you want to store you data entries from above step.

In bid example app, we also have two different data store structs, bid conversation store and bid offer store. Let's use bid conversation store as an example.

.
├── bid(example project folder)  
│   ├── bid_data
│   │   ├── errors.go
│   │   ├── init.go
│   │   ├── bid_conversation.go
│   │   ├── bid_conversation_store.go
│   │   └── types.go
│   └── bid_error
├── common(common utility folder)
└── init.go

Inside bid_conversation_store.go, we first define our data store

type BidConvStore struct {
	state *storage.State
	szlr  serialize.Serializer

	prefix []byte //Current Store Prefix

	prefixActive        []byte
	prefixSucceed       []byte
	prefixRejected      []byte
	prefixCancelled     []byte
	prefixExpired       []byte
	prefixExpiredFailed []byte
}

todo explain state

• state is where this store is saved, as mentioned before, data in every store is saved in a key-value database, and this database exists in the state of block chain network. We need this state to put our data stores.
• szlr is the serializer we used to handle (de)serialization
• prefix is a list of prefix will be used to store data entries that you can choose per external app. The reason to separate data into different prefixs is to make the getting and iterating process more intuitive.
  • You can define your own prefix type and value as you want.

And we add the constructor for this store:

func NewBidConvStore(prefixActive string, prefixSucceed string, prefixCancelled string, prefixExpired string, prefixRejected string, state *storage.State) *BidConvStore {
	return &BidConvStore{
		state:           state,
		szlr:            serialize.GetSerializer(serialize.LOCAL),
		prefix:          []byte(prefixActive),
		prefixActive:    []byte(prefixActive),
		prefixSucceed:   []byte(prefixSucceed),
		prefixCancelled: []byte(prefixCancelled),
		prefixExpired:   []byte(prefixExpired),
		prefixRejected:  []byte(prefixRejected),
	}
}

For serializer, normally we can just use serialize.PERSISTENT, which will do the (de)serialization using JSON. As mentioned above, here serialize.LOCAL is used to support generic bid asset in this example app.

Method GetState and WithState are used to correctly pass/get the state to/from the store.

func (bcs *BidConvStore) GetState() *storage.State {
	return bcs.state
}

func (bcs *BidConvStore) WithState(state *storage.State) *BidConvStore {
	bcs.state = state
	return bcs
}

Method WithPrefixType behaves like a filter, that will return the store with only selected prefix.

func (bcs *BidConvStore) WithPrefixType(prefixType BidConvState) *BidConvStore {
	switch prefixType {
	case BidStateActive:
		bcs.prefix = bcs.prefixActive
	case BidStateSucceed:
		bcs.prefix = bcs.prefixSucceed
	case BidStateRejected:
		bcs.prefix = bcs.prefixRejected
	case BidStateCancelled:
		bcs.prefix = bcs.prefixCancelled
	case BidStateExpired:
		bcs.prefix = bcs.prefixExpired

	}
	return bcs
}

Method Set is to set data into the store

func (bcs *BidConvStore) Set(bid *BidConv) error {
	prefixed := append(bcs.prefix, bid.BidConvId...)
	data, err := bcs.szlr.Serialize(bid)
	if err != nil {
		return ErrFailedInSerialization.Wrap(err)
	}

	err = bcs.state.Set(prefixed, data)

	return ErrSettingRecord.Wrap(err)
}

Here we first append the bid conversation id as a part of key to the prefixed key, and serialize the bid conversation into the store. You can design your own key pattern.

When we design the key pattern, the most important point is to create every data entry with a unique key. In this example, this is achieved by letting bid conversation id be the hash of multiple factors.

Method Get is to get data from the store

func (bcs *BidConvStore) Get(bidId BidConvId) (*BidConv, error) {
	bid := &BidConv{}
	prefixed := append(bcs.prefix, bidId...)
	data, err := bcs.state.Get(prefixed)
	if err != nil {
		return nil, ErrGettingRecord.Wrap(err)
	}
	err = bcs.szlr.Deserialize(data, bid)
	if err != nil {
		return nil, ErrFailedInDeserialization.Wrap(err)
	}

	return bid, nil
}

First we create empty bid conversation object and construct our key, and get the corresponding data from the state. After this, we finally deserialize data into our object.

Method Exist will be able to tell if a data entried can be found in the data store.

func (bcs *BidConvStore) Exists(key BidConvId) bool {
	active := append(bcs.prefixActive, key...)
	succeed := append(bcs.prefixSucceed, key...)
	rejected := append(bcs.prefixRejected, key...)
	cancelled := append(bcs.prefixCancelled, key...)
	expired := append(bcs.prefixExpired, key...)
	expiredFailed := append(bcs.prefixExpiredFailed, key...)
	return bcs.state.Exists(active) || bcs.state.Exists(succeed) || bcs.state.Exists(rejected) || bcs.state.Exists(cancelled) || bcs.state.Exists(expired) || bcs.state.Exists(expiredFailed)
}

Here we check if it exists in stores with any our pre-defined prefixs.

Method Delete is to delete data from the store

func (bcs *BidConvStore) Delete(key BidConvId) (bool, error) {
	prefixed := append(bcs.prefix, key...)
	res, err := bcs.state.Delete(prefixed)
	if err != nil {
		return false, ErrDeletingRecord.Wrap(err)
	}
	return res, err
}

Method Iterate is to iterate all or part of data in the store to get what we want.

⚠️ Iterate method can only be used in rpc package, not action package. Anything calls this method cannot be a part of action package.

func (bcs *BidConvStore) Iterate(fn func(id BidConvId, bid *BidConv) bool) (stopped bool) {
	return bcs.state.IterateRange(
		bcs.prefix,
		storage.Rangefix(string(bcs.prefix)),
		true,
		func(key, value []byte) bool {
			id := BidConvId(key)
			bid := &BidConv{}
			err := bcs.szlr.Deserialize(value, bid)
			if err != nil {
				return true
			}
			return fn(id, bid)
		},
	)
}

In this method, we will return the result from IterateRange function in the state, we first pass the prefix of the store we want to iterate, and generate the range prefix from it, using assending direction when iterating.

And we use a function to get info from the key and value.

At the end we pass a customized function fn into the Iterate method, after we successfully get and deserialize the key and value for each data entry in the data store, we will call this fn function to do the logic we want.

As an example, we call Iterate method in FilterBidConvs method:

func (bcs *BidConvStore) FilterBidConvs(bidState BidConvState, owner keys.Address, assetName string, assetType BidAssetType, bidder keys.Address) []BidConv {
	prefix := bcs.prefix
	defer func() { bcs.prefix = prefix }()

	bidConvs := make([]BidConv, 0)
	bcs.WithPrefixType(bidState).Iterate(func(id BidConvId, bidConv *BidConv) bool {
		if len(owner) != 0 && !bidConv.AssetOwner.Equal(owner) {
			return false
		}
		if len(bidder) != 0 && !bidConv.Bidder.Equal(bidder) {
			return false
		}
		if bidConv.AssetType != assetType {
			return false
		}
		if len(assetName) != 0 && !cmp.Equal(assetName, bidConv.AssetName) {
			return false
		}

		bidConvs = append(bidConvs, *bidConv)
		return false
	})
	return bidConvs
}

todo figure out this why we need this defer stuff In this method, we pass a function that will check it the given query info is a match of the data entry that being iterated, and here return false means continue next round of iteration.

If the key includes multiple infomation, we can use the iterate method in bid_offer_store.go as another example.

func (bos *BidOfferStore) iterate(fn func(bidConvId BidConvId, offerType BidOfferType, offerTime int64, offer BidOffer) bool) bool {
	return bos.State.IterateRange(
		assembleInactiveOfferPrefix(bos.prefix),
		storage.Rangefix(string(assembleInactiveOfferPrefix(bos.prefix))),
		true,
		func(key, value []byte) bool {
			offer := &BidOffer{}
			err := serialize.GetSerializer(serialize.PERSISTENT).Deserialize(value, offer)
			if err != nil {
				return true
			}
			arr := strings.Split(string(key), storage.DB_PREFIX)
			// key example: bidOffer_INACTIVE_bidConvId_offerType_offerTime
			bidConvId := arr[2]
			offerType, err := strconv.Atoi(arr[3])
			if err != nil {
				fmt.Println("Error Parsing Offer Type", err)
				return true
			}
			offerTime, err := strconv.ParseInt(arr[len(arr)-1], 10, 64)
			if err != nil {
				fmt.Println("Error Parsing Offer Time", err)
				return true
			}
			return fn(BidConvId(bidConvId), BidOfferType(offerType), int64(offerTime), *offer)
		},
	)
}

In bid offer section, we use offer status(ACTIVE/INACTIVE), related bid conversation id, offer type and offer time as key to make it unique.

And in the function where we get info from the key, we split the key into different components, which is separated by underscore.

Create unit test for data stores

Creating unit test is recommanded for data stores, it's better to fix the problem at data layer before going forward.

.
├── bid(example project folder)  
│   ├── bid_data
│   │   ├── errors.go
│   │   ├── init.go
│   │   ├── bid_conversation.go
│   │   ├── bid_conversation_store.go
│   │   ├── bid_conversation_store_test.go
│   │   └── types.go
│   └── bid_error
├── common(common utility folder)
└── init.go

In the bid_conversation_store_test.go file, we need to setup the testing environment in init function.

We first define some variables and constants that will be used in the test

const (
	numPrivateKeys = 10
	numBids        = 10
	testDeadline   = 1596740920
	height         = 111
)

var (
	addrList []keys.Address
	bidConvs []*BidConv

	bidConvStore *BidConvStore

	assetNames []string
)

And next is to create an init function to prepare the test

func init() {
	fmt.Println("####### TESTING BID CONV STORE #######")

	//Generate key pairs
	for i := 0; i < numPrivateKeys; i++ {
		pub, _, _ := keys.NewKeyPairFromTendermint()
		h, _ := pub.GetHandler()
		addrList = append(addrList, h.Address())
	}

	//Create new bid conversations
	for i := 0; i < numBids; i++ {
		j := i / 2                  //owner address list ranges from 0 - 4
		k := numPrivateKeys - 1 - j //bidder address list ranges from 9 - 5

		owner := addrList[j]
		assetNames = append(assetNames, "test"+strconv.Itoa(i)+".ol")
		bidder := addrList[k]

		bidConvs = append(bidConvs, NewBidConv(owner, assetNames[i], BidAssetOns,
			bidder, testDeadline, height))
	}

	//Create Test DB
	newDB := db.NewDB("test", db.MemDBBackend, "")
	cs := storage.NewState(storage.NewChainState("chainstate", newDB))

	//Create bid conversation store
	bidConvStore = NewBidConvStore("p_active", "p_succeed", "p_cancelled", "p_expired", "p_rejected", cs)
}

In the init function, we first create some addresses from key pairs, and create some bid conversations.

Next is to create a test db, as mentioned before, data in every store is saved in one key-value database universally, here we create the database and from which we get a chainstate.

Then we use the chainstate to create our store.

After the environment is set, we call our functions in the test to see if we can get expected results.

func TestBidConvStore_Set(t *testing.T) {
	err := bidConvStore.Set(bidConvs[0])
	assert.Equal(t, nil, err)

	bidConv, err := bidConvStore.Get(bidConvs[0].BidConvId)
	assert.Equal(t, nil, err)

	assert.Equal(t, bidConv.BidConvId, bidConvs[0].BidConvId)
}

For example, here we set a data entry and try to get it.

And for iterate test, we need one more step

func TestBidConvStore_Iterate(t *testing.T) {
	for _, val := range bidConvs {
		_ = bidConvStore.Set(val)
	}
	bidConvStore.state.Commit()

	bidConvCount := 0
	bidConvStore.Iterate(func(id BidConvId, bidConv *BidConv) bool {
		bidConvCount++
		return false
	})

	assert.Equal(t, numBids, bidConvCount)
}

Before we try to iterate the store, we need to commit the current state using bidConvStore.state.Commit(), this is the reason why we shouldn't use iterate method in action layer, if we do so, we may not get all the data we want.

4. Create action package

Action pacakge handles all the transactions in the app. In the blockchain network, every action is achieved by a transaction, such as sending tokens to another address, creating a domain, adding data to stores...

When a fullnode in the blockchain network receives a transaction, it will first do basic validation, then it will be passed into the network. Since blockchain network is a decentralized system, the consensus established among all the nodes is essential.

After this transaction is passed into the network, it will be excuted multiple times in different nodes to achieve the consensus. That means EVERYTHING in the transaction should be deterministic, everything should follow the same step for one transaction in multiple nodes. No random number, no random sequence and so on.

Create action folder as below

.
├── bid(example project folder)  
│   ├── bid_action
│   ├── bid_data
│   └── bid_error
├── common(common utility folder)
└── init.go

Create initialization file in action package

.
├── bid(example project folder)  
│   ├── bid_action
│   │   └── init.go
│   ├── bid_data
│   └── bid_error
├── common(common utility folder)
└── init.go

todo confirm the transaction type code Inside this init.go, first we will define some constants as the transaction type. You will be provided with a range of three-digit transaction type codes in hex. For example 0x910 to 0x91F.

const (
	//Bid
	BID_CREATE          action.Type = 0x901
	BID_CONTER_OFFER    action.Type = 0x902
	BID_CANCEL          action.Type = 0x903
	BID_BIDDER_DECISION action.Type = 0x904
	BID_EXPIRE          action.Type = 0x905
	BID_OWNER_DECISION  action.Type = 0x906
)

5. Create block function package

6. Create RPC package

7. Register the app into OneLedger Blockchain

8. Create integration test using python or any script you like