|
1 |
| -# ORM |
2 |
| - |
3 |
| -The Cosmos SDK ORM is a state management library that provides a rich, but opinionated set of tools for managing a |
4 |
| -module's state. It provides support for: |
5 |
| - |
6 |
| -* type safe management of state |
7 |
| -* multipart keys |
8 |
| -* secondary indexes |
9 |
| -* unique indexes |
10 |
| -* easy prefix and range queries |
11 |
| -* automatic genesis import/export |
12 |
| -* automatic query services for clients, including support for light client proofs (still in development) |
13 |
| -* indexing state data in external databases (still in development) |
14 |
| - |
15 |
| -## Design and Philosophy |
16 |
| - |
17 |
| -The ORM's data model is inspired by the relational data model found in SQL databases. The core abstraction is a table |
18 |
| -with a primary key and optional secondary indexes. |
19 |
| - |
20 |
| -Because the Cosmos SDK uses protobuf as its encoding layer, ORM tables are defined directly in .proto files using |
21 |
| -protobuf options. Each table is defined by a single protobuf `message` type and a schema of multiple tables is |
22 |
| -represented by a single .proto file. |
23 |
| - |
24 |
| -Table structure is specified in the same file where messages are defined in order to make it easy to focus on better |
25 |
| -design of the state layer. Because blockchain state layout is part of the public API for clients (TODO: link to docs on |
26 |
| -light client proofs), it is important to think about the state layout as being part of the public API of a module. |
27 |
| -Changing the state layout actually breaks clients, so it is ideal to think through it carefully up front and to aim for |
28 |
| -a design that will eliminate or minimize breaking changes down the road. Also, good design of state enables building |
29 |
| -more performant and sophisticated applications. Providing users with a set of tools inspired by relational databases |
30 |
| -which have a long history of database design best practices and allowing schema to be specified declaratively in a |
31 |
| -single place are design choices the ORM makes to enable better design and more durable APIs. |
32 |
| - |
33 |
| -Also, by only supporting the table abstraction as opposed to key-value pair maps, it is easy to add to new |
34 |
| -columns/fields to any data structure without causing a breaking change and the data structures can easily be indexed in |
35 |
| -any off-the-shelf SQL database for more sophisticated queries. |
36 |
| - |
37 |
| -The encoding of fields in keys is designed to support ordered iteration for all protobuf primitive field types |
38 |
| -except for `bytes` as well as the well-known types `google.protobuf.Timestamp` and `google.protobuf.Duration`. Encodings |
39 |
| -are optimized for storage space when it makes sense (see the documentation in `cosmos/orm/v1/orm.proto` for more details) |
40 |
| -and table rows do not use extra storage space to store key fields in the value. |
41 |
| - |
42 |
| -We recommend that users of the ORM attempt to follow database design best practices such as |
43 |
| -[normalization](https://en.wikipedia.org/wiki/Database_normalization) (at least 1NF). |
44 |
| -For instance, defining `repeated` fields in a table is considered an anti-pattern because breaks first normal form (1NF). |
45 |
| -Although we support `repeated` fields in tables, they cannot be used as key fields for this reason. This may seem |
46 |
| -restrictive but years of best practice (and also experience in the SDK) have shown that following this pattern |
47 |
| -leads to easier to maintain schemas. |
48 |
| - |
49 |
| -To illustrate the motivation for these principles with an example from the SDK, historically balances were stored |
50 |
| -as a mapping from account -> map of denom to amount. This did not scale well because an account with 100 token balances |
51 |
| -needed to be encoded/decoded every time a single coin balance changed. Now balances are stored as account,denom -> amount |
52 |
| -as in the example above. With the ORM's data model, if we wanted to add a new field to `Balance` such as |
53 |
| -`unlocked_balance` (if vesting accounts were redesigned in this way), it would be easy to add it to this table without |
54 |
| -requiring a data migration. Because of the ORM's optimizations, the account and denom are only stored in the key part |
55 |
| -of storage and not in the value leading to both a flexible data model and efficient usage of storage. |
56 |
| - |
57 |
| -## Defining Tables |
58 |
| - |
59 |
| -To define a table: |
60 |
| - |
61 |
| -1) create a .proto file to describe the module's state (naming it `state.proto` is recommended for consistency), |
62 |
| -and import "cosmos/orm/v1/orm.proto", ex: |
63 |
| - |
64 |
| -```protobuf |
65 |
| -syntax = "proto3"; |
66 |
| -package bank_example; |
67 |
| -
|
68 |
| -import "cosmos/orm/v1/orm.proto"; |
69 |
| -``` |
70 |
| - |
71 |
| -2) define a `message` for the table, ex: |
72 |
| - |
73 |
| -```protobuf |
74 |
| -message Balance { |
75 |
| - bytes account = 1; |
76 |
| - string denom = 2; |
77 |
| - uint64 balance = 3; |
78 |
| -} |
79 |
| -``` |
80 |
| - |
81 |
| -3) add the `cosmos.orm.v1.table` option to the table and give the table an `id` unique within this .proto file: |
82 |
| - |
83 |
| -```protobuf |
84 |
| -message Balance { |
85 |
| - option (cosmos.orm.v1.table) = { |
86 |
| - id: 1 |
87 |
| - }; |
88 |
| - |
89 |
| - bytes account = 1; |
90 |
| - string denom = 2; |
91 |
| - uint64 balance = 3; |
92 |
| -} |
93 |
| -``` |
94 |
| - |
95 |
| -4) define the primary key field or fields, as a comma-separated list of the fields from the message which should make |
96 |
| -up the primary key: |
97 |
| - |
98 |
| -```protobuf |
99 |
| -message Balance { |
100 |
| - option (cosmos.orm.v1.table) = { |
101 |
| - id: 1 |
102 |
| - primary_key: { fields: "account,denom" } |
103 |
| - }; |
104 |
| -
|
105 |
| - bytes account = 1; |
106 |
| - string denom = 2; |
107 |
| - uint64 balance = 3; |
108 |
| -} |
109 |
| -``` |
110 |
| - |
111 |
| -5) add any desired secondary indexes by specifying an `id` unique within the table and a comma-separate list of the |
112 |
| -index fields: |
113 |
| - |
114 |
| -```protobuf |
115 |
| -message Balance { |
116 |
| - option (cosmos.orm.v1.table) = { |
117 |
| - id: 1; |
118 |
| - primary_key: { fields: "account,denom" } |
119 |
| - index: { id: 1 fields: "denom" } // this allows querying for the accounts which own a denom |
120 |
| - }; |
121 |
| -
|
122 |
| - bytes account = 1; |
123 |
| - string denom = 2; |
124 |
| - uint64 amount = 3; |
125 |
| -} |
126 |
| -``` |
127 |
| - |
128 |
| -### Auto-incrementing Primary Keys |
129 |
| - |
130 |
| -A common pattern in SDK modules and in database design is to define tables with a single integer `id` field with an |
131 |
| -automatically generated primary key. In the ORM we can do this by setting the `auto_increment` option to `true` on the |
132 |
| -primary key, ex: |
133 |
| - |
134 |
| -```protobuf |
135 |
| -message Account { |
136 |
| - option (cosmos.orm.v1.table) = { |
137 |
| - id: 2; |
138 |
| - primary_key: { fields: "id", auto_increment: true } |
139 |
| - }; |
140 |
| -
|
141 |
| - uint64 id = 1; |
142 |
| - bytes address = 2; |
143 |
| -} |
144 |
| -``` |
145 |
| - |
146 |
| -### Unique Indexes |
147 |
| - |
148 |
| -A unique index can be added by setting the `unique` option to `true` on an index, ex: |
149 |
| - |
150 |
| -```protobuf |
151 |
| -message Account { |
152 |
| - option (cosmos.orm.v1.table) = { |
153 |
| - id: 2; |
154 |
| - primary_key: { fields: "id", auto_increment: true } |
155 |
| - index: {id: 1, fields: "address", unique: true} |
156 |
| - }; |
157 |
| -
|
158 |
| - uint64 id = 1; |
159 |
| - bytes address = 2; |
160 |
| -} |
161 |
| -``` |
162 |
| - |
163 |
| -### Singletons |
164 |
| - |
165 |
| -The ORM also supports a special type of table with only one row called a `singleton`. This can be used for storing |
166 |
| -module parameters. Singletons only need to define a unique `id` and that cannot conflict with the id of other |
167 |
| -tables or singletons in the same .proto file. Ex: |
168 |
| - |
169 |
| -```protobuf |
170 |
| -message Params { |
171 |
| - option (cosmos.orm.v1.singleton) = { |
172 |
| - id: 3; |
173 |
| - }; |
174 |
| - |
175 |
| - google.protobuf.Duration voting_period = 1; |
176 |
| - uint64 min_threshold = 2; |
177 |
| -} |
178 |
| -``` |
179 |
| - |
180 |
| -## Running Codegen |
181 |
| - |
182 |
| -NOTE: the ORM will only work with protobuf code that implements the [google.golang.org/protobuf](https://pkg.go.dev/google.golang.org/protobuf) |
183 |
| -API. That means it will not work with code generated using gogo-proto. |
184 |
| - |
185 |
| -To install the ORM's code generator, run: |
186 |
| - |
187 |
| -```shell |
188 |
| -go install cosmossdk.io/orm/cmd/protoc-gen-go-cosmos-orm@latest |
189 |
| -``` |
190 |
| - |
191 |
| -The recommended way to run the code generator is to use [buf build](https://docs.buf.build/build/usage). |
192 |
| -This is an example `buf.gen.yaml` that runs `protoc-gen-go`, `protoc-gen-go-grpc` and `protoc-gen-go-cosmos-orm` |
193 |
| -using buf managed mode: |
194 |
| - |
195 |
| -```yaml |
196 |
| -version: v1 |
197 |
| -managed: |
198 |
| - enabled: true |
199 |
| - go_package_prefix: |
200 |
| - default: foo.bar/api # the go package prefix of your package |
201 |
| - override: |
202 |
| - buf.build/cosmos/cosmos-sdk: cosmossdk.io/api # required to import the Cosmos SDK api module |
203 |
| -plugins: |
204 |
| - - name: go |
205 |
| - out: . |
206 |
| - opt: paths=source_relative |
207 |
| - - name: go-grpc |
208 |
| - out: . |
209 |
| - opt: paths=source_relative |
210 |
| - - name: go-cosmos-orm |
211 |
| - out: . |
212 |
| - opt: paths=source_relative |
213 |
| -``` |
214 |
| -
|
215 |
| -## Using the ORM in a module |
216 |
| -
|
217 |
| -### Initialization |
218 |
| -
|
219 |
| -To use the ORM in a module, first create a `ModuleSchemaDescriptor`. This tells the ORM which .proto files have defined |
220 |
| -an ORM schema and assigns them all a unique non-zero id. Ex: |
221 |
| - |
222 |
| -```go |
223 |
| -var MyModuleSchema = &ormv1alpha1.ModuleSchemaDescriptor{ |
224 |
| - SchemaFile: []*ormv1alpha1.ModuleSchemaDescriptor_FileEntry{ |
225 |
| - { |
226 |
| - Id: 1, |
227 |
| - ProtoFileName: mymodule.File_my_module_state_proto.Path(), |
228 |
| - }, |
229 |
| - }, |
230 |
| -} |
231 |
| -``` |
232 |
| - |
233 |
| -In the ORM generated code for a file named `state.proto`, there should be an interface `StateStore` that got generated |
234 |
| -with a constructor `NewStateStore` that takes a parameter of type `ormdb.ModuleDB`. Add a reference to `StateStore` |
235 |
| -to your module's keeper struct. Ex: |
236 |
| - |
237 |
| -```go |
238 |
| -type Keeper struct { |
239 |
| - db StateStore |
240 |
| -} |
241 |
| -``` |
242 |
| - |
243 |
| -Then instantiate the `StateStore` instance via an `ormdb.ModuleDB` that is instantiated from the `SchemaDescriptor` |
244 |
| -above and one or more store services from `cosmossdk.io/core/store`. Ex: |
245 |
| - |
246 |
| -```go |
247 |
| -func NewKeeper(storeService store.KVStoreService) (*Keeper, error) { |
248 |
| - modDb, err := ormdb.NewModuleDB(MyModuleSchema, ormdb.ModuleDBOptions{KVStoreService: storeService}) |
249 |
| - if err != nil { |
250 |
| - return nil, err |
251 |
| - } |
252 |
| - db, err := NewStateStore(modDb) |
253 |
| - if err != nil { |
254 |
| - return nil, err |
255 |
| - } |
256 |
| - return Keeper{db: db}, nil |
257 |
| -} |
258 |
| -``` |
259 |
| - |
260 |
| -### Using the generated code |
261 |
| - |
262 |
| -The generated code for the ORM contains methods for inserting, updating, deleting and querying table entries. |
263 |
| -For each table in a .proto file, there is a type-safe table interface implemented in generated code. For instance, |
264 |
| -for a table named `Balance` there should be a `BalanceTable` interface that looks like this: |
265 |
| - |
266 |
| -```go |
267 |
| -type BalanceTable interface { |
268 |
| - Insert(ctx context.Context, balance *Balance) error |
269 |
| - Update(ctx context.Context, balance *Balance) error |
270 |
| - Save(ctx context.Context, balance *Balance) error |
271 |
| - Delete(ctx context.Context, balance *Balance) error |
272 |
| - Has(ctx context.Context, account []byte, denom string) (found bool, err error) |
273 |
| - // Get returns nil and an error which responds true to ormerrors.IsNotFound() if the record was not found. |
274 |
| - Get(ctx context.Context, account []byte, denom string) (*Balance, error) |
275 |
| - List(ctx context.Context, prefixKey BalanceIndexKey, opts ...ormlist.Option) (BalanceIterator, error) |
276 |
| - ListRange(ctx context.Context, from, to BalanceIndexKey, opts ...ormlist.Option) (BalanceIterator, error) |
277 |
| - DeleteBy(ctx context.Context, prefixKey BalanceIndexKey) error |
278 |
| - DeleteRange(ctx context.Context, from, to BalanceIndexKey) error |
279 |
| -
|
280 |
| - doNotImplement() |
281 |
| -} |
282 |
| -``` |
283 |
| - |
284 |
| -This `BalanceTable` should be accessible from the `StateStore` interface (assuming our file is named `state.proto`) |
285 |
| -via a `BalanceTable()` accessor method. If all the above example tables/singletons were in the same `state.proto`, |
286 |
| -then `StateStore` would get generated like this: |
287 |
| - |
288 |
| -```go |
289 |
| -type BankStore interface { |
290 |
| - BalanceTable() BalanceTable |
291 |
| - AccountTable() AccountTable |
292 |
| - ParamsTable() ParamsTable |
293 |
| -
|
294 |
| - doNotImplement() |
295 |
| -} |
296 |
| -``` |
297 |
| - |
298 |
| -So to work with the `BalanceTable` in a keeper method we could use code like this: |
299 |
| - |
300 |
| -```go |
301 |
| -func (k keeper) AddBalance(ctx context.Context, acct []byte, denom string, amount uint64) error { |
302 |
| - balance, err := k.db.BalanceTable().Get(ctx, acct, denom) |
303 |
| - if err != nil && !ormerrors.IsNotFound(err) { |
304 |
| - return err |
305 |
| - } |
306 |
| -
|
307 |
| - if balance == nil { |
308 |
| - balance = &Balance{ |
309 |
| - Account: acct, |
310 |
| - Denom: denom, |
311 |
| - Amount: amount, |
312 |
| - } |
313 |
| - } else { |
314 |
| - balance.Amount = balance.Amount + amount |
315 |
| - } |
316 |
| -
|
317 |
| - return k.db.BalanceTable().Save(ctx, balance) |
318 |
| -} |
319 |
| -``` |
320 |
| - |
321 |
| -`List` methods take `IndexKey` parameters. For instance, `BalanceTable.List` takes `BalanceIndexKey`. `BalanceIndexKey` |
322 |
| -let's represent index keys for the different indexes (primary and secondary) on the `Balance` table. The primary key |
323 |
| -in the `Balance` table gets a struct `BalanceAccountDenomIndexKey` and the first index gets an index key `BalanceDenomIndexKey`. |
324 |
| -If we wanted to list all the denoms and amounts that an account holds, we would use `BalanceAccountDenomIndexKey` |
325 |
| -with a `List` query just on the account prefix. Ex: |
326 |
| - |
327 |
| -```go |
328 |
| -it, err := keeper.db.BalanceTable().List(ctx, BalanceAccountDenomIndexKey{}.WithAccount(acct)) |
329 |
| -``` |
0 commit comments