mst perf fiddling

[bnewbold]

Run bench code like: cd atproto/repo; go test -bench=.

I swapped out the small .car file for a full dump of my personal CAR for testing.

The current tests are around the most basic MST operations: loading from a CAR slice ("read") or computing hashes without saving blocks ("hash").

My initial numbers with larger CAR file:

goos: linux
goarch: amd64
pkg: github.com/bluesky-social/indigo/atproto/repo
cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
BenchmarkCreateAndHash-8         68  17126866 ns/op 9480340 B/op     201558 allocs/op
BenchmarkLoadFromCAR-8           22  49191077 ns/op 50775918 B/op    668644 allocs/op
BenchmarkCommitFile-8         84013     14754 ns/op    8598 B/op        141 allocs/op

fiddling a bit shaved some percentage but not a lot:

BenchmarkCreateAndHash-8         72  15713848 ns/op 7586793 B/op     187968 allocs/op
BenchmarkLoadFromCAR-8           24  48864745 ns/op 50244724 B/op    653971 allocs/op
BenchmarkCommitFile-8         80757     14595 ns/op    8603 B/op        141 allocs/op

to count allocations:

go test -bench=. -memprofile memprofile.out

go tool pprof memprofile.out

(pprof) sample_index = alloc_objects

(pprof) top 25
Showing nodes accounting for 33505915, 87.61% of 38245112 total
Dropped 116 nodes (cum <= 191225)
Showing top 25 nodes out of 101
      flat  flat%   sum%        cum   cum%
   4899703 12.81% 12.81%    9277220 24.26%  github.com/bluesky-social/indigo/atproto/repo/mst.(*Node).insert
   3602829  9.42% 22.23%    8602645 22.49%  github.com/bluesky-social/indigo/atproto/repo/mst.(*Node).insertChild
   2850983  7.45% 29.69%    2850983  7.45%  github.com/ipfs/go-ipfs-ds-help.NewKeyFromBinary
   2637981  6.90% 36.58%    2637981  6.90%  github.com/ipfs/go-datastore.Key.Child (inline)
   1682178  4.40% 40.98%    2624315  6.86%  github.com/ipfs/go-cid.CidFromReader
   1496471  3.91% 44.90%    1496471  3.91%  bytes.NewReader (inline)
   1452778  3.80% 48.69%    1452778  3.80%  github.com/ipfs/go-block-format.NewBlockWithCid
   1452777  3.80% 52.49%    1452777  3.80%  github.com/ipfs/go-cid.Cid.Bytes (inline)
   1386185  3.62% 56.12%    2107100  5.51%  github.com/bluesky-social/indigo/atproto/repo/mst.(*NodeData).Node
   1201545  3.14% 59.26%    1201545  3.14%  strings.(*Builder).grow
   1168776  3.06% 62.31%    1168776  3.06%  github.com/multiformats/go-multihash.Encode
   1130529  2.96% 65.27%    1130529  2.96%  crypto/internal/fips140/sha256.(*Digest).Sum
   1060992  2.77% 68.04%    1060992  2.77%  crypto/internal/fips140/sha256.New (inline)
   1032690  2.70% 70.74%    1032690  2.70%  github.com/ipld/go-car/util.LdRead
    942137  2.46% 73.21%     942137  2.46%  github.com/ipfs/go-cid.toBufByteReader (inline)
    917543  2.40% 75.61%     917543  2.40%  github.com/whyrusleeping/cbor-gen.ReadByteArray
    917541  2.40% 78.01%     917541  2.40%  github.com/ipfs/go-cid.CidFromBytes
    720915  1.88% 79.89%     720915  1.88%  slices.Grow[go.shape.[]uint8,go.shape.uint8] (inline)
    540687  1.41% 81.31%    6997119 18.30%  github.com/bluesky-social/indigo/atproto/repo/mst.loadNodeFromStore
    539473  1.41% 82.72%     539473  1.41%  github.com/bluesky-social/indigo/atproto/repo/mst.(*Node).NodeData
    491528  1.29% 84.00%    1998919  5.23%  github.com/bluesky-social/indigo/atproto/repo/mst.(*EntryData).UnmarshalCBOR
    376837  0.99% 84.99%    9588521 25.07%  github.com/bluesky-social/indigo/atproto/repo/mst.LoadTreeFromMap
    347465  0.91% 85.89%     347465  0.91%  github.com/bluesky-social/indigo/atproto/repo/mst.(*Node).splitEntries
    327687  0.86% 86.75%     415072  1.09%  reflect.Value.call
    327685  0.86% 87.61%   11375954 29.74%  github.com/bluesky-social/indigo/atproto/repo.VerifyCommitMessage

Note this is all for synthetic benchmarks, not real world usage.

Some quick notes (I haven't done much alloc reduction like this in go before):

• surely for relay use case the biggest win would be an arena or similar allocation pool, scoped to the event: all the allocations for processing an event can be free'd at once at the end of the event. go didn't end up with arena allocation, but surely there is something we can take advantage of?
• just tons of []byte passed around, more explicit buffers might help in some situations
• a little worried the design decision to use []byte for strings instead of string might be resulting in an alloc in there somewhere? but that only happens at the repo abstraction layer, not simply working with MST nodes
• parsing out to NodeData (and EntryData), then re-allocating higher-level separate structs, is kind of inefficient, but I don't see how to get around that
• maybe worth trying sync.Pool with sha256; though I read sometimes the SHA internal struct gets optimized away. the main place we use SHA-256 currently is working with "key" strings, not just doing CIDs

[bnewbold]

Another thing to think about would be using sync.Pool with Node and NodeEntry, with a tree.Free() method which would push all those structs back in to the pool. Would pull the pre-allocated structs out of the pool, call a .Reset() to clear fields, and then start using it. I'm not sure if that really helps?

[bnewbold]

(tests are failing because b.Loop() is a go v1.24 thing)