Running Locally

[clarionchain]

Great project, I've been working on something similar, I think you've got a better approach here.

I'm trying to get this running locally on my server, the parser is running, but I'm not sure it's correct.

Are there step by step instructions to get it fully up and running?

I also see you are trying to get funding for a server, after I get this running and unpack how you built it maybe I can help out with the server.

[drgarlic]

Thank you ! There are many of us trying to do this it seems

Sure, what seems wrong ? It should be very clear that it's running. What do you see in the terminal ?

There were (./run.sh --datadir=$HOME/Developer/bitcoin) but I changed how things are running and forgot to update the readme. At the top of my had you also need to specify the rpcuser and rpcpassword. You can run ./run.sh -h for help I think. Everything is saved in a file though so you only need to do this once. Oh and you must run the program from the root of the project.
With everything said though, a huge update is coming (in a month hopefully) which will change many things and should make things clearer and easier

That would be very appreciated, thank you very much ! I'll be more helpful once I'll be able to be in front of my laptop

[clarionchain]

I ran it differently. From the /satonomics/parser dir I ran:

cargo build --release
cargo run --release -- --datadir /mnt/bitcoin --rpcport 8332 --rpcuser --rpcpassword

It's running for a 3-4 days, but I'm a bit unclear where it's storing the data, how to see it and measure the progress.

I this output at the terminal.

2024-09-03 05:03:05 - binance: fetch 1mn
2024-09-03 05:03:55 - binance: fetch 1d
2024-09-03 05:05:36 - binance: fetch 1mn
2024-09-03 05:06:26 - binance: fetch 1d
2024-09-03 05:08:06 - binance: fetch 1mn
2024-09-03 05:08:57 - binance: fetch 1d
2024-09-03 05:10:37 - binance: fetch 1mn
2024-09-03 05:11:27 - binance: fetch 1d
2024-09-03 05:13:08 - binance: fetch 1mn
2024-09-03 05:13:58 - binance: fetch 1d
2024-09-03 05:15:38 - binance: fetch 1mn
2024-09-03 05:16:29 - binance: fetch 1d

Are you planning to use docker containers in the future?

[drgarlic]

The run.sh script is important to increase the limit of opened files and the stack size but it seems to not be a problem on your side since the parser would've just crashed

It's very weird, your output reminds me of a bug that I fixed a while ago, are you sure that you're up to date ? I just ran the parser with a clean slate and everything seems fine on my end

What you should see is something like this:

2024-09-03 09:26:48 - Processing 2009-04-19 (height: 11450)...
2024-09-03 09:26:48 - Processing 2009-04-20 (height: 11569)...
2024-09-03 09:26:48 - Processing 2009-04-21 (height: 11688)...
2024-09-03 09:26:48 - Processing 2009-04-22 (height: 11811)...
2024-09-03 09:26:48 - Processing 2009-04-23 (height: 11932)...
2024-09-03 09:26:49 - Processing 2009-04-24 (height: 12007)...
2024-09-03 09:26:49 - Processing 2009-04-25 (height: 12122)...
2024-09-03 09:26:49 - Processing 2009-04-26 (height: 12240)...
2024-09-03 09:26:49 - Processing 2009-04-27 (height: 12352)...
2024-09-03 09:26:49 - Processing 2009-04-28 (height: 12456)...
2024-09-03 09:26:49 - Processing 2009-04-29 (height: 12573)...
2024-09-03 09:26:49 - Processing 2009-04-30 (height: 12701)...
2024-09-03 09:26:49 - Parsing month took 1.8030503 seconds (last height: 12831)
2024-09-03 09:26:50 - Computing datasets: 1.0522441 seconds
2024-09-03 09:26:50 - Exporting...
2024-09-03 09:26:50 - WARNING: NOT SAFE TO STOP !!!
2024-09-03 09:26:52 - Datasets saved: 2.313546 seconds
2024-09-03 09:26:52 - States saved: 0.004140708 seconds
2024-09-03 09:26:52 - > Database txout_index_to_amount: 0.004773917 seconds
2024-09-03 09:26:53 - > Database txid_to_tx_data: 0.4433055 seconds
2024-09-03 09:26:53 - > Database address_index_to_empty_address_data: 0.002523292 seconds
2024-09-03 09:26:53 - > Database txout_index_to_address_index: 0.003224792 seconds
2024-09-03 09:26:53 - > Database address_index_to_address_data: 0.08278046 seconds
2024-09-03 09:26:53 - > Database address_to_address_index: 0.08280196 seconds
2024-09-03 09:26:53 - Databases saved: 0.52655804 seconds
2024-09-03 09:26:53 - Total save time: 2.8403459 seconds
2024-09-03 09:26:53 - Export done - Safe to stop now

2024-09-03 09:26:53 - Processing 2009-05-01 (height: 12832)...
2024-09-03 09:26:53 - Processing 2009-05-02 (height: 12953)...
2024-09-03 09:26:53 - Processing 2009-05-03 (height: 13079)...
2024-09-03 09:26:53 - Processing 2009-05-04 (height: 13185)...

Yes docker support is definitely on the roadmap ! Just have few things to do first

EDIT: Oh and the datasets are stored in satonomics/datasets. Though, they're compressed and so the easiest way to see the data is via the server

[tjw74]

Thanks! I have it running now, parser is running, server is running and app is available in cloudflare.

Super cool.

If the primary goal is to build this system and make it FOSS for anyone to run how are you viewing the need for the server?

Is it to improve your ability to develop on a dedicated machine making it easier for you to push new updates?

[tjw74]

Another question...are you planning to create UTXO based profit and loss metrics, coin days destroyed, etc type metrics?

I started working on building a historical view of all UTXOs by going through each block, recording inputs, outputs, price at the time UTXOs are created and spent.

It has been a bit difficult to get that working correctly. I see you have a UTXO folder, perhaps it's planned for the future.

Great work by the way...this is awesome.

[drgarlic]

Thanks! I have it running now, parser is running, server is running and app is available in cloudflare.

Super cool.

If the primary goal is to build this system and make it FOSS for anyone to run how are you viewing the need for the server?

Is it to improve your ability to develop on a dedicated machine making it easier for you to push new updates?

Amazing ! As you saw, the app is a bit different from the main instance, still working on it before doing an official release

The server is needed for several things:

• Having an API which makes allows for an easy access to the datasets which are in a compressed binary format to save space
• And soon host the app directly on it, which will remove the need for npm/pnpm/etc
• Finally, in the end a server is just a program that runs continuously similar to the parser and on a computer many things are a server without people even realizing so it doesn't really impact the FOSS-ness in any way really

Another question...are you planning to create UTXO based profit and loss metrics, coin days destroyed, etc type metrics?

I started working on building a historical view of all UTXOs by going through each block, recording inputs, outputs, price at the time UTXOs are created and spent.

It has been a bit difficult to get that working correctly. I see you have a UTXO folder, perhaps it's planned for the future.

Great work by the way...this is awesome.

Not really sure what you mean, could you provide an example ?

Most of the charts and thus datasets that aren't in the Addresses folder are UTXO based (and also based on their age in the Hodlers folder).

For example:

• Date to coin days destroyed: https://api.satonomics.xyz/date-to-coindays-destroyed
• Date to coin blocks destroyed: https://api.satonomics.xyz/date-to-coinblocks-destroyed
• Date to input count: https://api.satonomics.xyz/date-to-input-count

But many more datasets related to inputs and outputs will come with time.

And thank you !

EDIT: You can see a list of all the datasets available here, though I would recommend the open the link in a Firefox based browser for improved visibility

[mroxso]

I'm really interest in this project as well! I could write a Dockerfile and compose with no time. I just need this new version you said would come hopefully soon :)

[clarionchain]

Thanks! I have it running now, parser is running, server is running and app is available in cloudflare.
Super cool.
If the primary goal is to build this system and make it FOSS for anyone to run how are you viewing the need for the server?
Is it to improve your ability to develop on a dedicated machine making it easier for you to push new updates?

Amazing ! As you saw, the app is a bit different from the main instance, still working on it before doing an official release

The server is needed for several things:

* Having an API which makes allows for an easy access to the datasets which are in a compressed binary format to save space

* And soon host the app directly on it, which will remove the need for npm/pnpm/etc

* Finally, in the end a server is just a program that runs continuously similar to the parser and on a computer many things are a server without people even realizing so it doesn't really impact the FOSS-ness in any way really

Another question...are you planning to create UTXO based profit and loss metrics, coin days destroyed, etc type metrics?
I started working on building a historical view of all UTXOs by going through each block, recording inputs, outputs, price at the time UTXOs are created and spent.
It has been a bit difficult to get that working correctly. I see you have a UTXO folder, perhaps it's planned for the future.
Great work by the way...this is awesome.

Not really sure what you mean, could you provide an example ?

Most of the charts and thus datasets that aren't in the Addresses folder are UTXO based (and also based on their age in the Hodlers folder).

For example:

* Date to coin days destroyed: https://api.satonomics.xyz/date-to-coindays-destroyed

* Date to coin blocks destroyed: https://api.satonomics.xyz/date-to-coinblocks-destroyed

* Date to input count: https://api.satonomics.xyz/date-to-input-count

But many more datasets related to inputs and outputs will come with time.

And thank you !

EDIT: You can see a list of all the datasets available here, though I would recommend the open the link in a Firefox based browser for improved visibility

Regarding the UTXO metrics, I am thinking of profit and loss based UTXO metrics, Glassnode.com has these for example, percentage of UTXOs in profit. Maybe you already have this in there and I just didn't see it yet.

I need to spend more time looking at all the metrics you do have. Quite impressive, keep it up!

[clarionchain]

Dropped 300k sats in your geyser! LFG!

[drgarlic]

Regarding the UTXO metrics, I am thinking of profit and loss based UTXO metrics, Glassnode.com has these for example, percentage of UTXOs in profit. Maybe you already have this in there and I just didn't see it yet.

I need to spend more time looking at all the metrics you do have. Quite impressive, keep it up!

Right ! Then yes they're here too and you can have raw numbers (absolute) and percentages (relative):

• Absolute supply in profit and loss: https://app.satonomics.xyz/date-to-profit-and-loss?from=2024-05-13&to=2024-11-12
• Same relative to circulating: https://app.satonomics.xyz/date-to-profit-and-loss-relative-to-circulating-supply?from=2024-05-13&to=2024-11-12
• Same relative to own supply: https://app.satonomics.xyz/date-to-supply-in-profit-and-loss-relative-to-own-supply?from=2024-05-13&to=2024-11-12

The last two are the same since it's for the whole supply (all utxos) anyway

Now if you want those but for the STH:

• Absolute: https://app.satonomics.xyz/date-to-short-term-holders-profit-and-loss?from=2024-05-13&to=2024-11-12
• Relative to circulating: https://app.satonomics.xyz/date-to-short-term-holders-profit-and-loss-relative-to-circulating-supply?from=2024-05-13&to=2024-11-12
• Relative to own: https://app.satonomics.xyz/date-to-short-term-holders-supply-in-profit-and-loss-relative-to-own-supply?from=2024-05-13&to=2024-11-12

Though I do need to make the names clearer

Dropped 300k sats in your geyser! LFG!

Thank you so much ! Really appreciate it

[clarionchain]

You are welcome dude! I'm looking forward to the next release with the docker containers.

[drgarlic]

I'm really interest in this project as well! I could write a Dockerfile and compose with no time. I just need this new version you said would come hopefully soon :)

Hey @mroxso ! happy to say that the new version has been released 🤙

[clarionchain]

When initially running the parser, as one would expect, it takes some time to process the chain and produce the dataset.

After it finishes, and you restart it or if it crashes and restarts is it designed to pick up where it left off or does the parser start over from the beginning?

[drgarlic]

When initially running the parser, as one would expect, it takes some time to process the chain and produce the dataset.

After it finishes, and you restart it or if it crashes and restarts is it designed to pick up where it left off or does the parser start over from the beginning?

To answer your question it is designed to pick up where it left off but it all depends on the scenario.

If you stop it properly (ctrl+c, or kill PID after a recent update) and run it again, it will be fine.

But there are not only datasets but also databases which are exported and used and they only hold the latest state, otherwise you'd need many many TB of disk space to store everything and if they get corrupted (by killing the program during their export), the program will detect that and recompute them which takes pretty much the same time as running the program for the first time.

If you run it after an update, if there are no new datasets, it will be all good.
If there are new datasets it will depend on their type (inserted (very slow to slowish) vs computed (very fast)) and what they need and it gets a little complicated. I try to not compute stuff that's not needed but it's not perfect and I'm sure that there is room for a lot of improvements

[clarionchain]

Right on, thanks for replying so quickly. I'll keep playing around, trying to get into the details of how designed it all. Good stuff.

Any updates on when the docker integration?

[drgarlic]

Hopefully soon but honestly it's better to run it natively, you only need rust and the performance will be better

[drgarlic]

@mroxso did you have a chance to look into it ? I'm close but not quite there yet

[mroxso]

Nah sorry. Had no time yet. Do you have a Work in Progress branch? Maybe I can take a look at it.

[drgarlic]

Nah sorry. Had no time yet. Do you have a Work in Progress branch? Maybe I can take a look at it.

No worries and yes but only the parser container not yet the server (which will be much easier) and not in a branch but in the docker folder

build.sh to build the image and run.sh to run it

It seems to work but I haven't yet found how to set ulimit via docker run, my usual values make it crash

What's really important and that's why there is an exec is that is responds correctly to SIGINT (ctrl+c) and SIGTERM (kill) to not corrupt the databases. I think there is a timeout when stopping a container, which kills it if it's taking too long ? If there a way to increase it that'd be great, to avoid any frustration from users

[mroxso]

I took a look at it.
It is possible to go this way, but it is definitely not the best one. You are compiling on runtime, which leads to big container images. This can be improved by building the rust application while building the docker image. I need more time to do this.

For now, I think the compiling on runtime is okay.

You can take a look at the PR (#5 )

[clarionchain]

After cloning the directory onto my system, I run the parser giving it my bitcoin node credentials, datadir, etc. Parser starts but immediately says datasets are missing but continues to run.

I then start the server and it panic crashes saying it can't find datasets and some directories.

Is it expected for the parser to report missing datasets because it hasn't created them yet?

Will the server also crash until the parser has created some minimum amount of datasets?

[drgarlic]

So when it says dataset missing it means that it'll need to compute (or recompute if the version changed) it, which is one of the reasons why it can restart from the beginning. I understand how it can be confusing, would you word it differently ?

When it panics it's because the parser hasn't had the time to generate a particular file (server/in/disk_path_to_type.json which should only take seconds) before you running the server but if you run it with ./run.sh instead of cargo run -r it will reload by itself

[clarionchain]

Got it, because it just started it's looking for certain metrics/files that haven't been created yet but will be created later.

Seeing "missing" on the initial run had me thinking maybe I missed something, or a misconfiguration.

All good, thanks for confirming.

This build I used ./run.sh, good to know it restarts.

[clarionchain]

Where can I find the code that orders the metric list for the front end?

The metric tree in the side bar for the charts.

Thx!

[drgarlic]

It's here: https://github.com/kibo-money/kibo/blob/main/website/scripts/options.js (createPartialOptions the first function)

but please contact me on nostr for something like this