Satonomics - A FOSS on-chain data suite of tools (aka free Glassnode)satonomics.xyz
4646 sats \ 34 comments \ @kibo 19 Jun bitcoin freebie
Hi y'all,
I'm very pleased to finally and officially share with the world Satonomics which is a suite of tools that compute, distribute and display on-chain data. The generated datasets are very heterogeneous and can be used for many different purposes. In a nutshell, you can view this as the FOSS and self-hostable version of glassnode and thus a complimentary tool to mempool.space.
Link to the repository
The project is split in 3 parts:
  • A parser: The backbone of the project, it does most of the work by parsing and then computing datasets from the timechain.
  • A small server which automatically creates routes to access through an API all created datasets.
  • A web app which displays the generated datasets in various charts (and hopefully soon dashboards)
Everything is fully free even the API, there I think more than 10 000 routes with many more to come but the project is very much still in its infancy, many things (including routes) will change, some things might be broken, the data might be false, etc, so please be careful !
I've been working on for a looooong time now and quit my (first) job to work on this full-time since November 2023 ! It's quite a relief to finally have something to show for it.
Anyhow, feel free to ask any questions
PS: If the website doesn't work or crashes, especially on Safari iOS just refresh it should work then (like I said it's still quite early haha)
PS2: The Realized Cap and Price for hodlers isn't yet up to date, still computing them (will probably finish in 1 to 2 hours)
write
preview
related posts
142 sats \ 2 replies \ @dontforgetthekeys 19 Jun
nice work + keep it up.
might be worth your while to reach out to james check at https://charts.checkonchain.com/ he used to do their weekly analysis newsletter/youtube video. he left to start his own thing.
reply
21 sats \ 1 reply \ @kibo OP 19 Jun
Thanks !
I've been following James's work for quite a while now but like you said he has his own thing with his own website and paid substack (which I understand), thus I don't think he'd have any interest in this but I'll try, we never know !
By the way, Glassnode (his old employer) is mostly the reason Satonomics exists because of theirs outrageous prices !
reply
21 sats \ 0 replies \ @dontforgetthekeys 19 Jun
you never know. you took one leap, what's one (million) more?
reply
363 sats \ 3 replies \ @nullcount 19 Jun
Very impressive to roll-your-own indexer instead of just using electrum server. Is there a reason you wrote your own indexer for this tool?
reply
152 sats \ 2 replies \ @kibo OP 19 Jun freebie
While I haven't even considered using electrum (and I'm not very familiar with how it works internally), to have all the data that you can from the chain you really need to read each block and keep track of every state (addresses/outputs/blocks/transactions) at every single block. This can be very slow very easily, it might've been possible but it would've been slower by magnitude I think and even then I would've needed most of the code I have now
reply
21 sats \ 1 reply \ @nullcount 19 Jun
It makes sense. Electrs is too general purpose. You're building an optimized tool.
Maybe I'll try to write an indexer too: https://learnmeabitcoin.com/technical/block/blkdat/
reply
30 sats \ 0 replies \ @kibo OP 19 Jun
Oh ! To read the .blk files and convert them to usable structs, I’m using a strip down version this library which is sadly archived and doesn’t work with latest version of bitcoin rust. I had so much on my plate already that I just took what I needed and updated it to make it work
What’s nice is that I uses a parr sync iterator to read in advance the block files that you’ll read later while still being synchronous which makes the parsing of the files 60% faster I believe.
You can find the code with credits in parser/bitcoin/db
reply
31 sats \ 1 reply \ @nullcount 19 Jun
Information wants to be free. Tools like this are putting pressure on the information traffickers (like Glassnode) to find a new business model.
reply
20 sats \ 0 replies \ @kibo OP 19 Jun
deleted by author
reply
100 sats \ 1 reply \ @kibo OP 19 Jun
@Car Tagging you in case you're still interested !
reply
21 sats \ 0 replies \ @Car 20 Jun
Nice! I'll check it out this week!
reply
420 sats \ 4 replies \ @anon 21 Jun
https://www.bitesizebitcoin.xyz/news/satonomics-the-open-source-alternative-to-glassnode
reply
0 sats \ 3 replies \ @kibo OP 21 Jun
Amazing !
But those scrollbars don’t look great, what’s your setup ? I need to tweak the design a bit
reply
0 sats \ 2 replies \ @anon 23 Jun
Ah man, don't worry about that - it's explained in the article about it being a work in progress - it's the idea of FOSS that's exciting. Congrats. My set up is just me trying to build a magazine that offers pure signal to the readers. So far I have some of the team from Looking Glass helping with the educational content. I edit the content into a digital magazine format - working on a printed version once I've got the funds and logistics right - to put in every bitcoin meet up that will have it. Working with just 5 Bitcoin businesses per month to amplify their businesses throughs ads - then it's recent news, tech updates, education and some fun stuff. It's completely free to the readers.
I'd love to keep up to date with the progress of this project, so keep in touch sean@bitsizebitcoin.xyz if you don't mind.
reply
0 sats \ 0 replies \ @kibo OP 23 Jun
That’s great ! Good luck !
Not an email guy, you can hit me up here or on nostr
If you want to follow updates you can change subscribe to this page via RSS
https://codeberg.org/satonomics/satonomics/src/branch/main/CHANGELOG.md
reply
0 sats \ 0 replies \ @anon 23 Jun
sean@bitesizebitcoin.xyz*** i missed an 'e'
reply
21 sats \ 1 reply \ @Coinsreporter 19 Jun
Seems quite interesting at first look.
Great job! Keep it up and update us from time to time...
reply
0 sats \ 0 replies \ @kibo OP 19 Jun
Will do !
reply
21 sats \ 0 replies \ @Rsync25 19 Jun
Wow
reply
21 sats \ 1 reply \ @IamSINGLE 19 Jun
Great job! I appreciate your commitment and hard work. Hope you had plans to survive before quitting your job..
reply
131 sats \ 0 replies \ @kibo OP 19 Jun
Kind of ! At first I had several ideas for monetization but I decided that it was too important to have this fully free and accessible to all and I really don't see how I can keep that without degrading the software
I will probably apply for a grant at OpenSats though
But in the meantime, I still have some savings to survive for a few more months, so we'll see !
reply
21 sats \ 1 reply \ @DiedOnTitan 20 Jun
Very impressive. Had a quick tour of the site and bookmarked it. I do have a question, how are these sizes tiered, other than by sea creature?
https://m.stacker.news/36107
reply
50 sats \ 0 replies \ @kibo OP 20 Jun freebie
My bad ! I've seen them so often that I completely forgot to add their range. So:
  • Plankton - 1 sat < 0.1 btc
  • Shrimp - 0.1 btc < 1 btc
  • Crab - 1 btc < 10 btc
  • Fish - 10 btc < 100 btc
  • Shark - 100 btc < 1 000 btc
  • Whale - 1 000 btc < 10 000 btc
  • Humpback - 10 000 btc < 100 000 btc
  • Megalodon - >= 100 000 btc
reply
21 sats \ 1 reply \ @SatsMate 20 Jun
Whoa, this is awesome! I've favorited this. From the analysis side of things, I'll use this to compliment my bitfeed, and mempool. Love it, customizing dashboards in the future will be sick!
reply
0 sats \ 0 replies \ @kibo OP 20 Jun
That's the goal :)
Just to be clear, by dashboards I meant something like clark's one but if you thought about having multiple charts on the same page it definitely could also be doable
reply
0 sats \ 8 replies \ @alexbit 5 Jul
Tried to run the parser, but ended up with this error. What could it be?
thread 'main' has overflowed its stack fatal runtime error: stack overflow ./run.sh: line 25: 1954942 Aborted (core dumped) cargo run -r -- "$HOME/.bitcoin/"
reply
0 sats \ 0 replies \ @kibo OP 5 Jul
I pushed an update of run.sh, it should be better now !
If you end up using Satonomics for your website, a mention would be cool and appreciated 🤙
reply
0 sats \ 0 replies \ @kibo OP 5 Jul
My bad, it's not ulimit -n 1000000 (but you probably still need that line, it's for the maximum number of files that can be opened at a time) but ulimit -s $(ulimit -Hs) which increases the stack size
reply
0 sats \ 5 replies \ @kibo OP 5 Jul
So basically the default stack size allowed is too small for all the different things that are needed in the program. (the structs tree is an absolute monster !)
If you open the run.sh there is a hack for that but I only enabled it for Mac OS as I didn't know how it would react on Linux.
What you can do is try ulimit -n 1000000 && ./run.sh (or just move the line inside the run.sh file) and see if it helps, otherwise don't hesitate to ping me
reply
0 sats \ 4 replies \ @alexbit 6 Jul
I already had a big max open file limit on my linux server. The missing command was ulimit -s $(ulimit -Hs) to increse the stack size.
Now it worked, the parser is running, thanks. I report back to you when it finishes.
reply
0 sats \ 3 replies \ @kibo OP 6 Jul
Sure ! Like I said in the other replies, which you seem to have missed, I inverted the two lines
reply
0 sats \ 0 replies \ @alexbit 8 Jul
I messaged you on Nostr, too many doubts.
reply
0 sats \ 0 replies \ @alexbit 8 Jul
Apparently I cannot go further 2024-07-05 in the parser, due to the lack of price data:
`2024-07-08 14:28:19 - Processing 2024-07-05 (height: 850735)... 2024-07-08 14:28:19 - fetch kraken daily 2024-07-08 14:32:44 - kraken: fetch 1mn 2024-07-08 14:32:44 - binance: fetch 1mn 2024-07-08 14:32:45 - binance: read har file The application panicked (crashed). Message: Can't find price for 850839 - 1720211220 - 2024-07-05, please update binance.har file Location: src/datasets/price/mod.rs:306
Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it. Run with RUST_BACKTRACE=full to include source snippets.`
reply
0 sats \ 0 replies \ @alexbit 8 Jul
The parser just finished, it took 52 hours in a powerful linux machine.
To update its database all I have to do is to run the parser again, and it will pickup in the last height it processed?
How can I access the generated datasets?
reply