New Version of Blockchain Scalpel Now with XOR-key Support

Hey everyone, Just launched v2.0.0 of my blockchain parser script, now called Blockchain Scalpel! It's designed to directly parse raw Bitcoin blk*.dat files from your disk, mainly for research or analyzing blockchain data. Here’s what’s new in this version: - Support for handling XOR-keys in blocksdir.dat files (if there’s no key, it parses directly) - Incremental parsing: it recognizes processed files and only parses new ones - It’ll throw an error if there’s an incomplete.dat file. Excited to hear your feedback!

Interesting stuff! How likely is it to use this for an SPV client instead of relying on the usual RPC calls from bitcoind? Those calls feel kinda clunky and incomplete at times.

Yeah, you could do it, but it’s not as simple as swapping out RPC. You’d basically have to recreate a chunk of a node. Blk*.dat files give you the raw data, but an SPV client needs a nice header chain and must handle reorgs. Plus, you’d need merkle proofs for specific transactions or a way to find relevant transactions without re-scanning everything. If you’re just parsing from disk, you still need indexes headers, transaction lookups, proof building, and some awareness of chainstate for smooth reorgs and rescans.

Hmm, I see your point. What if I get one of those massive HPC servers with tons of RAM? I could fit the whole blockchain in memory. Then it should be faster, right? No disk lookups unless a new block is generated. So, I'd manage the chainstate in RAM too. But if, say, block 900010 is invalidated later on, I'd have to recalculate the entire utxo set, which sounds costly. But I mean, I guess that would be a trade-off.