8

After update Bitcoin Core to version 0.15, chainstate database changed format of data.

Old prefix of key has been 0x63, now 0x43, and key/value are different

Where is the new format described, or can someone have work on this?

UPDATED

Sample

Key:

43000000dcf3713a29f3da53dbde07ea93ace52a4393ba9a009d8ac17c853c18e500

Value:

d42ade0dfd6a93b6ecbc6098336feedaa21b79128e29d5b7ee2d42

Prefix BYTE:

0x43

TransactionId:

e5183c857cc18a9d009aba93432ae5ac93ea07dedb53daf3293a71f3dc000000

Index VARINT:

0x00 -> 0x00(0)

Code VARINT:

0xD42A -> 0x2AAA(10922)

Block Height:

Code >> 1 = 0x1555(5461)

Is Coinbase:

Code & 1 = 0x00(0)

Value VARINT:

0xDE0D -> 0x2F8D(12173) -> decompressMSB128 -> 0x21084(135300)

Type VARINT:

0xFD6A -> 0x3F6A(16234)
Ninazu
  • 195
  • 6
  • If you want to find UTXOs, I find scanning the blockchain to be a better option: https://bitcoin.stackexchange.com/a/65083/2075 – Jus12 Dec 29 '17 at 05:17

1 Answers1

6

Each entry in the new 0.15 format is defined as outpoint:coin, and has the following structure:

Outpoint is formed by: key | tx_hash | index.

Where the key corresponds to b'C', or 43 in hex. The transaction hash in encoded in Little endian, and the index is a base128 varint. The corresponding Bitcoin Core source code can be found here.

On the other hand, a coin is formed by: code | value | out_type | script.

Where code encodes the block height and whether the transaction is coinbase or not, as 2*height + coinbase, the value is a txout compressed base128 varint, the out_type is also a base128 varint, and the script is the remaining data. The corresponding Bitcoin Core source code can be found here.

Version 0.15 still used an obfuscation key to avoid triggering anti-virus software, you can check this answer to know how it is stored in the database and how to use it to de-obfuscate you data.

Finally, if you need a decoder for both versions 0.14 (0.8-0.14) and the new 0.15 format, you can check this GitHub repo (particularly functions decode_utxo and decode_utxo_v08_v014 from utils.py).

Disclaimer: I'm the author of the repo.

sr_gi
  • 3,220
  • 1
  • 13
  • 37
  • I parsed Key, But with Value i have a problem. Question updated – Ninazu Nov 04 '17 at 10:51
  • What's the problem? – sr_gi Nov 04 '17 at 15:25
  • There can be still something? Data does not converge. In question i try parse example from db. I get txhash and index from outpoint. https://blockchain.info/tx/e5183c857cc18a9d009aba93432ae5ac93ea07dedb53daf3293a71f3dc000000?format=json. But any other fields in coin invalid. The simplest field is the script, since it remains unchanged. I cant find any part of script in transaction – Ninazu Nov 04 '17 at 15:53
  • For what I see, the coin and the outpoint you are giving as example do not match. The corresponding coin for '43000000dcf3713a29f3da53dbde07ea93ace52a4393ba9a009d8ac17c853c18e500' is 'bafe78849575007b8268c6115b707d17cccfdf9be636467a80f9e4'. Either way, the data you have decoded in your updated question match with the coin you provided, but since the coin seems not to be correct, the parsing goes wrong at the end. – sr_gi Nov 04 '17 at 15:54
  • Very strange. I wait full synced of my wallet and double checked value for 43000000dcf3713a29f3da53dbde07ea93ace52a4393ba9a009d8ac17c8‌ in two different leveldb driver https://github.com/syndtr/goleveldb and https://github.com/golang/leveldb and my value still is D42ADE0DFD6A93B6ECBC6098336FEEDAA21B79128E29D5B7EE2D42. Data is somehow compressed, encrypted? Is the database unique for each node? – Ninazu Nov 04 '17 at 17:14
  • I've update the answer with information about the obfuscation key and links to a repo with tools for utxo set decoding. – sr_gi Nov 04 '17 at 17:31
  • Thx! Obfuscation key - i confused. What's the point in this? This is simply XOR who stored in this db. – Ninazu Nov 05 '17 at 13:41
  • 1
    The chainstate use to trigger some anti-virus software in windows systems, by XORing the information the signature of the data changes and there is no more false positive with AV. – sr_gi Nov 05 '17 at 14:33
  • I followed your instructions and successfully decoded all but the script part: TxID: e5183c857cc18a9d009aba93432ae5ac93ea07dedb53daf3293a71f3dc000000 DecodedValue: bafe78849575007bba135794915ad83d8f25cbd406b9c92b52963a Code: 983032 Height: 491516 Amount: 942800 (84853) OutType: 0 (P2PKH) Script: 7bba135794915ad83d8f25cbd406b9c92b52963a The actual hash160 must be 7b8268c6115b707d17cccfdf9be636467a80f9e4 Is the script encoded somehow? – Andrey Dzhukov Nov 13 '17 at 13:53
  • How are you getting 7bba135794915ad83d8f25cbd406b9c92b52963a? That's the response I'm getting: {'coinbase': 0, 'tx_id': '000000dcf3713a29f3da53dbde07ea93ace52a4393ba9a009d8ac17c853c18e5', 'height': 491516, 'outs': [{'amount': 942800, 'out_type': 0, 'data': '7b8268c6115b707d17cccfdf9be636467a80f9e4'}], 'index': 0} – sr_gi Nov 13 '17 at 17:32