Parsing blk.dat file fails, is this block legitimate?

Question

I'm parsing in blocks from my local blk*.dat files, using this excellent website as a guide.

All goes well until file blk00976.dat, block 000000000000000000daf7a26d903543377d5cdddb962077e58fd11212479eea

In my copy of the file, when the parser reaches byte 27871335, it is about to iterate through a set of 64 transaction outputs.

When it reaches the 48th of these, it gets the value c4f1c2b3b69c42c5 and then reads a VarInt of fe80ffffff. This is the number of bytes to read for the next scriptPubKey.

The integer value of the VarInt is 2164260863 bytes (larger than the blk.dat file of 133768162 bytes).

Note the transaction begins with ffffffff58. Implying a version of ffffffff.

A hexdump of the block in question can be found here: https://blockchain.info/rawblock/000000000000000000daf7a26d903543377d5cdddb962077e58fd11212479eea?format=hex (Find "fe80ffffff" to see where I am looking)

A. Is this transaction and/or the block legitimate?

B. If not, why is it in my blk.dat archive?

C. What does a transaction version of ffffffff imply?

D. Is the scriptPubKey size VarInt fe80ffffff correct?

E. What is the story behind this block?

We can't know what is at byte position 27871335 in your blk00976.dat, as the contents of those files depends on the order in which peers gave you historical blocks. Can you perhaps include the full transaction hexdump in your question? — Pieter Wuille, Dec 08 '22 at 15:13
@PieterWuille thanks, the hexdump wouldn't fit in the question (too many chars). I've put on github: https://github.com/4c656554/SETransactionHex/blob/main/hex_dump . This is the transaction as constructed until meeting the fe80ffffff VarInt. — Lee, Dec 08 '22 at 15:54
Can you compute the hash of the block it's in? (Not the claimed hash, but recompute it from the bytes in the blk file). — Pieter Wuille, Dec 08 '22 at 17:03
@PieterWuille do you mean the hash of the header? Would I not need to recompute the merkle root including the above problem tx? (which I can't figure out how to do due to big varint)... — Lee, Dec 08 '22 at 19:22
Oh, right. My guess is just that you're seeing garbage bytes in the blk.dat file, which you should skip. — Pieter Wuille, Dec 08 '22 at 19:24
@PieterWuille perhaps, though seems odd that the block is listed at blockchain.info too in that case? — Lee, Dec 08 '22 at 19:53
Oh, the block you link is the first segwit block (segregated witness), a softfork and associated protocol change that introduced a new block and transaction serialization format. The details are specified in BIP144: https://github.com/bitcoin/bips/blob/master/bip-0144.mediawiki. I strongly suspect that's the cause of what throws off your parser. — Pieter Wuille, Dec 08 '22 at 20:07
@PieterWuille got it; with help from your other answers on here as well, thank you. — Lee, Dec 09 '22 at 04:55

score 5 · Accepted Answer · edited Mar 19 '24 at 16:56

A. Is this transaction and/or the block legitimate?

Yes - assuming you accept the segwit soft fork and resulting option for BIP144 format transaction serialisation.

B. If not, why is it in my blk.dat archive?

Not applicable: see above.

C. What does a transaction version of ffffffff imply?

This is noise resulting from attempting to read a segwit tx with pre-segwit parsing schema.

D. Is the scriptPubKey size VarInt fe80ffffff correct?

No, this is also noise resulting from attempting to read a segwit tx with pre-segwit parsing schema.

E. What is the story behind this block?

Per Pieter Wuille's comment:

...the block you link is the first segwit block (segregated witness), a softfork and associated protocol change that introduced a new block and transaction serialization format. The details are specified in BIP144: github.com/bitcoin/bips/blob/master/bip-0144.mediawiki. I strongly suspect that's the cause of what throws off your parser.

Note, the block I linked is the second segwit block, but it must have been the first segwit block saved in my local .dat store so my parser hit it first.

BIP144 link

To adjust my parser to accept segwit transactions I had to add a conditional statement after reading the Input Count byte. If a value of zero (0x00) appears at its position, it is the "marker" byte which indicates that we are dealing with a segwit transaction. We then read the "flag" byte (which should be one (0x01)), and then the actual Input Count byte. Finally, we must also read the script witnesses:

Before reading the locktime, for each transaction input, read:

 NumberStackItems <VarInt>
 For each stack item:
    SizeOfStackItem <VarInt>
    StackItem <SizeofStackItem bytes>

Parsing blk.dat file fails, is this block legitimate?

1 Answers1