Possible optimisation to bitcoin propagation

Question

One of the main arguments for not raising the block size in Bitcoin is the fact that larger block size means slower propagation.

The most space consuming section of the Bitcoin block is the data of the transactions. Assuming a large portion of the nodes is full nodes, there is a large redundancy in the block data.

When a miner successfully mines a block, he needs to send it to the other nodes in the network. Many of those nodes already possess most of the transactions data that is included in the block.

Maybe it is better to send a hash of each transaction instead of the data itself. In case the receiving node doesn't have this data, it can request it from the sending node.

This might compress the size of a block by 5-10 times (depending on the size of the hash) and allow significantly larger block size.

Answer 1

My concern would be that in the case where the recipient doesn't have all the transactions, the extra request(s) to get them would end up taking more time than just sending them in the first place. Miners are scattered all around the world but are likely to be able to pay for fast connections, so I would expect connections between them to have high latency and high bandwidth. In such a case it's better to preemptively send a larger amount of data (fast because of high bandwidth) than to risk a request/reply round trip (slow because of high latency).

Also, miners pretty often include transactions that they originated, or were sent to them privately, or otherwise have not been broadcast on the peer-to-peer network. Consider mining pool payouts, nonstandard transactions, low-fee transactions which the miner has agreed to "accelerate", etc. So I think this extra request protocol would be needed more often than not.

This could be a reasonable optimization for full nodes who are not miners, and thus don't need to receive the block urgently. They might save some bandwidth this way. But on the other hand, since they don't need to receive the block urgently, we don't care so much about them, and it's hard to justify a major p2p protocol change just for this.

Answer 2

Not sure how this didn't get more completely answered previously.

Bitcoin has not sent the full block as the primary propagation method for a couple years now, see BIP 152. What gets sent is the block header, 6 bytes per transaction, and the coinbase transaction. Optionally, additional transactions may be sent that the sender predicted the receiver won't know.

The very short hash is possible because the hash is salted distinctly by each transmitting node. This makes it impossible for an attacker to intentionally create collisions, and any collisions that happen by chance get 'routed around' in network propagation by virtue of the fact that blocks just flow faster on links where there wasn't a collision.

Because this message is so small BIP152 is able to frequently eliminate one round trip by not performing an INV. So the fact that an extra roundtrip is sometimes needed to fill in missing transactions only leaves it with the same number of round trips as the original protocol.

There is also fibre which unconditionally reduces the transmission to 0.5 round-trips, but at a cost of wasting a considerable amount of bandwidth. For fast links this is the fastest thing to do.

For the concerns of network security and stability, it isn't just a question of how slow propagation is on average but how slow it can be made by a miner in the first case. BIP152 does a great job on the average, but doesn't actually improve the worst case Also the amount of data in a block also controls the amount of data outside of blocks, at most compact blocks can only reduce the total bandwidth of a node to half over the original scheme.

BIP152's design was selected as a trade-off between many factors-- latency, bandwidth, computational load, implementation complexity. Fibre represents a different tradeoff (favoring the absolute lowest delay instead of less bandwidth). We know how to get blocks down to about 500 bytes typically, with considerably more implementation complexity and computation, but it doesn't seem like there is much cause to do so relative to the importance of improving other areas of the protocol. This is doubly true because these improvements don't help the worst case. I believe FIBRE already achieves very close to the best possible worst case performance.