Limit size of data files

Question

As the mere size of the constantly changing data file tampers with my backup concept, I would like to ask if there is a way to split this file into many smaller pieces. I have no problem with the size as such, but with the amount of changes.

Background is, I do my backups with rsnapshot. This links files which haven't been changed on the target together, while changed files are copied in full.

That's why I would imagine to split the currently about 763 MB big blk0001.dat into two files:

blk0001.dat having maybe 700 MB, and not changing (frequently) in the future.
blk0002.dat holding the rest.

As the 2nd file grows steadily, I would, after it has reached a certain size, put more data from it into blk0001.dat, and eventually stop changing this file at all, starting a 3rd one.

Has someone done this already?

Just in case you're unaware, backing up your bitcoins only requires wallet.dat, not the block chain data. — Meni Rosenfeld, Nov 21 '11 at 14:07
Yes, but I don't want to download all that stuff in a case of emergency... that needed me 2 days or so the last time... — glglgl, Nov 21 '11 at 14:38
OTOH: as I have it on 3 computers, I exclude it everywhere from the backup. Good point! — glglgl, Nov 21 '11 at 14:39
@glglgl You can always use e-wallets if you really want the emergency access time - https://en.bitcoin.it/wiki/Category:EWallets — ThePiachu, Nov 21 '11 at 22:50
Backing up my wallet is no problem. I am just worried about the huge files. But if I just exclude them from backup, all is fine. — glglgl, Nov 22 '11 at 08:58
And, as I just noticed: if I lose them all, I can re-download them as an archive. — glglgl, Nov 24 '11 at 08:04

Pieter Wuille · Answer 1 · 2012-05-31T08:59:55.680

5

This is actually already implemented: blk0001.dat contains the first two gigabytes of blockchain data, blk0002.dat contains the second two gigabytes, and so on. We just haven't hit two gigabytes of data yet.

edited May 31 '12 at 08:59

answered Nov 22 '11 at 17:31

Pieter Wuille

105,497
9
194
308

one gigabyte is much stuff... – glglgl Nov 22 '11 at 17:49
1

@glglgl You can request a feature for the Main Client to offer variable size of .dat dumps here: https://github.com/bitcoin/bitcoin , although I don't think it will be a high priority, unless it turns out that a lot of other people want the same. – ThePiachu Nov 22 '11 at 21:34

score 2 · Accepted Answer · answered Nov 21 '11 at 10:59

If you want custom file handling, you'll need to alter the client code, or write some application that would be reading from the dat file and splitting it accordingly. I haven't heard of any existing offline solutions for alternative handling of the files.

score 2 · Answer 3 · answered May 31 '12 at 06:06

From https://github.com/bitcoin/bitcoin/issues/1400#issuecomment-6028443, Gregory Maxwell explains that blk0001.dat will contain closer to 2GB of blockchain data. He also explained that a gambling site and a miner with an inefficient payout system are responsible for the fast increase in file size.

Limit size of data files

3 Answers3