10

What buffer size should I choose for read/write files via POSIX fread/fwrite functions?

  • This question is too localized. It only pertains to whatever particular program you are coding at the moment. –  Oct 25 '12 at 07:04
  • I mean general case. Many programs needs to read/write whole files with very unknown size (like grep, cp) –  Oct 25 '12 at 07:09
  • Do you mean POSIX read/write functions or ANSI/ISO/IEC fread/fwrite functions? – Jan Hudec Oct 25 '12 at 08:19

1 Answers1

6

The ANSI/ISO fread/fwrite functions are buffered. The buffer is usually 8 KiB and that gives the granularity independent of what you use in your code. It might make sense to increase the buffer a bit, perhaps to the value below. For bulk transfer they will always be a tiny bit slower though due to the extra copies.

For the POSIX read/write functions it depends on operating system and device and a lot of other things, but practical experience is that you don't get any performance improvement by increasing the buffer beyond tens of KiB, so 32 or 64 KiB is about right.

On some systems the dependency is larger than on others. On Linux the difference is usually minimal above 8 KiB (so the default buffer is fine), e.g. on Windows CE (using native API; they don't have POSIX) even bigger than 64 KiB still help. It might also depend on the device.

Jan Hudec
  • 18,340
  • so 8k for buffered I/O (i.e. fread, fwrite) and 32/64K for unbuffered? If sending image data is there any specific rationale to choose buffered or unbuffered? – Francesco Boi Mar 05 '19 at 10:45
  • 1
    @FrancescoBoi, 8k is only default for buffered that you can change. The point of buffering is to avoid all the context switches when you process a few bytes at a time. For image you usually need or have all of it and then unbuffered is better as it skips a bit of processing. – Jan Hudec Mar 06 '19 at 07:00