3

I'm syncing a node right now from scratch. It seems to be going slower than I expected. CPU/Memory/IO/Network all do not seem fully utilized. What is the limiting factor here?

My assumption is it might be it's doing a lot of random IO which is a lot slower than sequential IO.

Chris Stryczynski
  • 141
  • 3
  • I am seeing the same thing. Network is well below 1Mb/s of 96 Mb/s (per speedtest-cli), Disk is ~300 KB/s of 17.1 MB/s (using dd), CPU ~25% (renice -10), RAM ~25%. Peers: 36 (47 connections). Only thing I can think of is that peers are just very slow? – Jonathan Cross Oct 12 '20 at 16:18
  • It was significantly faster with a NVME ssd. So highly likely to be random IO bottlenecked. – Chris Stryczynski Oct 12 '20 at 17:35
  • Yes, this was mentioned by selsta and others. – Jonathan Cross Oct 12 '20 at 21:25

1 Answers1

2

Possible causes explained to me in #monero:matrix.org:

  1. Verification tends to be limiting factor.
  2. Using an HDD is known to be significantly slower than SSD/NVMe.
  3. Apparently HDD seek time is a significant bottleneck.
  4. Verification operations are not very parallelizable therefore if a single CPU core is spiked to 100%, that could be limiting factor, even if others are idle. Monero code doesn't seem to properly utilize multiple CPU cores.
  5. A single slow peer might be able to slow things down overall.
  6. Wallet syncing at the same time as blockchain syncing -> very slow.

Notes:

  • I was just syncing the most recent few thousand blocks.
  • I was using an HDD rather than SSD/NVMe, so my bottleneck may not apply to you.

[1] Per selsta: "yes, because bottleneck is verification not network speed"

[2] Per selsta: "yes, HDD will always be extremely slow during verification."

[3] Per moneromooo: "It's seek time that matters, not contiguous r/w speed."

[4] Per Inge-: "disk queue length is indicative (while transfers/sec will be very low for random reads on a HDD) and if any single cpu core is pinned at 100% (overall CPU utilization could be very low as there is a limit to how parallelizable the workload is))" and "the number of cores that can be utilized are limited. I would expect parallel verification of transactions is possible, while parallel validation of blocks is more difficult"

Q: Wouldn't users complain about multiple cores not being used?
A: hyc: "you probably would never have noticed. I have 8 cores and pretty much never see more than 4 cores used
there's definitely something artificially limiting throughput
I just don't know how to pinpoint what it is"

[5] Per moneromooo: "If none is used, it's possibly waiting on data from a peer. If that peer's busy, it might wait a bit before sending."

[6] Per moneromooo.

Jonathan Cross
  • 623
  • 5
  • 19