11

Last night one of my nodes feel behind sync. Someone suggested I set the heap size explicitly when running the iri with -Xmx6500m.

java -Xmx6500m -jar iri-1.4.1.4.jar -c iri.ini

I did that and both my nodes started consuming more RAM. However, it seems that neither of nodes are syncing at all now. They have booth fell way behind in terms of milestones. 307935 and 308195.

Does anyone know what's happening here or how I can resolve this?

Both nodes seem to be working fine otherwise:

curl http://173.212.242.212:14265 -X POST -H 'Content-Type: application/json' -H 'X-IOTA-API-Version: 1.4' -d '{"command": "getNodeInfo"}'

curl http://144.217.243.6:14265 -X POST -H 'Content-Type: application/json' -H 'X-IOTA-API-Version: 1.4' -d '{"command": "getNodeInfo"}'

Both nodes are running iri 1.4.1.4, nelson 0.2.4 on Debian 9 VPS with at least 2 cores, 8Gb of RAM and, 30 GB SSD.

currenthandle
  • 331
  • 1
  • 8
  • Does the problem disappear for a while if you restart the node? Do you have any OutOfMemoryError in the logfiles? – mihi Dec 21 '17 at 23:57
  • On the restarting, it's hard to say but I'll give it another try.

    Which logs specifically? /opt/iri/target# grep -r 'OutOfMemoryError' mainnet.log/ gives no output.

     – currenthandle Dec 22 '17 at 00:03
  • Restarting seems to have brought the milestones much closer now behind by ~20. – currenthandle Dec 22 '17 at 00:48
  • You'll have to check the script that starts your iri, where it redirects stdout and stderr of the Java process to. The OutOfMemoryError (if any) would be on stderr. – mihi Dec 22 '17 at 14:05
  • so upon running java -Xmx7g -jar iri-1.4.1.4.jar -c iri.ini > out.log 2> error.log here is my error.log and my output.log (https://gist.github.com/anonymous/daeb6b4223ef124e9005a1fedcb5f5d9)

    There is no instance of the string memory in either.

    Could it be another issue?

     – currenthandle Dec 25 '17 at 00:49
  • I cannot see any obvious error, but when grepping the out.log for milestone tracker it is obvious that around 05:40:14 the solid milestone tracker must have somehow died or gone into an infinite loop as it did not log anything more in the next 14 hours although milestones were received. Perhaps open an issue on github? – mihi Dec 25 '17 at 12:02

1 Answers1

1

You have 2 Nodes, both are not(!) synchronous:

curl http://173.212.242.212:14265 -X POST -H 'Content-Type: application/json' -H 'X-IOTA-API-Version: 1.4' -d '{"command": "getNodeInfo"}' | jq

curl http://144.217.243.6:14265 -X POST -H 'Content-Type: application/json' -H 'X-IOTA-API-Version: 1.4' -d '{"command": "getNodeInfo"}' | jq

Have a look at your 1. node neighbor:

curl http://173.212.242.212:14265 -X POST -H 'Content-Type: application/json' -H 'X-IOTA-API-Version: 1.4' -d '{"command": "getNeighbors"}' | jq

The first is also not synchronous:

curl http://6.ip-144-217-243.net:14265 -X POST -H 'Content-Type: application/json' -H 'X-IOTA-API-Version: 1.4' -d '{"command": "getNodeInfo"}' | jq

This is not seen :(

curl http://173.249.16.125.net:14265 -X POST -H 'Content-Type: application/json' -H 'X-IOTA-API-Version: 1.4' -d '{"command": "getNodeInfo"}' | jq

Then have a look at your 2. node neighbor:

curl http://144.217.243.6:14265 -X POST -H 'Content-Type: application/json' -H 'X-IOTA-API-Version: 1.4' -d '{"command": "getNeighbors"}' | jq

The first is also not synchronous:

curl http://vmi154840.contaboserver.net:14265 -X POST -H 'Content-Type: application/json' -H 'X-IOTA-API-Version: 1.4' -d '{"command": "getNodeInfo"}' | jq

This is not seen :(

curl http://your.brokenfuture.com:14265 -X POST -H 'Content-Type: application/json' -H 'X-IOTA-API-Version: 1.4' -d '{"command": "getNodeInfo"}' | jq

So you have neighbors that are not synchronous or not visible. And so I don't expect your node to be in sync. Search better for other neighbors but ask the unvisible neighbors if they are sync.

Mike H.
  • 11
  • 2