base node slow sync

[AlexanderBarkovsky]

Hello everyone!
I’m facing an issue with slow blockchain sync on a mainnet base node.

I deployed the base node on-premise with the following configuration:

Disks: RAID-5 (NVME SSD INTEL SSDPF2KX153T)
VM: 16 vCPUs and 32 GB RAM
I downloaded the full mainnet snapshot for the base and started the node with the snapshot data. However, the sync process is very slow, and I noticed the ETA keeps increasing.

INFO [12-23|07:38:31.127] Imported new potential chain segment     number=23,836,828 hash=48bcc7..bbb4f8 blocks=1          txs=172         mgas=52.506 elapsed=6.721s       mgasps=7.811   age=5d12h21m snapdiffs=397.35MiB triedirty=0.00B
WARN [12-23|07:38:39.631] Ignoring already known beacon payload    number=23,836,828 hash=48bcc7..bbb4f8 age=5d12h21m
INFO [12-23|07:38:39.640] Chain head was updated                   number=23,836,828 hash=48bcc7..bbb4f8 root=7c64dc..6593ba elapsed=7.781567ms   age=5d12h21m
INFO [12-23|07:38:44.427] Starting work on payload                 id=0x03170834000a8ac0
INFO [12-23|07:38:48.900] Imported new potential chain segment     number=23,836,829 hash=aa9b64..5cdef9 blocks=1          txs=140         mgas=27.996 elapsed=4.454s       mgasps=6.285   age=5d12h22m snapdiffs=397.40MiB triedirty=0.00B
INFO [12-23|07:38:48.908] Chain head was updated                   number=23,836,829 hash=aa9b64..5cdef9 root=54ed48..cd89bc elapsed=6.00817ms    age=5d12h22m
INFO [12-23|07:38:56.109] Starting work on payload                 id=0x03f54a8b2e39dfde
INFO [12-23|07:39:10.298] Imported new potential chain segment     number=23,836,830 hash=6e427b..1d5643 blocks=1          txs=188         mgas=39.307 elapsed=6.184s       mgasps=6.356   age=5d12h22m snapdiffs=397.45MiB triedirty=0.00B
WARN [12-23|07:39:19.201] Ignoring already known beacon payload    number=23,836,830 hash=6e427b..1d5643 age=5d12h22m
INFO [12-23|07:39:19.210] Chain head was updated                   number=23,836,830 hash=6e427b..1d5643 root=319e91..a2ed18 elapsed=7.384527ms   age=5d12h22m
INFO [12-23|07:39:24.591] Starting work on payload                 id=0x032f4a6d94c7eaeb
INFO [12-23|07:39:39.271] Imported new potential chain segment     number=23,836,831 hash=0af3d2..c245b2 blocks=1          txs=179         mgas=34.157 elapsed=4.757s       mgasps=7.179   age=5d12h22m snapdiffs=397.51MiB triedirty=0.00B
INFO [12-23|07:39:39.282] Chain head was updated                   number=23,836,831 hash=0af3d2..c245b2 root=6e1885..18b071 elapsed=8.379856ms   age=5d12h22m
INFO [12-23|07:39:45.404] Starting work on payload                 id=0x031dd651d0308981
INFO [12-23|07:39:59.623] Imported new potential chain segment     number=23,836,832 hash=1174e7..75e904 blocks=1          txs=153         mgas=26.762 elapsed=5.203s       mgasps=5.143   age=5d12h23m snapdiffs=397.55MiB triedirty=0.00B
WARN [12-23|07:40:09.536] Ignoring already known beacon payload    number=23,836,832 hash=1174e7..75e904 age=5d12h23m
INFO [12-23|07:40:09.544] Chain head was updated                   number=23,836,832 hash=1174e7..75e904 root=94aaa8..56ef86 elapsed=6.444021ms   age=5d12h23m
INFO [12-23|07:40:13.733] Starting work on payload                 id=0x03b7c289509d104a
INFO [12-23|07:40:17.789] Imported new potential chain segment     number=23,836,833 hash=6405a8..4ff9a8 blocks=1          txs=154         mgas=29.458 elapsed=4.037s       mgasps=7.297   age=5d12h23m snapdiffs=397.61MiB triedirty=0.00B
INFO [12-23|07:40:17.796] Chain head was updated                   number=23,836,833 hash=6405a8..4ff9a8 root=90d2a8..21ff18 elapsed=5.119713ms   age=5d12h23m
INFO [12-23|07:40:26.440] Starting work on payload                 id=0x0322362d71750d50
INFO [12-23|07:40:40.512] Imported new potential chain segment     number=23,836,834 hash=1e14f3..d85679 blocks=1          txs=171         mgas=39.376 elapsed=7.629s       mgasps=5.161   age=5d12h23m snapdiffs=397.66MiB triedirty=0.00B
WARN [12-23|07:40:48.122] Ignoring already known beacon payload    number=23,836,834 hash=1e14f3..d85679 age=5d12h23m
INFO [12-23|07:40:48.130] Chain head was updated                   number=23,836,834 hash=1e14f3..d85679 root=fd8224..bfc704 elapsed=6.108157ms   age=5d12h23m
INFO [12-23|07:40:52.344] Starting work on payload                 id=0x0345731952208713
INFO [12-23|07:40:56.825] Imported new potential chain segment     number=23,836,835 hash=bfba79..12e183 blocks=1          txs=154         mgas=29.979 elapsed=4.460s       mgasps=6.721   age=5d12h23m snapdiffs=397.70MiB triedirty=0.00B
INFO [12-23|07:40:56.832] Chain head was updated                   number=23,836,835 hash=bfba79..12e183 root=e8fe03..6ff13d elapsed=5.000728ms   age=5d12h23m
INFO [12-23|07:41:02.105] Starting work on payload                 id=0x03fa962cc29bdc3b
INFO [12-23|07:41:16.962] Imported new potential chain segment     number=23,836,836 hash=d025b0..519bba blocks=1          txs=156         mgas=51.048 elapsed=5.027s       mgasps=10.154  age=5d12h24m snapdiffs=397.74MiB triedirty=0.00B
WARN [12-23|07:41:27.179] Ignoring already known beacon payload    number=23,836,836 hash=d025b0..519bba age=5d12h24m
INFO [12-23|07:41:27.256] Chain head was updated                   number=23,836,836 hash=d025b0..519bba root=70e341..ac55dc elapsed=75.999861ms  age=5d12h24m
INFO [12-23|07:41:33.485] Starting work on payload                 id=0x03f03a08fa307c4c
INFO [12-23|07:41:49.467] Imported new potential chain segment     number=23,836,837 hash=e034f0..6a1494 blocks=1          txs=178         mgas=33.738 elapsed=6.823s       mgasps=4.945   age=5d12h24m snapdiffs=397.79MiB triedirty=0.00B
WARN [12-23|07:41:57.703] Ignoring already known beacon payload    number=23,836,837 hash=e034f0..6a1494 age=5d12h24m
INFO [12-23|07:41:57.712] Chain head was updated                   number=23,836,837 hash=e034f0..6a1494 root=22e19e..073985 elapsed=7.761638ms   age=5d12h24m
INFO [12-23|07:42:03.005] Starting work on payload                 id=0x036797e925bd08f5
INFO [12-23|07:42:17.442] Imported new potential chain segment     number=23,836,838 hash=4b21fb..b59257 blocks=1          txs=183         mgas=32.270 elapsed=4.441s       mgasps=7.266   age=5d12h25m snapdiffs=397.84MiB triedirty=0.00B
INFO [12-23|07:42:17.450] Chain head was updated                   number=23,836,838 hash=4b21fb..b59257 root=ad591b..43b6f1 elapsed=5.834872ms   age=5d12h25m
INFO [12-23|07:42:23.558] Starting work on payload                 id=0x03475b54c9711a49
INFO [12-23|07:42:39.260] Imported new potential chain segment     number=23,836,839 hash=858ce2..1d78ff blocks=1          txs=137         mgas=38.144 elapsed=6.704s       mgasps=5.690   age=5d12h25m snapdiffs=397.89MiB triedirty=0.00B
WARN [12-23|07:42:47.655] Ignoring already known beacon payload    number=23,836,839 hash=858ce2..1d78ff age=5d12h25m
INFO [12-23|07:42:47.665] Chain head was updated                   number=23,836,839 hash=858ce2..1d78ff root=d7bd0e..ad65a4 elapsed=9.011718ms   age=5d12h25m
INFO [12-23|07:42:53.177] Starting work on payload                 id=0x0300e12a3f378477

[AlexanderBarkovsky]

Hello @AlexanderBarkovsky You might need to restart the whole process because this error is due to incorrect synchronization process I hope this helps

Thanks for that, but I think it won't help... I started syncing from the genesis block, but I noticed that the sync speed gradually slowed down over time. To address this, I decided to download a snapshot instead. However, it seems that the sync speed decreases on its own as time progresses.

For example, this graph shows the sync process when I started from the genesis block.

This one shows the continued sync from a snapshot.

The sync speed is not good in both cases

[Mukzid]

Increasing the cache size might help to speed up the processing of blocks

[AlexanderBarkovsky]

Increasing the cache size might help to speed up the processing of blocks

I've increased the cache size from 2048 to 8192. However, the speed is still low—around 3 requests per second.

I also added the variable OP_NODE_L1_TRUST_RPC=true.

I read somewhere that this can help increase speed if I’m using a trusted Geth node. In my case, I am using a self-hosted Geth archive node.

[Mukzid]

Have you Experiment with increasing the number of peers (--maxpeers) in your Geth configuration. This can help speed up the sync by downloading blocks and states from multiple sources simultaneously

-If you haven't already, try using the --syncmode "fast" or --syncmode "snap" options. These modes are designed to speed up the initial sync process.
using Geth's pruning settings can reduce the burden on your system, allowing faster performance

Since you're using OP_NODE_L1_TRUST_RPC=true with a self-hosted Geth archive node, this setting should theoretically help. However, the performance gains can also depend on the efficiency of the archive node itself.

Given that you've increased the cache size and are still experiencing slow speeds, it might be helpful to examine the log files for any potential issues or errors that could be contributing to the slowdown.

Have you considered any of these approaches before?

[ianhe8x]

We got same problem. The syncmode used in op-node is execution-layer. had similar log says Ignoring already known beacon payload and the syncing speed from geth is basically 1 block a time.
I'm wondering is it an issue mostly from geth or op-node? because it seems geth can not get a batch of recent blocks but have to discover 1 at a time.

[AlexanderBarkovsky]

Hello! Sorry for the late reply

Have you Experiment with increasing the number of peers (--maxpeers) in your Geth configuration. This can help speed up the sync by downloading blocks and states from multiple sources simultaneously

No, because my base node is not able to reach the maximum number of peers. For example, I only have 31 peers even when --maxpeers is set to 100.

-If you haven't already, try using the --syncmode "fast" or --syncmode "snap" options. These modes are designed to speed up the initial sync process.

The point in that I don't have the problem with leak of resources. I can increase CPU or Memory twice, but it doesn't boost the
node speed. This way I expect the base should have enough resources for working properly using archive mode

However, the performance gains can also depend on the efficiency of the archive node itself.

I will try switching the Geth archive node to another one. Maybe that will help. Thank you for the advice!

Given that you've increased the cache size and are still experiencing slow speeds, it might be helpful to examine the log files for any potential issues or errors that could be contributing to the slowdown.

The strangest thing is that I don’t see any suspicious logs. Everything seems to be working correctly from the logs' perspective.

[orange723]

same issues

[orange723]

My machine's disk I/O is too low. Improving I/O and throughput should fix the slow sync issue.

[mohsinriaz17]

@AlexanderBarkovsky hi
1- how did you get those graphs?

2- did you find any definitive answer to increasing the sync speed?

i made a small script to fetch latest block every 10 sec 5 times and then average the speed of blocks sync
for me it comes out to be around 1.5-1.7 blocks per second

i am on a 6th gen xeon vm with 16vcpu and 32gb ram and 2TB nvme disk
and for becon node i am using chainstack, it has a 25 api calls per sec throughput, however, i see on there dashboard the request made are less than 2 per second. which kind of matches with the number of blocks i am syncing per second

so in my theory, the node is not even attempting to fetch blocks faster

[haardikk21]

A 2tb nvme disk is not enough to sync a Base mainnet node. You need significantly more than that. Bumping up RAM and having faster disks is the best way to improve your sync speed. The issue isn't the network's sync speed but rather blocked by I/O on the hardware.