Node synchronization is painfully slow

[Ryan Butler]

Why is this so slow?  I setup a new xt node a while back.  It took almost 2 weeks to download the blockchain that is around 50GB?  Surely there are some optimizations here.  Does the node download blocks synchronously?  As in does it...download block, verify, download block, verify...?  Couldn't it have two streams of work such as one downloading and buffering a set of blocks and another validating them?  Yes in theory some of the download could be wasted if it turns out one of the blocks in the buffer doesn't validate, but I don't really see that as an issue.

[Ryan Butler]

The work stream that downloads blocks and buffers them could even request sequential blocks from different nodes, further speeding up the download.  The verification stream could even be threaded out.  Is any multithreading being utilized here at all?

[NxtChg]

>Why is this so slow? I setup a new xt node a while back. It took
>almost 2 weeks to download the blockchain that is around 50GB?

2 weeks is abnormal. On a good connection and a decent computer it takes about 1-3 days.

There must be a problem with your particular environment.

[Ryan Butler]

It's possible, I did it on a 2011 era mac mini which only has 2GB of ram.  I noticed the cpu hovered around 25% during the synchronization process.  Also bitnodes reported my latency as HUGE during this process.

[Ryan Butler]

I should also note that i have a 45Mbit internet.

On Saturday, September 12, 2015 at 2:43:57 PM UTC-5, nxtchg wrote:

[NxtChg]

>I should also note that i have a 45Mbit internet.

The system is as slow as its slowest element.

So if your bottleneck is, for example, an old HDD, fast internet connection won't help.

[Peter Tschipper]

I reported a problem a few weeks ago over at Bitcoin Core where nodes would get hung on occassion and then restart after a few mintues.  It happens on Windows but not Linux.  I don't know about Mac.  I also notice on my node, every day when I start up and it resyncs for the last 8 hours that there are large gaps in the sync process even though I might have 10 peers connected.  I'll see several blocks get downloaded and then nothing for 2 or 3 minutes or more sometimes...then it restarts again. 

This has nothing to do with XT it's also happening on Core...It also has nothing to do with a network connection because it happens when running the regression test suite which is all local traffic.  There's a problem there.  I've been trying to track it down myself but it's intermittent and quite vexxing to reproduce it on demand.

I don't know if you're experiencing the exact same thing but just giving you some background.

--
You received this message because you are subscribed to the Google Groups "bitcoin-xt" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcoin-xt+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tom Zander]

On Saturday 12. September 2015 12.43.17 Ryan Butler wrote:
> The verification stream could even be threaded out. Is any multithreading
> being utilized here at all?

yes, it uses all cores.

Problem is that most of the time is spent in IO-wait. Unless you have a really
fast HD or an SSD (or really slow CPU).
--
Tom Zander

[Ryan Butler]

I wish I could answer this myself, but c++ code is extremely foreign to me.  I only know c#.  Does the software utilize multithreading at all?  Does it have a multithreaded block download buffer that pulls from separate nodes and a multithreaded validation buffer?  As a developer this is how I would first look at tackling any performance problem in this area.

[Neil]

For intiial download, increase the cache in the settings to 500MB or 1GB if you can.  Reduce it again once synced.  It  makes a huge difference to sync time because more UTXOs can be kept in memory.  Once you get to block 200k, with the default cache size, it was being filled and cleared to zero every few blocks.

[Ryan Butler]

This makes sense, this potentially explains why the first part of the sync is pretty darn fast.

[G. Andrew Stone]

were you running a different node in the same home network (thru a gateway)?  From casual observations I'm getting the sense that that does not work well...

[Ryan Butler]

No, this is the only node.

[GGG 2014]

On Saturday, September 12, 2015 at 8:40:23 PM UTC+1, Ryan Butler wrote:

Why is this so slow?

I've been trying stuff out and report here.  My goal is to end up with a full node bitcoin-qt XT 0.11.0 fully synched but I'm not there yet.


i) the nasty old desktop
single core 1.8GHz i386 with 512MiB RAM and several GiB SWAP
never came close to synch
the "why is it slow" answer is that it spends minutes shunting a GiB in and out of swap space for every few seconds of block synch processing.

ii) the raspberryPi B2 quad-core ARM7 with 1024MiB RAM
adding a 64GiB usb flash disk for storage of the blockchain was a mistake.
swappy rw churn might be incompatible with flash drive technology.  that usb stick was on the blink copying its incomplete blockchain to another pc and now is unfound by automount for file managers and fdisk to try a clean reformat.
That same rPi with altcoins using their smaller blockchains and much less memory is happily running several different altcoin-qt full nodes, so the rPi is ok; just not well suited to the huge blockchain data storage and memory use.

iii) the i5 quad-core desktop PC with 6GiB memory
that caught up with the blockchain running bitcoin-qt 0.9.2.4 so it seems like it should work with bitcoin-qt XT when I upgrade to that

iv) the older twin-core 2.8GHz desktop with 2GB memory and 90GB of spare space on hdd (for <50GB blockchain and data) 32bit os
This has caught up from bootstrap.dat to present blockchain of bitcoin-qt 0.9.2.4 in about two weeks, and might be the minimum recommended spec for a full node running XT.  Most of the time is spent verifying blocks; looking through >30GiB of prior block data to check that every transaction in a new block is a valid one. 

The "why so slow" question is "why is it slow to check a validity of a few hundred transactions by going back to the complete record and finding bitcoin inputs which funded these claimed transactions?".  Estimate how many Mr. Benn characters with desks filing cabinets and quill pens it would take to do the same, and I think that you find two weeks catching up a 39GiB blockchain to be remarkably quick for the task. 

[Tom Zander]

On Thursday 1. October 2015 05.14.42 GGG 2014 wrote:
> The "why so slow" question is "why is it slow to check a validity of a few
> hundred transactions by going back to the complete record and finding
> bitcoin inputs which funded these claimed transactions?". Estimate how
> many Mr. Benn characters with desks filing cabinets and quill pens it would
> take to do the same, and I think that you find two weeks catching up a
> 39GiB blockchain to be remarkably quick for the task.

This is still bugging me;
the unspent output database (UXTO IIRC) is really the only thing that needs to
be kept up-to-date and a new block needs to only really check that.
Now, that database is not small. But as databases go, its really not a biggy
ether. Is it a gigabyte?

So the question then becomes, why is it so slow to validate a block (and its
purely IO wait) when we already have an up-to-date uxto?

Or am I missing something in the process of validation?

[Peter Tschipper]

what was your db cache setting? I find the 100mb default not large
enough. There are many blocks (particularly the newer ones) now that
require 200 - 300mb of cache space. (You can look through your logs and
find the "update tip" and look for the amount of cache it used for each
block). I find there is far too much paging going on with just a 100mb
cache and results in very slow sync.

I believe we really should be upping the default value to 300mb at minimum.

[G. Andrew Stone]

I wonder if we could look at free memory and dynamically adjust the cache size... is this for leveldb or the older bdb?

[Mike Hearn]

Hm, the cache size printed to the logs when a block is processed is a running total, not the amount used by that block. Are you sure a single block can use more than 200-300mb of cache? That sounds implausible to me.

[Peter Tschipper]

Yeah, I didn't realize it was a running total...I'm just taking a closer look and see the largest cache usage in my current log at 25MB.   But, I'm wondering why the sync seems better each morning with a 300MB cache.   Maybe there's something in the mechanism for recycling the cache when it's full?  Maybe this is another Windows specific thing? Or maybe, it's just a lack of repeatable performance metrics and what I think I'm seeing isn't there.  I'll  make a note of it and profile this part of the app when I get a chance. 

[Mike Hearn]

Yes, please do. I can't think of any obvious reason why sync would be faster in the morning than afternoon, except for external factors like heat differences causing CPU throttling, or background maintenance being done in the afternoon.