Thin blocks patch

[Mike Hearn]

I've uploaded a patch set that implements thin block relay:

   https://github.com/bitcoinxt/bitcoinxt/pull/91

Thin blocks are where you download a block from a peer as a list of hashes rather than full block contents. Therefore it eliminates the double transmission of tx data.

The patch works by using full match Bloom filters. Therefore, you can download thin blocks from any peer on the network. However you can't upload thin blocks unless you connect to peers running this patch.

For me it's hard to see much difference, but I only have access to fast internet connections that can shift 900kb of data so fast it's not worth worrying about. This code would make more of a difference with bigger blocks and on slow/flaky connections.

I will be busy with other things for a while. I'm looking for someone to take this code and finish it off, by ensuring it's tested better and by handling the case of Core nodes that have disabled Bloom filtering (checking the service bits). There are probably other issues too. I'm running it on my nodes and they seem fine, but I haven't spent a whole lot of time thinking about edge cases.

[Peter Tschipper]

Mike,

I tried this out on Windows 7...

syncing a 600mb blockchain on a fast wireless LAN...blocks are a 50 - 50 mix of either small 181 byte or  large 980KB blocks

time to sync chains is as follows:

ThinBlockclient to Thinblockclient - 5:00 mins
Thinblockclient to XT 11B -  3:23
XT 11B to Thinblockclient - 4:50

When syncing the chain *to* a thinkblock client, this test setup shows a performance degradation

Looking in the logs I had some questions:   first, there seems to be a ping/pong for every single transaction.  Every time a getdata is sent out there is a ping right after? The
second question is, what are all those "received:tx of 67647 bytes, I thought it only needed the merkelblock?

2015-11-03 15:55:23 sending: getdata (37 bytes) peer=1
2015-11-03 15:55:23 received: pong (8 bytes) peer=1
2015-11-03 15:55:23 pong peer=1 /Bitcoin XT:0.11.0B/: Unsolicited pong without ping, 0 expected, b821356f24a98336 received, 8 bytes
2015-11-03 15:55:23 received: merkleblock (570 bytes) peer=1
2015-11-03 15:55:23 received thin block 1d3e5a6a33f251feb1deb0e3fe17826dd8c7597da1c53056a45e3a9a8c52d6e0 peer=1
2015-11-03 15:55:23 sending: ping (8 bytes) peer=1
2015-11-03 15:55:23 received: tx (100 bytes) peer=1
2015-11-03 15:55:23 received: tx (67648 bytes) peer=1
2015-11-03 15:55:23  25: For conf success > 0.85 need FeeRate >:           -1 from buckets 2.1e+015 - 2.1e+015  Cur Bucket stats    nan%       0.0/(0.0+0 mempool)
2015-11-03 15:55:23 received: tx (67648 bytes) peer=1
2015-11-03 15:55:23 received: tx (67648 bytes) peer=1
2015-11-03 15:55:23 received: tx (67647 bytes) peer=1
2015-11-03 15:55:23 received: tx (67647 bytes) peer=1
2015-11-03 15:55:23 received: tx (67648 bytes) peer=1
2015-11-03 15:55:23 received: tx (67647 bytes) peer=1
2015-11-03 15:55:23 received: tx (67614 bytes) peer=1
2015-11-03 15:55:23 received: tx (67648 bytes) peer=1
2015-11-03 15:55:23 received: tx (67647 bytes) peer=1
2015-11-03 15:55:23 received: tx (67648 bytes) peer=1
2015-11-03 15:55:23 received: tx (67647 bytes) peer=1
2015-11-03 15:55:23 received: tx (67647 bytes) peer=1
2015-11-03 15:55:23 received: tx (67647 bytes) peer=1
2015-11-03 15:55:23 Reassembled thin block for 1d3e5a6a33f251feb1deb0e3fe17826dd8c7597da1c53056a45e3a9a8c52d6e0 (947212 bytes)
2015-11-03 15:55:23 Received block 1d3e5a6a33f251feb1deb0e3fe17826dd8c7597da1c53056a45e3a9a8c52d6e0 in 6.57 seconds

--
You received this message because you are subscribed to the Google Groups "bitcoin-xt" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcoin-xt+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mike Hearn]

I would expect it to get slower during IBD, that's part of the work that's needed. The bandwidth savings only occur when you're in the steady state and listening to broadcasts. The code is supposed to avoid using thin blocks when IsInitialBlockDownload() returns true, but I haven't tested that.

It looks like you have -debug=net enabled?

[Peter Tschipper]

debug is set to

-debug=

[Mike Hearn]

Check your bitcoin.conf file. You're seeing debug spew that isn't on by default.

At any rate. Try syncing until you're at head using the -nouse-thin-blocks flag. Then restart without it, and see if things work better.

[Peter Tschipper]

I have my debug set that way so i can see most everything...it's doesn't bog the system down at all.
---
Have had it up on Testnet all afternoon...seems to work functionally but once the chain is sync'd it's hard to say anything about performance.


By the way, hats off to you, this is a great piece of work! 

On 03/11/2015 8:50 AM, Mike Hearn wrote:

Check your bitcoin.conf file. You're seeing debug spew that isn't on by default.

At any rate. Try syncing until you're at head using the -nouse-thin-blocks flag. Then restart without it, and see if things work better.

[Mike Hearn]

Yeah, logging won't cause any slowdowns at Bitcoin's present level of traffic, that's not the issue. The issue is that when you don't have any of the transactions in a block, the old format is more efficient. Thin blocks work better when you have already downloaded the transactions in the block at broadcast time and want to avoid the duplication.

What we really need is testing from people behind very slow or crappy internet connections.

[Jonathan Toomim]

On Nov 4, 2015, at 3:18 AM, Mike Hearn <he...@vinumeris.com> wrote:

What we really need is testing from people behind very slow or crappy internet connections.

http://stackoverflow.com/questions/3536249/simulating-slow-internet-connection

Or you could ask Luke-jr to test it.

[G. Andrew Stone]

On Wednesday, November 4, 2015 at 6:18:43 AM UTC-5, Mike Hearn wrote:

Yeah, logging won't cause any slowdowns at Bitcoin's present level of traffic, that's not the issue. The issue is that when you don't have any of the transactions in a block, the old format is more efficient. Thin blocks work better when you have already downloaded the transactions in the block at broadcast time and want to avoid the duplication.

What we really need is testing from people behind very slow or crappy internet connections.

 Use the traffic shaper

[Robin]

Maybe you can try on an artificially restrained connection as well so you can select a few different bandwidths to run the same tests on.

What would help is:
What are the performance measurements you are looking for? Block throughput? Bytes saved? Propagation times?

Mike Hearn <he...@vinumeris.com> schreef op 4 november 2015 12:18:42 CET:

[Peter Tschipper]

the data transmitted is very small, what's needed more than bandwidth restriction is to increase latency ...

[G. Andrew Stone]

I can easily give you a random latency injection patch if it will help your testing.

You received this message because you are subscribed to a topic in the Google Groups "bitcoin-xt" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bitcoin-xt/enX-pRQ46OU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bitcoin-xt+...@googlegroups.com.

[Mike Hearn]

Yeah, there are probably also kernel patches that can simulate packet loss.

It'd be good to adjust the patch to print the amount of downloaded data avoided. That'd probably have more psychological impact that timings.