bitcoinj disconnecting/connecting from node continuously

[Sumanth Yalagandula]

Hi, 
    bitcoinj(0.16.1) is frequently disconnecting/connecting from the node(own node) by printing the below logs. This is causing high CPU utilization in my app and node. Please help solve the issue.

Logs: 

2023-04-03 11:38:51.482 INFO 1 --- [ioClientManager] org.bitcoinj.core.Peer : Peer{[10.17.10.81]:8333, version=70016, subVer=/Satoshi:22.0.0/, services=1037 (NETWORK, BLOOM, WITNESS, NETWORK_LIMITED), time=2023-04-03 11:38:44, height=783756}: Too many pending transactions, disconnecting

2023-04-03 11:38:51.482  INFO 1 --- [ioClientManager] org.bitcoinj.core.PeerGroup : [10.17.10.81]:8333: Peer died      (0 connected, 0 pending, 1 max)

2023-04-03 11:38:51.482  INFO 1 --- [ioClientManager] org.bitcoinj.core.PeerGroup: Download peer died. Picking a new one.

2023-04-03 11:38:51.482  INFO 1 --- [ioClientManager] org.bitcoinj.core.PeerGroup : Unsetting download peer: Peer{[10.17.10.81]:8333, version=70016, subVer=/Satoshi:22.0.0/, services=1037 (NETWORK, BLOOM, WITNESS, NETWORK_LIMITED), time=2023-04-03 11:38:44, height=783756

2023-04-03 11:38:51.482 INFO 1 --- [eerGroup Thread] org.bitcoinj.core.PeerGroup: Peer discovery took 296.0 ns and returned 0 items from 0 discoverers

2023-04-03 11:38:51.482  INFO 1 --- [eerGroup Thread] org.bitcoinj.core.PeerGroup: Waiting 1500 ms before next connect attempt to [10.17.10.81]:8333

2023-04-03 11:38:52.983  INFO 1 --- [ioClientManager] org.bitcoinj.core.Peer: Announcing to btc-node.prod.local/10.17.10.81:8333 as: /bitcoinj:0.16.1/

and then again disconnects and logs repeat. Transactions are downloaded correctly but continuous disruptions are causing high CPU usage. Please help to solve the issue. Thanks.

[googl...@schildbach.de]

Hmm, this could be caused by a high number of unconfirmed transactions in the mempool of your bitcoind right now, plus your node not responding quickly. For whatever reason, bitcoinj has a hard limit of pending requests for transactions set at 100.

Questions: Is your PeerGroup connected to a Wallet? Do you have bloom filtering enabled?

As a quick & dirty fix you could raise the constant: Peer.PENDING_TX_DOWNLOADS_LIMIT

[Sumanth Yalagandula]

Thanks for your response.
 Yes, PeerGrooup is connected to a wallet. Bloom filtering is enabled in Node.

Will increasing PENDING_TX_DOWNLOADS_LIMIT cause any issues in the future?

[Sumanth Yalagandula]

Also can you please let me know the reasonable value that can be used for PENDING_TX_DOWNLOADS_LIMIT. Currently, I see 35K transactions in mempool of node.

[googl...@schildbach.de]

You can set it to any value that your RAM (and Java heap size) allows. I'd probably try 1000, but it's not really reasonable in the sense that in should stay like this.

The real question is why are these requests piling up?

If you use bloom filtering, these 35k pending transactions should be filtered by the server and thus you'd get only a small fraction. Another question, does your wallet have many transactions? How many? I'm asking because the more pubkeys and outpoints are registered in the bloom filter, the more false positives you'd get.

[Sumanth Yalagandula]

I have two wallets running in two different apps(both connected to the same node). One has 2k transactions and another has 24 transactions. Both wallets are loaded with 13k addresses based on our requirements. 

I tried increasing the number to 1k and then 10k but got the same error. When I tried 40k it worked and the error was gone. All my config is the same in testnet and mainnet but I am facing this issue only in the mainnet.  Can you please suggest a way forward? I am not sure if I should keep 40k as the limit as it is too large and may cause other issues.

[Andreas Schildbach]

If "wallets are loaded with 13k addresses" means you derived 13k
addresses from the wallet, that's really a lot! I doubt we have ever
tested it with that many addresses. Under normal usage, one address
would be created for each incoming payment (for receiving) and one
address for most outgoing payments (for change).

In your case, the bloom filter is probably ineffective and you could
just as well receive full blocks.

It's pretty clear we need to improve the handling of those tx inv's /
getdata's. We should probably not throw if that limit (whatever it is)
is reached. I think there is nothing bad about dropping tx inv's – we're
not guaranteed to receive them in the first place. The worst thing that
can happen is that we miss an incoming transaction, but we'll learn of
it with the block that confirms it.

Still, I think you should try to find out why your Bitcoin Core is slow
to respond to getdata requests.

If your heap size allows, you can go with the large limit. But I think
it should only be a workaround.

> <http://10.17.10.81:8333> as: /bitcoinj:0.16.1/

>
> and then again disconnects and logs repeat.
> Transactions are downloaded correctly but continuous
> disruptions are causing high CPU usage. Please help
> to solve the issue. Thanks.
>

> --
> You received this message because you are subscribed to the Google
> Groups "bitcoinj" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to bitcoinj+u...@googlegroups.com
> <mailto:bitcoinj+u...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/bitcoinj/ac81a8a9-738a-4421-a770-345535b871ecn%40googlegroups.com <https://groups.google.com/d/msgid/bitcoinj/ac81a8a9-738a-4421-a770-345535b871ecn%40googlegroups.com?utm_medium=email&utm_source=footer>.