BloomFilter issue with segwit addresses

[Anton]

I'm currently experimenting with making a native segwit wallet here: https://github.com/btcontract/bitcoinj by merging current master with segwit address support and a https://github.com/bitcoinj/bitcoinj/tree/segwit branch.
I've got to a point where wallet can receive and send transactions but have stumbled into an issue which may become relevant once bitcoinj master starts supporting native segwit wallets.

The issue is that once a transaction without change (the one which empties a wallet) is sent out it won't be seen by wallet again after blockchain rescan or restoring from seed.

As far as I understand the issue is two-fold:
1. tx scriptSig is empty and does not contain a relevant `pubKey` (it's located in a witness).
2. none of tx outputs contain `pubKeyHash`es relevant to a wallet.

Any thoughts if my understanding is correct and what can be done about it?

[Andreas Schildbach]

[+cc Jonas Schnelli, as I discussed this topic with him last year.]

Hmmm, indeed that sounds bad. I totally overlooked this case. Seems like
BIP37 needs to be amended to include the witness in filter matching.
However, on nodes that prune the witnesses this won't work either.

P2WPKH-P2SH doesn't seem to have this problem, as it contains the
20-byte key hash in its scriptSig.

Of course as a workaround you could make sure there is always a change
back to you. Maybe to not waste coins/UTXOs create an OP_RETURN
containing one of the pubkeys?

I wonder if the new client-side filtering (BIP157) has the same problem?

> --
> You received this message because you are subscribed to the Google
> Groups "bitcoinj" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to bitcoinj+u...@googlegroups.com
> <mailto:bitcoinj+u...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[Anton]

Thanks for the suggestion,

tx.addOutput(Coin.ZERO, new ScriptBuilder().op(OP_RETURN).data(pubKeyHash).build());

does the job.

Of course, I'm interested to know whether a less hacky solution exists.

вторник, 10 апреля 2018 г., 18:42:10 UTC+3 пользователь Andreas Schildbach написал:

[Andreas Schildbach]

Just for clarification, did you test both cases against real bitcoind?
Or did you just look at the specification, as I did?

API-wise, you can use ScriptBuilder.createOpReturnScript(). Maybe you
want to randomize which pubkey is used in the OP_RETURN data, in case
there are multiple candidates.

Maybe you'd like to publish your finding on the bitcoin-dev mailing
list? I'm sure they are interested, because it affects SegWit efficiency
and maybe adoption.

> <https://groups.google.com/d/optout>.

[Anton]

Yes, I've tried OR_RETURN case with emptying tx on a testnet and segwit wallet does correctly catch it on blockchain rescans.
As far as I understand the best choice for OP_RETURN `pubKeyHash` is one of the signing keys used in tx inputs (in order to not give away more info).

OK, I'll definitely put this on mailing list.

вторник, 10 апреля 2018 г., 23:43:05 UTC+3 пользователь Andreas Schildbach написал:

[Andreas Schildbach]

Investigating more into this... actually the correct way to deal with
bloom filters is inserting UTXO OutPoints into the filter.

And bitcoinj already has a mechanism for this, which is currently only
used for watched scripts.

Can you the top commit from my branch?
https://github.com/schildbach/bitcoinj/tree/p2wpkh-bloom-filter

(Obviously for isRequiringUpdateAllBloomFilter() we need a more
appropriate solution, rather than just returning true. But for a quick
test it should do.)

I'm looking forward to your findings.

On 04/10/2018 06:51 PM, Anton wrote:

> <https://groups.google.com/d/optout>.

[Anton]

Yes, changes introduced in https://github.com/schildbach/bitcoinj/commit/66d76131a6b453b27d1344a71665b6050c201c43 also fix this issue.
Just in case, I see "plus the BLOOM_UPDATE_ALL flag set" in commit comments but no such changes in 66d76131a6b453b27d1344a71665b6050c201c43 itself, am I missing something?

суббота, 14 апреля 2018 г., 3:40:52 UTC+3 пользователь Andreas Schildbach написал:

[Andreas Schildbach]

Thanks, great to hear!

If isRequiringUpdateAllBloomFilter() returns true a BLOOM_UPDATE_ALL
filter is constructed. It won't stay like this though. I guess in future
we will ask a Wallet's KeyChainGroups if at least one of them requires
an 'update all' filter.

[Andreas Schildbach]

I've worked the learning from this thread into this PR:

https://github.com/bitcoinj/bitcoinj/pull/1563

I invite you to use the code from the PR. Look at the provided test
cases for how to derive segwit addresses. Feek free to report and
problems you have, I'm interested in how it works for you -- especially
the bloom filter problem.

[Anton]

Thanks! Upon further testing it turned out an issue still remains in one specific case.

Scenario 1: empty wallet in a single tx, wait until it gets at least one confirmation, rescan blockchain: everything is fine, wallet sees that tx after rescan.

Scenario 1: empty wallet in a single tx and start blockchain rescan immediately, such that after rescan is done a tx is still unconfirmed: wallet won't ever see it but this can be fixed by doing a second rescan once missing tx gets confirmed (this should be checked externally).

[Andreas Schildbach]

On 06/06/2018 07:31 PM, Anton wrote:
> Thanks! Upon further testing it turned out an issue still remains in one
> specific case.
>

> Scenario 1: empty wallet in a single tx and start blockchain rescan
> immediately, such that after rescan is done a tx is still unconfirmed:
> wallet won't ever see it

I'm wondering is this actually a regression related to SegWit? Did you
try the same in the non-SegWit case?

> but this can be fixed by doing a second rescan
> once missing tx gets confirmed (this should be checked externally).

It would be bad if we'd need to rely on a second scan. And on an
"external" (centralized?) check.

[Anton]

Yes, seems like it is. Never happens in non-segwit case, consistently happens in SegWit case.

[Andreas Schildbach]

Damn, in this case we need to find the root cause of this issue.

It might be related to the following issue which we had to deal with
when implementing BIP37:

* Received pending relevant tx which was thrown away (I guess because of
risk analysis)
* Same transaction was confirmed, partial merkle tree was received but
not the transaction itself again. This is due to how bitcoind works, it
sends only txns it hasn't sent over the same connection.
* Result: Tx was missing from the wallet and never received again
(unless the wallet was replayed).

Have you looked closely at the logfile?

[Anton]

I was able to isolate an issue, here's a transaction sent at Jun 8 whose 2nd output of belongs to a wallet:

https://www.smartbit.com.au/tx/6e0a1a32c981703b4df3210deabd1c9cc87ac8683fdfc904f6db7db443b3bb2b

Here's a second transaction sent at Jun 11 which spends that output without change:

https://www.smartbit.com.au/tx/513833bad28407a1a63740a136d5534603f3e6a68af69678ebc0cec273079176

If I start a blockchain resync and do nothing, then I can see an incoming tx on Jun 8 but no spending tx is found and

суббота, 23 июня 2018 г., 21:14:03 UTC+3 пользователь Andreas Schildbach написал: