unreadable wallet corruption after crash

[Jen]

I know there was a known bug before with unreadable wallet exceptions caused by failure to connect, but i think this is different. It also may not be a bug, however i see no possible way to handle this error with the built in functionality of master.

So i posted the error below, but basically im running a full pruned store/chain in conjunction with a regtest bitcoind node. On 3 seperate occasions my server crashed, and upon reboot the wallets do not load due to an unreadable wallet exception. if i shutdown the server, or terminate bitcoind, it loads fine. I do push low level/raw transactions with manual input/output designation and manual signing of the TX.I sometimes get a "can not decode DER" error that i plan to fix soon, i feel this may be related because the transactions are being improperly signed (im signing manually), however even so, i should be able to fix this can not connect error regardless of the cause.

If this "unreadable wallet exception" error happens, is there a way to resolve it by getting the block from store and repopulating transactions? or some other way?
my second question is, if this is related to the DER error, is there any other way for this error to occur without mistakes being made on my end? should i be concerned about not being able to resolve the unreadablewalletexception once the root cause is fixed?


the error:

org.bitcoinj.wallet.UnreadableWalletException: Could not connect 94d81e2f8f48b1d897d096f7b7df446aca512c68755825093a256aaac50e8631 to a2388837a858cd0644b5ad660c228726aa1bd36b2724ea4fafa7653974293ef1
    at org.bitcoinj.wallet.WalletProtobufSerializer.connectTransactionOutputs(WalletProtobufSerializer.java:731)
    at org.bitcoinj.wallet.WalletProtobufSerializer.readWallet(WalletProtobufSerializer.java:529)
    at org.bitcoinj.wallet.WalletProtobufSerializer.readWallet(WalletProtobufSerializer.java:438)
    at org.bitcoinj.wallet.WalletProtobufSerializer.readWallet(WalletProtobufSerializer.java:412)
    at org.bitcoinj.wallet.Wallet.loadFromFileStream(Wallet.java:1605)
    at org.bitcoinj.wallet.Wallet.loadFromFile(Wallet.java:1511)

[Bobby_Bouche]

is this on testnet? If so perhaps just delete the wallet and remake?

[Jen]

It is regtest not mainnet or testnet, i can re-create the wallet to resolve the issue, but that is not the point. If it can happen on testnet/regtest enviornment it can happen on mainnet. so while i can resolve it by recreating a wallet (which is what i have done the past 2 times) in production that is not a viable option.

[Jen]

ok im 99% certain it is a bug in bitcoinj, the commit 943a139d0846afb4f1432a1cfcd3949f52dede28 does not solve it (i added changes to wallet.java manually even to check), error still occurs because the spentBy of tx outputs going out from wallet #2 is set, yet wallet #1 has no need to hold that transaction as relevant. on the bug fix it only disconnects the irrelevant TXs inputs and not the spentBy of the outputs on the previous TX that is serialized along with the wallet. As i am using fullPrunedBlockStore as a UTXO provider, something is not adding up.

This is a major issue as wallet movement between 2 bitcoinj wallets seems crucial. I worry about modifying too much in fear of creating vulnerabilities with double spending or allowing invalid transactions to be spent. is there any known work around for this?

[Andreas Schildbach]

Your issue sounds like at some point the transaction is not properly
cloned. As far as I can remember we decided that the same transaction
(as per object identity), or any of the sub-messages (input, output,
...) cannot be part of two or more wallets at the same time.

Before saving your wallet, I suggest studying your two object graphs for
identical references.

> --
> You received this message because you are subscribed to the Google
> Groups "bitcoinj" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to bitcoinj+u...@googlegroups.com
> <mailto:bitcoinj+u...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[Jen]

Thank you for responding! this is very helpful.

So the one thing i found that could be an issue is that i have 3 wallet #1 #2 and #3 number 1 sends some coins to #2 and rest to #3, that means the same tx is on both wallet #2 and #3. Basically, this is not possible since they share the same TX verbatim as received from the broadcast?

This feature seems it could have horrible consequences, considering malicious users could destroy an application by sending a tx with outputs to 2 seperate wallets.

Am i understanding this correctly? we cant have 2 outputs to 2 seperate bitcoinj addresses if they belong to different wallets? what is the purpose here? i have been having good luck with this aside from the "spent by" transaction hash issue. or is this just a bottleneck that came up over time? 

[Jen]

Oh id like to add since i put this in the "bug report" that you directed here, i did "solve" this by covering up the issue, im still on regtest so its hard to tell any issues that could come from mainnet. but i set the wallet to check if the TX has a value sent from us before it recognizes and loads the "spent by tx hash", it does not modify the TX in the utxo provider, but it does modify how the wallets parse the TXs. This may have big performance kicks on a full node, im not sure.

It seems the spent by tx hash is the only non-TX original datapoint, aka the only thing added by bitcoinj not present in the original tx. so part of me thinks this solution is viable.