Design notes for supporting married wallets

[Mike Hearn]

Lately there's been a flowering of many third party risk analysis services. These services typically will ask the user to verify via an SMS when sending coins, or perhaps do some more complex risk analysis like skipping the SMS for known-safe recipients (identified via BIP 70). On IRC Johann asked about how best to integrate them into bitcoinj, so here are some initial design thoughts. Feedback welcome!

I will call these married wallets. I do not use the more common term "multi-sig wallet" because multi-sig is a rather general feature and the details can get complicated quickly: for instance, a common use case for a multi-sig wallet is something like a board of directors in a company controlling funds, which has a rather different structure and API needs from the always-online 1-party risk analysis case. I'm also not calling them risk analyzed wallets because the Wallet code already has a notion of risk analysis, which is about trying to determine whether a pending transaction is likely to get double spent (it's not very useful today to be honest). Also the term married wallet correctly conjures the image of the wallet being required to get the significant others permission to spend money ;) But other suggestions are welcome.

Be warned: this is quite a complex project!

Goals

• Allow a wallet to be linked to a single third party server, such that the server is needed to sign transactions for multisig 2-of-2 or 2-of-3 coins. 
  
  
• Create a framework that allows competing RA providers to provide simple extension libraries for bitcoinj. That is, these services are not expected to provide compatible wire protocols at the moment, just Java classes that wallet app authors can bundle if they wish to support the third party in question.
  
  
• Nicely support TREZOR and other hardware wallets within this framework. This case differs slightly because there's no multisig, but it should hopefully be a subset of the functionality.
  
  
• Allow seamless marriage of existing wallets.
  
  
• Stretch goal: allow a user to undo that upgrade by going back to single-signer mode. This is a form of key rotation.

Non-goals

• Mixed wallets, i.e. that have some local single-signer coins and then start receiving multisig coins such that a spend can contain them in arbitrary combinations. This might turn out to be not very hard to support, but in case it is, this would not be a required feature. Instead apps are expected to do a key rotation from the old keys to the set of new keys shared with the service when it's set up.
  
  
• Complete transparency for wallet authors. It's OK to require wallet authors to do manual work+wiring for each RA service they wish to support. In particular, some RA services may wish to present custom UI or interact with the user on their regular computing device in some way, like to ask the user for the code that was sent to their phone. It's up to wallet authors to implement this UI themselves, and up to the RA plugins to ask the wallet app to do it for them. Likewise, for configuring the plugin, associating it with some server side account etc, all punted to the wallet app developers themselves. BitcoinJ should make the parts it's responsible for easy, but it has nothing to say about all the extra stuff involved.
  
  
• Complete transparency for RA providers. It's OK to require RA providers to make tweaks to their service to support bitcoinj based wallets. For example, we can assume that each user of the RA service gets his own unique HD tree and thus, that the keys used are predictable:  we do not support things like requiring an RPC to the server each time the user wants a new address.
  
  
• Support for or dependency on the Ibrahim/Ali/Princeton threshold ECDSA technique. Although threshold ECDSA is extremely cool and has various advantages, code for it is not yet available and RA providers want to add support sooner than this will be practical. Also the technique is quite complicated and for people to learn to trust it will take some time, vs CHECKMULTISIG which is conceptually very simple. If we can leave doors open to make a future upgrade easier though, we should.
  
  

Overview

Supporting married wallets breaks down into several subcomponents:

1. Supporting P2SH addresses more thoroughly.
2. Supporting a "shadow key tree", this is a watching DeterministicKeyChain (DKC) that is kept in sync with the active DKC.
3. Extending coin selection and adjusting the fee calculation.
4. Making the completion process know how to gather signatures from multiple pluggable sources, let's call them TransactionSigners.
5. Making the completion process asynchronous.

Almost all of this code ends up being done inside the Wallet class.

Better P2SH support

In an ideal world, BIP 70 would be more widely adopted by now and P2SH would not be required or useful. But the payment protocol is still very new and it'll be a long time before most payments use it (assuming success!), so for now most money will enter a married wallet via a static P2SH address.

1. Add a method to Wallet that vends Address objects instead of Key objects. I think most applications actually just immediately turns the returned key into an Address for rendering/display/placement into a p2address output anyway. In a married wallet the freshReceiveKey/currentReceiveKey methods would start throwing exceptions. In a regular wallet the freshReceiveAddress would just return a regular 1AbC style address and in a married wallet it'd return a P2SH address.
   
   
2. Introduce a notion of a shadow keychain, which is a watching keychain that tracks the active keychain. This can all go into KeyChainGroup. The Wallet just forwards address requests to the KCG in the same style as it does currently, and the KCG picks a key from the active keychain (for which we have the private part), and the corresponding key from the shadow keychain, combines them together into a CHECKMULTISIG script, and then hashes it to yield a P2SH address. It must then store the P2SH address internally, so it can use them when constructing Bloom filters, and map the hashes back to the original keys.
   
   Then the serialization format needs to be extended to support saving the shadow keychain. I suggest a small extension to the DETERMINISTIC_KEY protobuf message to note that it's "following" an earlier/later chain, and the details of why it's following can be stored elsewhere.
   
   Note that we prefer to store EC public keys in memory once derived, because the calculations to derive them are slow, and we need fast access to them in order to do things like buildin Bloom filters and identifying relevant transactions out of the false positives. In an ideal world we'd be able to rederive them on demand whenever we wanted, and perhaps one day we'll have an ECC implementation that's fast enough to do that.
   
   
3. Bloom filters in the P2SH setup need to contain two things: the script hash, and the redeemScript itself. So KCG needs to understand that in a setup where one keychain is following the other, rather than ask the two chains for the Bloom filters itself, it should combine the keys into a script, serialize the script, add that to the filter, then add the hash of the script to the filter as well.

By the end of this you should be able to initialise a wallet that can fill itself with transactions sent to P2SH addresses that are backed by valid scripts that the RA server knows how to process.

Note that in the above arrangement, KeyChainGroup ends up with quite a bit of extra logic, and the DeterministicKeyChain classes no longer have all their functionality used: in particular, they are no longer being asked to look up keys given addresses, etc. If we wanted to support mixed wallets where you could get single-signer addresses as well, then that's OK, this functionality would still get used, but if a wallet can only be married then the DKC will be building hashmaps that are never needed, which is a waste. Later on the code can be refactored to save memory.

Extending coin selection and fee calculation

TransactionOutput.isMine() decides whether an output is a candidate for selection. It matches against pay-to-address and pay-to-pubkey script types, asking the wallet whether it owns the key in question.

It needs to be extended to recognise P2SH outputs, and the wallet needs to be able to answer whether it can find both keys in the shadow/active keychain. If it can then it's eligible for spending and should be considered "mine" by Wallet.calculateAllSpendCandidates.

When calculating the fee, we have to estimate how big the input will be before we signed it. This is done in Wallet.estimateBytesForSigning. This has to be extended for P2SH too: for a P2SH output we have to ask the KCG to give us back the original script, then recurse, and add the size of the redeemScript to the size needed to satisfy the redeemScript. Then the method needs to be taught how to read CHECKMULTISIG outputs and understand how many signatures are needed, to find the byte size. Finally, at least a part of this method should probably be moved inside Script itself, as a refactoring.

Pluggable signing

Currently signing is done on Wallet.java line 2292 (on hdw-alpha), where we call req.tx.signInputs from completeTx. The transaction is expected to sign itself, calling back into the Wallet to get the keys it needs.

This design doesn't really jive with what we need these days though. Signing is getting too complicated to be something a Transaction does to itself.

Instead, Wallet should examine the selected outputs and ask the KeyChainGroup to provide the redeemScripts and ECKeys that are involved for each output. Of course some of these will be watching ECKeys but that's OK. Then a signing engine that implements TransactionSigner is given:

• The unsigned Transaction object
• A set of (TransactionOutput, redeemScript, [ECKey]) tuples. For regular pay-to-address/pay-to-pubkey outputs there is no redeemScript of course and just one ECKey. For a multisig coin, there's a redeemScript, one watching key and one full key.
• In future, BIP70 PaymentRequest data. The PaymentSession API isn't really fully integrated into the Wallet API at the moment, so this would require some additional data and replumbing, but knowledge of the recipient of the money is obviously critical to perform any kind of smart/adaptive risk analysis in future.

At this stage in the project, don't worry about long delays. Just block the calling thread until things are done. This is obviously incompatible with real apps that will typically be calling from their GUI thread, but fixing it can come in the next stage.

For some things, like the TREZOR, the signing engine also needs full copies of the transactions being spent, so it can calculate the fee. The API Should probably take that into account.

I'd suggest making a mock server that just always signs immediately for this step, so you can write a signing engine that bundles up the needed data, POSTs it to the server, gets back the signatures, hands them back to the Wallet, etc.

The current code would then be refactored out into a SimpleTransactionSigner object is constructed by default. It would throw an exception if it got handed redeemScripts or >1 ECKey object, because it only knows how to sign for pay-to-address or pay-to-pubkey outputs.

The Wallet can then have an addTransactionSigner method that just adds the given signer to the list. Each signer can be tried in turn, until one is found that doesn't throw any exceptions.

It's expected that TransactionSigner plugins will want to store data, and that the most natural place to do this is in the wallet file. We might need to refactor the serialization/extensions API a bit more, so that there is a TransactionSignerFactory that knows how to take in byte arrays from extensions and create a TransactionSigner implementation that is ready to go. We don't usually want a Wallet that's in a partially constructed/unusable state though, so this has to be done during construction time.

Risk analysis services provide their own implementation of TransactionSigner which may require arbitrary things to be done before they're in a state where they can be added to the wallet. So the interface should have an isReady() method that the Wallet calls, and it will throw an exception if a signer is added before it's ready. A signer must always be ready after deserialization from an existing wallet: it gets one chance to prepare itself, and that's before it's ever added to any wallet. A signer may only be added to a single Wallet class, if it's already owned, attempting to add it to another Wallet throws.

To better support merge avoidance in future, we need to be passing around lists of Transactions rather than a single Transaction for signing.

Making the completion process asynchronous

The above design works OK for very simple services that can do risk analysis quickly and with no user intervention. Real services don't fit yet.

Android poses special design issues. There, we may need to actually shut down the wallet app whilst the process of creating the spend is in process, then load back from disk and carry on later. The TransactionSigner plugin may be designed to work with an additional app via IPC, which Android is very good at, rather than relying on the hosting wallet app to do all the UI work. For instance, if you sign up for MoneyMoat(tm) then you would start by installing the Android MoneyMoat app from the Play Store. Then your Bitcoin Wallet app would notice the presence of the new app and allow activation of the already-bundled MoneyMoatTransactionSigner. But all the UI for interacting with the user, getting the SMS code, signing up for the service and so on would be handled by the dedicated app. In this case some phones won't have enough RAM to run both the wallet and the other app simultaneously, so the wallet would be asked to shut down to free up resources until the user is done.

So the final stage of the project is to make the whole process interruptable, serializable and asynchronous. Wallet.completeTx should be changed to return a new PendingSignatureOp class which can be asked to return a list of unsigned transactions immediately, or signed transactions later. It might be smart to make it implement the ListenableFuture interface so straight-forward ordinary code doesn't get much more complicated, and so we can do things like signing on a background thread later. PendingSignatureOp should be serializable, but it's up to the app to save the data: it won't be saved in the wallet. Outputs are not locked because this would complicate things quite dramatically and the current API doesn't work that way: wallet apps are still expected to ensure there's only one outstanding pending spend being created at once, and apps are expected to be able to tolerate that a created transaction fails to commit because some outputs it used were double spent in the background.

On Android, a good place for the PendingSignatureOp to be saved would be in activity state or in an intent. A mobile wallet app could then send the entire serialized PendingSignatureOp to the third party app via an intent that starts a new activity on the same task stack, thus ensuring that if the user task switches away and comes back later, all the needed data is restored from disk and the back button behaves as expected.

PendingSignatureOp is in fact an interface, with a concrete implementation provided by the RA plugin. Apps are allowed to cast it to the provider specific type in order to do things like add user-specific auth details or whatever else might be needed.

The PendingSignatureOp keeps track of the signatures it's been able to gather and becomes complete when all the signatures needed have been obtained. Once complete, a list of transactions becomes available which can be broadcast to the network, thus committing them to the wallet once a quorum of peers has been seen propagating it. Alternatively, they can be sent to a server via a BIP70 Payment message.

[Dev Random]

Mike, thank you for writing this up! It looks very good.

Here are my comments:

* Input transactions should be provided as you mention, otherwise the
coins can be burned as fees by an attacker.

* Input and output derivation paths should be provided. The input
derivation paths lets the third-party signer use the right signing key.
The output derivation paths are used to prove that some of the coins are
being returned as change, so that just the net amount is used in risk
calculations.

* Both TrustedCoin and ourselves (CryptoCorp) incorporate delays as a
security measure. There can also be delays caused by the user being
called out of band. Not locking outputs will have a UX impact (can't
pay for coffee after paying my rent). Maybe do locking in a second
phase?

* The doc focuses mostly on the 2-of-2 case. The 2-of-3 is basically
just adding another remote (shadow) keychain.

* When generating the p2sh addresses, we should probably sort the
pubkeys in lexicographic order, like TrustedCoin does. This way we
don't have to worry about the order of the keychains.

* When constructing the shadow keychains, we should probably use an
BIP-0032 xpub, since that's the standard way of communicating HD
pubkeys.

As to names, we could call these "married keychains" - since the third
party signer isn't really a full wallet - only one keychain in the p2sh
marriage. I see the user facing piece as the "wallet".

On Fri, 2014-04-04 at 13:19 +0200, Mike Hearn wrote:
> Lately there's been a flowering of many third party risk analysis
> services. These services typically will ask the user to verify via an
> SMS when sending coins, or perhaps do some more complex risk analysis
> like skipping the SMS for known-safe recipients (identified via BIP
> 70). On IRC Johann asked about how best to integrate them into
> bitcoinj, so here are some initial design thoughts. Feedback welcome!
>
>
> I will call these married wallets. I do not use the more common term
> "multi-sig wallet" because multi-sig is a rather general feature and
> the details can get complicated quickly: for instance, a common use
> case for a multi-sig wallet is something like a board of directors in
> a company controlling funds, which has a rather different structure
> and API needs from the always-online 1-party risk analysis case. I'm
> also not calling them risk analyzed wallets because the Wallet code
> already has a notion of risk analysis, which is about trying to
> determine whether a pending transaction is likely to get double spent
> (it's not very useful today to be honest). Also the term married
> wallet correctly conjures the image of the wallet being required to
> get the significant others permission to spend money ;) But other
> suggestions are welcome.
>
>
>
>
> Be warned: this is quite a complex project!
>
>
> Goals

> * Allow a wallet to be linked to a single third party server,

> such that the server is needed to sign transactions for
> multisig 2-of-2 or 2-of-3 coins.
>

> * Create a framework that allows competing RA providers to

> provide simple extension libraries for bitcoinj. That is,
> these services are not expected to provide compatible wire
> protocols at the moment, just Java classes that wallet app
> authors can bundle if they wish to support the third party in
> question.
>

> * Nicely support TREZOR and other hardware wallets within this

> framework. This case differs slightly because there's no
> multisig, but it should hopefully be a subset of the
> functionality.
>

> * Allow seamless marriage of existing wallets.
>
> * Stretch goal: allow a user to undo that upgrade by going back

> to single-signer mode. This is a form of key rotation.
> Non-goals

> * Mixed wallets, i.e. that have some local single-signer coins

> and then start receiving multisig coins such that a spend can
> contain them in arbitrary combinations. This might turn out to
> be not very hard to support, but in case it is, this would not
> be a required feature. Instead apps are expected to do a key
> rotation from the old keys to the set of new keys shared with
> the service when it's set up.
>

> * Complete transparency for wallet authors. It's OK to require

> wallet authors to do manual work+wiring for each RA service
> they wish to support. In particular, some RA services may wish
> to present custom UI or interact with the user on their
> regular computing device in some way, like to ask the user for
> the code that was sent to their phone. It's up to wallet
> authors to implement this UI themselves, and up to the RA
> plugins to ask the wallet app to do it for them. Likewise, for
> configuring the plugin, associating it with some server side
> account etc, all punted to the wallet app developers
> themselves. BitcoinJ should make the parts it's responsible
> for easy, but it has nothing to say about all the extra stuff
> involved.
>

> * Complete transparency for RA providers. It's OK to require RA

> providers to make tweaks to their service to support bitcoinj
> based wallets. For example, we can assume that each user of
> the RA service gets his own unique HD tree and thus, that the
> keys used are predictable: we do not support things like
> requiring an RPC to the server each time the user wants a new
> address.
>

> * Support for or dependency on the Ibrahim/Ali/Princeton

> threshold ECDSA technique. Although threshold ECDSA is
> extremely cool and has various advantages, code for it is not
> yet available and RA providers want to add support sooner than
> this will be practical. Also the technique is quite
> complicated and for people to learn to trust it will take some
> time, vs CHECKMULTISIG which is conceptually very simple. If
> we can leave doors open to make a future upgrade easier
> though, we should.
>
> Overview
>
>
> Supporting married wallets breaks down into several subcomponents:

> 1. Supporting P2SH addresses more thoroughly.
> 2. Supporting a "shadow key tree", this is a watching

> DeterministicKeyChain (DKC) that is kept in sync with the
> active DKC.

> 3. Extending coin selection and adjusting the fee calculation.
> 4. Making the completion process know how to gather signatures

> from multiple pluggable sources, let's call them
> TransactionSigners.

> 5. Making the completion process asynchronous.

> Almost all of this code ends up being done inside the Wallet class.
>
>
>
>
> Better P2SH support
>
>
> In an ideal world, BIP 70 would be more widely adopted by now and P2SH
> would not be required or useful. But the payment protocol is still
> very new and it'll be a long time before most payments use it
> (assuming success!), so for now most money will enter a married wallet
> via a static P2SH address.

> 1. Add a method to Wallet that vends Address objects instead of

> Key objects. I think most applications actually just
> immediately turns the returned key into an Address for
> rendering/display/placement into a p2address output anyway. In
> a married wallet the freshReceiveKey/currentReceiveKey methods
> would start throwing exceptions. In a regular wallet the
> freshReceiveAddress would just return a regular 1AbC style
> address and in a married wallet it'd return a P2SH address.
>

> 2. Introduce a notion of a shadow keychain, which is a watching

> keychain that tracks the active keychain. This can all go into
> KeyChainGroup. The Wallet just forwards address requests to
> the KCG in the same style as it does currently, and the KCG
> picks a key from the active keychain (for which we have the
> private part), and the corresponding key from the shadow
> keychain, combines them together into a CHECKMULTISIG script,
> and then hashes it to yield a P2SH address. It must then store
> the P2SH address internally, so it can use them when
> constructing Bloom filters, and map the hashes back to the
> original keys.
>
> Then the serialization format needs to be extended to support
> saving the shadow keychain. I suggest a small extension to the
> DETERMINISTIC_KEY protobuf message to note that it's
> "following" an earlier/later chain, and the details of why
> it's following can be stored elsewhere.
>
> Note that we prefer to store EC public keys in memory once
> derived, because the calculations to derive them are slow, and
> we need fast access to them in order to do things like buildin
> Bloom filters and identifying relevant transactions out of the
> false positives. In an ideal world we'd be able to rederive
> them on demand whenever we wanted, and perhaps one day we'll
> have an ECC implementation that's fast enough to do that.
>

> 3. Bloom filters in the P2SH setup need to contain two things:

> * The unsigned Transaction object
> * A set of (TransactionOutput, redeemScript, [ECKey]) tuples.

> For regular pay-to-address/pay-to-pubkey outputs there is no
> redeemScript of course and just one ECKey. For a multisig
> coin, there's a redeemScript, one watching key and one full
> key.

> * In future, BIP70 PaymentRequest data. The PaymentSession API

> --
> You received this message because you are subscribed to the Google
> Groups "bitcoinj" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to bitcoinj+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
Miron / devrandom

[Mike Hearn]

Thanks Miron,

Agree on all points except this one:

On Fri, Apr 4, 2014 at 7:32 PM, Dev Random <c1.dev...@niftybox.net> wrote:

* Both TrustedCoin and ourselves (CryptoCorp) incorporate delays as a
security measure.  There can also be delays caused by the user being
called out of band.  Not locking outputs will have a UX impact (can't
pay for coffee after paying my rent).  Maybe do locking in a second
phase?

I question the value of delays, though it's not really my place to say what you should or should not do. But locking outputs would expose the underlying protocol to users in the UI, which we should avoid as much as possible. Even the existing pending/confirmed stuff is bad enough. But with long delays on a spend, you might find your spendable balance hit zero because the change isn't available yet, even if you spent only a small amount. There's no good way to explain this to end users. So I feel like having overlapping spends can never work well and it should be up to the RA service to ensure the user is never waiting to make a spend.

 

[Dev Random]

You are right, locking outputs is not a good approach. Here's a better
scheme: index pending authorizations on the set of outputs (address,
amount), excluding fees and change. This way transactions can be
recreated from scratch after the delay passes, but the user intent still
applies to the new transaction so the authorization is valid. Does this
sound reasonable?

I think this can be implemented by the app without requiring API
support.

By the way, the value of delays is in letting the user know of unusual
activity through multiple channels and giving them a chance to cancel.
For example, if your savings are about to be drained, your spouse and
accountant might be notified and a 48 hour delay started. The delay
gives people a chance to abort or use the 3rd key (recovery) to move the
funds to a safe location in case the transaction was actually a result
of the main wallet being compromised.

[Jim]

I think any sort of earmarking of tx outputs as "do not use" for long periods will confuse users, especially the change output.


For SPV wallets change availability is critical for the wallet to be usable. If the change output of a married tx is unavailable / marked as 'reserved' the user could sit there twiddling their thumbs with all their funds locked up until a decision was made.


It was painful enough when, ages ago, users had to wait for the next block to spend change. That was only ten minutes compared to the 48 hours we are discussing now.

http://bitcoin-solutions.co.uk

[Dev Random]

Jim,

The new scheme I'm proposing doesn't involve locking. I apologize I
didn't describe it clearly.

What I'm suggesting is that the 3rd party signer internally indexes
authorizations by the *new* outputs. This is a sorted set of
destinations for the BTC - [<address, amount>, ...].

By indexing authorizations to this set, the transaction can be recreated
with new UTXO inputs at any time, so no locking is required.

Does this make sense?

--
Miron / devrandom

[Jim]

Yes that makes sense.

Another way you could do it (at the cost of an extra tx fee) is to create an extra tx that has an output of the perfect size. Here's an example:

Say the co-signing service knows the user's master public key and has a rule "Every spend away from the wallet of 1 BTC or more gets delayed 24 hours).

The user has a single tx output of 100 BTC.

User creates 2 tx:

** tx 1**
100 BTC output -> tx output A for 1.001 BTC
tx output B (change) for 98.998 BTC
(0.001 BTC fee)

** tx 2 **
tx output A (worth 1.001 BTC) -> 1 BTC to an address outside the wallet
(0.001 BTC fee)


The co-signing service signs tx 1 IMMEDIATELY, as it knows all the outputs are in the user's wallet.
The tx 2 it will only co-sign after 24 hours.

From the user's wallet's perspective, the first output will confirm as normal - they can spend their 98.998 BTC change immediately. The second tx is marked as "waiting until now+24 hours for approval". As it is the perfect size there is no change to worry about its status.

The second tx only has one input and one output. Pretty unusual !

[Johann Barbie]

 Hi Mike,

Thanks for the writeup. I tried myself at improving P2SH support: https://github.com/Btc4All/bitcoinj/commit/5d2c81c3b8851e98a3906d849b14476b861a8390

Half through it I'm battling with Bloom filters, and a potentially misconstructed P2SH address. Here a few things I was forced to do:

- when serialising shadow keys I had to reference the master chain somehow, I chose the public key hash of the masters extended public key.

- when constructing bloom filters I had reference NetworkParameters within the DKC, to be able to generate the addresses.

- because a master chain has to know all shadow chains to pull from, I had to reference the group, so I can delegate to it for selection.

In general I got the feeling that it would make sense to have a keychain interface and have specific implementations of GKC, DKC, watch-only KC and Master/Shadow KC. Encryption, serialisation and bloom-filtering could be extracted and implemented in a visitor pattern?

regards

Johann

[Mike Hearn]

Hi Johann, I will take a look later. However params are not necessary for bloom filter construction because what's added to the filter are the hash160s and not the checksummed and versioned address bytes.

There is already a KeyChain interface. DKC delegates to BKC for parts of serialisation, filter construction and so on. I don't know if a different  arrangement would work better for this use case - try it and see I guess.

Please forgive me if my responses are slow. I have my own project too and there is still much work to be done on basic HD wallets. If you're working on this full time you are very likely to run ahead of me.

[Mike Hearn]

Hi Johann,

I haven't done a full review of the code as it's clearly not ready yet. Just a few things to bear in mind as you work:

1) Watch your formatting and spelling. Your current code is all over the place: are you using an IDE? If not I recommend IntelliJ. Please try hard to match the regular code style you see around you, it's basically the normal Sun/Java style with a few tweaks:

https://code.google.com/p/bitcoinj/wiki/CodingConventions

2) Please avoid pointless renaming of things, e.g. Wallet.keychain -> Wallet.keychainGroup. Either name works but it pollutes the diff and makes it harder to read if you do things like that.

3) Don't copy and paste JavaDocs unless you're going to actually update them. I noticed a few things you added had JavaDocs but they were simply wrong or misleading because they were copied from elsewhere.

You probably want to implement a shadow keychain as a descendent of DeterministicKeyChain rather than trying to put all the functionality into DKC.

Ultimately, have fun and good luck with this, but it's a large complex project and I'm not going to merge code to such a critical part of the codebase unless I'm happy with it. Your best bet is to start out by making a series of small changes so you can get used to passing code review and getting changes into the core, then work your way up to bigger ones. I appreciate that might be frustrating if you've got limited time/funding, but in the past I accepted code I wasn't totally happy with and it sometimes ended up being buggy or hard to maintain. So I won't do that anymore.

[Johann Barbie]

Hi Mike,

thanks for the comments. I'm aware of the issues with the committed code. I took too big of a bite and then got all confused, that's why I didn't send a pull request. It was very educational though, and also disillusioning. Would I sound pessimistic if I say that we are at least a few month away from production-ready multi-sig code?

I'll set out with smaller steps and try to produce something acceptable.

cheers

Johann

[Mike Hearn]

I think it's a rather large project, yes. I'm not sure about a few months, but given that plain old single-sig HD wallets aren't launched yet, it could be.

These things are always more complex than they look, especially in the decentralised world.

[Kosta Korenkov]

Working on fee calculation, I'm kind of stuck on Wallet.estimateBytesForSigning. It is not clear for me what we are measuring here exactly. Why this method returns pubkey.length + 75 bytes for send-to-address case or 74 bytes for send-to-pubkey? If we're calculating TxIn size, shouldn't this be calculated as (schematically): length(previous_output_reference)+script_length_byte+length(script_sig)+sequence_length?

I admit I don't have solid theory background here, so I'd appreciate any help

[Mike Hearn]

Working on fee calculation, I'm kind of stuck on Wallet.estimateBytesForSigning. It is not clear for me what we are measuring here exactly.

Size of the scriptSig.

 

Why this method returns pubkey.length + 75 bytes for send-to-address case or 74 bytes for send-to-pubkey?

A scriptSig for a pay to address output contains the signature (~75 bytes), plus a copy of the public key.

A scriptSig for a pay to pubkey output only contains the signature (75 bytes).

[Luke Dickinson]

Hello, I understand that married wallets are not complete and maybe that is the issue I am running into, but I figured I'd try asking here.

I have been experimenting with the married wallets idea for couple days now. My goal is to get a multisig 2-of-2 transaction to work. I have got a multi sig transaction to work, using the contracts tutorial on the bitcoinj page, but I as I was using a regular wallet, the address it self was not a multisig address, which was going to lead me to problems.

So If i understand right, the married wallet allows you to create an address that people can send to normally, but to send out, you would need two signatures. Here is where my issue is. I got it working up to signing the transaction.

I can create the addresses, build a transaction to send the bit coins out, but can't sign it because, I don't have a private key. When I search the wallet with the public keys of the transaction, I find one of the eckeys (this is good), but it has only the public part of the key, and not the private part (this is bad), so I cannot sign the transaction. The Eckey's parent keys, do have a private and public key.

So my question is, what am I to do to get the signatures? Or am I going about this the wrong way?

I actually have another question, that might should be another post, but might be related as well. When I am creating this wallet, I create a master key with HDKeyDerivation.createMasterPrivateKey, which I then import into a new wallet. Then I the wallet to updateToDeteministic(). Then (here is my questionable action, where I think I am doing it wrong), I need to create a watching key, so I can call wallet.addFollowingAccontKeys(watchingKey) in order to create a married wallet. The red flag that I am doing this wrong is, In order to create a watchingKey, I need to create a new Wallet with another master private key, (do importKey with the masterKey, then upgradeToDeteministic, before I can call serverWallet.getWatchingKey() without errors being thrown.

Wouldn't it sort of defeat the purpose of having a multisig 2-2 sort of thing, if one of the applications already had both master keys? This is why I think I am doing it wrong. OR is there a way to serialize the watchingKey so I could send it from another application? I have not been able to successfully serialize an eckey before.

Thanks!

[Mike Hearn]

Hello, I understand that married wallets are not complete and maybe that is the issue I am running into, but I figured I'd try asking here.

Yes, indeed. It's not that you can't do it, but the convenience code to do it for you isn't there yet (nor is any documentation or tutorials).

 

Wouldn't it sort of defeat the purpose of having a multisig 2-2 sort of thing, if one of the applications already had both master keys? This is why I think I am doing it wrong.

Yes, you are :) I'm afraid you seem to have some deep conceptual confusion. The goal of a multisig transaction is that the keys are in different places. It does indeed defeat the point if both keys are in the same app at the same time, which is why your code isn't working well.

What actual problem are you trying to solve here? Do you want to build a risk analyzing wallet service, do mobile authentication, etc ... ?

[Luke Dickinson]

Hey, 

First, thanks for being quick and trying to answer my question. For now, as like you said the married wallets aren't ready yet, Ill go another route.

But I would like to let you know what I was trying to do, and how far I got. (maybe it will be useful for someone reading in the future).

I believe I was not as clear as I could have been in what wrote last time. 

What I was trying to do is have a client be able to receive bitcoins to an address, but not be able to send them without another authentication from another source (the same idea as mobile authentication. A server with rules to validate someones purchase).

I want to have the private keys (key sets, as in the married wallet), separate, only sharing the public keys. I had this done, except I couldn't correctly create p2sh address (I mean an address where two or more keys are required to send the bitcoins)

I tried the married wallet route, and got close, but was stopped at a couple points. 

first being, bitcoinj would throw an error if I tried to create a married wallet, with a watching key containing only the public master key. (This is what I meant with one application having both the master keys). I am unsure why.

Another route might might be a possible by serializing the watching key and loading it in, which I see there may be support for, but I have not figure it out yet. 

the second being, after I do have a married wallet set up, I need to set up a transaction, sign it and then send it.

In order to sign it, I would need two private keys, or signatures. I would assume, I should get one of the private keys from the wallet, and one from another source.

When I searched the wallet for keys, I found one of them, but it only had a public key, and no private key.

So it would be impossible to send the transaction, even if I get the 2nd signature from the remote wallet.

Since my last post, In order to test my curiosity, I made an edit to the bitcoinj code, in DeterministircKeyChain. on about line 888. by commenting out  "key = key.getPubOnly();" in maybeLookAhead,

I was then able to get the local private key I needed. With my 2nd remote private key/signature I imported, I was able to go through the whole, receive, and send process successfully with a married wallet.

I do understand though, that line is probably there for a reason, and this will not work for me in the long term. 

Thanks again

Luke

[Mike Hearn]

On Tue, Jul 8, 2014 at 8:25 PM, 'Luke Dickinson' via bitcoinj <bitc...@googlegroups.com> wrote:

What I was trying to do is have a client be able to receive bitcoins to an address, but not be able to send them without another authentication from another source (the same idea as mobile authentication.

OK, then I take it back, this is indeed what married wallets are all about.

 

first being, bitcoinj would throw an error if I tried to create a married wallet, with a watching key containing only the public master key.

Hmm, well this should work - take a look at WalletTool and the marry command it has. Are you doing the same thing? What error did you get?

Given the total absence of documentation for this bleeding edge feature it would not surprise me if you had gone slightly wrong somewhere through no fault of your own.

 

Since my last post, In order to test my curiosity, I made an edit to the bitcoinj code, in DeterministircKeyChain. on about line 888. by commenting out  "key = key.getPubOnly();" in maybeLookAhead,

As an optimisation leaf keys all have their private key bytes removed, because deriving them on demand is so fast. When you have a DeterministicKey calling getPrivKey() should calculate the private bytes on the fly from the parent keys, so I'm not sure how you got into this state. Additionally, DeterministicKey.sign will do the right thing for encrypted wallets.

How exactly are you trying to calculate the signature? Can you show your code?

[Luke Dickinson]

Well for starters, the DeterministicKey deriving its own private key is awesome. I did not notice that it would do that automatically when .sign() is called. My mistake there was I had been finding the key I needed to sign with with wallet.findKeyFromPubKey(), which returned a ECkey. Today, after casting it into a DeterministicKey, everything works great. The best thing about this is now I am back to the master branch.

Ok (I am editing my code as I write this), I was not creating a married wallet in the same way at the walletTool.

I had been creating a "serverWallet," getting a watching key, and using that. The issue seemed to be when I created the watching key from a wallet created in this a round about way.

I created the wallet by importing a deterministicKey created by, HDKeyDerivation.createMasterPublicKey(). Then when I did get watching key, or maybe it was when I tried to put it into the wallet, it would throw some exception.

When I created the wallet by importing a deterministicKey created by, HDKeyDerivation.createMasterPrivateKey(), it worked fine, so that's where I got my confusion.

Ether way, using deserializeB58/serializeB58 like the walletTool does, it works great. makes a lot more sense too.

Looks like my personal next couple steps, is to look through the deterministicKey class, and see what it can all do now. Also maybe the tools package, I didn't know that existed. This would have saved me a couple headaches. I had been using an older build of master until last week.

About how I am signing the has been through a few iterations, from which started with the example on the website about on multisig contracts. I can't say this is the cleanest way to do it, but I am happy to share what I have working for me.

Currently I realize I have a bug in my code, so later when I have fixed it, I will post again, with a bit code of how I am creating the transaction all the way to where I am signing and sending the transaction.

Here I have where I am just getting the signature(s) that exist in the local wallet. (When I originally was using this, I had multiple signatures coming from the local wallet)

This is with the change I got from your last suggestion

    private void addInWalletSignatures(ArrayList<String> publicKeysNeededToBeSigned, ECKey.ECDSASignature[] signatureArray, Sha256Hash sighash) {

        int i = 0;

        for (String publicKeyNeededToBeSigned : publicKeysNeededToBeSigned) {

            byte[] publicKeyBytes = Utils.parseAsHexOrBase58(publicKeyNeededToBeSigned));

            try {

                ECKey eckey = wallet().findKeyFromPubKey(publicKeyBytes);

                if(eckey != null)

                {

                    DeterministicKey deterministicKey = (DeterministicKey) privateECKey;

                    ECKey.ECDSASignature ecdsaSignature = deterministicKey.sign(sighash);

                    if(verifyTransaction(ecdsaSignature, deterministicKey.getPubKey(), sighash)) {

                        signatureArray[i] = ecdsaSignature;

                    }

                }

            }

            catch (IllegalAccessException e){}

            i++;

        }

    }

Its not pretty, but this is working. the signatureArray once it is full, I use it to create a inputScript.

    private Script getInputScript(ECKey.ECDSASignature[] signatureArray, byte[] program)

    {

        ImmutableList.Builder<TransactionSignature> buildingSignatures= ImmutableList.builder();

        for (int i = 0; i < signatureArray.length; i++)

        {

            if(signatureArray[i] != null) {

                TransactionSignature transactionSignature = new TransactionSignature(signatureArray[i], Transaction.SigHash.ALL, false);

                buildingSignatures.add(transactionSignature);

            }

        }

        List<TransactionSignature> signatureList = buildingSignatures.build();

        //Script inputScript = ScriptBuilder.createMultiSigInputScript(signatureList);

        Script inputScript = ScriptBuilder.createP2SHMultiSigInputScript(signatureList, program);

        return inputScript;

    }

 

My current server is actually not using bitcoinj, and I am able to get the signatures I need for the server through a c++ library. my client asks the server for the signatures.

When I get my code working I would be happy to show what else I am doing to get from step A to step Z.

[Mike Hearn]

On Thu, Jul 10, 2014 at 3:02 AM, 'Luke Dickinson' via bitcoinj <bitc...@googlegroups.com> wrote:

I needed to sign with with wallet.findKeyFromPubKey(), which returned a ECkey. Today, after casting it into a DeterministicKey, everything works great. The best thing about this is now I am back to the master branch.

Glad to hear it's working now, but there must have been some other issue. Casting the object to the subclass will not change its behaviour like that, as all methods in Java are virtual. What you got back from findKeyFromPubKey was always a DeterministicKey even if the type was ECKey. So I'd try taking out the cast and seeing if it's still working.

    private Script getInputScript(ECKey.ECDSASignature[] signatureArray, byte[] program)

Take a look at ScriptBuilder for utility methods that do this.

[Luke Dickinson]

Yes, Mike you were right. I must have some other issue, and when I fixed that I also thought I needed to change it into a DeterministicKey. Java isn't my strongest language, but I am learning.

I did some more clean up of my code and had realized that many of the issues I came across were because of different sized transactions, (more than one input, etc).

Thanks again for all the help, I now have a solution from A to Z.

Did you want to a copy of the section of the code I am using to use the married wallet, create the transaction, sign it, and send it? And if so, where would you want me to post it?

I am sure there are some things that could be improved in it, but I'm fairly sure its working good now.

As a plus for me, if you saw things in it that I am doing, that bitcoinj already does, I could factor out some of my code.

As a plus for others and maybe you, it may give you an ideas of how the married wallet can be used.

I appreciate all the help you've given me this week.

Luke

[Nate B]

** Pluggable Signer **

Would Married Wallets support the implementation of a signer that Wallet Protection Services can offer their users to "plug in"?

I have heard the term "pluggable signer" before, but I'm not sure I understand it, entirely.

[Mike Hearn]

Not offer to users but to developers. A user-exposed plugin system is probably too ambitious for now.

--

[Jim]

RE: pluggability.

For MultiBit we decided that the runtime loading of classes was too much of a security risk to implement. There isn't really any protection in the JVM once a class is running.

It would be pretty easy to dress up a wallet stealer as 'some useful little widget' and get it running on lots of users' machine is we had the runtime loading of classes.

--
http://bitcoin-solutions.co.uk

[Mike Hearn]

For MultiBit we decided that the runtime loading of classes was too much of a security risk to implement. There isn't really any protection in the JVM once a class is running.

Yes that's certainly a risk. A bigger and more obvious problem is that the plugins would probably need to display service specific UI and that's not something that's easy to do cross platform. A simple per-wallet integration is good enough for now, if you can add a new risk analysis provider in say an hour or twos worth of work, that sounds OK to me. If need be we can standardise some protocols and more stuff later.

[Mike Hearn]

Making the completion process know how to gather signatures from multiple pluggable sources, let's call them TransactionSigners.

troggy has completed this step too, leaving only:

Making the completion process asynchronous.

left! 

And actually you could probably make a wallet that uses a remote server already now, if you're willing to block a thread and know your app won't die during the signing process (so: not on Android).

[Kosta Korenkov]

Current implementation doesn't allow to specify multisig threshold. Only majority of keys required to spend (2-of-2, 2-of-3, 3-of-4 and so on). To support mutisigs like 3-of-3, threshold needs to be stored in KeyChainGroup. Is there an easy way to serialize such a value? Most streamlined way is to introduce new protobuf message for KeyChainGroup to keep threshold and keys, but this would not be backward compatible with existing wallets.

[Mike Hearn]

I guess it can be serialized to the DETERMINISTIC_SEED key entry as that's where the rest of the data is. Or yes just introduce an extension to the protobuf format. It can be easily backwards compatible for existing non-married wallets.

On Sun, Aug 24, 2014 at 4:01 PM, Kosta Korenkov <7r0...@gmail.com> wrote:

Current implementation doesn't allow to specify multisig threshold. Only majority of keys required to spend (2-of-2, 2-of-3, 3-of-4 and so on). To support mutisigs like 3-of-3, threshold needs to be stored in KeyChainGroup. Is there an easy way to serialize such a value? Most streamlined way is to introduce new protobuf message for KeyChainGroup to keep threshold and keys, but this would not be backward compatible with existing wallets.

--

[Kosta Korenkov]

It would be logical to keep keys in introduced KeyChainGroup proto instead of the Wallet. Doesn't it break backward compatibility? Introducing KeyChainGroup proto would make serialization messier (and it is not perfect already) and I would prefer not doing this.

--
You received this message because you are subscribed to a topic in the Google Groups "bitcoinj" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bitcoinj/Uxl-z40OLuQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bitcoinj+u...@googlegroups.com.

[Mike Hearn]

How do you mean, keep it in introduced KeyChainGroup proto. You mean in a new file?

[Kosta Korenkov]

I mean create a KeyChainGroup message in wallet.proto. It would be referenced in Wallet message then

[Mike Hearn]

Sure, a separate message is fine. Probably don't call it KeyChainGroup though but rather name it after the contents of what data it describes.

[ia...@iang.org]

On Friday, April 4, 2014 12:19:30 PM UTC+1, Mike Hearn wrote:

Lately there's been a flowering of many third party risk analysis services. These services typically will ask the user to verify via an SMS when sending coins, or perhaps do some more complex risk analysis like skipping the SMS for known-safe recipients (identified via BIP 70). On IRC Johann asked about how best to integrate them into bitcoinj, so here are some initial design thoughts. Feedback welcome!

I will call these married wallets. I do not use the more common term "multi-sig wallet" because multi-sig is a rather general feature and the details can get complicated quickly: for instance, a common use case for a multi-sig wallet is something like a board of directors in a company controlling funds, which has a rather different structure and API needs from the always-online 1-party risk analysis case. I'm also not calling them risk analyzed wallets because the Wallet code already has a notion of risk analysis, which is about trying to determine whether a pending transaction is likely to get double spent (it's not very useful today to be honest). Also the term married wallet correctly conjures the image of the wallet being required to get the significant others permission to spend money ;) But other suggestions are welcome.

Just commenting on the terms here:   the more common terms in banking would be dual factor or dual channel authentication, leading up to dual authorisation.

Dual factor authentication would be like the RSA SecureId number generator, which is "something the user has" aka a second factor.  However, in the advent of MITB attacks around 2007, merely authenticating the user was found to be weak, because the number generated by the person could be used to authenticate any transaction.

The Europeans then moved on to dual channel authorisation, which was the sense of sending the entire transaction across SMS to her phone, and displaying a code (hash over transaction) to authorise that transaction only.  This then required the attacker to take over both the browser and the phone, a higher order of task, albeit an attack that has been tried in recent times.

With that in mind, moving on to the term married wallets.  Such a term does not reflect the European model above, but it does come a lot closer to models I have seen/anticipated in Africa.  There, instead of the user having a laptop plus phone, the user has one phone and works closely with other people with other phones.  So, there is less point to dual factor authentication, and dual channel authorisation doesn't make sense on the face of it because we're already in the domain of multiple parties.  E.g. marriages.

But going beyond that, there are specific arrangements that they are familiar with there:  One model is to transact through the treasurer who has the job of deploying the actual funds on behalf of the owner.  Another is to share with best friends on some issue, so, working with close peers.  What might surprise here is that one thing that doesn't happen so often is trust between spouses.  In typical fashion, the financial trust is more between peers on a horizontal basis (gender, school, age, tribe) less on a household basis because the peers hold stronger shared beliefs whereas the households are often as separated by different beliefs along those lines.

iang

ps; just as a personal comment, I find the use of a mailing list for project descriptions to be quite confusing.  I had somehow got it into my head that this post was from the last week, was working through it, and now I've discovered that there is a big long thread and it was 6 months ago.... hence, I don't know now whether my comments make any sense!  A wiki format would help a lot to organise this project and others forward into time.