You're correct that the next logical step here is to extend btcwallet to
allow a watch-only mode. Digging in one level deeper, this entails extending
the waddrmgr[1] to allow xpub/xpriv key import. As is today, the waddrmgr is
only able to be _converted_ into a watch-only wallet via this method [2].
However, this requires the wallet to already be fully initialized with the
routine private key information, which is then expunged within the method,
leaving only public key data remaining in the database.
I've thought about the process to allowing lnd to better support external
signing abstractions (as well as just general extensions) in detail. In the
remainder of this email, I'll walk through what in my eyes is the ideal path to
achieving this goal. During the initial design phase of lnd, decisions were
made as to ensure a clear path existed to further signing abstractions, even if
the initial version didn't support it fully. The current path may not be very
clear as is to an outside party, but all the pieces are there, they simply need
to be assembled together. Hopefully after reading this email, the path is clear
to you and the other developers which are looking to extend lnd to better
support external signing abstractions (and beyond!).
The path to implementing proper watch-only support (xpub/xpriv) import to
the waddrmgr looks something like the following:
* create a _new_ nested bucket hierarchy that mirrors the main top-level
bucket structure [3] kin the database
* for each new xpub/xpriv imported into the database, a new sub-bucket
hierarchy will be created allowing all the normal functionality
(sub-scoped waddrmgr's, etc)k
* extend the main public exported struct to expose [Export+Import]Xpub/Xpriv
methods likely using the existing btcutil.hdkeychain [4] data structures
* Import keys can either use the same base set of encryption keys for
storing data at rest, or ideally use a disticnt set of keys which allows
easy import/export of the database state into other wallet instances
With the above done, it would be possible to create a new instance of btcwallet
backed with the full private keys derived from a seed, then use the exported
xpub information to import the public data into a newly created btcwallet
instance. The instance with private keys actually doesn't need the shell of
btcwallet at all, and can simply be a waddrmgr instance. From there, the
existing public methods of btcwallet can be used to perform coin selection to
construct an unsigned transaction, with the transaction being passed to the
fully-loaded waddrmgr instance which is then able to populate the relevant
witnesses, etc.
However, one question that arises from the workflow above is: how do you
communicate _which_ keys to use when signing? lnd already has a nice
abstraction for that that we use within the project [5]: KeyDescriptors! These
are similar (but predate) bitcoind's concept of address descriptors, but are
concerned with only proper key derivation rather than generic output script
satisfiability. All SignDescriptors [6] within `lnd` already carry embedded
KeyDescriptors, so all the piece of the puzzle fit rather nicely!
Once btcwallet is able to perform xpub/xpriv import+export, and is made aware
of something similar to KeyDescriptors, then the next episode in the saga
requires removing all instance of raw private key manipulation from lnd.
Currently, in a few areas within the codebase, the Signer/KeyChain abstractions
are broken in order to give a caller raw access to a private key [7]. Clearly
this behavior must be removed in order to allow fully externally signing for
lnd. Thankfully, non of the current sub-systems that current manipulate raw
private keys fundamentally need direct access to the raw key material. The
sub-systems that current require raw private key access are the following:
* The brontide handshake. Currently we pass the entire private key into the
brontide state machine. However, it can get by with simply being able to
perform an ECDH against our long-tern static identity key. From there on,
it can carry out the remainder of the handshake using this resulting state,
as well as the ephemeral key it generates for the handshake. The current
SecretKeyChain interface already has a ScalarMult [8] method which is meant
to be used for ECDH, so we have that base covered already.
* The watch tower's brontide handshake. Same thing as the above, just uses a
distinct key for any created watch towers.
* shachain root creation and secret generation. Fundamentally, it seems that
whatever external signer abstraction that is created will need to also be
able to manipulate and derive shachain leaves. Thankfully, we already have
an interface used.
* SCB blob encryption/decryption. When creating the SCB format, we didn't
want to require that some future external singing abstraction had to be
aware of the encryption scheme we used. As a result, we ended up obtaining
the raw private key, and using that directly for encryption/decryption.
Instead, we can start to use the ECDH method from the SecretKeyRing
interface to obtain a new shared secret which is then used directly. We can
bump the SCB version (the Multi specifically) to handle the transition from
the old to the new format.
With the above areas expunged of raw private key manipulation, we should then
be able to _remove_ the DerivePrivKey method from the SecretKeyChain interface.
The final chapter in the saga is to extend the current `lnd` package to create
a proper lnd.Daemon struct that allows a caller to easily embed lnd in a new Go
program. Atm, we only expose a Main() method, which was enough to implement
support for mobile builds using the gomobile tool. With what I envision, we'll
need to go a step further, and move most of the config parsing into new methods
which will allow a caller to fully assemble all the fundamental interfaces [11]
that lnd needs to function. I have a local branch that makes some steps towards
working things into this direction, and aim to have it finished (PR proposed)
sometime before the end of the year.
Putting it all together, after this saga is complete, you'll be able to have
your external signer implement the Signer+WalletKit gRPC services [12][13],
then pass in a concrete implementation of the Signer+KeyChain interfaces that
simply proxy all calls to your external signer, communicating over gRPC. The
remote server will more or less be some additional business logic wrapped
around a waddrmgr instance. This would all live in either in lnd, or a new
package that wraps lnd with this new functionality, thereby _extending_ the set
of capabilities that lnd offers. Once this refactoring of the main lnd package
is complete, other developers will be able to make similar extensions such as
dropping in a replicated database, or using Electrum as a BlockChainIO chain
backend.
Let me know if you have any questions w.r.t the steps mentioned above!
-- Laolu
[1]:
https://godoc.org/github.com/btcsuite/btcwallet/waddrmgr[2]:
https://godoc.org/github.com/btcsuite/btcwallet/waddrmgr#Manager.ConvertToWatchingOnly[3]:
https://github.com/btcsuite/btcwallet/blob/master/waddrmgr/db.go#L139[4]:
https://godoc.org/github.com/btcsuite/btcutil/hdkeychain[5]:
https://github.com/lightningnetwork/lnd/blob/master/keychain/derivation.go#L138k[6]
https://github.com/lightningnetwork/lnd/search?p=1&q=DerivePrivKey&unscoped_q=DerivePrivKey[7]:
https://github.com/lightningnetwork/lnd/blob/77fde0f201e929d659430a823f54bfacd3f39f09/input/signdescriptor.go#L23:6[8]:
https://github.com/lightningnetwork/lnd/blob/master/keychain/derivation.go#L186[9]:
https://github.com/lightningnetwork/lnd/blob/master/shachain/producer.go#L9[10]:
https://godoc.org/github.com/lightningnetwork/lnd[11]:
https://github.com/lightningnetwork/lnd/blob/master/chainregistry.go#L126[12]:
https://github.com/lightningnetwork/lnd/blob/master/lnrpc/signrpc/signer.proto[13]:
https://github.com/lightningnetwork/lnd/blob/master/lnrpc/walletrpc/walletkit.proto