Neutrino, Taproot, and The Evolution of BiPs 157/158

[Olaoluwa Osuntokun]

Hi y'all,

If you're an active user of neutrino [8], then you probably heard about what
went down over the past week or so on testnet as related to Taproot. First,
i just wanted to reassure everyone that nothing is fundamentally broken with
BIP 157/158 as it relates to taproot, and we already have a mitigation patch
in place for the issue we encountered.

The rest of this mail is structured in a FAQ style to make it easy to skim
and extract the information that may be relevant to the reader.

## What happened on testnet?

Neutrino nodes on testnet rejected a filter (thinking it was invalid) due
to this transaction spending a taproot input [1]. This was due to a faulty
heuristics in the neutrino _client code_ (not part of the protocol) that
attempted to verify the contents of a filter more completely.

In retrospect, the heuristic in question wasn't full proof, as it attempted
to derive the _pk script_ of a transaction based on its input
witness/sigScript. This worked pretty well in the context of segwit v0, but
it isn't possible to exhaustively do as we don't know what future spends
will look like.

## Is neutrino broken?

No, the client side is fine, and the protocol is fine.

The problematic heuristic has been removed in this PR [2], which will be
included in lnd 0.14, and has been tagged with neutrino 0.13 [3].

To dig into _why_ we attempted to use such a heuristic, we'll need to
revisit how BIP 158 works briefly. For each block, we insert the `pkScript`s
of all the outputs, and also the prev out's pkScript (the script being
spent) as well. This lets the filter compress script re-use in both inputs
and outputs, and also makes it possible to implement some protocols in a
more light-client friendly manner (for example Loop uses this to has the
client watch HTLC _scripts_ being spent, as it doesn't necessarily know the
txid/outpoint).

The one issue with this, is that while clients can ensure that all the
`pkScripts` of outputs have been inserted, they can't do the same for the
inputs (which is why we added that heuristic in the client code). Luckily we
know how to properly fix this at the protocol level, more on that below.

## How can I make sure my neutrino clients handle the Taproot upgrade on mainnet smoothly?

Upgrade to 0.14 (assuming it's out in time), or apply this small patch [4].
The patch just demotes an error case to a warning message, so anyone running
a fork should be able to easily apply the fix.

Alongside, optionally extend these filter header guides [7].

We're looking into some intermediate ground where we can verify the scripts
that we know are relevant to the node.

## How will BIP 158/157 evolve post taproot?

In terms of adding more taproot specific functionality, I've had a number of
items in my laundry list such as:

  * creating new segwit-only filters with re-parameterized fp rates (also
    examine other filter types such as pure outpoints, etc)

  * creating filters that include witness data to allow matching on
    internal/external keys, the control block, merkle root, annex, etc

  * add a new protocol extension to btcd (with a corresponding BIP) to
    allow notes to fetch block undo data (as described here [5]) to fully
    verify fetched filters or a node needs to reconcile conflicting filters

  * new filters that span across multiple blocks as previously worked on by
    Kalle Alm (couldn't find a link to his PR when typing this...)

Make further progress towards a proposal that allows filters to be committed
either as a soft-fork, or a "velvet fork" [6] where miners optionally commit to
the past filter header chain.


-- Laolu

[1]: https://mempool.space/testnet/tx/4b425a1f5c0fcf4794c48b810c53078773fb768acd2be1398e3f561cc3f19fb8
[2]: https://github.com/lightninglabs/neutrino/pull/234
[3]: https://github.com/lightninglabs/neutrino/releases/tag/v0.13.0
[4]: https://github.com/lightninglabs/neutrino/pull/234/files
[5]: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2019-February/016649.html
[6]: https://eprint.iacr.org/2018/087
[7]: https://github.com/lightninglabs/neutrino/blob/5e09bd9b5d65e90c6ff07aa11b3b9d80d42afb86/chainsync/filtercontrol.go#L15
[8]: https://github.com/lightninglabs/neutrino

[Olaoluwa Osuntokun]

> In terms of adding more taproot specific functionality, I've had a number of
> items in my laundry list such as:

  

Forgot to add this other item (also the list wasn't meant to be only tapoot stuff):
  * reviving old projects to include a micropayment-for-data layer to
    incentivize nodes to serve the filters and other data

[David A. Harding]

On Thu, Nov 04, 2021 at 03:01:13PM -0700, Olaoluwa Osuntokun wrote:
> * add a new protocol extension to btcd (with a corresponding BIP) to
> allow notes to fetch block undo data (as described here [5]) to fully
> verify fetched filters or a node needs to reconcile conflicting filters
>

> [5]: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2019-February/016649.html

I had to think about this for a bit. Could you confirm that this is
what you're thinking:

1. Bob gets different filter commitmentments from Alice and Mallory.

2. Bob downloads both complete filters from Alice and Mallory.

3. Bob downloads the undo data from both Alice and Mallory (for the
inputs) as well as also downloading the complete block (for the
outputs). Bob then verifies that both Alice and Mallory generated a
filter congruent with the block and their individual undo data.

4. Bob then finds the first difference in spent UTXO entries between
Alice's and Mallory's undo data.

5. For that first difference, Bob downloads the block which created that
UTXO and uses it to determine whether Alice or Mallory is the liar.
(Note, for those unaware, UTXO entries must include at least the height
of the block which created them in order for coinbase maturity and BIP68
relative timelocks to work, so that information is part of the undo
data; this makes it possible to request the block which created a
particular UTXO).

If I understand correctly, that seems pretty sensible to me. It's
efficient in the normal case, not terrible in the worst case I think
(details below), and doesn't affect privacy AFAICT. One challenge
though might be that I think the worst-case undo file size is about 238
MB[1], which I believe is larger than Bitcoin Core's maximum P2P message
size (32 MiB, IIRC).

-Dave

[1] Over a series of blocks, an attacker creates 23,800 outputs each
with a scriptPubKey the maximum 10,000 bytes in size. Then all of those
UTXOs are spent in a single block: 23,800 utxos * (32 txids + 4 vout + 4
nSequence + 1 push + 1 OP_TRUE) = 999600. The undo data is thus 23,800
* (10,000 scriptPubKey + some extra stuff).