Settled HTLCs accounting

[Joost Jager]

Hi,

Recently I've been looking into the accounting of settled htlcs in lnd. Is it possible to get a definitive answer from lnd to the question of whether a htlc was settled or not?

With lightning payment amounts going up, this becomes increasingly important. For a $100k incoming payment, you'd want to be as sure as you can be that it was actually settled.

Of course final settlement in lightning may take a long time. Only when a channel closes, the balance in the channel becomes truly yours. This is inherent to lightning and there is no way around that. Risk can be mitigated by actively managing the total amount of funds in channels.

What is possible though is to make sure that the settlement of an htlc is locked in on the commitment transaction. Or, in the on-chain resolution flow, make sure that the on-chain htlc is swept successfully. Because as long as the htlc is pending, there is the strict requirement to act before the htlc expires. With lnd's default 40 blocks final cltv delta for invoices, this means that a few hours of downtime may already lead to the loss of htlcs.

One potential problem in this context is that when lnd marks an invoice as 'settled', the htlc is not yet locked in. It would probably be more appropriate to call the settled state 'settle requested'. It isn't guaranteed that the htlc will be locked-in, because downtime may lead to the remote party claiming the htlc via the timeout path.

If this timeout occurs, there won't be an update to the invoice state. The invoice will remain 'settled' even though the payment wasn't received. I think a subsequent channel force-close is the only indication that you'll get and you'd need to manually trace the on-chain resolutions back to the invoices. At that point, the damage may have been done already. In a custodial wallet environment for example, the incoming amount may have been credited to a user who spent it already. For onchain/offchain swaps, a similar outcome is possible.

The lack of a definitive settle signal can also be a problem for htlc interceptor applications. Via the htlc interceptor api, settle requests can be sent to lnd. But again there is no confirmation when the settle was locked-in on the commitment transaction. After an application has requested a settle, it must assume that the lock-in will happen eventually even though it may not.

What I think would really help here is a way in lnd to look up the true settle status of an htlc. Applications for which this is critical could use this lookup and only release funds to a user when it indicates 'settled'. I've done a little exploration of what this could look like code-wise in https://github.com/lightningnetwork/lnd/pull/6517. Note that the settle information should remain available after the closure of a channel. Channels may close without operator involvement, but the need to find out the outcome of an htlc remains.

A lookup rpc is an invitation to lots of polling, so perhaps a notification stream would be a better option. If you want lightning payments to remain fast, polling in the critical path isn't a good thing.

With this information available, it also becomes possible to add a new state `settle_requested` to lnd's invoice database.

Interested to hear your opinions on this problem and ideas for solutions.

Joost

Related issues:

* https://github.com/lightningnetwork/lnd/issues/4088

* https://github.com/lightningnetwork/lnd/issues/6208

[Olaoluwa Osuntokun]

Hi Joost,

> Only when a channel closes, the balance in the channel becomes truly
> yours.

Not related to the main issue here, but: not necessarily, there're ways to
deliver BTC on chain without closing the channel nor modifying the funding
output itself.

> One potential problem in this context is that when lnd marks an invoice as
> 'settled', the htlc is not yet locked in.

If by locked-in, you mean the HTLC hasn't been fully removed, then this is
accurate: the link calls into the invoice registry for a resolution, the
registry updates its state then responds back. On restart any unresolved
resolutions will be replayed
(https://github.com/lightningnetwork/lnd/blob/e98a739130a4364e2b9302e98ed30a49881502d5/htlcswitch/link.go#L1040-L1047)..

Ultimately though, I agree that things should be in more of a partial state
"settlement pending" or w/ until the HTLC is actually removed from our
commitment transaction.

> A lookup rpc is an invitation to lots of polling, so perhaps a
> notification stream would be a better option. If you want lightning
> payments to remain fast, polling in the critical path isn't a good thing.

Which is related to the existing issue to add a single global notification
stream of all invoices, and some of the existing technical debt in that area
re the add/settle indexes...

Ideally with this issue
(https://github.com/lightningnetwork/lnd/issues/6288), we can move to
implement a more future proofed event sourcing based notification system,
which also makes future changes like this easier.

Taking a look at that initial implementation you have, it would appear to
introduce another bucket of unbounded size (grows w/ the number of settled
invoices). Instead, what if we just stored this information within the
existing invoices? So things would first go to that "settle pending" state,
which is then ultimately finalized either: the HTLC times out
(reverts), or is fully settled.

In terms of an implementation path, every time we process a revocation from
the remote party (only after they revoke the state w/o that HTLC can it
actually be considered to be "settled"), we'll go to compact the logs
(remove adds that have a settled fully locked in, etc):
https://github.com/lightningnetwork/lnd/blob/08e4e665741c14854f3cf227ba3777a443bce932/lnwallet/channel.go#L1171.

At this point, we know that the HTLC is no longer present, and also that as
long as its present, on restart we'll re-run this routine. This is where the
logic to "up all" back to the invoice registry could live potentially (in
the form of extending what's returned to include the HTLCs that are fully
resolved at that height.

This would let us avoid the addition of yet-another-mega-bucket, and keeps
the state where it should be: in the invoice registry. This also means we
don't need to worry about more historical channel information, and if to
delete this state or note when a channel is abandoned.

For the on-chain resolution case, a similar upcall can be used back to the
invoice registry that the on-chain state is fully finalized.

-- Laolu

[Joost Jager]

Hi Laolu,

> A lookup rpc is an invitation to lots of polling, so perhaps a
> notification stream would be a better option. If you want lightning
> payments to remain fast, polling in the critical path isn't a good thing.

Which is related to the existing issue to add a single global notification
stream of all invoices, and some of the existing technical debt in that area
re the add/settle indexes...

Ideally with this issue
(https://github.com/lightningnetwork/lnd/issues/6288), we can move to
implement a more future proofed event sourcing based notification system,
which also makes future changes like this easier.

Taking a look at that initial implementation you have, it would appear to
introduce another bucket of unbounded size (grows w/ the number of settled
invoices). Instead, what if we just stored this information within the
existing invoices? So things would first go to that "settle pending" state,
which is then ultimately finalized either: the HTLC times out
(reverts), or is fully settled.

If it was just about invoices, I think that could work. But that wouldn't provide reliable accounting for forwarded and intercepted htlcs.

 

It's true that that implementation introduces another bucket of unbounded size, but the actual size may be negligible compared to what's already stored in the revocation log. For open channels at least.

Looking at the analysis in https://github.com/lightningnetwork/lnd/pull/6347, it seems that an add+settle of an htlc roughly uses up 2 commit txes with the first containing the htlc info. So 2*38 + 58 + some tlv overhead = ~150 bytes. I haven't thought about the most efficient way to store the ids of the htlcs that settled. Even using a list of 4 bytes per settled htlcs is small compared to the 150 bytes that are stored already. Perhaps a different encoding can work out even better (bitmask, sparse tree something something?).

A delete rpc that deletes the list for a particular (closed) channel isn't difficult to add either. Can be called after the data has been transferred to an external system.

In terms of an implementation path, every time we process a revocation from
the remote party (only after they revoke the state w/o that HTLC can it
actually be considered to be "settled"), we'll go to compact the logs
(remove adds that have a settled fully locked in, etc):
https://github.com/lightningnetwork/lnd/blob/08e4e665741c14854f3cf227ba3777a443bce932/lnwallet/channel.go#L1171.

At this point, we know that the HTLC is no longer present, and also that as
long as its present, on restart we'll re-run this routine. This is where the
logic to "up all" back to the invoice registry could live potentially (in
the form of extending what's returned to include the HTLCs that are fully
resolved at that height.

I can indeed see that this as a potential solution to avoid having to implement an atomic transaction that updates both the channel state and the invoice registry.

 

This would let us avoid the addition of yet-another-mega-bucket, and keeps
the state where it should be: in the invoice registry. This also means we
don't need to worry about more historical channel information, and if to
delete this state or note when a channel is abandoned.

For the on-chain resolution case, a similar upcall can be used back to the
invoice registry that the on-chain state is fully finalized.

As mentioned above, I think this problem deserves a full solution that fixes accounting for every htlc, not just the exit hop ones. And for that, the invoice registry hardly seems the right place.

Joost

[Olaoluwa Osuntokun]

Hi Joost,

> If it was just about invoices, I think that could work. But that wouldn't
> provide reliable accounting for forwarded and intercepted htlcs.

Ah yeah true...

One other alternative would be to re-purpose the existing WitnessCache:
anytime a forwarded HTLC is settled (also if we discover the pre-image on
chain), all links will add the hash+secret information to this DB. In theory
we could either add another index here, or extend the set of values stored.
On the API side, we'd then expose querying into this (which is already keyed
by payment hash).

I took another look at your draft, impl:
```
message IsHtlcSettledRequest {
    ChannelPoint channel_point = 1;

    uint64 htlc_index = 2;
}
```

Thinking of the case where a service wants to make sure a given payment is
actually sent/received, Wouldn't it be more natural to instead expose a look
up by _payment hash_?  Otherwise, I think the only way one could get that
hltc_index information would be to introspect into the revocation log, or
look up the full HTLC information. This would also make it easier to verify
that all the splits of an MPP payment have been fully sent/received.

-- Laolu

[Joost Jager]

Hi Laolu,

One other alternative would be to re-purpose the existing WitnessCache:
anytime a forwarded HTLC is settled (also if we discover the pre-image on
chain), all links will add the hash+secret information to this DB. In theory
we could either add another index here, or extend the set of values stored.

You mean to keep using the hash as the key and extend the value that is stored?

I think that in addition to the secret that currently makes up the value, there would need to be a variable length list of (channel, htlc_id). Adding a newly settled htlc to the list would require reading the full value, appending the new settle and writing back. Maybe not the most efficient for hashes that have a lot of settles?

Also there would probably be quite a bit of duplication of the channel id on disk. In a scheme keyed by channel id, it would only be stored once.

With a repurposed witness cache it may also be more difficult to guarantee atomicity.

I understand the potential advantage of extending an existing bucket that has a lifetime beyond the life of a channel already, but not sure if in this case it is actually better. It seems that a dedicated bucket requires less disk space.

 

On the API side, we'd then expose querying into this (which is already keyed
by payment hash).

I took another look at your draft, impl:
```
message IsHtlcSettledRequest {
    ChannelPoint channel_point = 1;

    uint64 htlc_index = 2;
}
```

Thinking of the case where a service wants to make sure a given payment is
actually sent/received, Wouldn't it be more natural to instead expose a look
up by _payment hash_?

I think for looking up the settlement status on the payment/invoice level, it may be more suitable to add a flag to the existing `lnrpc.InvoiceHTLC` and `lnrpc.HTLCAttempt` messages? Or a single higher level per payment/invoice field on `lnrpc.Invoice` and `lnrpc.Payment`.

Of course a lookup by hash works too, but it may increase the size of the database unnecessarily.

 

Otherwise, I think the only way one could get that
hltc_index information would be to introspect into the revocation log, or
look up the full HTLC information.

We'd use the proposed `IsHtlcSettled` call in combination with the htlc interceptor, which is keyed by (chan,htlc_id).

Also for invoices (chan,htlc_id) is currently already exposed (`lnrpc.InvoiceHTLC`). For payment htlcs (`lnrpc.HTLCAttempt`) it could be added too.

One final thing to note is that the PoC PR implements `IsHtlcSettled` as a single call, but we probably want to make that a subscription. If payments are only considered done when all htlcs are truly settled, polling `IsHtlcSettled` would be on the critical path of keeping lightning payments fast. An event seems to be better suited for that.

Joost