Regarding commitIndex and lastAppliedIndex

[Srinivas V]

Hi All,

I had a quick design question:

Why did we decide to maintain commitIndex and lastAppliedIndex in volatile storage? if we had maintained then in non volatile storage, we could have avoided re applying logs that have been applied before follower failures right?

Am I missing something here?

Thanks,

Regards,

V Srinivas.

[Free Ekanayaka]

Hello,

Srinivas V <sriniv...@gmail.com> writes:

> Hi All,
> I had a quick design question:
>
> Why did we decide to maintain commitIndex and lastAppliedIndex in volatile
> storage?

Because there's no need for commitIndex to be stored. In other words,
there's no loss of information if you don't store it.

> if we had maintained then in non volatile storage, we could have
> avoided re applying logs that have been applied before follower failures
> right?

If your FSM is memory-based (volatile), then you will have to re-apply
logs anyway, whether you store commitIndex or not.

If your FSM is disk-based, then yes, you normally need to persist
lastAppliedIndex indeed (but not necessarily commitIndex), to avoid
applying entries twice.

Cheers,

Free

[Konstantin Osipov]

* Srinivas V <sriniv...@gmail.com> [23/07/03 09:55]:

The algorithm doesn't formally gain from storing it neither in
safety nor in liveness.

Most raft state machine implementations use key/value store under
the hood where updating the commit index is an extra key write.

If you need to update a commit index frequently,
it becomes expensive.

Having said that, maintaining the commit or apply index simplifies
reasoning about joint change configuration transition during
recovery, as well as it helps speed up re-joining the cluster
during restart.


--
Konstantin Osipov, Moscow, Russia