[PATCH] raft: add more raft metrics to make debug easier

[Gleb Natapov]

New metrics add more visibility to the raft internal state.

---

Also in scylla-dev gleb/raft-metrics

CI: https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/9858/

diff --git a/raft/fsm.hh b/raft/fsm.hh
index 93cf1686b31..0e8153a043d 100644
--- a/raft/fsm.hh
+++ b/raft/fsm.hh
@@ -359,6 +359,9 @@ class fsm {
bool is_prevote_candidate() const {
return is_candidate() && std::get<candidate>(_state).is_prevote;
}
+ size_t state_to_metric() const {
+ return _state.index();
+ }
index_t log_last_idx() const {
return _log.last_idx();
}
diff --git a/raft/server.cc b/raft/server.cc
index fa166a4cfb2..4e30af4a6b4 100644
--- a/raft/server.cc
+++ b/raft/server.cc
@@ -1775,6 +1775,20 @@ void server_impl::register_metrics() {
sm::description("size of in-memory part of the log"), {server_id_label(_id)}),
sm::make_gauge("log_memory_usage", [this] { return _fsm->log_memory_usage(); },
sm::description("memory usage of in-memory part of the log in bytes"), {server_id_label(_id)}),
+ sm::make_gauge("log_last_term", [this] { return _fsm->log_last_idx(); },
+ sm::description("term of the last log entry"), {server_id_label(_id)}),
+ sm::make_gauge("log_last_index", [this] { return _fsm->log_last_term(); },
+ sm::description("index of the last log entry"), {server_id_label(_id)}),
+ sm::make_gauge("snapshot_last_term", [this] { return _fsm->log_last_snapshot_idx(); },
+ sm::description("term of the snapshot"), {server_id_label(_id)}),
+ sm::make_gauge("snapshot_last_index", [this] { return _fsm->log_term_for(_fsm->log_last_snapshot_idx()).value(); },
+ sm::description("index of the snapshot"), {server_id_label(_id)}),
+ sm::make_gauge("state", [this] { return _fsm->state_to_metric(); },
+ sm::description("current state: 0 - follower, 1 - candidate, 2 - leader"), {server_id_label(_id)}),
+ sm::make_gauge("commit_index", [this] { return _fsm->commit_idx(); },
+ sm::description("commit index"), {server_id_label(_id)}),
+ sm::make_gauge("apply_index", _applied_idx,
+ sm::description("applied index"), {server_id_label(_id)}),
});
}

--
Gleb.

[Kamil Braun]

log term and index implementations are switched

> + sm::make_gauge("snapshot_last_term", [this] { return _fsm->log_last_snapshot_idx(); },
> + sm::description("term of the snapshot"), {server_id_label(_id)}),
> + sm::make_gauge("snapshot_last_index", [this] { return _fsm->log_term_for(_fsm->log_last_snapshot_idx()).value(); },
> + sm::description("index of the snapshot"), {server_id_label(_id)}),

snapshot term and index implementations are switched

[Gleb Natapov]

New metrics add more visibility to the raft internal state.

---

v1->v2:
- fix index/term metric mismatch

Also in scylla-dev gleb/raft-metrics-v2
CI: https://jenkins.scylladb.com/job/scylla-master/job/scylla-ci/9995/

diff --git a/raft/fsm.hh b/raft/fsm.hh
index 93cf1686b31..0e8153a043d 100644
--- a/raft/fsm.hh
+++ b/raft/fsm.hh
@@ -359,6 +359,9 @@ class fsm {
bool is_prevote_candidate() const {
return is_candidate() && std::get<candidate>(_state).is_prevote;
}
+ size_t state_to_metric() const {
+ return _state.index();
+ }
index_t log_last_idx() const {
return _log.last_idx();
}
diff --git a/raft/server.cc b/raft/server.cc

index fa166a4cfb2..6eaec6f9d47 100644

--- a/raft/server.cc
+++ b/raft/server.cc
@@ -1775,6 +1775,20 @@ void server_impl::register_metrics() {
sm::description("size of in-memory part of the log"), {server_id_label(_id)}),
sm::make_gauge("log_memory_usage", [this] { return _fsm->log_memory_usage(); },
sm::description("memory usage of in-memory part of the log in bytes"), {server_id_label(_id)}),

+ sm::make_gauge("log_last_index", [this] { return _fsm->log_last_idx(); },

+ sm::description("term of the last log entry"), {server_id_label(_id)}),

+ sm::make_gauge("log_last_term", [this] { return _fsm->log_last_term(); },

+ sm::description("index of the last log entry"), {server_id_label(_id)}),

+ sm::make_gauge("snapshot_last_index", [this] { return _fsm->log_last_snapshot_idx(); },

+ sm::description("term of the snapshot"), {server_id_label(_id)}),

+ sm::make_gauge("snapshot_last_term", [this] { return _fsm->log_term_for(_fsm->log_last_snapshot_idx()).value(); },

+ sm::description("index of the snapshot"), {server_id_label(_id)}),

[Gleb Natapov]

Ping.

--
Gleb.

[Commit Bot]

From: Gleb Natapov <gl...@scylladb.com>
Committer: Kamil Braun <kbr...@scylladb.com>
Branch: next

raft: add more raft metrics to make debug easier

---
diff --git a/raft/fsm.hh b/raft/fsm.hh
--- a/raft/fsm.hh
+++ b/raft/fsm.hh
@@ -359,6 +359,9 @@ public:

bool is_prevote_candidate() const {
return is_candidate() && std::get<candidate>(_state).is_prevote;
}
+ size_t state_to_metric() const {
+ return _state.index();
+ }
index_t log_last_idx() const {
return _log.last_idx();
}
diff --git a/raft/server.cc b/raft/server.cc

[Kamil Braun]

Qd

[Commit Bot]

From: Gleb Natapov <gl...@scylladb.com>
Committer: Kamil Braun <kbr...@scylladb.com>

Branch: master

raft: add more raft metrics to make debug easier

---
diff --git a/raft/fsm.hh b/raft/fsm.hh

--- a/raft/fsm.hh
+++ b/raft/fsm.hh
@@ -359,6 +359,9 @@ public:

bool is_prevote_candidate() const {
return is_candidate() && std::get<candidate>(_state).is_prevote;
}
+ size_t state_to_metric() const {
+ return _state.index();
+ }
index_t log_last_idx() const {
return _log.last_idx();
}
diff --git a/raft/server.cc b/raft/server.cc