sshd-SshServer vs sshd-SshDaemon threads in Gerrit 3.2

174 views
Skip to first unread message

Nuno Costa

unread,
Mar 30, 2021, 3:33:46 PM3/30/21
to Repo and Gerrit Discussion

Hi All,

After upgrading 2 instances to from 2.16.17 to 3.2.7, we noticed that sshd_log now shows sshd-SshDaemon instead of sshd-SshServer.

2.16.17
[2021-03-14 00:00:00,094 +0530] 3ba09891 [sshd-SshServer[57ed8f95]-nio2-thread-32] anotheruser LOGIN FROM anotherip

3.2.7
[2021-03-14T23:59:59.805+0530] f4512207 [sshd-SshDaemon[37a14aa1](port=22)-nio2-thread-57] someuser LOGIN FROM someip

gerrit show-caches --show-threads command in 3.2.7 is also not showing sshd-SshServer (nor sshd-SshDaemon which I would assume to be the replacement now).

Same for REST API call "/a/config/server/summary/". It will no longer return sshd-SshServer as used to.

I see that the documentation[1] is still mentioning sshd-SshServer but did not see  any mention to this specific change.

Does anyone have some tips?

Thanks,
Nuno

Nuno Costa

unread,
Mar 31, 2021, 5:25:57 AM3/31/21
to Repo and Gerrit Discussion
What I meant regarding the tips was if someone knows if this is something to worry about or if is working as expected, since I did not found anything related to this change :)

Thanks

Luca Milanesio

unread,
Mar 31, 2021, 5:32:49 AM3/31/21
to Repo and Gerrit Discussion, Luca Milanesio

On 31 Mar 2021, at 10:25, Nuno Costa <nunoco...@gmail.com> wrote:

What I meant regarding the tips was if someone knows if this is something to worry about or if is working as expected, since I did not found anything related to this change :)

I believe the logging format has the thread name on it, so what you see isn’t a difference in the way Gerrit logs its SSHD data but rather the name of the current thread service that log line print.
Bottom line: it’s nothing you should be worried about :-)

HTH

Luca.


Thanks

On Tuesday, 30 March 2021 at 20:33:46 UTC+1 Nuno Costa wrote:

Hi All,

After upgrading 2 instances to from 2.16.17 to 3.2.7, we noticed that sshd_log now shows sshd-SshDaemon instead of sshd-SshServer.

2.16.17
[2021-03-14 00:00:00,094 +0530] 3ba09891 [sshd-SshServer[57ed8f95]-nio2-thread-32] anotheruser LOGIN FROM anotherip

3.2.7
[2021-03-14T23:59:59.805+0530] f4512207 [sshd-SshDaemon[37a14aa1](port=22)-nio2-thread-57] someuser LOGIN FROM someip

gerrit show-caches --show-threads command in 3.2.7 is also not showing sshd-SshServer (nor sshd-SshDaemon which I would assume to be the replacement now).

Same for REST API call "/a/config/server/summary/". It will no longer return sshd-SshServer as used to.

I see that the documentation[1] is still mentioning sshd-SshServer but did not see  any mention to this specific change.

Does anyone have some tips?

Thanks,
Nuno


-- 
-- 
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/50072599-9bed-4566-907c-b7f75c7d7044n%40googlegroups.com.

Nuno Costa

unread,
Mar 31, 2021, 6:19:48 AM3/31/21
to Repo and Gerrit Discussion
Hi Luca, thanks for the reply :)

I believe the logging format has the thread name on it, so what you see isn’t a difference in the way Gerrit logs its SSHD data but rather the name of the current thread service that log line print.
Bottom line: it’s nothing you should be worried about :-)

To be honest, I was not worried about the log format change. I used those to show the change.

My worry is if it was expected to stop showing in the output of "gerrit show-caches --show-threads" and REST API call "/a/config/server/summary/"

Those commands provides some metrics that we were monitoring and "sshd-SshServer"(nor "sshd-SshDaemon") is no longer there.
Are this metrics now being sent to other place?

2.16.17
$ curl --netrc-file ~/.netrc https://gerrit-serverA/a/config/server/summary/
)]}'
{
  "task_summary": {
    "total": 79,
    "running": 74,
    "sleeping": 5
  },
  "mem_summary": {
    "total": "248.84g",
    "used": "77.99g",
    "free": "145.85g",
    "buffers": "25.00g",
    "max": "256.00g",
    "open_files": 2447
  },
  "thread_summary": {
    "cpus": 48,
    "threads": 668,
    "counts": {
      "ReceiveCommits": {
        "WAITING": 48
      },
      "SshCommandStart": {
        "WAITING": 2
      },
      "SSH-Interactive-Worker": {
        "WAITING": 80
      },
      "sshd-SshServer": {
        "RUNNABLE": 1,
        "TIMED_WAITING": 1,
        "WAITING": 48
      },
      "HTTP": {
        "RUNNABLE": 5,
        "TIMED_WAITING": 46,
        "WAITING": 1
      },
      "H2": {
        "TIMED_WAITING": 22
      },
      "SSH git-upload-pack": {
        "RUNNABLE": 34,
        "BLOCKED": 23,
        "TIMED_WAITING": 8,
        "WAITING": 8
      },
      "Other": {
        "RUNNABLE": 2,
        "TIMED_WAITING": 28,
        "WAITING": 271
      },
      "SSH-Stream-Worker": {
        "WAITING": 40
      }
    }
  }
}

3.2.7
$ curl --netrc-file ~/.netrc https://gerrit-serverB/a/config/server/summary/
)]}'
{"task_summary":{"total":13,"running":8,"sleeping":5},"mem_summary":{"total":"46.36g","used":"4.95g","free":"10.80g","buffers":"30.61g","max":"99.56g","open_files":1024},"thread_summary":{"cpus":56,"threads":578,"counts":{"ReceiveCommits":{"WAITING":56},"SshCommandStart":{"WAITING":5},"SSH-Interactive-Worker":{"WAITING":91},"HTTP":{"TIMED_WAITING":44,"RUNNABLE":5,"WAITING":1},"H2":{"TIMED_WAITING":26},"SSH git-upload-pack":{"RUNNABLE":1,"WAITING":3},"Other":{"TIMED_WAITING":31,"RUNNABLE":5,"WAITING":254},"SSH-Stream-Worker":{"WAITING":56}}}}


Snippet of cli "gerrit show-caches --show-thread"

2.16.17 gerrit-serverA
Threads: 48 CPUs available, 666 threads
                                    NEW       RUNNABLE        BLOCKED        WAITING  TIMED_WAITING     TERMINATED
  ReceiveCommits                      0              0              0             48              0              0
  SshCommandStart                     0              0              0              2              0              0
  SSH-Interactive-Worker              0              0              0             70              0              0
  sshd-SshServer                      0              0              0             49              1              0
  HTTP                                0              4              0              0             46              0
  H2                                  0              0              0              0             22              0
  SSH git-upload-pack                 0             29              9             10             17              0
  Other                               0              3              0            288             28              0
  SSH-Stream-Worker                   0              0              0             40              0              0


3.2.7 gerrit-serverB
Threads: 56 CPUs available, 575 threads
                                    NEW       RUNNABLE        BLOCKED        WAITING  TIMED_WAITING     TERMINATED
  ReceiveCommits                      0              0              0             56              0              0
  SshCommandStart                     0              0              0              5              0              0
  SSH-Interactive-Worker              0              0              0             92              0              0
  HTTP                                0              4              0              1             45              0
  H2                                  0              0              0              0             26              0
  SSH git-upload-pack                 0              2              0              1              0              0
  Other                               0              3              0            256             28              0
  SSH-Stream-Worker                   0              0              0             56              0              0



PS: Did API also changed from pretty json to oneliner json?

Nuno Costa

unread,
Apr 8, 2021, 5:39:09 AM4/8/21
to Repo and Gerrit Discussion
Hi Luca,

Were the above outputs expected in 3.2.7?

Thanks,
Nuno

Luca Milanesio

unread,
Apr 8, 2021, 8:30:49 AM4/8/21
to Repo and Gerrit Discussion, Luca Milanesio
Can you explain what is end user issue with the outputs? What isn’t working in v3.2.7?
I have honestly lost the focus of the discussion :-)

With regards to the changes, the list of changes between 2.16.7 and 3.2.7 is *huge*, over 9k commits between the two releases.
See the full list of changes on the release notes pages at [1], [2] and [3].

Should you feel that anything isn’t working or isn’t reported there, feel free to raise an issue at [4].

Luca.



Thanks,
Nuno

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

Nuno Costa

unread,
Apr 8, 2021, 8:54:04 AM4/8/21
to Repo and Gerrit Discussion
Can you explain what is end user issue with the outputs? What isn’t working in v3.2.7?
I have honestly lost the focus of the discussion :-)
Sorry, some times I want to give as much info as possible and with that some times it does not help :)

To sum up, our issue is that "sshd-SshServer" is no longer part of cli command "gerrit show-caches --show-thread" and REST API call "/a/config/server/summary/" outputs.
We were monitoring that specific metric and with 3.2.7 there is no longer that metric and we wanted to understand why.
 
With regards to the changes, the list of changes between 2.16.7 and 3.2.7 is *huge*, over 9k commits between the two releases.
See the full list of changes on the release notes pages at [1], [2] and [3].

I went through all of them and did not find anything related. Maybe I did not search for the right keywords.
I found documentation still mentioning "sshd-SshServer" and not "sshd-SshDaemon, as I mention in the first post.

I'm just wondering if this is really a bug and we are the only ones affected by it before opening a bug report :)

Matthias Sohn

unread,
Apr 8, 2021, 10:28:03 AM4/8/21
to Nuno Costa, Repo and Gerrit Discussion
I think what you are observing is that sshd thread names changed between 2.16 and 3.2.
I can confirm that this is the case by looking at some of our own servers, but I didn't try to find the change
which did this renaming. I don't think we consider thread names to be API.

-Matthias

Nuno Costa

unread,
Apr 8, 2021, 10:51:07 AM4/8/21
to Repo and Gerrit Discussion
Hi Matthias,

Thanks for the reply and checking on your environment.

If that is "working as designed", it is ok for us. Just wanted to make sure of it and pass the information to the team.
Since we are just passing the json file returned by the API call to the monitoring system, we don't need to do anything besides removing a graph that have an empty timeline after the upgrade :)


Luca Milanesio

unread,
Apr 8, 2021, 1:50:58 PM4/8/21
to Repo and Gerrit Discussion, Luca Milanesio

On 8 Apr 2021, at 15:51, Nuno Costa <nunoco...@gmail.com> wrote:

Hi Matthias,

Thanks for the reply and checking on your environment.

If that is "working as designed", it is ok for us. Just wanted to make sure of it and pass the information to the team.
Since we are just passing the json file returned by the API call to the monitoring system, we don't need to do anything besides removing a graph that have an empty timeline after the upgrade :)

Hi believe there is a misconception here: you are considering the “show-caches” a monitoring API whilst it really isn’t. 
Also, you are using the thread name of the output as “key” for a metric, which isn’t the purpose of the command.
Using a ’non-feature’ is risky because it may change at any time and we do not have any functional test for it.

Why don’t you rely on Gerrit metrics [1] for extracting that information about SSH?

HTH

Luca.

[4] https://gerrit-documentation.storage.googleapis.com/Documentation/3.3.3/metrics.html#_ssh_sessions 




On Thursday, 8 April 2021 at 15:28:03 UTC+1 Matthias Sohn wrote:
On Thu, Apr 8, 2021 at 2:54 PM Nuno Costa <nunoco...@gmail.com> wrote:

Can you explain what is end user issue with the outputs? What isn’t working in v3.2.7?
I have honestly lost the focus of the discussion :-)
Sorry, some times I want to give as much info as possible and with that some times it does not help :)

To sum up, our issue is that "sshd-SshServer" is no longer part of cli command "gerrit show-caches --show-thread" and REST API call "/a/config/server/summary/" outputs.
We were monitoring that specific metric and with 3.2.7 there is no longer that metric and we wanted to understand why.
 
With regards to the changes, the list of changes between 2.16.7 and 3.2.7 is *huge*, over 9k commits between the two releases.
See the full list of changes on the release notes pages at [1], [2] and [3].

I went through all of them and did not find anything related. Maybe I did not search for the right keywords.
I found documentation still mentioning "sshd-SshServer" and not "sshd-SshDaemon, as I mention in the first post.

I'm just wondering if this is really a bug and we are the only ones affected by it before opening a bug report :)

I think what you are observing is that sshd thread names changed between 2.16 and 3.2.
I can confirm that this is the case by looking at some of our own servers, but I didn't try to find the change
which did this renaming. I don't think we consider thread names to be API.

-Matthias

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

Nuno Costa

unread,
Apr 8, 2021, 2:19:17 PM4/8/21
to Repo and Gerrit Discussion
Hi believe there is a misconception here: you are considering the “show-caches” a monitoring API whilst it really isn’t. 
Also, you are using the thread name of the output as “key” for a metric, which isn’t the purpose of the command.
Using a ’non-feature’ is risky because it may change at any time and we do not have any functional test for it.

Why don’t you rely on Gerrit metrics [1] for extracting that information about SSH?

I cannot agree more with you on that but that's how it was implemented at the time.

I see that elastic search one is deprecated and that would be useful because we use the ELK stack (being fed by json/log files) for the monitoring/analytics.

Without going the custom plugin path, JMX seems to be promising.

Any other tips are appreciated :)

Nuno
 

Luca Milanesio

unread,
Apr 8, 2021, 2:29:34 PM4/8/21
to Repo and Gerrit Discussion, Luca Milanesio

On 8 Apr 2021, at 19:19, Nuno Costa <nunoco...@gmail.com> wrote:

Hi believe there is a misconception here: you are considering the “show-caches” a monitoring API whilst it really isn’t. 
Also, you are using the thread name of the output as “key” for a metric, which isn’t the purpose of the command.
Using a ’non-feature’ is risky because it may change at any time and we do not have any functional test for it.

Why don’t you rely on Gerrit metrics [1] for extracting that information about SSH?

I cannot agree more with you on that but that's how it was implemented at the time.

I see that elastic search one is deprecated and that would be useful because we use the ELK stack (being fed by json/log files) for the monitoring/analytics.

The ELK stack is deprecated as “index backend for Gerrit” but it isn’t deprecated at all for analytics.
See the Gerrit Analytics presentation at [5].


Without going the custom plugin path, JMX seems to be promising.

The Prometheus/Grafana stack is the one you should be looking at [6].

HTH

Luca.



Any other tips are appreciated :)

Nuno
 

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

Nuno Costa

unread,
Apr 8, 2021, 2:40:25 PM4/8/21
to Repo and Gerrit Discussion
Hi believe there is a misconception here: you are considering the “show-caches” a monitoring API whilst it really isn’t. 
Also, you are using the thread name of the output as “key” for a metric, which isn’t the purpose of the command.
Using a ’non-feature’ is risky because it may change at any time and we do not have any functional test for it.

Why don’t you rely on Gerrit metrics [1] for extracting that information about SSH?

I cannot agree more with you on that but that's how it was implemented at the time.

I see that elastic search one is deprecated and that would be useful because we use the ELK stack (being fed by json/log files) for the monitoring/analytics.

The ELK stack is deprecated as “index backend for Gerrit” but it isn’t deprecated at all for analytics.
 
The ES metrics plugin page[7] mentions that is deprecated, so that's what I meant :)
 
See the Gerrit Analytics presentation at [5].

I will, thanks.

Without going the custom plugin path, JMX seems to be promising.

The Prometheus/Grafana stack is the one you should be looking at [6].

Yes, I quite like Grafana. I will definitely take a look on that one, thanks.


Matthias Sohn

unread,
Apr 8, 2021, 3:57:22 PM4/8/21
to Nuno Costa, Repo and Gerrit Discussion
and here is a full kubernetes based monitoring setup with prometheus to collect metrics, loki to collect logs and Grafana for dashboards [8]

Reply all
Reply to author
Forward
0 new messages