fresh Jenkins installation does not present the dashboard

[Matthias Apitz]

Hello,

I've installed a fresh Jenkins jenkins-2.401.2-1.1.noarch.rpm on a RH8
server without the old jobs and config (not moved from the old server).

The server starts fine and when I connect with the browser to port 8080,
I'm asked for a first administrator password from the file
/var/lib/jenkins/secrets/initialAdminPassword
When I provide it, the page in the browser only presents some ongoing
pattern, three small vertical bars which are moving up and down, but
nothing more.

The configured log shows:

...
Jenkins initial setup is required. An admin user has been created and a password generated.
Please use the following password to proceed to installation:

74565d94443f4ed08790bf86da7953fd

This may also be found at: /var/lib/jenkins/secrets/initialAdminPassword

*************************************************************
*************************************************************
*************************************************************

2023-07-24 11:04:09.232+0000 [id=34] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization
2023-07-24 11:04:09.251+0000 [id=23] INFO hudson.lifecycle.Lifecycle#onReady: Jenkins is fully up and running
2023-07-24 11:05:32.147+0000 [id=17] WARNING h.i.i.InstallUncaughtExceptionHandler#handleException
java.lang.IllegalStateException: Response is committed
at org.eclipse.jetty.server.Request.getSession(Request.java:1549)
at org.eclipse.jetty.server.Request.getSession(Request.java:1531)
at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:229)
...
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078)
at java.base/java.lang.Thread.run(Thread.java:829)

(end of log)

What is missing or what I could have done wrong?

Thanks

matthias

--
Matthias Apitz, ✉ gu...@unixarea.de, http://www.unixarea.de/ +49-176-38902045
Public GnuPG key: http://www.unixarea.de/key.pub

[Matthias Apitz]

El día Monday, July 24, 2023 a las 01:30:56PM +0200, Matthias Apitz escribió:

>
> Hello,
>
> I've installed a fresh Jenkins jenkins-2.401.2-1.1.noarch.rpm on a RH8
> server without the old jobs and config (not moved from the old server).
>
> The server starts fine and when I connect with the browser to port 8080,
> I'm asked for a first administrator password from the file
> /var/lib/jenkins/secrets/initialAdminPassword
> When I provide it, the page in the browser only presents some ongoing
> pattern, three small vertical bars which are moving up and down, but
> nothing more.
>

> ...

UPDATE:

I spent a lot of time in tracing with tcpdump and Linux' strace to
understand what is going on between the browser and Jenkins, and also
between Jenkins and resources in Internet:

After providing the initial admin password from the browser to Jenkins,
there is some HTTP dialog between the browser and Jenkins server;
the browser sends a lot of all the same requests of the form
GET /updateCenter/connectionStatus?siteId=default...

# egrep 'HTTP: GET /update' tcp2 | wc -l
2283

i.e. thousands of such requests!

The jenkins server itself is requesting data from two servers in
Internet:

# grep 142.250.181.228 tcp2 | grep '\[S'
10:51:26.222340 IP 10.201.64.28.48674 > 142.250.181.228.https: Flags [S], seq 2438595054, win 29200, options [mss 1460,sackOK,TS val 1475725650 ecr 0,nop,wscale 7], length 0

# grep 52.202.51.185 tcp2 | grep '\[S'
10:51:26.236951 IP 10.201.64.28.41834 > 52.202.51.185.https: Flags [S], seq 3893898537, win 29200, options [mss 1460,sackOK,TS val 1476244307 ecr 0,nop,wscale 7], length 0
10:51:26.327508 IP 52.202.51.185.https > 10.201.64.28.41834: Flags [S.], seq 351494233, ack 3893898538, win 62643, options [mss 1460,sackOK,TS val 3802601119 ecr 1476244307,nop,wscale 7], length 0
11:07:24.055143 IP 10.201.64.28.46718 > 52.202.51.185.https: Flags [S], seq 3933852832, win 29200, options [mss 1460,sackOK,TS val 1477202126 ecr 0,nop,wscale 7], length 0
11:07:24.146115 IP 52.202.51.185.https > 10.201.64.28.46718: Flags [S.], seq 2048943343, ack 3933852833, win 62643, options [mss 1460,sackOK,TS val 3803558940 ecr 1477202126,nop,wscale 7], length 0

The latter is to be configured in the file:

# cat hudson.model.UpdateCenter.xml
<?xml version='1.1' encoding='UTF-8'?>
<sites>
<site>
<id>default</id>
<url>https://updates.jenkins.io/update-center.json</url>
</site>
</sites>

The first I have no idea, where is this configured.

The box is sitting in some data center and is not fully connected to
Internet... How could those external request to Internet be switched off?

The browser is still sending those GET requests, already 3181 for 30
minutes or so.

[Daniel Beck]

On Wed, Jul 26, 2023 at 11:21 AM Matthias Apitz <gu...@unixarea.de> wrote:

The jenkins server itself is requesting data from two servers in
Internet:

…

The first I have no idea, where is this configured.

That's google.com, something set up long ago as an attempt to determine whether there's any internet connectivity. See https://github.com/jenkinsci/jenkins/blob/7fb5f24efd92eda2461e9bf2f1fdc50295e12573/core/src/main/java/hudson/model/UpdateCenter.java#L1414 or https://github.com/jenkinsci/jenkins/blob/7fb5f24efd92eda2461e9bf2f1fdc50295e12573/core/src/main/java/hudson/model/UpdateSite.java#L437 (the amount of unused/legacy code in this area makes it difficult to tell at a glance which is called) or an equivalent field in the JSON data retrieved from updates.jenkins.io.

 

The box is sitting in some data center and is not fully connected to
Internet... How could those external request to Internet be switched off?

This looks like a bug. The setup wizard should identify that the requests fail, and offer to configure a proxy, or let you skip plugin installation. Do some of the requests pass? If so, it might be different from a basic "no networking" situation in ways not anticipated.

That said, the setup wizard has fairly little value if Jenkins is offline. You can bypass it, but note that it'll start with authorization + authentication set so that anonymous users are admins; so you'll want to customize things quickly on startup, e.g. with an init.groovy script, ideally before exposing the HTTP port to anyone other than localhost.

[Matthias Apitz]

El día miércoles, julio 26, 2023 a las 11:49:51 +0200, 'Daniel Beck' via Jenkins Users escribió:

> On Wed, Jul 26, 2023 at 11:21 AM Matthias Apitz <gu...@unixarea.de> wrote:
>
> > The jenkins server itself is requesting data from two servers in
> > Internet:
> >
> > …
> >
> > The first I have no idea, where is this configured.
> >
>

> ...

>
>
> > The box is sitting in some data center and is not fully connected to
> > Internet... How could those external request to Internet be switched off?
> >
>
> This looks like a bug. The setup wizard should identify that the requests
> fail, and offer to configure a proxy, or let you skip plugin installation.
> Do some of the requests pass? If so, it might be different from a basic "no
> networking" situation in ways not anticipated.

Thanks for your kind and helping reply.

Jenkins tries more servers with HTTPS (grepped from a tcpdump file):

142.250.186.100.https: Flags [S]
162.159.140.26.https: Flags [S]
193.149.136.101.https: Flags [S]
193.149.136.134.https: Flags [S]
193.149.136.16.https: Flags [S]
193.149.136.168.https: Flags [S]
34.226.68.35.https: Flags [S]
34.236.161.191.https: Flags [S]
52.202.51.185.https: Flags [S]
52.203.25.223.https: Flags [S]
54.183.140.32.https: Flags [S]
54.241.197.58.https: Flags [S]
54.67.26.184.https: Flags [S]
54.67.54.116.https: Flags [S]
54.67.92.206.https: Flags [S]

From the TCP/IP it looks like the connections get established and data is coming
down. Due to HTTPS I can say what's in the packages. In any case, the
directory plugins remains empty.


Because I spent some time reading the tcpdump, I see now that the
browser shows a page saying:


Offline
Offline

This Jenkins instance appears to be offline.

For information about installing Jenkins without an internet connection, see the Offline Jenkins Installation Documentation.

You may choose to continue by configuring a proxy or skipping plugin installation.
...

Is there a way to see in logs what is missing? I have already level set
to 9. The last line in the log is:

[2023-07-26 14:14:16][INFO ][hudson.lifecycle.Lifecycle onReady] Jenkins is fully up and running

before typing in the password into the browser. I'd need this to talk to
Internal IT Departement.

[Matthias Apitz]

El día miércoles, julio 26, 2023 a las 11:49:51 +0200, 'Daniel Beck' via Jenkins Users escribió:

> On Wed, Jul 26, 2023 at 11:21 AM Matthias Apitz <gu...@unixarea.de> wrote:
>
> > The jenkins server itself is requesting data from two servers in
> > Internet:
> >
> > …
> >
> > The first I have no idea, where is this configured.
> >
>
> That's google.com, something set up long ago as an attempt to determine
> whether there's any internet connectivity. See
> https://github.com/jenkinsci/jenkins/blob/7fb5f24efd92eda2461e9bf2f1fdc50295e12573/core/src/main/java/hudson/model/UpdateCenter.java#L1414
> or
> https://github.com/jenkinsci/jenkins/blob/7fb5f24efd92eda2461e9bf2f1fdc50295e12573/core/src/main/java/hudson/model/UpdateSite.java#L437
> (the amount of unused/legacy code in this area makes it difficult to tell
> at a glance which is called) or an equivalent field in the JSON data
> retrieved from updates.jenkins.io.

If I look into the above Java files, they have a lot of log statements,
like

LOGGER.log(Level.FINE, "Failed to parse releaseTimestamp for "

I don't get why I'm not able to get all logs into the configured file
/var/log/jenkins/jenkins.log; my used log configuration is:

cat logging.properties
.level=DEBUG
handlers=java.util.logging.ConsoleHandler

java.util.logging.ConsoleHandler.level=ALL
# java.util.logging.ConsoleHandler.formatter=java.util.logging.SimpleFormatter
java.util.logging.SimpleFormatter.format = [%1$tF %1$tT][%4$-6s][%2$s] %5$s %6$s %n

hudson.security.csrf.CrumbFilter.level = DEBUG
hudson.plugins.git.GitStatus.level = DEBUG

and for 'FINE' I only see:

grep FINE /var/log/jenkins/jenkins.log
[2023-07-27 08:39:06][FINER ][winstone.Logger logInternal] Using JAVA_HOME=/usr/lib/jvm/java-17-openjdk-17.0.7.0.7-1.el8_7.x86_64
[2023-07-27 08:39:06][FINER ][winstone.Logger logInternal] No common lib folder found
[2023-07-27 08:39:06][FINER ][winstone.Logger logInternal] Deploying host found at default
[2023-07-27 08:39:06][FINER ][winstone.Logger logInternal] ArgumentsRealm initialised: users: 0
[2023-07-27 08:39:06][FINER ][winstone.Logger logInternal] The webroot supplied already exists - overwriting where newer (/var/cache/jenkins/war)
[2023-07-27 08:39:06][FINER ][winstone.Logger logInternal] Access logging disabled - no logger class defined
[2023-07-27 08:39:06][FINER ][winstone.Logger logInternal] Initialized 1 webapps: prefixes - [/]
[2023-07-27 08:39:06][FINER ][winstone.Logger logInternal] Initialized in non-virtual-host mode

Something must be missing in the log config.

[Daniel Beck]

On Thu, Jul 27, 2023 at 10:15 AM Matthias Apitz <gu...@unixarea.de> wrote:

I don't get why I'm not able to get all logs into the configured file
/var/log/jenkins/jenkins.log; my used log configuration is:

cat logging.properties
.level=DEBUG

DEBUG isn't a legal level. There should be a corresponding error logged. 

[Matthias Apitz]

What would be a legal level?

[Thomas Markus]

Hi,

Am 27.07.23 um 11:21 schrieb Matthias Apitz:

DEBUG isn't a legal level. There should be a corresponding error logged.

What would be a legal level?

	matthias

see java.util.logging.Level

• SEVERE (highest value)
• WARNING
• INFO
• CONFIG
• FINE
• FINER
• FINEST (lowest value)

best regards
Thomas

[Matthias Apitz]

I used the moment when Jenkins in the browser said to me, that I'm
offline and if I want to continue without updating the plugins. I
created an account and when I now update plugins manually, some work
fine, sometimes it fails with a message in the log file, like:

[2023-07-28 09:50:31][SEVERE][hudson.model.UpdateCenter$DownloadJob run] Failed to install scm-api
java.io.IOException: Failed to dynamically deploy this plugin
at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:2204)
at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1850)
...

It even says sometimes in the browser (like right now) that the update
of a given plugin was successful, but that a test to Google.com failed:

Download progress
Preparation

Checking internet connectivity
Checking update center connectivity
Success
Failed to connect to https://www.google.com/. Perhaps you need to configure HTTP proxy?

Mina SSHD API :: Core Downloaded Successfully. Will be activated during the next boot

When I watch such a plugin update with tcpdump for all traffic to/from
port 443 such fetch is very slow and lazy. I looked in the file for
the update: the JSON file in https://updates.jenkins.io/update-center.json
it has 3413 URLs for downloading and I understand now why this took
always so long in the first initialization of Jenkins.

What should I do. Our IT department will say: 443 outbound is all open
...

Thanks

[Daniel Beck]

On Fri, Jul 28, 2023 at 10:30 AM Matthias Apitz <gu...@unixarea.de> wrote:

[2023-07-28 09:50:31][SEVERE][hudson.model.UpdateCenter$DownloadJob run] Failed to install scm-api
java.io.IOException: Failed to dynamically deploy this plugin
        at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:2204)
        at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1850)
        ...

Might have a number of causes, not all of them network-related.

 

When I watch such a plugin update with tcpdump for all traffic to/from
port 443 such fetch is very slow and lazy. I looked in the file for
the update: the JSON file in https://updates.jenkins.io/update-center.json
it has 3413 URLs for downloading and I understand now why this took
always so long in the first initialization of Jenkins.

None of them are actually downloaded. The JSON file is basically all that's being downloaded.

 

Our IT department will say: 443 outbound is all open

Can you curl from the Jenkins controller system to e.g. google.com?

Any exceptions getting logged?

[Matthias Apitz]

El día viernes, julio 28, 2023 a las 10:37:54 +0200, 'Daniel Beck' via Jenkins Users escribió:

> > Our IT department will say: 443 outbound is all open
> >
>
> Can you curl from the Jenkins controller system to e.g. google.com?
>
> Any exceptions getting logged?

Please be more specific what you want me to test from the Jenkins
controller.

When I curl from the Linux cmd line:

$ curl --verbose https://www.google.com/bla
* Trying 142.250.186.100...
* TCP_NODELAY set
* Connected to www.google.com (142.250.186.100) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):

it hangs. Sould it says '404 bla not found"

the JSON is downloaded fine and fast:

$ curl https://updates.jenkins.io/current/update-center.json | wc -c
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 2770k 100 2770k 0 0 2823k 0 --:--:-- --:--:-- --:--:-- 2820k
2836543

[Daniel Beck]

On Fri, Jul 28, 2023 at 11:21 AM Matthias Apitz <gu...@unixarea.de> wrote:

* TLSv1.3 (OUT), TLS handshake, Client hello (1):

it hangs. Sould it says '404 bla not found"

This indicates that something other than Jenkins is broken, so it's a system/environment issue. Fix that first, then try again with Jenkins.

[Matthias Apitz]

At the end of the day it turned out that IT had to enable a service "app
google-base" to let curl and openssl to www.google.com working as they
should, dont know what the diff is between HTTPS and "google-base" is
and dont have access to the firewall.

Daniel, thanks for your helping hints;

--
Matthias Apitz
E-mail: gu...@unixarea.de
WWW: http://www.unixarea.de/
phone: +49-170-4527211

> --
> You received this message because you are subscribed to the Google
> Groups "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jenkinsci-use...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jenkinsci-users/CAMo7Pt%2Bx9SKZcEifv4JPOJBNBd1WydQYcaMGcrXfBiNGwKWi%2Bg%40mail.gmail.com
> [1].
>
>
> Links:
> ------
> [1]
> https://groups.google.com/d/msgid/jenkinsci-users/CAMo7Pt%2Bx9SKZcEifv4JPOJBNBd1WydQYcaMGcrXfBiNGwKWi%2Bg%40mail.gmail.com?utm_medium=email&utm_source=footer

[Matthias Apitz]

El día miércoles, julio 26, 2023 a las 11:49:51a. m. +0200, 'Daniel Beck' via Jenkins Users escribió:

> On Wed, Jul 26, 2023 at 11:21 AM Matthias Apitz <gu...@unixarea.de> wrote:
>
> > The jenkins server itself is requesting data from two servers in
> > Internet:
> >
> > …
> >
> > The first I have no idea, where is this configured.
> >
>
> That's google.com, something set up long ago as an attempt to determine
> whether there's any internet connectivity. See
> https://github.com/jenkinsci/jenkins/blob/7fb5f24efd92eda2461e9bf2f1fdc50295e12573/core/src/main/java/hudson/model/UpdateCenter.java#L1414
> or
> https://github.com/jenkinsci/jenkins/blob/7fb5f24efd92eda2461e9bf2f1fdc50295e12573/core/src/main/java/hudson/model/UpdateSite.java#L437
> (the amount of unused/legacy code in this area makes it difficult to tell
> at a glance which is called) or an equivalent field in the JSON data
> retrieved from updates.jenkins.io.

I've pulled the sources from github.com because I was curious and
because the Java code uses HTTP, while the problem was with HTTPS. It must
come from this entry in the JSON file:

updateCenter.post(
{"connectionCheckUrl":"https://www.google.com/","core":{"buildDate":"Jul
24,
2023","name":"core","sha1":"U9iGDX21CIj9lQ1P/ejIS1F/GRY=","sha256":"zrPBceJXKLR2RYSNriY+QaEsSRGscRVJhfBlPn+iEcg=","size":89541474,
...