Issues with GoCD db.properties file and LDAP after upgrade to 20.4.

[Funkycybermonk]

Hello!

I'm working on an upgrade path from 20.1 through 20.4 and a db migration up to 22.2. A long path and a lot of complicated things changing. The issues started with 20.4 and after a couple of hours of work I decided to go for broke and push through to 22.2. It worked but didn't improve anything.

I feel like something relatively simple is going on but I am missing something and not finding it. 

I'm getting an error on service start "ConnectionManager:117 - The file config\db.properties specified by `go.db.config` does not exist." The file db.properties does exist in the config folder but I'm unable to see what the path is that its looking for prior to config. My installation folder after the two upgrade stages is still "C:\Program Files (x86)\Go Server" and I don't know if that means something is looking in the wrong place or not. I expected 22.2 to migrate it to "C:\Program Files\Go Server" from reading the documentation. 

I also have an issue with LDAP successfully authenticating and logging a success but always sending me back to login. I have removed authentication temporarily to let me work on troubleshooting but still haven't found a solution. 

The LDAP log shows:
2022-10-27 00:22:57,095 INFO  [Thread-79] LdapPlugin:72 - Loading plugin null version 2.2.1-168
2022-10-27 00:22:57,231 ERROR [Thread-79] LdapPlugin:127 - Error while executing request go.plugin-settings.get-configuration
com.thoughtworks.go.plugin.api.exceptions.UnhandledRequestTypeException: This is an invalid request type :go.plugin-settings.get-configuration

Followed by 2022-10-27 00:23:38,205 INFO  [qtp1522792394-33] LdapPlugin:72 - [Authenticate] User `<deleted>` successfully authenticated using auth_config: Domain-LDAP

I'm sort of wondering if something doesn't know where everything is at and is trying to load from incorrect locations but this is the test run for an upgrade on several systems that I can't do a clean wipe/restart on.

Any ideas?

Thanks!

[Chad Wilson]

Hiya

On Thu, Oct 27, 2022 at 8:26 AM Funkycybermonk <chan...@gmail.com> wrote:

Hello!

I'm working on an upgrade path from 20.1 through 20.4 and a db migration up to 22.2. A long path and a lot of complicated things changing. The issues started with 20.4 and after a couple of hours of work I decided to go for broke and push through to 22.2. It worked but didn't improve anything.

I feel like something relatively simple is going on but I am missing something and not finding it. 

I'm getting an error on service start "ConnectionManager:117 - The file config\db.properties specified by `go.db.config` does not exist." The file db.properties does exist in the config folder but I'm unable to see what the path is that its looking for prior to config. My installation folder after the two upgrade stages is still "C:\Program Files (x86)\Go Server" and I don't know if that means something is looking in the wrong place or not. I expected 22.2 to migrate it to "C:\Program Files\Go Server" from reading the documentation. 

GoCD 22.2.0 shouldn't have changed the install location on Windows, especially not for upgrades. It just corrects some permissions, but shouldn't affect you if you are installing in the default Program Files location which it looks like. The docs might be misleading about which Prog Files variant it goes into for various Windows types.

In any case, if the behaviour you see is/was the same between GoCD 20.5.0 (post db migration) and 22.2.0 then we can probably rule out any change for Windows permissions as a cause of your challenges.

In your case, post-20.5.0 migration, are you trying to use a migrated inbuilt H2 database, or a separate Postgres/Mysql install?

• Generally speaking, if you are not trying to either override some H2 configuration nor use an external DB, config\db.properties doesn't exist and isn't required.
• That error message is just a warning. It doesn't mean there is a problem, unless you are expecting it to use it, to know how to connect to an external DB.
  
• The path should be relative to the working directory which should be set by the wrapper at startup. if you want to confirm, that should be logged within logs\go-server-wrapper.log (search for "Working directory") before it launches GoCD.
  

Nevertheless it's weird if you have config\db.properties and it's not working. Perhaps you can check the permissions on the file and see if there is anything unusual, or different than, say a "known good" config that is in the same folder, e.g config\config.xml. Assuming you are running GoCD via a windows service with the default user account of "Local System", it should be able to see the file.

Having said all this, I am not sure I understand what actually happens with your GoCD server. Does it fail to start? it starts, but with a new/fresh local file DB rather than what you expect? It's missing data that you expect to be there?

If there are database issues, I'd have thought it might not be wise to move to the next step of checking LDAP before ensuring database stuff is OK.....

 

I also have an issue with LDAP successfully authenticating and logging a success but always sending me back to login. I have removed authentication temporarily to let me work on troubleshooting but still haven't found a solution. 

The LDAP log shows:
2022-10-27 00:22:57,095 INFO  [Thread-79] LdapPlugin:72 - Loading plugin null version 2.2.1-168
2022-10-27 00:22:57,231 ERROR [Thread-79] LdapPlugin:127 - Error while executing request go.plugin-settings.get-configuration
com.thoughtworks.go.plugin.api.exceptions.UnhandledRequestTypeException: This is an invalid request type :go.plugin-settings.get-configuration

Followed by 2022-10-27 00:23:38,205 INFO  [qtp1522792394-33] LdapPlugin:72 - [Authenticate] User `<deleted>` successfully authenticated using auth_config: Domain-LDAP

I'm sort of wondering if something doesn't know where everything is at and is trying to load from incorrect locations but this is the test run for an upgrade on several systems that I can't do a clean wipe/restart on.

Unfortunately that doesn't tell us too much. :( What this looks like is a configuration problem with the authorization config or an "unknown user"; it has authenticated the user, but the GoCD server isn't allowing that user to login. Do you have "Allow only known users to login" in your Domain-LDAP auth config? If so, is the user you are logging in as listed and enabled in Users Management?

The allowed/permitted users and roles are persisted in the DB (or dynamically added if that option is unchecked) so if you have issues with GoCD connecting to the right DB it might not have loaded the previously "allowed" users you expect - and thus maybe best to ensure the DB is in the right state first?

 

-Chad

[Chantry Conkle]

The upgrade path was migrating from h2 to the postgresql database. It seemed to be recommended and I was hoping for better performance. Everything else that logs shows the full path to the file. This one only shows what I pasted. I pulled the source code and I can find where the error is thrown but I can't seem to find where the initial data is pulled/stored for that path. 

I can go back to h2 if that ends up being the same performance but we have frequent pipelines running at the same time so I was hoping to head off performance issues with the postgresql features.

It's just really odd because 20.1 was running great, 20.4 blew up on start but didn't have any issues during the upgrade that turned out to be the ldap failure and a git url that was a unc path to improve performance and avoid lots of consecutive pulls from the internet repo. Removed the unc path and oddly go seems to work fine with no Auth and no db but I'm not sure how or why. I thought at first maybe it was false-erroring but nothing is updated in the database although no h2 dB is being created. Is there a way to force the path using the wrapper-properties file? I've confirmed permissions are everyone - read-execute and other files in that same folder are being read. 

It seems like something is off with plug-ins though so I'm not sure if that's the core issue but the articles I've seen were suggesting those errors should be warnings. It's those getting 500 errors from the api. 

I can post the debug part if the logs if that helps as well. I still don't understand where it decides the path for that file though. It seems to be in the source but I couldn't find the class that is referenced from other files in the project.

I can roll back and start over but this is virtually a pure install/ upgrade with a clean setup of 20.1 that runs perfectly followed by the 20.4 upgrade, dB migration and then 22.2 upgrade. Failing at 20.4 on leap with no settings changes was odd though. I rebuilt it multiple times and the strict option is disabled so any Auth is supposed to be accepted.

Is there a way to increase logging or anything to provide more detail? (Although I do have a good bit in logs already. 

Thanks! 

--
You received this message because you are subscribed to a topic in the Google Groups "go-cd" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-cd/i3cY4DEA7Dw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAA1RwH_41SFPdJ_Lnn_bJqv%2B%3DUNpGOf0DrXg9NnPQnMsWfah0A%40mail.gmail.com.

[Aravind SV]

Hello,

Is there a way to increase logging or anything to provide more detail?

You can access: http://your-gocd-server:8153/go/api/support (basically /go/api/support) to get more information about paths + much more. Search for: “loc.config.dir”.

You can increase the logging levels (see docs) but it can quickly get overwhelming. I’d suggest that the current level of logs should be enough.

I agree with Chad. I haven’t ever seen config/db.properties not working.

Cheers,
Aravind

You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAHmR7StQUXKMQyGavKbm8krwob98VFOx711X0T9zmWxZw9h1wg%40mail.gmail.com.

[Chad Wilson]

Sorry, one major thing to clarify - when you say 20.4 did you actually mean 20.5? The db migration and OSS support for postgres via config/db.properties is only in 20.5 so if you were getting errors on startup on 20.4, that would be unrelated to any database change.

If we stick to trying to target getting things working on 20.5.0 it might be easier to follow what you've tried. Output from the API Aravind mentions would be useful to avoid confusion.

-Chad

You received this message because you are subscribed to the Google Groups "go-cd" group.

To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAHmR7StQUXKMQyGavKbm8krwob98VFOx711X0T9zmWxZw9h1wg%40mail.gmail.com.

[Chantry Conkle]

What I had done was go to 20.4, then upgrade the DB as a preparation step for continuing to 22.2. I have this time rolled back to 20.1, confirmed it was active and then went directly to 20.5. As expected, the DB was not compatible so I again installed and configured PostgreSQL per docs and upgraded the database from H2->PostgreSQL.

 

I did run into the issue again with the git url being rejected so I’ve done the same fix there but I’m hoping that hasn’t been removed as an allowed feature since we do too many large repo pulls across multiple machines to do over the internet every time.

 

I’m at 20.5 now and back to roughly the same place. LDAP is blowing up with the messages after which authentication is successful but doesn’t redirect into the dashboard.:

I went through the previous process of removing the security section from the cruise config and adding the authorization provider back in as LDAP without checking the box for strict access. It redirects back to login and the loop restarts.

 

I’m still not getting anything that I can see logged into the database, for example running a backup doesn’t add a record to the serverbackups table in the database.

Is there any chance there is a required intermediate step of upgrading to 20.3 or something before going higher?

 

I can also try doing an H2 upgrade instead of going to PostgreSQL but that won’t solve the DB issue, that would just be getting a DB working at all.

 

 I'm happy to send my support page (sanitized) if that helps.

Thanks! 

 

From: go...@googlegroups.com <go...@googlegroups.com> On Behalf Of Chad Wilson
Sent: Thursday, October 27, 2022 3:20 AM
To: go...@googlegroups.com
Subject: Re: [go-cd] Issues with GoCD db.properties file and LDAP after upgrade to 20.4.

 

Sorry, one major thing to clarify - when you say 20.4 did you actually mean 20.5? The db migration and OSS support for postgres via config/db.properties is only in 20.5 so if you were getting errors on startup on 20.4, that would be unrelated to any database change.

.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAA1RwH_ynJ5fVL_w6%2BD9UVvV2n8aefpMdzcq%3DiTk19D0_zAMFQ%40mail.gmail.com.

[Funkycybermonk]

As an add-on to this for diagnostic purposes, I reset back to 20.1 and its literally on 20.2 that the issue occurs so nothing beyond that matters much. Its the version that SSL changed so I don't think that matters but just wanted to add that. I'm trying to get some more debug logging to get a better idea whats going on under the hood when the authentication is finished. Interestingly enough, if I go to /go/api/support I'm prompted for authentication and the LDAP that doesn't seem to do anything on the web server dashboard login, works perfectly there. I know this isn't the case, but this feels sort-of like I'd expect if an authentication token wasn't getting passed or a cookie was being lost on a .net site I was working with.

And to go back to the method of starting, this was a pure 20.1 install with nothing changed except for some templates copied over from another working server. Every agent registration, pipeline group, environment, etc. was all set up by hand so there shouldn't be any garbage in play. I can remove the templates from the config and see if that does anything but I'd think that would result in an error message or loading a blank config rather than always redirecting back to login on successful login. This is with no database change (not that its expected with 20.2 but just to clarify state) so I've quite literally just double clicked on the installer and let it run and then tried logging back in. This server I don't mind doing a bit of diagnostic changes but this is prepping for some pretty complex servers so I'm trying to sort out all the complexities on the side before I start with a production system that is going to be pathing the same (20.1+). 

I did try adding a config line for wrapper.java.additional.100=-Dplugin.cd.go.authentication.ldap.log.level=debug but that only expanded out the steps the ldap plugin was taking to get a successful authentication, it doesn't say what happens afterwards and nothing is being logged to the go-server.log or go-server-wrapper.log when the authentication completes. I also tried going the extra step of recording the session in chrome to see if I could see the redirect happening at the url level but as far as the recording was concerned it just reloads the page.

If there are any tips on logging in the logback-include.xml I'd love to hear it. I've tried just the default xml from the documentation for logging. I'll update as I have more information, I'm just basically trying to adjust all the settings I can find to debug to get as much information as possible. I'm not opposed to doing heavy lifting in the logs, I just haven't yet found the proper thing to give me all the details I'd like to see.

I really appreciate the responses and assistance!!

[Funkycybermonk]

This might be simplified massively. It seems that it will function properly on the local machine as localhost and from a remote machine as the machine name. We alias them as a cname since our servernames are pretty complex so perhaps the issue is that its not properly authenticating/redirecting because it needs to have alternate site name bindings entered somewhere?

I believe I've discovered that the missing config file prompts are not always valid because I've had that message right before logging indicating it loaded the file it thinks it couldn't find. I may try running this on up to 22.2 to see if this is completely just issues with aliasing the site. If that is a known issue/resolution that might solve my problem to know the workaround.

Thanks!

[Aravind SV]

Hello,

We alias them as a cname since our servernames are pretty complex so perhaps the issue is that its not properly authenticating/redirecting because it needs to have alternate site name bindings entered somewhere?

Hmm, I wonder if the site URL and secure site URL are set wrong, and the cookies are being set on the wrong domain? Worth checking.

https://docs.gocd.org/current/installation/configuring_server_details.html#configure-site-urls

Cheers,
Aravind

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/cf3d4c97-0233-42cb-892e-f97c03862c16n%40googlegroups.com.

[chan...@gmail.com]

That did help. It appears that starting in version 20.2 having those fields blank doesn’t allow it as *, it allows only localhost and %servername%. I added the cname url andthat seems to have worked. I think that is a solution honestly. I would have liked to be able  to allow multiple custom names so I can have a friendly direct-to-server as well as a reverse proxy but once in production its unlikely that the direct name will be needed without also having retrieved the server name as well to use for the url.

 

I appreciate the help on this! I think we can call this one solved and I’ll start a DB specific thread if my DB migration still has issues. I think I’ve done enough source crawling and reading documentation to be much better off in that area and worst case I’ll just H2>H2 to make it easier. 😊

 

Thanks to all involved for the help on solving this!

[Chad Wilson]

Good that you're unblocked on this part at least.

Since 20.2 involved the disabling of direct TLS support on the GoCD server itself (unless you temporarily opt-in, which would be a bit pointless in your case since it was completely removed in a later release you intend to upgrade past), any direct to server comms would be HTTP-only on port 8153.

Assuming 8154 is disabled on your GoCD server, and you require TLS via your reverse proxy (which is its core role) you might be able to get away with only setting the secure site URL to your RP addressable name and leaving the other one blank for the localhost/direct to server HTTP connections?

Speaking generally without having dug into GoCD specifics, generally the way of handling this stuff when reverse proxied without server configuration for proxied traffic is to rely upon reverse-proxy-set headers such as X-Forwarded-Host/Port/Protocol or the standard Forwarded HTTP header to determine the "real" URL but it does require some plumbing and I'm not sure if all of GoCD's traffic routes do that consistently. Would be interesting to know if your reverse proxy is setting/propagating those standard headers.

-Chad

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/03f901d8ee0a%24bb01a340%243104e9c0%24%40gmail.com.