Config Repository Not Parsing When Credentials Supplied via Secret (v24.3.0)

[Jason Smyth]

Hello community,

 

I encountered a strange issue whereby config repositories don’t seem to work properly when we try to supply the password via a secret instead of directly in the config repo config. The connection test succeeds, so the system is fetching the password from the secret at that point, but once saved, the config repo fails to parse.

 

The error message (URL redacted) I see in the UI is:

 

There was an error parsing this configuration repository:

MODIFICATION CHECK FAILED FOR MATERIAL: URL: HTTPS://DEV.AZURE.COM/ORGANIZATION/TEAMPROJECT/_GIT/SANDBOX-JASONS, BRANCH: GOCD-PIPELINE-TEST

NO PIPELINES ARE AFFECTED BY THIS MATERIAL, PERHAPS THIS MATERIAL IS UNUSED.

 

Failed to load pipelines defined in this repository: There was an unknown error performing the operation. Possible reason (Not Found)

 

The GoCD server log shows the following warning:

 

2024-10-18 10:47:01 jvm 1    | 2024-10-18 14:47:01,263 WARN  [143@MessageListener for ConfigMaterialUpdateListener] ConfigMaterialUpdateListener:65 - [Config Material Update] Cannot update configuration part because material update has failed. Reason:

 

When I switched from using a secret to directly supplying the password via the UI, the configuration repository started working as intended.

 

I’m reasonably certain that this is a bug, but wanted to check with the community to confirm that using secrets in this way is supposed to be a supported use-case.

 

Any thoughts or guidance would be appreciated.

 

Regards,

Jason Smyth

 

[Chad Wilson]

This sounds weird/unexpected but haven't had time to try reproducing this myself. Which secrets plugin are you using? What's in the logs for the server or the secret plugin itself?

Is the same repo url also used for other materials (whether config repos or normal materials)?

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/DM6PR16MB36715673328A38D68EA7AE8ECF402%40DM6PR16MB3671.namprd16.prod.outlook.com.

[Jason Smyth]

Hi Chad,

My tests were done using the "AWS Secrets Manager plugin for GoCD". There is nothing in go-server.log aside from what I posted in my initial message, and the secret plugin logs are empty. I can try turning up logging verbosity and repeating the tests, but I'm not sure how to do that in my Docker Compose test environment.

I know that the secret plugin is working. I have pipelines configured in a working (because I used a password) config repo, and those pipelines depend on the same secret plugin. Additionally, the test connection button in the config repo modal returns errors if I intentionally misconfigure the secret.

I don't think it has anything to do with the repository contents. I have replicated the behaviour in 2 different test systems, with multiple source repos, including some that are known to be good because they are accessed by other GoCD instances (albeit without secrets plugin references). I have also replicated the issue with both JSON and YAML repos, so I don't think the issue is with the individual configuration repository plugins.

The config repos are not used as additional materials. I tested adding a config repo that pointed to a repo that _is_ used as a material for pipelines, and it works properly. I suspect that this means that the existing material configuration takes precedence and GoCD doesn't bother trying to re-clone, but I could be mistaken there.

If I get a chance, I will try to see if I can replicate the issue with the "GoCD file based Secrets Plugin". That should provide some indication of whether the issue is in the secrets plugin or the core GoCD configuration repo module.

Any other thoughts on things to try?

Regards,

Jason Smyth

[Chad Wilson]

OK, thanks.

Are you working with a "modern" GoCD version, or is this still some older 19.x version?

-Chad

To view this discussion visit https://groups.google.com/d/msgid/go-cd/5e1c44b3-be46-4764-8a39-fb45703f9193n%40googlegroups.com.

[Jason Smyth]

Hi Chad,

 

This is on the latest (I believe) version: v24.3.0.

 

Regards,

Jason Smyth

--
You received this message because you are subscribed to a topic in the Google Groups "go-cd" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-cd/JlzHTa-Vy_0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to go-cd+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/go-cd/CAA1RwH-qHq8U1qXRiZuupuh-T0ojU8j%3DTSbAt4VveEahAP8DMg%40mail.gmail.com.

[Jason Smyth]

Hi Chad,

Here is the error as it appears in the GoCD UI:

I tried the "GoCD file based Secrets Plugin" and replicated the issue with that secret manager as well. I think this suggests that the issue is in the core GoCD configuration repo module. Unless there is something wrong with my test cases.

Thoughts?

Regards,

Jason Smyth

[Chad Wilson]

Not really, other than to raise a proper bug/enhancement report at https://github.com/gocd/gocd/issues preferably with the easiest way to replicate this so it can be tracked. I'd be digging into the GoCD issue history to see if this was expected to work (or worked at some point) the same as you :-)

-Chad

To view this discussion visit https://groups.google.com/d/msgid/go-cd/8b6f8c9a-78fd-45c2-badb-3a46befff0dcn%40googlegroups.com.

[Chad Wilson]

The error you see seems to happen because when validating secret lookup references (to check the rules configured on the secret config) the logic seems to incorrectly assume that any ScmMaterial with secrets will have 1+ pipeline using that secret. It then incorrectly assumes that "all pipelines have secret lookup errors" when it compares # pipelines and # pipelines with errors (0 == 0, oops) and fails.

That's why you get the useless/empty error message and broken resolution. If you turn on DEBUG logging for com.thoughtworks.go.server.materials.MaterialDatabaseUpdater you'll see the below with no error message:

2024-11-03 00:14:00,477 DEBUG  [147@MessageListener for MaterialUpdateListener] MaterialDatabaseUpdater:128 - [Material Update] Modification check failed for material: URL: https://github.com/gocd/aws, Branch: master
com.thoughtworks.go.server.exceptions.RulesViolationException:
        at com.thoughtworks.go.server.service.RulesService.validateSecretConfigReferences(RulesService.java:82)
        at com.thoughtworks.go.server.service.SecretParamResolver.resolve(SecretParamResolver.java:77)
        at com.thoughtworks.go.server.service.SecretParamResolver.resolve(SecretParamResolver.java:67)
        at com.thoughtworks.go.server.service.MaterialService.resolveSecretParams(MaterialService.java:163)
        at com.thoughtworks.go.server.service.MaterialService.latestModification(MaterialService.java:126)
        at com.thoughtworks.go.server.materials.LegacyMaterialChecker.findLatestModification(LegacyMaterialChecker.java:50)
        at com.thoughtworks.go.server.materials.ScmMaterialUpdater.insertLatestOrNewModifications(ScmMaterialUpdater.java:55)
        at com.thoughtworks.go.server.materials.ScmMaterialUpdater.addNewMaterialWithModifications(ScmMaterialUpdater.java:70)
        at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.addNewMaterialWithModifications(MaterialDatabaseUpdater.java:179)
        at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.initializeMaterialWithLatestRevision(MaterialDatabaseUpdater.java:137)
        at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater$1.doInTransaction(MaterialDatabaseUpdater.java:95)
        at com.thoughtworks.go.server.transaction.TransactionCallback.doWithExceptionHandling(TransactionCallback.java:23)
        at com.thoughtworks.go.server.transaction.TransactionTemplate.lambda$executeWithExceptionHandling$1(TransactionTemplate.java:43)
        at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
        at com.thoughtworks.go.server.transaction.TransactionTemplate.executeWithExceptionHandling(TransactionTemplate.java:40)
        at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.updateMaterial(MaterialDatabaseUpdater.java:92)
        at com.thoughtworks.go.server.materials.MaterialUpdateListener.onMessage(MaterialUpdateListener.java:64)
        at com.thoughtworks.go.server.materials.MaterialUpdateListener.onMessage(MaterialUpdateListener.java:32)
        at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:83)
        at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:63)
        at java.base/java.lang.Thread.run(Thread.java:1583)

If I fix the error handling so it doesn't incorrectly consider it a rules violation it seems to work OK. However I'll need to think a little bit more about whether it should. :-)

I think if you add at least 1 pipeline that refers to the same logical material (same type, URL, branch, username) you might find it works as expected?

Test Connection works because it seems to resolve secrets without validating the references in the same way .... for better or worse. Arguably it should be validating references here also.

-Chad

[Chad Wilson]

Did another few checks and it did indeed work OK if there is 1+ pipeline sharing the material so no harm fixing the case you discovered - have also improved the error message and the logging associated to be a bit less misleading. Fixed the underlying problem here - release in 24.4.0 (out now) - give it a go?

-Chad

[Jason Smyth]

Hi Chad,

Thank you so much for this fix!

I tested in v24.4.0 and confirm that this is now working.

Cheers,

Jason Smyth