SCM Import via HTTPS with Git/AWS CodeCommit throwing exception

[pmelling...@gmail.com]

Hello!

I'm going to tell you everything I know about this issue. Hopefully it isn't too much detail. Alex-SF on IRC asked me to post this here.

I installed Rundeck 2.9.3 on Centos 7.3 with the RPM and tried to set up SCM Import to pull in jobs and things. Things originally exported from our staging host - also Rundeck 2.9.3 but on Centos 6.

The staging host was set up to use SSH with SCM Export, because I didn't know that outbound SSH is blocked in our prod datacenters. I'm the new guy. *shrug*

I followed these instructions to configure GIT. http://docs.aws.amazon.com/codecommit/latest/userguide/setting-up-https-unixes.html

It works from the command line. I can clone and do git commands just fine.

I configured SCM Import like this:

Import UUID Behavior: archive 

File Path Template: jobs/${job.group}${job.name}-${job.id}.${config.format} 

Base Directory: /var/rundeck/projects/rundeckAdmin/scm 

Git URL: https://user...@git-codecommit.us-east-1.amazonaws.com/v1/repos/reponame 

Branch: master 

SSH: Strict Host Key Checking: no 

Password Storage Path: keys/git/pw

Format: xml 

Fetch Automatically: false

When I try to enable the import, i get the following error in the UI:

Failed fetch from the repository: https://user...@codecommiturl.com: cannot open git-upload-pack; sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

I looked into it, and it seems this can usually be solved by importing new certs.

I checked the keystore and the root cert was in there already. So i imported the intermediate and site certs. Same error.

Is it possible I imported then into the wrong cacerts? Is there one specific to Rundeck that's not the global java one? 

Any help with this would be greatly appreciated. Thank you in advance!

Here's the full stack from /var/log/rundeck/service.log:

ERROR ScmService: Failed to initialize SCM import plugin git-import for rundeckAdmin: Failed fetch from the repository: https://user...@git-codecommit.us-east-1.amazonaws.com/v1/repos/reponame : cannot open git-upload-pack; sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

com.dtolabs.rundeck.plugins.scm.ScmPluginException: Failed fetch from the repository: https://user...@git-codecommit.us-east-1.amazonaws.com/v1/repos/reponame : cannot open git-upload-pack; sun.security.validator.ValidatorException: PKIX path building failed: s$n.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

        at org.rundeck.plugin.scm.git.BaseGitPlugin.cloneOrCreate(BaseGitPlugin.groovy:552)

        at org.rundeck.plugin.scm.git.GitImportPlugin.setup(GitImportPlugin.groovy:121)

        at org.rundeck.plugin.scm.git.GitImportPlugin.initialize(GitImportPlugin.groovy:78)

        at org.rundeck.plugin.scm.git.GitImportPluginFactory.createPlugin(GitImportPluginFactory.groovy:77)

        at rundeck.services.ScmService.loadImportPluginWithConfig(ScmService.groovy:738)

        at rundeck.services.ScmService.loadPluginWithConfig(ScmService.groovy:706)

        at rundeck.services.ScmService.initPlugin(ScmService.groovy:435)

        at rundeck.services.ScmService.initProject(ScmService.groovy:145)

        at rundeck.services.ScmService.initProject(ScmService.groovy:122)

        at rundeck.services.ScmService.projectHasConfiguredExportPlugin(ScmService.groovy:205)

        at rundeck.controllers.MenuController.jobsFragment(MenuController.groovy:418)

        at rundeck.controllers.MenuController.jobs(MenuController.groovy:264)

        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:696)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1515)

        at grails.plugin.cache.web.filter.PageFragmentCachingFilter.doFilter(PageFragmentCachingFilter.java:198)

        at grails.plugin.cache.web.filter.AbstractFilter.doFilter(AbstractFilter.java:63)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)

        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:519)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138)

        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:582)

        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213)

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1097)

        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:448)

        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175)

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1031)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)

        at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:261)

        at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:101)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)

        at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)

        at com.dtolabs.rundeck.server.filters.AuthFilter.doFilter(AuthFilter.java:74)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1486)

        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:519)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138)

        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:529)

        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213)

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1097)

        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:448)

        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175)

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1031)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

        at org.eclipse.jetty.server.Server.handle(Server.java:446)

        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:271)

        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:246)

        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358)

        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601)

        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532)

        at java.lang.Thread.run(Thread.java:748)

Caused by: org.eclipse.jgit.api.errors.TransportException: https://user...@git-codecommit.us-east-1.amazonaws.com/v1/repos/reponame : cannot open git-upload-pack

        at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:139)

        at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:76)

        at org.rundeck.plugin.scm.git.BaseGitPlugin.fetchFromRemote(BaseGitPlugin.groovy:230)

        at org.rundeck.plugin.scm.git.BaseGitPlugin.cloneOrCreate(BaseGitPlugin.groovy:548)

        ... 55 more

Caused by: org.eclipse.jgit.errors.TransportException: https://user...@git-codecommit.us-east-1.amazonaws.com/v1/repos/reponame : cannot open git-upload-pack

        at org.eclipse.jgit.transport.TransportHttp.connect(TransportHttp.java:524)

        at org.eclipse.jgit.transport.TransportHttp.openFetch(TransportHttp.java:309)

        at org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:136)

        at org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:122)

        at org.eclipse.jgit.transport.Transport.fetch(Transport.java:1138)

        at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:130)

        ... 58 more

Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)

        at com.sun.net.ssl.internal.www.protocol.https.HttpsURLConnectionOldImpl.getResponseCode(HttpsURLConnectionOldImpl.java:308)

        at org.eclipse.jgit.transport.http.JDKHttpConnection.getResponseCode(JDKHttpConnection.java:98)

        at org.eclipse.jgit.util.HttpSupport.response(HttpSupport.java:168)

        at org.eclipse.jgit.transport.TransportHttp.connect(TransportHttp.java:475)

        ... 63 more

Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

        ... 68 more

Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

        at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)

        ... 68 more

[Luis Toledo]

Hi pmelling,

This worked for me:

1) On the IAM user I added the AWSCodeCommitFullAccess policy

2) On Security Credentials tags I uploaded a  "SSH keys for AWS CodeCommit" and following the instructions of the Step 3 on

http://docs.aws.amazon.com/codecommit/latest/userguide/setting-up-ssh-unixes.html

(you need to set up the ~/.ssh/config as described the document)

3) On Security Credentials tags  y generate a new credential on "HTTPS Git credentials for AWS CodeCommit"

4) I added the password generated on the "HTTPS Git credentials" on a "Password Key Storage" on Rundeck

5) I set the SCM import/export on Rundeck, eg:

 

Please let me know if this helps

Luis

On Friday, September 8, 2017 at 4:24:56 PM UTC-3, pmelling...@gmail.com wrote:

Hello!

I'm going to tell you everything I know about this issue. Hopefully it isn't too much detail. Alex-SF on IRC asked me to post this here.

I installed Rundeck 2.9.3 on Centos 7.3 with the RPM and tried to set up SCM Import to pull in jobs and things. Things originally exported from our staging host - also Rundeck 2.9.3 but on Centos 6.

The staging host was set up to use SSH with SCM Export, because I didn't know that outbound SSH is blocked in our prod datacenters. I'm the new guy. *shrug*

I followed these instructions to configure GIT. http://docs.aws.amazon.com/codecommit/latest/userguide/setting-up-https-unixes.html

It works from the command line. I can clone and do git commands just fine.

I configured SCM Import like this:

Import UUID Behavior: archive 

File Path Template: jobs/${job.group}${job.name}-${job.id}.${config.format} 

Base Directory: /var/rundeck/projects/rundeckAdmin/scm 

Git URL: https://username@git-codecommit.us-east-1.amazonaws.com/v1/repos/reponame 

Branch: master 

SSH: Strict Host Key Checking: no 

Password Storage Path: keys/git/pw

Format: xml 

Fetch Automatically: false

When I try to enable the import, i get the following error in the UI:

Failed fetch from the repository: https://username@codecommiturl.com: cannot open git-upload-pack; sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

I looked into it, and it seems this can usually be solved by importing new certs.

I checked the keystore and the root cert was in there already. So i imported the intermediate and site certs. Same error.

Is it possible I imported then into the wrong cacerts? Is there one specific to Rundeck that's not the global java one? 

Any help with this would be greatly appreciated. Thank you in advance!

Here's the full stack from /var/log/rundeck/service.log:

ERROR ScmService: Failed to initialize SCM import plugin git-import for rundeckAdmin: Failed fetch from the repository: https://username@git-codecommit.us-east-1.amazonaws.com/v1/repos/reponame : cannot open git-upload-pack; sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

com.dtolabs.rundeck.plugins.scm.ScmPluginException: Failed fetch from the repository: https://username@git-codecommit.us-east-1.amazonaws.com/v1/repos/reponame : cannot open git-upload-pack; sun.security.validator.ValidatorException: PKIX path building failed: s$n.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Caused by: org.eclipse.jgit.api.errors.TransportException: https://username@git-codecommit.us-east-1.amazonaws.com/v1/repos/reponame : cannot open git-upload-pack

        at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:139)

        at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:76)

        at org.rundeck.plugin.scm.git.BaseGitPlugin.fetchFromRemote(BaseGitPlugin.groovy:230)

        at org.rundeck.plugin.scm.git.BaseGitPlugin.cloneOrCreate(BaseGitPlugin.groovy:548)

        ... 55 more

Caused by: org.eclipse.jgit.errors.TransportException: https://username@git-codecommit.us-east-1.amazonaws.com/v1/repos/reponame : cannot open git-upload-pack

[pmelling...@gmail.com]

Luis, thanks for the response.

That's pretty much how I've got it set up too.

1) My user has AWSCodeCommitPowerUser policy instead. I would rather not elevate it's permissions.

2) Outbound SSH won't work on this host, but because of the way my deployment scripts work, it's configured for it. The SSH config deployed to this host works great in another environment.

3) Yeah, same.

4). Yeah, same.

5). Yeah, same.

So, I didn't change anything. And, was getting the same error.

I decided to start over. I very carefully went through these instructions again as the rundeck user: http://docs.aws.amazon.com/codecommit/latest/userguide/setting-up-https-unixes.html

Still, the same error. No change.

I then had the idea to disable ssl cert verification in git. https://stackoverflow.com/questions/41930608/jenkins-git-integration-how-to-disable-ssl-certificate-validation

So, this is what the rundeck user's .gitconfig looks like:

[credential]

        helper = !aws codecommit credential-helper $@

        UseHttpPath = true

[http]

sslVerify = false

And, I'm happy to say the SSL error is gone, and has been replaced with a new error that I can't figure out.

"Failed fetch from the repository: com.sun.net.ssl.internal.www.protocol.https.HttpsURLConnectionOldImpl cannot be cast to javax.net.ssl.HttpsURLConnection"

When I change "UseHttpPath = true" to "UseHttpPath = false", the error message is a little bit more informative, perhaps:

"Failed cloning the repository from https://user...@git-codecommit.us-east-1.amazonaws.com/v1/repos/reponame.git: com.sun.net.ssl.internal.www.protocol.https.HttpsURLConnectionOldImpl cannot be cast to javax.net.ssl.HttpsURLConnection"

If you're still reading this, I appreciate it a lot. Thank you.

I'm so stumped. Any help is welcome.