Code_manager checkout fails during puppet run with a 401/404 error

[evan....@noaa.gov]

I had Codemanger and git_lab running well, but had to move change the server name. After the name change, Git_lab is running fine from the GUI, and for code check-in. However the puppetmaster when it does its puppet run fails to connect to the control-repo. I am pretty sure the errors are being triggered by the fact puppet is getting 401 error from the git host. And I am pretty sure this means I have corrected an entry somewhere in the manifest or need to change keys somewhere. Suggestions?

Debug log Message from the Puppet run:

Debug: gitlab_deploy_key::api_call: REST API GET Endpoint: http://nwcal-srv07.nwc.nws.noaa.gov/api/v3/projects/puppet%2Fcontrol-repo
Debug: gitlab_deploy_key::api_call: REST API GET Request: #<Net::HTTP::Get GET>
opening connection to nwcal-srv07.nwc.nws.noaa.gov:80...
opened
<- "GET /api/v3/projects/puppet%2Fcontrol-repo HTTP/1.1\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nAccept: */*\r\nUser-Agent: Ruby\r\nContent-Type: application/json\r\nPrivate-Token: ***************************\r\nConnection: close\r\nHost: nwcal-srv07.nwc.nws.noaa.gov\r\n\r\n"
-> "HTTP/1.1 404 Not Found\r\n"
-> "Date: Thu, 17 Mar 2016 16:41:06 GMT\r\n"
-> "Server: Apache/2.4.6 (CentOS)\r\n"
-> "Cache-Control: no-cache\r\n"
-> "Content-Length: 35\r\n"
-> "Content-Type: application/json\r\n"
-> "Status: 404 Not Found\r\n"
-> "Vary: Origin\r\n"
-> "X-Request-Id: aa9af9c9-9e59-4a2b-bcc8-c97be8f8035c\r\n"
-> "X-Runtime: 0.028008\r\n"
-> "Connection: close\r\n"
-> "\r\n"
reading 35 bytes...
-> "{\"message\":\"404 Project Not Found\"}"
read 35 bytes
Conn close
Debug: gitlab_deploy_key::api_call: REST API GET Response: #<Net::HTTPNotFound 404 Not Found readbody=true>
Debug: gitlab_deploy_key::api_call: REST API GET Endpoint: http://nwcal-srv07.nwc.nws.noaa.gov/api/v3/projects/0/keys
Debug: gitlab_deploy_key::api_call: REST API GET Request: #<Net::HTTP::Get GET>
opening connection to nwcal-srv07.nwc.nws.noaa.gov:80...
opened
<- "GET /api/v3/projects/0/keys HTTP/1.1\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nAccept: */*\r\nUser-Agent: Ruby\r\nContent-Type: application/json\r\nPrivate-Token: ********************\r\nConnection: close\r\nHost: nwcal-srv07.nwc.nws.noaa.gov\r\n\r\n"
-> "HTTP/1.1 404 Not Found\r\n"
-> "Date: Thu, 17 Mar 2016 16:41:06 GMT\r\n"
-> "Server: Apache/2.4.6 (CentOS)\r\n"
-> "Cache-Control: no-cache\r\n"
-> "Content-Length: 35\r\n"
-> "Content-Type: application/json\r\n"
-> "Status: 404 Not Found\r\n"
-> "Vary: Origin\r\n"
-> "X-Request-Id: bd185376-d065-4b4d-b2be-6bec9000ce35\r\n"
-> "X-Runtime: 0.023668\r\n"
-> "Connection: close\r\n"
-> "\r\n"
reading 35 bytes...
-> "{\"message\":\"404 Project Not Found\"}"
read 35 bytes
Conn close
Debug: gitlab_deploy_key::api_call: REST API GET Response: #<Net::HTTPNotFound 404 Not Found readbody=true>
Error: /Stage[main]/Profile::Git_webhook::Code_manager/Git_deploy_key[add_deploy_key_to_puppet_control-nwcal-srv06.nwc.nws.noaa.gov]: Could not evaluate: no implicit conversion of String into Integer
Debug: gitlab_webhook::api_call: REST API GET Endpoint: http://nwcal-srv07.nwc.nws.noaa.gov/api/v3/projects/puppet%2Fcontrol-repo
Debug: gitlab_webhook::api_call: REST API GET Request: #<Net::HTTP::Get GET>
opening connection to nwcal-srv07.nwc.nws.noaa.gov:80...
opened
<- "GET /api/v3/projects/puppet%2Fcontrol-repo HTTP/1.1\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nAccept: */*\r\nUser-Agent: Ruby\r\nContent-Type: application/json\r\nPrivate-Token: **********************\r\nConnection: close\r\nHost: nwcal-srv07.nwc.nws.noaa.gov\r\n\r\n"
-> "HTTP/1.1 404 Not Found\r\n"
-> "Date: Thu, 17 Mar 2016 16:41:06 GMT\r\n"
-> "Server: Apache/2.4.6 (CentOS)\r\n"
-> "Cache-Control: no-cache\r\n"
-> "Content-Length: 35\r\n"
-> "Content-Type: application/json\r\n"
-> "Status: 404 Not Found\r\n"
-> "Vary: Origin\r\n"
-> "X-Request-Id: b3f757d2-7ca7-42bf-8a62-bb67facfa7da\r\n"
-> "X-Runtime: 0.013062\r\n"
-> "Connection: close\r\n"
-> "\r\n"
reading 35 bytes...
-> "{\"message\":\"404 Project Not Found\"}"
read 35 bytes
Conn close
Debug: gitlab_webhook::api_call: REST API GET Response: #<Net::HTTPNotFound 404 Not Found readbody=true>
Debug: gitlab_webhook::api_call: REST API GET Endpoint: http://nwcal-srv07.nwc.nws.noaa.gov/api/v3/projects/0/hooks
Debug: gitlab_webhook::api_call: REST API GET Request: #<Net::HTTP::Get GET>
opening connection to nwcal-srv07.nwc.nws.noaa.gov:80...
opened
<- "GET /api/v3/projects/0/hooks HTTP/1.1\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nAccept: */*\r\nUser-Agent: Ruby\r\nContent-Type: application/json\r\nPrivate-Token: ***********************\r\nConnection: close\r\nHost: nwcal-srv07.nwc.nws.noaa.gov\r\n\r\n"
-> "HTTP/1.1 404 Not Found\r\n"
-> "Date: Thu, 17 Mar 2016 16:41:06 GMT\r\n"
-> "Server: Apache/2.4.6 (CentOS)\r\n"
-> "Cache-Control: no-cache\r\n"
-> "Content-Length: 35\r\n"
-> "Content-Type: application/json\r\n"
-> "Status: 404 Not Found\r\n"
-> "Vary: Origin\r\n"
-> "X-Request-Id: a1885023-728f-47f3-a0d8-b5d9dd3cf62b\r\n"
-> "X-Runtime: 0.015593\r\n"
-> "Connection: close\r\n"
-> "\r\n"
reading 35 bytes...
-> "{\"message\":\"404 Project Not Found\"}"
read 35 bytes
Conn close
Debug: gitlab_webhook::api_call: REST API GET Response: #<Net::HTTPNotFound 404 Not Found readbody=true>
Error: /Stage[main]/Profile::Git_webhook::Code_manager/Git_webhook[code_manager_post_receive_webhook-nwcal-srv06.nwc.nws.noaa.gov]: Could not evaluate: no implicit conversion of String into Integer
Debug: Class[Profile::Git_webhook::Code_manager]: Resource is being skipped, unscheduling all events

[Nick Walker]

Hi Evan,

It looks like you still need to change the gitlab server url via your hieradata.  

This is the example, but you'll need to change it in the file that corresponds to your master.  

https://github.com/puppetlabs/control-repo/blob/production/hieradata/nodes/example-puppet-master.yaml#L3

[Andrew Brader]

Evan,

    If you read the output closely, you'll see the following line:

"{\"message\":\"404 Project Not Found\"}"

That's a pretty clear indication of the following:

1) Project doesn't exist

2) Project exists but you made a typo in the 'project_name' param 

3) The GitLab user you are using to access the REST API doesn't have sufficient rights

Can you confirm project_name is correct and the GitLab has correct permissions: https://github.com/abrader/abrader-gms#permissions-to-use-api

-Andrew