cf target failed: 401 Unauthorized; CC: Authorization is required' (NATS::ServerError)

[Iwan Winoto]

Hi.
I followed the steps in http://docs.cloudfoundry.com/docs/running/deploying-cf/openstack/install_cf_openstack.html.
I have OpenStack Folsom with Nova networking. Deployment with micro-bosh works, but when I try to target CF, I get:

$ cf target http://api.192.168.200.2.xip.io:8080
Setting target to http://api.192.168.200.2.xip.io:8080... FAILED
CFoundry::Unauthorized: 401: 401 Unauthorized

Checking vms, the cloud_controller job is showing failing intermittently.

$ bosh vms
Deployment `cf-demo'

Director task 191

Task 191 done

+---------------------+---------+---------------+-------------------------------+
| Job/index           | State   | Resource Pool | IPs                           |
+---------------------+---------+---------------+-------------------------------+
| cloud_controller/0  | failing | common        | 192.168.100.8                 |
| dea/0               | running | large         | 192.168.100.11                |
| health_manager/0    | running | common        | 192.168.100.9                 |
| nats/0              | running | common        | 192.168.100.4                 |
| nfs_server/0        | running | common        | 192.168.100.6                 |
| postgres/0          | running | common        | 192.168.100.2                 |
| router/0            | running | common        | 192.168.100.10, 192.168.200.2 |
| syslog_aggregator/0 | running | common        | 192.168.100.5                 |
| uaa/0               | running | common        | 192.168.100.7                 |
+---------------------+---------+---------------+-------------------------------+

VMs total: 9

At other times I run bosh vms and all jobs show as running.

I downloaded the logs for cloud_controller and in cloud_controller_ng/cloud_controller_ng.stderr.log I see a lot of entries of:

/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/nats-0.4.26/lib/nats/client.rb:567:in `block in connection_completed': 'Authorization is required' (NATS::ServerError)
        from /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/nats-0.4.26/lib/nats/client.rb:506:in `call'
        from /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/nats-0.4.26/lib/nats/client.rb:506:in `receive_data'
        from /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in `run_machine'
        from /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in `run'
        from /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/runner.rb:92:in `run!'
        from /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/bin/cloud_controller:12:in `<main>'

I saw an entry from Dr Nic with the same log output (https://groups.google.com/a/cloudfoundry.org/forum/#!topicsearch/Authorization$20is$20required/vcap-dev/sjPlwv1km-U). The suggested fix was to remove dns from the network in the deployment yml. Dr. Nic's post didn't specify which job, so I tried for just cloud_controller as well as all jobs. Neither changes made any difference.

cloud_controller_ng/cloud_controller_ng.log showing NATS registration log:

{"timestamp":1379299306.676946,"message":"reusing default serving domain: 192.168.200.2.xip.io","log_level":"info","source":"cc.db.domain","data":{},"thread_id":14223020,"fiber_id":37402740,"process_id":26794,"file":"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/core/domain.rb","lineno":149,"method":"block in find_or_create_shared_domain"}
{"timestamp":1379299306.9081013,"message":"Connected to NATS - router registration","log_level":"info","source":"cf.registrar","data":{},"thread_id":14223020,"fiber_id":37402740,"process_id":26794,"file":"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/vcap_common-2.2.1/lib/cf/registrar.rb","lineno":62,"method":"register_with_router"}
{"timestamp":1379299306.9115846,"message":"Sending registration: {:host=>\"192.168.100.8\", :port=>9022, :uris=>[\"ccng.192.168.200.2.xip.io\", \"api.192.168.200.2.xip.io\"], :tags=>{:component=>\"CloudController\"}, :index=>0, :private_instance_id=>nil}","log_level":"debug","source":"cf.registrar","data":{},"thread_id":14223020,"fiber_id":37402740,"process_id":26794,"file":"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/vcap_common-2.2.1/lib/cf/registrar.rb","lineno":96,"method":"send_registration_message"}

Stemcells I've used:

$ bosh stemcells

+---------------+---------+--------------------------------------+
| Name          | Version | CID                                  |
+---------------+---------+--------------------------------------+
| bosh-stemcell | 877     | 2302e4fc-38ae-4d69-9319-ea9afde70770 |
| bosh-stemcell | 939     | 3ca5db47-6ac0-4dc5-9f0b-0127b08cfcb5 |
| bosh-stemcell | 962     | f717056b-ba51-4cf4-aea0-910157e63f4a |
| bosh-stemcell | 991     | 9581def9-66e7-47c2-950b-a586dded7898 |
+---------------+---------+--------------------------------------+

CF release:

$ bosh releases

+------+----------+-------------+
| Name | Versions | Commit Hash |
+------+----------+-------------+
| cf   | 138*     | adca9c45+   |
+------+----------+-------------+
(*) Currently deployed
(+) Uncommitted changes

Releases total: 1

Deployment yaml are as per instructions from 13/09/13, except persistent disks are 15GB: demo.yml

Any help appreciated.

[Dr Nic Williams]

Can you try targeting the API url without :8080? As an aside, where did this port come from so I can fix the docs?

The access to CCNG is via the router, not directly (which would be port 9022) which watches for HTTP traffic on port 80.

Nic

To unsubscribe from this group and stop receiving emails from it, send an email to bosh-users+...@cloudfoundry.org.

--

Dr Nic Williams

Stark & Wayne LLC - consultancy for Cloud Foundry users

http://drnicwilliams.com

http://starkandwayne.com

cell +1 (415) 860-2185

twitter @drnic

[Dr Nic Williams]

I see that you've hardcoded 8080 into the demo.yml - any reason you prefer 8080 over the default 80?

[Dr Nic Williams]

My gut feeling is that overriding the gorouter port isn't supported really. All the other internal jobs that want to connect to ccng & uaa will do so via the router and will assume (through omission) that its port 80.

[Ferran Rodenas]

Yep, the gorouter port is fixed at port 80: https://github.com/cloudfoundry/cf-release/blob/master/jobs/gorouter/templates/gorouter.yml.erb#L34. The 8080 port is the default port for the gorouter varz/healthz endpoint.

- Ferdy

2013/9/16 Dr Nic Williams <drnicw...@gmail.com>

[Iwan Winoto]

Thanks Nic,

I cut & pasted the demo.yml from the docs. The only thing I changed was persistent disk size.
I just checked the docs, and in the properties section, it has:

  router:
    status:
      port: 8080
      user: gorouter
      password: <%= common_password %>

I'll change this to port 80 and redeploy.

Cheers,
e1

[Ferran Rodenas]

You don't need to modify/redeploy. Just target your CF without the 8080 port: cf target http://api.192.168.200.2.xip.io

- Ferdy

2013/9/16 Iwan Winoto <iwi...@gmail.com>

[Iwan Winoto]

Hi Nic, Ferdy.
Thanks for you help to date. Sorry I haven't responded till now. I've discovered some issues with the Folsom installation which could be related to thhe CF deploy. So I'm resolving those and will then redeploy.
Hopefully I won't be back on this thread.
Thanks again for your help.
Cheers,
e1