Proxy problems with CF + possible bug in cf cli?
271 views
Skip to first unread message
yaco...@gmail.com
Aug 28, 2014, 2:42:50 AM8/28/14
to vcap...@cloudfoundry.org
Hi all.
1)
I have a CF system installed with nise bosh (latest version) which is running on a server behind a port-forwarding gateway that forwards port X to 80, and port Y to 443 (https).
The CF was an old version (how do I check the version?) which didn't support HTTPS and I decided to upgrade it to the latest version so it'd support HTTPS and be up to date.
The problem is that when I try to do a cf target while setting HTTP_PROXY to the gateway's host:port the cf cli fails targetting with an error: Error performing request: Get https://api.[server-internal-
1)
I have a CF system installed with nise bosh (latest version) which is running on a server behind a port-forwarding gateway that forwards port X to 80, and port Y to 443 (https).
The CF was an old version (how do I check the version?) which didn't support HTTPS and I decided to upgrade it to the latest version so it'd support HTTPS and be up to date.
The problem is that when I try to do a cf target while setting HTTP_PROXY to the gateway's host:port the cf cli fails targetting with an error: Error performing request: Get https://api.[server-internal-
ip].xip.io/v2/info: unexpected EOF.
I get a similar error (EOF) when I try to use my own java CF client (java.net.SocketException: Unexpected end of file from server)
The port forwarding works (I checked it by turning off the CF environment, running nc -l 443 on the server and doing telnet to the external port of the gateway.
I have no idea why the CF login fails, and I suspect it to be related to SSL. When the old CF version was running which accepts HTTP, the port forwarding and cf target\cf login always worked.
Does anyone have any idea what's the problem and how do I fix it?
2)
I really need this thing to work because clients\project partners are using the CF environment and they don't have ssh access to the machine, they can only use the cf cli through the port forwarding gateway that now doesn't work.
So I tried bypassing the problem in the following way:
We have a tomcat server running in the internal network behind the gateway and the gateway forwards another port to the tomcat server's port.
So I wrote a simpl servlet reverse proxy that forwards every request that starts with "/v2/" to api.[server-internal-ip].xip.io, and other requests to uaa.[server-internal-ip].xip.io (for the authentication).
The tomcat server shares different apps on it, so I can't make the app take the "/" path and I must use some string (like "/proxy").
So I target the servlet and the "API endpoint"s URL is the following:
So when I for example query the apps, the cf cli goes to /proxy/v2/spaces:
But when I CF push, after the push is done and the CF cli queries the staging process, it goes to /v2/jobs instead of /proxy/v2/jobs.
Is this a bug in CF CLI?
I get a similar error (EOF) when I try to use my own java CF client (java.net.SocketException: Unexpected end of file from server)
The port forwarding works (I checked it by turning off the CF environment, running nc -l 443 on the server and doing telnet to the external port of the gateway.
I have no idea why the CF login fails, and I suspect it to be related to SSL. When the old CF version was running which accepts HTTP, the port forwarding and cf target\cf login always worked.
Does anyone have any idea what's the problem and how do I fix it?
2)
I really need this thing to work because clients\project partners are using the CF environment and they don't have ssh access to the machine, they can only use the cf cli through the port forwarding gateway that now doesn't work.
So I tried bypassing the problem in the following way:
We have a tomcat server running in the internal network behind the gateway and the gateway forwards another port to the tomcat server's port.
So I wrote a simpl servlet reverse proxy that forwards every request that starts with "/v2/" to api.[server-internal-ip].xip.io, and other requests to uaa.[server-internal-ip].xip.io (for the authentication).
The tomcat server shares different apps on it, so I can't make the app take the "/" path and I must use some string (like "/proxy").
So I target the servlet and the "API endpoint"s URL is the following:
D:\cfv6\cf.exe target
VERSION:
6.5.0-9838b31
API endpoint: http://*******/proxy (API version: 2.11.0)
User: *****
Org: *****
Space: *****
So when I for example query the apps, the cf cli goes to /proxy/v2/spaces:
D:\cfv6\cf.exe apps
VERSION:
6.5.0-9838b31
Getting apps in org ***** / space ***** as admin...
REQUEST: [2014-08-28T09:17:52+03:00]
GET /proxy/v2/spaces/85a5ee1b-94ad-447b-9ad5-b92394d56f1f/summary HTTP/1.1
Host: *********
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Content-Type: application/json
User-Agent: go-cli 6.5.0-9838b31 / windows
RESPONSE: [2014-08-28T09:17:53+03:00]
HTTP/1.1 200 OK
Content-Length: 2426
Date: Thu, 28 Aug 2014 06:18:37 GMT
Server: Apache-Coyote/1.1
But when I CF push, after the push is done and the CF cli queries the staging process, it goes to /v2/jobs instead of /proxy/v2/jobs.
![]()
REQUEST: [2014-08-28T09:06:07+03:00]
GET /v2/jobs/8068cf85-f333-4346-b546-bdb26f62b7e3 HTTP/1.1
Host: *******
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Content-Type: application/json
User-Agent: go-cli 6.5.0-9838b31 / windows
RESPONSE: [2014-08-28T09:06:08+03:00]
HTTP/1.1 404 Not Found
Content-Length: 1039
Content-Language: en
Content-Type: text/html;charset=utf-8
Date: Thu, 28 Aug 2014 06:06:52 GMT
Server: Apache-Coyote/1.1
<html><head><title>Apache Tomcat/7.0.54 - Error report</title><style><!--H1 {font-family:T
ahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-fami
ly:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-
family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:T
ahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,
sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color :
black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 404 - /v2/jobs/8068cf
85-f333-4346-b546-bdb26f62b7e3</h1><HR size="1" noshade="noshade"><p><b>type</b> Status re
port</p><p><b>message</b> <u>/v2/jobs/8068cf85-f333-4346-b546-bdb26f62b7e3</u></p><p><b>de
scription</b> <u>The requested resource is not available.</u></p><HR size="1" noshade="nos
hade"><h3>Apache Tomcat/7.0.54</h3></body></html>
FAILED
Error uploading application.
Server error, status code: 404, error code: 0, message:
FAILED
Error uploading application.
Server error, status code: 404, error code: 0, message:
Is this a bug in CF CLI?
Greg Oehmen
Aug 28, 2014, 1:36:53 PM8/28/14
to vcap...@cloudfoundry.org, yaco...@gmail.com
You can determine what release of CF you are using via BOSH.
1. Verify you have `bosh target` set to the BOSH you used to deploy CF.
2. Use `bosh deployments` to see the release version of CF that is current.
Greg
Greg Oehmen
Aug 28, 2014, 7:59:55 PM8/28/14
to vcap...@cloudfoundry.org, yaco...@gmail.com
Also, the CLI team is looking into a similar issue via this story: https://www.pivotaltracker.com/story/show/77846300
yacov manevich
Aug 29, 2014, 10:33:29 AM8/29/14
to Greg Oehmen, vcap...@cloudfoundry.org
Greg, I'm using nise-bosh, I don't think I can do these bosh target and bosh deployments.
In the directory "releases" the latest release yml file is 180, so I guess that's the release. I upgraded the CF this week, so it should've downloaded the latest release.
yaco...@gmail.com
Aug 31, 2014, 3:16:38 AM8/31/14
to vcap...@cloudfoundry.org, yaco...@gmail.com
I have another problem I discovered after the upgrade:
when I login to some orgs I get an error:
FAILED
Error finding available spaces
Server error, status code: 500, error code: 10001, message: An unknown error occurred.
The cloud_controller_ng logs reads:
{"timestamp":1409468665.087893,"message":"Request failed: 500: {\"code\"=>10001, \"description\"=>\"bad decrypt\", \"error_code\"=>\"CF-CipherError\", \"backtrace\"=>[\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud
_controller/encryptor.rb:32:in `final'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/encryptor.rb:32:in `block in run_cipher'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/clou
d_controller/encryptor.rb:32:in `tap'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/encryptor.rb:32:in `run_cipher'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controll
er/encryptor.rb:20:in `decrypt'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/runtime/app.rb:310:in `environment_json'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/sequel_plugins/vcap_s
erialization.rb:24:in `block in to_hash'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/sequel_plugins/vcap_serialization.rb:22:in `each'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/sequel_plu
gins/vcap_serialization.rb:22:in `to_hash'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/runtime/app.rb:544:in `to_hash'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/re
st_controller/preloaded_object_serializer.rb:38:in `to_hash'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/preloaded_object_serializer.rb:108:in `block (2 levels) in serialize_relati
onships'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/preloaded_object_serializer.rb:107:in `map'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controlle
r/rest_controller/preloaded_object_serializer.rb:107:in `block in serialize_relationships'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/preloaded_object_serializer.rb:81:in `each'\"
, \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/preloaded_object_serializer.rb:81:in `serialize_relationships'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_
controller/rest_controller/preloaded_object_serializer.rb:69:in `block in relations_hash'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/preloaded_object_serializer.rb:59:in `tap'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/preloaded_object_serializer.rb:59:in `relations_hash'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/
rest_controller/preloaded_object_serializer.rb:37:in `to_hash'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/preloaded_object_serializer.rb:29:in `serialize'\", \"/var/vcap/packages/
cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/paginated_collection_renderer.rb:71:in `block in render_json'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controll
er/paginated_collection_renderer.rb:71:in `map'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/paginated_collection_renderer.rb:71:in `render_json'\", \"/var/vcap/packages/cloud_contr
oller_ng/cloud_controller_ng/app/controllers/base/model_controller.rb:132:in `enumerate_related'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/base/base_controller.rb:76:in `dispatch'\", \"/var/vcap/pack
ages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/routes.rb:52:in `block (2 levels) in define_to_many_routes'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_contr
oller/routes.rb:18:in `yield'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/routes.rb:18:in `block in define_route'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/ve
ndor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1540:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1540:in `block in compile!'\", \
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:950:in `[]'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra
-1.4.3/lib/sinatra/base.rb:950:in `block (3 levels) in route!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:966:in `route_eval'\", \"/var/vcap/packages/cl
oud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:950:in `block (2 levels) in route!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinat
ra-1.4.3/lib/sinatra/base.rb:987:in `block in process_route'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:985:in `catch'\", \"/var/vcap/packages/cloud_con
troller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:985:in `process_route'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra
/base.rb:948:in `block in route!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:947:in `each'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_
ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:947:in `route!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1059:in `block in dispatch
!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `block in invoke'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ru
by/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `catch'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `invoke'\", \"/var/vcap/packages/clou
d_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1056:in `dispatch!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinat
ra/base.rb:882:in `block in call!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `block in invoke'\", \"/var/vcap/packages/cloud_controller_ng/clou
d_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `catch'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `invo
ke'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:882:in `call!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/g
ems/sinatra-1.4.3/lib/sinatra/base.rb:870:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/xss_header.rb:18:in `call'\", \"/var/vcap/package
s/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/path_traversal.rb:16:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gem
s/rack-protection-1.5.0/lib/rack/protection/json_csrf.rb:18:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/base.rb:49:in `call'\", \"/var/
vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/base.rb:49:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/g
ems/rack-protection-1.5.0/lib/rack/protection/frame_options.rb:31:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/nulllogger.rb:9:in `call'\", \"/var/vcap/packag
es/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/head.rb:11:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/bas
e.rb:175:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1949:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bund
le/ruby/1.9.1/gems/rack-1.5.2/lib/rack/builder.rb:138:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/urlmap.rb:65:in `block in call'\", \"/var/vcap/packages/clo
ud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/urlmap.rb:50:in `each'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/urlmap.rb:50:
in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/builder.rb:138:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1
/gems/thin-1.6.2/lib/thin/connection.rb:86:in `block in pre_process'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/thin-1.6.2/lib/thin/connection.rb:84:in `catch'\", \"/var/vcap/packages/cl
oud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/thin-1.6.2/lib/thin/connection.rb:84:in `pre_process'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/li
b/eventmachine.rb:1037:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/lib/eventmachine.rb:1037:in `block in spawn_threadpool'\"]}","log_level":"error","source":"
cc.api","data":{"request_guid":"6ba1e100-9786-46d4-6963-60faa9fba84c::7aa5854f-a612-4341-95b8-54af8506cfce"},"thread_id":40010340,"fiber_id":41321960,"process_id":9686,"file":"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/li
b/sinatra/vcap.rb","lineno":53,"method":"block in registered"}
Does anyone know what could be the problem and how do I fix it? I currently have a service broker servlet (forwards requests to a backend service broker) in one of these un-accessible orgs, and I'd like to move the service broker's URL to a new one since the app is in an org that's not accessible.
I issued: cf update-service-broker broker-name user password url but got the same error:
Updating service broker universalbroker as admin...
FAILED
Server error, status code: 500, error code: 10001, message: An unknown error occurred.
{"timestamp":1409469193.1475592,"message":"Request failed: 500: {\"code\"=>10001, \"description\"=>\"bad decrypt\", \"error_code\"=>\"CF-CipherError\", \"backtrace\"=>[\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/clou
d_controller/encryptor.rb:32:in `final'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/encryptor.rb:32:in `block in run_cipher'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/clo
ud_controller/encryptor.rb:32:in `tap'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/encryptor.rb:32:in `run_cipher'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_control
ler/encryptor.rb:20:in `decrypt'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/services/service_broker.rb:29:in `auth_password'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ru
by/1.9.1/gems/sequel-4.11.0/lib/sequel/plugins/dirty.rb:198:in `change_column_value'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-4.11.0/lib/sequel/model/base.rb:1174:in `[]='\", \"
/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/sequel_plugins/vcap_normalization.rb:6:in `[]='\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-4.11.0/lib/sequel/model/b
ase.rb:850:in `auth_password='\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/services/service_broker.rb:39:in `auth_password='\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/rub
y/1.9.1/gems/sequel-4.11.0/lib/sequel/model/base.rb:2035:in `block in set_restricted'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-4.11.0/lib/sequel/model/base.rb:2032:in `each'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-4.11.0/lib/sequel/model/base.rb:2032:in `set_restricted'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby
/1.9.1/gems/sequel-4.11.0/lib/sequel/model/base.rb:1544:in `set'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/services/service_brokers_controller.rb:44:in `update'\", \"/var/vcap/packages/cloud_controll
er_ng/cloud_controller_ng/app/controllers/base/base_controller.rb:76:in `dispatch'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/routes.rb:16:in `block in define_route'\", \"/var/vca
p/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1540:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3
/lib/sinatra/base.rb:1540:in `block in compile!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:950:in `[]'\", \"/var/vcap/packages/cloud_controller_ng/clou
d_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:950:in `block (3 levels) in route!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/b
ase.rb:966:in `route_eval'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:950:in `block (2 levels) in route!'\", \"/var/vcap/packages/cloud_controller_ng/cl
oud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:987:in `block in process_route'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/bas
e.rb:985:in `catch'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:985:in `process_route'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:948:in `block in route!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:947:in `each'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:947:in `route!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1059:in `block in dispatch!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `block in invoke'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `catch'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `invoke'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1056:in `dispatch!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:882:in `block in call!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `block in invoke'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `catch'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `invoke'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:882:in `call!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:870:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/xss_header.rb:18:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/path_traversal.rb:16:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/json_csrf.rb:18:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/base.rb:49:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/base.rb:49:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/frame_options.rb:31:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/nulllogger.rb:9:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/head.rb:11:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:175:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1949:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/builder.rb:138:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/urlmap.rb:65:in `block in call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/urlmap.rb:50:in `each'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/urlmap.rb:50:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/builder.rb:138:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/thin-1.6.2/lib/thin/connection.rb:86:in `block in pre_process'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/thin-1.6.2/lib/thin/connection.rb:84:in `catch'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/thin-1.6.2/lib/thin/connection.rb:84:in `pre_process'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/lib/eventmachine.rb:1037:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/lib/eventmachine.rb:1037:in `block in spawn_threadpool'\"]}","log_level":"error","source":"cc.api","data":{"request_guid":"55f62dfa-598b-4b11-41cc-e6282c768a98::512d168d-ec2b-4a37-a7ef-e46428cd47ef"},"thread_id":40032640,"fiber_id":41336340,"process_id":9686,"file":"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/sinatra/vcap.rb","lineno":53,"method":"block in registered"}
What do I do now to clean up this mess? when I upgraded the CF I simply deployed the new version, without touching the /var/vcap directory. The old orgs\apps are still there, but some are un-accessible...
when I login to some orgs I get an error:
FAILED
Error finding available spaces
Server error, status code: 500, error code: 10001, message: An unknown error occurred.
The cloud_controller_ng logs reads:
{"timestamp":1409468665.087893,"message":"Request failed: 500: {\"code\"=>10001, \"description\"=>\"bad decrypt\", \"error_code\"=>\"CF-CipherError\", \"backtrace\"=>[\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud
_controller/encryptor.rb:32:in `final'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/encryptor.rb:32:in `block in run_cipher'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/clou
d_controller/encryptor.rb:32:in `tap'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/encryptor.rb:32:in `run_cipher'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controll
er/encryptor.rb:20:in `decrypt'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/runtime/app.rb:310:in `environment_json'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/sequel_plugins/vcap_s
erialization.rb:24:in `block in to_hash'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/sequel_plugins/vcap_serialization.rb:22:in `each'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/sequel_plu
gins/vcap_serialization.rb:22:in `to_hash'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/runtime/app.rb:544:in `to_hash'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/re
st_controller/preloaded_object_serializer.rb:38:in `to_hash'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/preloaded_object_serializer.rb:108:in `block (2 levels) in serialize_relati
onships'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/preloaded_object_serializer.rb:107:in `map'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controlle
r/rest_controller/preloaded_object_serializer.rb:107:in `block in serialize_relationships'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/preloaded_object_serializer.rb:81:in `each'\"
, \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/preloaded_object_serializer.rb:81:in `serialize_relationships'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_
controller/rest_controller/preloaded_object_serializer.rb:69:in `block in relations_hash'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/preloaded_object_serializer.rb:59:in `tap'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/preloaded_object_serializer.rb:59:in `relations_hash'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/
rest_controller/preloaded_object_serializer.rb:37:in `to_hash'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/preloaded_object_serializer.rb:29:in `serialize'\", \"/var/vcap/packages/
cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/paginated_collection_renderer.rb:71:in `block in render_json'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controll
er/paginated_collection_renderer.rb:71:in `map'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/paginated_collection_renderer.rb:71:in `render_json'\", \"/var/vcap/packages/cloud_contr
oller_ng/cloud_controller_ng/app/controllers/base/model_controller.rb:132:in `enumerate_related'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/base/base_controller.rb:76:in `dispatch'\", \"/var/vcap/pack
ages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/routes.rb:52:in `block (2 levels) in define_to_many_routes'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_contr
oller/routes.rb:18:in `yield'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/routes.rb:18:in `block in define_route'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/ve
ndor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1540:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1540:in `block in compile!'\", \
"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:950:in `[]'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra
-1.4.3/lib/sinatra/base.rb:950:in `block (3 levels) in route!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:966:in `route_eval'\", \"/var/vcap/packages/cl
oud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:950:in `block (2 levels) in route!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinat
ra-1.4.3/lib/sinatra/base.rb:987:in `block in process_route'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:985:in `catch'\", \"/var/vcap/packages/cloud_con
troller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:985:in `process_route'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra
/base.rb:948:in `block in route!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:947:in `each'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_
ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:947:in `route!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1059:in `block in dispatch
!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `block in invoke'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ru
by/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `catch'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `invoke'\", \"/var/vcap/packages/clou
d_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1056:in `dispatch!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinat
ra/base.rb:882:in `block in call!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `block in invoke'\", \"/var/vcap/packages/cloud_controller_ng/clou
d_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `catch'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `invo
ke'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:882:in `call!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/g
ems/sinatra-1.4.3/lib/sinatra/base.rb:870:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/xss_header.rb:18:in `call'\", \"/var/vcap/package
s/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/path_traversal.rb:16:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gem
s/rack-protection-1.5.0/lib/rack/protection/json_csrf.rb:18:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/base.rb:49:in `call'\", \"/var/
vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/base.rb:49:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/g
ems/rack-protection-1.5.0/lib/rack/protection/frame_options.rb:31:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/nulllogger.rb:9:in `call'\", \"/var/vcap/packag
es/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/head.rb:11:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/bas
e.rb:175:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1949:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bund
le/ruby/1.9.1/gems/rack-1.5.2/lib/rack/builder.rb:138:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/urlmap.rb:65:in `block in call'\", \"/var/vcap/packages/clo
ud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/urlmap.rb:50:in `each'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/urlmap.rb:50:
in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/builder.rb:138:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1
/gems/thin-1.6.2/lib/thin/connection.rb:86:in `block in pre_process'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/thin-1.6.2/lib/thin/connection.rb:84:in `catch'\", \"/var/vcap/packages/cl
oud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/thin-1.6.2/lib/thin/connection.rb:84:in `pre_process'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/li
b/eventmachine.rb:1037:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/lib/eventmachine.rb:1037:in `block in spawn_threadpool'\"]}","log_level":"error","source":"
cc.api","data":{"request_guid":"6ba1e100-9786-46d4-6963-60faa9fba84c::7aa5854f-a612-4341-95b8-54af8506cfce"},"thread_id":40010340,"fiber_id":41321960,"process_id":9686,"file":"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/li
b/sinatra/vcap.rb","lineno":53,"method":"block in registered"}
Does anyone know what could be the problem and how do I fix it? I currently have a service broker servlet (forwards requests to a backend service broker) in one of these un-accessible orgs, and I'd like to move the service broker's URL to a new one since the app is in an org that's not accessible.
I issued: cf update-service-broker broker-name user password url but got the same error:
Updating service broker universalbroker as admin...
FAILED
Server error, status code: 500, error code: 10001, message: An unknown error occurred.
{"timestamp":1409469193.1475592,"message":"Request failed: 500: {\"code\"=>10001, \"description\"=>\"bad decrypt\", \"error_code\"=>\"CF-CipherError\", \"backtrace\"=>[\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/clou
d_controller/encryptor.rb:32:in `final'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/encryptor.rb:32:in `block in run_cipher'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/clo
ud_controller/encryptor.rb:32:in `tap'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/encryptor.rb:32:in `run_cipher'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_control
ler/encryptor.rb:20:in `decrypt'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/services/service_broker.rb:29:in `auth_password'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ru
by/1.9.1/gems/sequel-4.11.0/lib/sequel/plugins/dirty.rb:198:in `change_column_value'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-4.11.0/lib/sequel/model/base.rb:1174:in `[]='\", \"
/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/sequel_plugins/vcap_normalization.rb:6:in `[]='\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-4.11.0/lib/sequel/model/b
ase.rb:850:in `auth_password='\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/services/service_broker.rb:39:in `auth_password='\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/rub
y/1.9.1/gems/sequel-4.11.0/lib/sequel/model/base.rb:2035:in `block in set_restricted'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-4.11.0/lib/sequel/model/base.rb:2032:in `each'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sequel-4.11.0/lib/sequel/model/base.rb:2032:in `set_restricted'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby
/1.9.1/gems/sequel-4.11.0/lib/sequel/model/base.rb:1544:in `set'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/services/service_brokers_controller.rb:44:in `update'\", \"/var/vcap/packages/cloud_controll
er_ng/cloud_controller_ng/app/controllers/base/base_controller.rb:76:in `dispatch'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/routes.rb:16:in `block in define_route'\", \"/var/vca
p/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1540:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3
/lib/sinatra/base.rb:1540:in `block in compile!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:950:in `[]'\", \"/var/vcap/packages/cloud_controller_ng/clou
d_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:950:in `block (3 levels) in route!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/b
ase.rb:966:in `route_eval'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:950:in `block (2 levels) in route!'\", \"/var/vcap/packages/cloud_controller_ng/cl
oud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:987:in `block in process_route'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/bas
e.rb:985:in `catch'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:985:in `process_route'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:948:in `block in route!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:947:in `each'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:947:in `route!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1059:in `block in dispatch!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `block in invoke'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `catch'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `invoke'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1056:in `dispatch!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:882:in `block in call!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `block in invoke'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `catch'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1041:in `invoke'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:882:in `call!'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:870:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/xss_header.rb:18:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/path_traversal.rb:16:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/json_csrf.rb:18:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/base.rb:49:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/base.rb:49:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-protection-1.5.0/lib/rack/protection/frame_options.rb:31:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/nulllogger.rb:9:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/head.rb:11:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:175:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/sinatra-1.4.3/lib/sinatra/base.rb:1949:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/builder.rb:138:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/urlmap.rb:65:in `block in call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/urlmap.rb:50:in `each'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/urlmap.rb:50:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/rack-1.5.2/lib/rack/builder.rb:138:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/thin-1.6.2/lib/thin/connection.rb:86:in `block in pre_process'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/thin-1.6.2/lib/thin/connection.rb:84:in `catch'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/thin-1.6.2/lib/thin/connection.rb:84:in `pre_process'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/lib/eventmachine.rb:1037:in `call'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/lib/eventmachine.rb:1037:in `block in spawn_threadpool'\"]}","log_level":"error","source":"cc.api","data":{"request_guid":"55f62dfa-598b-4b11-41cc-e6282c768a98::512d168d-ec2b-4a37-a7ef-e46428cd47ef"},"thread_id":40032640,"fiber_id":41336340,"process_id":9686,"file":"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/sinatra/vcap.rb","lineno":53,"method":"block in registered"}
What do I do now to clean up this mess? when I upgraded the CF I simply deployed the new version, without touching the /var/vcap directory. The old orgs\apps are still there, but some are un-accessible...
Scott Taggart
Jan 31, 2015, 1:17:43 PM1/31/15
to vcap...@cloudfoundry.org, yaco...@gmail.com
Did you ever solve your CF-CipherError issue? I'm having the same issue...
Scott Taggart
Jan 31, 2015, 1:58:56 PM1/31/15
to vcap...@cloudfoundry.org, yaco...@gmail.com
Never mind, just figured it out
For the benefit of anyone playing along at home - you can't change your DB encryption key in your manifest after the DB has been created!
Reply all
Reply to author
Forward
0 new messages