Making VCAP_SERVICES externally accessible.

[Jonathan Berkhahn]

Hello all,

I'm part of the IBM team that has been working on CloudFoundry. Some of our customers have expressed an interest in having the value of an applications VCAP_SERVICES be accessible from outside of just that particular app. We've written the code that would enable it to be accessible from the CLI as well as through REST api, but before submitting a pull request we wanted to get the input of the community on whether this is a direction we want to go.

Thoughts?

Jonathan Berkhahn
Software Engineer
IBM Cloud Labs

[Mike Youngstrom]

I'm interested in this space.  What are some of the use cases for this functionality?  Most of our customers have been looking for the exact opposite.  Wanting to lock down VCAP_SERVICES so it is as hard as possible for anything but the application to have access to this.

Mike

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

[James Bayer]

there is already a story for this in the CLI icebox marked as blocked on runtime.

https://www.pivotaltracker.com/story/show/55332776

i've bcc'd dekel who has been talking with many enterprise customers who have the same concern as mike, and they want to retrict VCAP_SERVICES visibility outside of the apps.

we make the env variables available to the application, and therefore also to any user that can issue the equivalent of:

/v2/apps/63f67dc5-79f8-49cb-98f6-29cc7d444444/instances/0/files/logs/env.log 

where the app_guid changes, but returns the contents of VCAP_SERVICES, then i am in favor of sharing this variable.

the cf/gcf command is "gcf files APPNAME logs/env.log". i see no reason not to return VCAP_SERVICES. one way to protect the credentials is to not allow the "files" command equivalent for anything but a user with "Developer" permissions on a Space. these are the trusted users on a space. to really secure an environment, you could restrict this to a CI deployment user that is not a human. furthermore, we can record/audit the usage of the "files" command in the event logs stored in the cloud controller as a significant event we want to track and retain for at least a period of time. we should make sure that the "Auditor" for a Space user cannot issue these commands.

so i'm in favor of sharing VCAP_SERVICES on the CLI. it's useful to developers and i believe the permissions model and auditing can address concerns about credentials being accessed.

--

Thank you,

James Bayer

[Shannon Coen]

A user can already get the value of VCAP_SERVICES with the ruby cli, via 'cf files APPNAME logs/env.log' (maybe go cli, also?) but it's not an intuitive user experience. The story we've talked about doing for so long would be to return VCAP_SERVICES along with user set env vars in response to 'cf env'. 

I'm with James, I'd like to see roles and permissions used to enable this feature for users who need it or lock it down as needed.

[James Bayer]

yes it works with gcf too like gcf files APPNAME logs/env.log

[Patrick Mueller]

I've been concerned about who can see my "goodies" as well; eg, I tend to have other secret-ish keys stashed in config files available in my droplet for things like Twitter API client secret, etc.  Restricting "files"-like capabilities with space permissions would make me feel a lot better.  

I've also been known to dump sensitive information in stdout while I'm debugging an app, and then forgetting to take it out again later - I assume stdout/stderr logs and whatever downstream log accessors (loggregator?) are or will be protected as well?

Patrick Mueller
http://muellerware.org

[James Bayer]

I assume stdout/stderr logs and whatever downstream log accessors (loggregator?) are or will be protected as well?

yes, we could say that the Auditor role does not have access either to "logs" or "files" commands. although i'm not so sure about telling an "Space Auditor" that they can't use "logs". if the auditor can't see logs then really what can they really do other than see stuff like app status, which services are bound to the app, stats, and little else. my sense is that the predefined "Space Auditor" role should have access to logs and app developers have to know not to put sensitive information in logs.

[Mike Youngstrom]

For us it would be nice if there was a role for people to monitor/notice potential issues through logging and such but not have the ability to see service credentials or change application deployment information.  So, I would like to keep log access for auditors. :)  Or create some other role that would allow for monitoring an application.

Mike

[Mark Kropf]

There is a story in the services backlog to address this. Thanks for everyone's input!

[Shannon Coen]

I wanted to provide an update on the PR from IBM to expose the value of VCAP_SERVICES in the cloud controller API (https://github.com/cloudfoundry/cloud_controller_ng/pull/146).

In investigating performance issues in cloud controller in January and early February, the Runtime team expressed concern with the performance impact of the implementation of this feature. This implementation was agreed on by Pivotal and IBM prior to the PR being submitted so no one is pointing fingers. However, after performance issues blocked our pipeline for three weeks, there's an effort for more attention on performance with each commit. 

The feature was not reverted, but we made a band-aid commit to hide the system_env_json field from the GET /v2/apps/:guid endpoint it was added to. There's a story in the Services team backlog to refactor the implementation, and expose the field on a different endpoint, GET /v2/apps/:guid/summary (https://www.pivotaltracker.com/story/show/55332776). If Jonathan (OP and PR submitter) would like to help us with that, we would likely be grateful for the help, otherwise we'll get to it as soon as we can considering other priorities.  

Thank you,

Shannon

[Shannon Coen]

With the release-candidate branch of cf-release and edge of the 'cf' CLI, I've just verified that the CLI command `cf env APPNAME` returns the value of VCAP_SERVICES, along with user-set environment variables. 

The CLI command now uses a new endpoint on cloud controller, `/v2/apps/:guid/env`, which returns an unauthorized error to users who do not have the SpaceDeveloper role in the space for the requested app. 

This feature should be available in the next final release of cf-release (173), and next stable version of the CLI.

We're also just wrapping up a related epic to lock down availability of binding credentials and environment variables in the cloud controller API to the SpaceDeveloper role. Where these objects are included in endpoints available to users without SpaceDeveloper, we're now redacting sensitive data by replacing this data with `{ "redacted_message": "[PRIVATE DATA HIDDEN]" }`. Our progress on this epic can be tracked here: https://www.pivotaltracker.com/epic/show/1207012.

Happy cloud smithing,

Shannon

[Mike Youngstrom]

This is excellent news!!!  Great work Shannon!

Is there an issue in the epic to remove the creation of an env.log for a running instance?

Mike

--
You received this message because you are subscribed to the Google Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/f964959d-166e-4e03-bb87-83542d00a4ba%40cloudfoundry.org.

[Shannon Coen]

Hi Mike,

We haven't considered removing this logfile. We can look into what the pros and cons of that would be.

Shannon Coen

Product Manager, Cloud Foundry

Pivotal, Inc.

mobile: 415.640.0272

skype: shannoncoen

--
You received this message because you are subscribed to a topic in the Google Groups "Cloud Foundry Developers" group.

To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAEoPEDrd1Gb2FQUXmK-%3D4wvNYM93ipbgxT5Zt9nxQHV6pVjYZQ%40mail.gmail.com.

[Shannon Coen]

Something to consider is that there are many system-set environment variables which are still only discoverable by looking at that logfile. I suspect there is still value is providing it.

Shannon Coen

Product Manager, Cloud Foundry

Pivotal, Inc.

mobile: 415.640.0272

skype: shannoncoen

[Mike Youngstrom]

I wouldn't mind the file staying as long as VCAP_SERICES and the app environment variables were redacted.

I'd like to keep viewing files a space viewer capability.  Having sensitive data available in that file kind of defeats the purpose of this work if we are to give viewers access to see files.

Mike

To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAJKcadXU_6_YPHf_uySiz3LEzTs2swoj7k1vd7dsBT%2BsuVRdBw%40mail.gmail.com.

[Shannon Coen]

Hi Mike,

It seems to me that an auditor should not have access to the runtime filesystem. If only a SpaceDeveloper can read these files, then while I agree it would be better that credentials weren't logged, it is less of an issue. 

Could you help me understand your use case for why a read-only user should have access to the runtime filesystem? 

Thanks!

[Mike Youngstrom]

Thanks for responding Shannon,

Our main use case is the following.  Some of our production spaces require limiting access to service credentials to a small number of operations personnel.  In these spaces we'd like to make the operations personnel "space developers" but give the actual developers "viewer" access so that they can diagnose issues and confirm configuration.  We view read only access to files as an important part of problem diagnosis.  It allows for confirming correct application bits, configuration, and downloading logs not available in loggregator. (e.g. App dynamics agent logs).

Are there particular issues you know of that giving "viewers" read only access to "files" can cause?  Assuming the env.log file were redacted?

BTW, I've brought this issue up a few times before and the idea of giving auditors/viewers access to files didn't seem like a problem.

https://groups.google.com/a/cloudfoundry.org/d/msg/vcap-dev/8Owzq9pzDSs/q4VGjRKMrzoJ

https://groups.google.com/a/cloudfoundry.org/d/msg/vcap-dev/W_0jaAJV2j0/m5BndGPl8SoJ

Thanks,

Mike

To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/b415a217-7d00-4ff6-a7fe-bd92ebf6ddbb%40cloudfoundry.org.