Attaching drain url to user-provided-service/CF Services frustrations

[Mike Youngstrom]

Thanks for the clarification.  I'm going to do some ranting.  I apologize in advance. :)

I don't really have a problem with the command itself.  I have more of a problem with the current state of services, user provided services, and custom service parameters generally in CF.  Attaching a drain url to user-provided-service, to me, just epitomizes the problems we have in the services space today.

Here are some of my thoughts:

* A service that drains logs is a certain type of service.  A first class parameter for a specific type of services, like drain url, clutters the service story to the user (CLI), to service authors, etc.  If I'm not creating a drain service why should I see "-l" as an option on my command?  It is confusing.

* Since user-provided-services exist to support providing custom credentials to a service, is there any value in attaching a drain url to a user provided service?  In the given example, who consumes the username/password? `gcf create-user-provided-service my-drain-service -p '{"username":"admin","password":"pa55woRD"}' -l syslog://example.com`  If CC/Loggregator are consuming the service what are they going to do with these random user provided credentials?

* This concept of using hard coded parameters attached to user-provided-services seems to contradict the currently discussed models of identifying service type. Tags or credential structure seems to be a much more robust way to provide this type of functionality as discussed here: https://groups.google.com/a/cloudfoundry.org/forum/#!searchin/vcap-dev/tags/vcap-dev/qOT2dwkcOnM/1udwt0fHT9EJ

Some concepts that I think would have worked better for this problem:

* Use Tags or credential key names for CC/Loggregator to find drain urls.

* If Tags/credential keys are used then a standard user-provided-service could be made without any special parameters for a small subclass of service types: `gcf create-user-provided-service my-drain-service -p '{"drain_url":"syslog://example.com"}' -tags log_drain,syslog`

Another possible solution could use custom parameters to services:

* We custom service creators have been pushing for custom parameters on non user-provided-services for a while now.  With custom service parameters a Syslog drain service broker could be created that consumes drain url.

* Then you could just use a custom command for creating a syslog drain services: `gcf create-log-drain-service my-drain-service -l syslog://example.com`.  No ambiguity about unused credentails also attached to the user-provided-service.  The service just needs a url.  So, that is what you give it.

* This service broker doesn't need to be its own job.  It could be co-located with CC but it should conform to the same services api that any service-broker uses.

On a bit of a tangent, IMO, user-provided-service should never have been created as a special CC API.  user-provided-service should have been created same as any other service broker that takes as a custom parameter a json string of credentials.  The service could be collocated with the CC but should use broker apis.  Creating special CC APIs for user-provided-service is yet another example of the design problems we have with services.

As a custom service creator I feel like the current services model and assumptions are WAY too restrictive for me to provide value to my customers.  Debates are underway to try and fix this in the V2 services doc.  The push back we've received there has been frustrating but we're still trying.  Anyway, seeing this drain url hack attached to services/user-provided-services that could have been fixed generally, for all custom service creators is kind of frustrating.  Especially in light of the very debates going on right now in the V2 services doc.

IMO the current services model, direction, and vision is broken and this -l parameter added to user-provided-service exemplifies, to me, just how broken the current CF services model is.

Thanks for listening.  Sorry about the rant. :)

Mike

On Wed, Nov 13, 2013 at 11:08 AM, Scott Truitt <str...@gopivotal.com> wrote:

Mike, I'm interested to hear about where this command breaks down for you, and how you might do it together differently... 

On Tue, Nov 12, 2013 at 6:05 PM, James Bayer <jba...@gopivotal.com> wrote:

that matches my understanding as to how it works. i haven't tried the drain url with a user provided service yet, but i have seen them drain into a different service that is specially configured for logging automatically without the user specifying the drain (pivotal analytics). it's up the the service author to say whether their service can support that. user provided is a work-around when those aren't available.

On Tue, Nov 12, 2013 at 3:01 PM, Mike Youngstrom <you...@gmail.com> wrote:

This `gcf -l` stuff confuses me.

I must be missing something.  So, I'm going to restate what I'm seeing here in the hope that I will be enlightened. :)

So, the question as I understand it is.  How can I drain my application's loggregator entries to an external log provider like splunk or syslog?

The answer as I see it here is:

* Create a user-provided-service

* Specify the url to the service I wish to drain to using -l

* Bind this service to my application.

* CC/Loggregator will then drain application logs to all drail url parameters associated with user-provided-services I've created and bound to my application.

Is that correct?  That seems bad to me in several ways.  But, I'd prefer to see if my understanding of this is correct before I get into the issues. :)

Mike

On Sat, Nov 9, 2013 at 8:32 AM, James Bayer <jba...@gopivotal.com> wrote:

well, we do have some additional functionality in v147, which is to register a syslog drain by binding a user-provided-service that was created with the gcf beta using the -l log drain parameter.

drain log messages are formatted according to RFC5424  [1]. the log messages are delivered over tcp as described in RFC6587 [2], using the octet counting framing method.

we have not put mileage on this yet, so there may be issues.

$ gcf h cups
NAME:
   create-user-provided-service - Make a user-provided service available to cf apps

ALIAS:
   cups

USAGE:
   gcf create-user-provided-service SERVICE_INSTANCE [-p PARAMETERS] [-l SYSLOG-DRAIN-URL]

   Pass comma separated parameter names to enable interactive mode:
   gcf create-user-provided-service SERVICE_INSTANCE -p "comma, separated, parameter, names"

   Pass parameters as JSON to create a service non-interactively:
   gcf create-user-provided-service SERVICE_INSTANCE -p '{"name":"value","name":"value"}'

EXAMPLE:
   gcf create-user-provided-service oracle-db-mine -p "host, port, dbname, username, password"
   gcf create-user-provided-service oracle-db-mine -p '{"username":"admin","password":"pa55woRD"}'
   gcf create-user-provided-service my-drain-service -l syslog://example.com
   gcf create-user-provided-service my-drain-service -p '{"username":"admin","password":"pa55woRD"}' -l syslog://example.com

OPTIONS:
   -p ''    Parameters
   -l ''    Syslog Drain Url

[1] https://tools.ietf.org/html/rfc5424

[2] https://tools.ietf.org/html/rfc6587#section-3.4.1

On Thu, Nov 7, 2013 at 4:26 PM, Tammer Saleh <tam...@pivotallabs.com> wrote:

We don't have that ability as of yet, without writing your own log drain broker.  Also, we don't have the ability to drain logs for all applications in a space - drains are bound to apps directly.

Cheers,

Tammer Saleh

On Thu, Nov 7, 2013 at 3:58 PM, Dr Nic Williams <drnicw...@gmail.com> wrote:

Tammer, perhaps an example for draining/multiplexing "all the apps" into logstash? Or is it trivial is I sit down to look at it?

On Thu, Nov 7, 2013 at 3:53 PM, Tammer Saleh <tam...@pivotallabs.com> wrote:

Not yet, but we're working with logging partners to get that into production as soon as possible.

Cheers,

Tammer Saleh

On Thu, Nov 7, 2013 at 3:45 PM, David Laing <da...@davidlaing.com> wrote:

Very exciting!  Thank you's all round.

Q:  The Loggregator readme [1] talks about "...drain their application logs to 3rd party log archive and analysis services"  

Has this functionality also been delivered in v147?

;D

[1] https://github.com/cloudfoundry/loggregator#features

On Thursday, 7 November 2013, James Bayer wrote:

Tuesday the CF team created cf-release v147 [1].

This release includes loggregator and we have turned that on for all users in run.pivotal.io, which now has v147 deployed. This means if you use the latest gcf [2] (the beta go cf cli) then you can see loggregator working with the gcf logs myapp command. Attached some example output. We're thrilled to have this per-app aggregated logging capability now available and recommended for production use. We will not be back-porting the loggregator functions to the ruby cf cli and the existing logs should continue to work there by connecting to a single app instance log unaggregated. Expect to see another announcement shortly from Scott Truitt about the gcf beta information.

There are several known issues with loggregator:

• staging logs will be truncated if they are too long
• during gcf push staging logs may miss some of the early part of the staging logs due to a race condition
• some log lines don’t recognize newlines

[1] https://github.com/cloudfoundry/cf-release/tree/v147
[2] https://github.com/cloudfoundry/cli#downloading-edge

--

Thank you,

James Bayer

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

--
David Laing
Open source @ City Index - github.com/cityindex
http://davidlaing.com
Twitter: @davidlaing

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

--

Thank you,

James Bayer

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

--

Thank you,

James Bayer

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

[Mike Youngstrom]

To help make my previous message a bit more digestible I thought I'd summarize by providing some bullets of things I would like to see considered as we evolve services, user-provided-services, and custom syslog drain in the future.

* Add support for creating services with custom parameters.  (e.g. allow a UI to provide more information than the plan to a service broker at service creation time)

* Evolve user-provided-service from the CC apis pseudo service it is today to an actual service broker that gets its credentials from custom parameters passed to it.

* Remove -l drain url configuration from user-provided-service.

* Create a Syslog-drain service that takes as a parameter a syslog drain url and configures itself correctly so that CC/Loggregator can identify the service as a drain service with a url.

* Figure out a more generic way for system explicit configuration to make it into a service without having to customize CC.  Custom service developers want add features to CC similar to loggregator drain.  Core CF developers and thirdparty custom service developers should be able to do this the same way without needing to customize CC.  This thread is a good discussion on the topic. https://groups.google.com/a/cloudfoundry.org/forum/#!searchin/vcap-dev/tags/vcap-dev/qOT2dwkcOnM/1udwt0fHT9EJ

Thanks guys.  I hope that feedback is a little easier to digest. :)

Mike

[Tammer Saleh]

Hi Mike,

I have little context on the first few points, but I'll chime in on the last:

* Figure out a more generic way for system explicit configuration to make it into a service without having to customize CC.  Custom service developers want add features to CC similar to loggregator drain.  Core CF developers and thirdparty custom service developers should be able to do this the same way without needing to customize CC.  This thread is a good discussion on the topic. https://groups.google.com/a/cloudfoundry.org/forum/#!searchin/vcap-dev/tags/vcap-dev/qOT2dwkcOnM/1udwt0fHT9EJ

The only thing to mention is that even with the abilities above, we wouldn't have wanted 3rd party service developers to have been able to add loggregator functionality without modifying CC, DEA, etc.  

The issue is that automatic stdout syslog draining can be a security issue if the user isn't aware that the service provides this functionality.  This is fundamentally different from a 3rd party DB, since the user must still explicitly use the DB after provisioning.  With stdout syslog draining, potentially sensitive information will be automatically sent to the service as soon as the service is provisioned.

To deal with this, we require a service to register itself as a syslog service, and we error out if the service attempts to set a drain url if it's not.  This allows us to control such access and ensure the end user is aware.

 

Cheers,

Tammer Saleh

[Mike Youngstrom]

The issue is that automatic stdout syslog draining can be a security issue if the user isn't aware that the service provides this functionality.  This is fundamentally different from a 3rd party DB, since the user must still explicitly use the DB after provisioning.  With stdout syslog draining, potentially sensitive information will be automatically sent to the service as soon as the service is provisioned.

Thanks Tammer.  That provides some good additional context.  I don't really have a problem with custom logic in CC to verify that this service can drain logs.  I'm more concerned with the precedence set of adding a drain service specific parameter to all services of which drain services will be a small fraction.  Tags and/or credential property keys could also be used as a key to validate drain url.  Regardless, if this is the best way to do it then great.  But, lets at least put some thought into how this requirement could potentially be generalized for other services too.  Maybe that is the plan when the team has more time.  If it is the plan then lets have some community discussion about it.

Thanks,