Showing 1-12 of 12 messages
Loggregator Tammer Saleh 6/25/13 1:35 PM
Hi to the Clound Foundry community!

I have product management duties for a new project that we've just kicked off -- Loggregator

I'd like to make sure there's a good avenue for community feedback around the architecture and features for the project.  To that end, I've opened a pull request for documenting the new component on docs.cloudfoundry.com.  Feel free to watch and/or comment on the pull request.

Tammer Saleh
Re: [vcap-dev] Loggregator Matt Stine 6/25/13 1:41 PM
Looks groovy. :-)
Matt Stine | Community Engineer, Cloud Foundry | Pivotal
901-493-5546 | mstine@goPivotal.com

Re: [vcap-dev] Loggregator Mike Youngstrom 6/25/13 1:57 PM
Awesome!  I've been hoping for something like this.

Re: [vcap-dev] Loggregator Dr Nic Williams 6/25/13 4:31 PM
Tammer & co, this will be great. Love the diagram. I look forward to a life size Loggerator in the Pivotal offices one day!

Dr Nic Williams
Stark & Wayne LLC - consultancy for Cloud Foundry users
cell +1 (415) 860-2185
twitter @drnic
Re: [vcap-dev] Loggregator Manikandan Lakshminarayaan 6/25/13 9:03 PM

Can we not use logstash for aggregating logs instead of a new project or how different is loggregator different from logstash

 - Mani
Re: [vcap-dev] Loggregator James Bayer 6/26/13 12:19 AM
This was one of Tammer's last activities before being out of the office a few days, so let me just jump in and say we did a review of the many of the other solutions in the logging space, there are many. The Cloud Foundry requirements are somewhat unique in that we require a multi-tenant solution that is dynamically configurable as new tenants are created and start sending logs immediately. We have not taken the decision to write something new lightly. If you look at the design documents, I think you'll find that the design is straight-forward. We believe our solution will be as well.
Thank you,

James Bayer
Re: [vcap-dev] Loggregator Kristian Øllegaard 6/26/13 7:39 AM
Hi there,

Did you also take https://github.com/heroku/logplex into consideration? They are more or less working with the same issues as Cloud Foundry.

Kind regards,

Re: [vcap-dev] Loggregator James Bayer 6/26/13 8:18 AM
Yes, we had a team that spent some hands-on time with LogPlex for several weeks. It's obviously a very capable project used at scale for Heroku and successfully implemented there.

Ultimately we have decided that while the goals are aligned, the team was not able to be very productive with LogPlex. Some of which was due to lack of Erlang familiarity on the team. Some of it was due to difficulty understand it from the at-the-time documentation and test coverage (which I think may have improved some since the first effort started). Some of it was due to perception that it was closely tied to some of Heroku's architecture, which was not as adaptable for our needs. Whether others feel those are accurate or not, these were our realized experiences after digging in on it, hence where we are now. 
Re: [vcap-dev] Loggregator Mike Youngstrom 6/26/13 10:41 AM
Is there an approximate time table for integration?

On Tue, Jun 25, 2013 at 2:35 PM, Tammer Saleh <m...@tammersaleh.com> wrote:

Re: [vcap-dev] Loggregator Tammer Saleh 6/27/13 10:34 AM
We don't have a public timetable, but I can say that this is actively worked on and that we're focused on getting something into production as soon as possible.

Tammer Saleh
Re: [vcap-dev] Loggregator Matt Stine 6/27/13 10:40 AM
Correct me if I'm wrong, but couldn't the Loggregator Drainer hook up to logstash should you desire to use it?

On Tue, Jun 25, 2013 at 9:03 PM, Manikandan Lakshminarayaan <man...@gmail.com> wrote:

Matt Stine | Community Engineer, Cloud Foundry | Pivotal
901-493-5546 | mstine@goPivotal.com

Re: [vcap-dev] Loggregator James Bayer 6/28/13 6:54 AM
Yes, we're trying to build a flexible building block that meets the needs of CF to aggregate logs into a single stream from multiple related sources in a dynamic way that has overload protection. That's the first part. The 2nd part is to provide a very generic drain API and protocol that can be used to retrieve this aggregated log that can then be fed to other places...something like LogStash if you like.
Thank you,

James Bayer