Loggregator

279 views
Skip to first unread message

Tammer Saleh

unread,
Jun 25, 2013, 4:35:12 PM6/25/13
to vcap...@cloudfoundry.org
Hi to the Clound Foundry community!

I have product management duties for a new project that we've just kicked off -- Loggregator

I'd like to make sure there's a good avenue for community feedback around the architecture and features for the project.  To that end, I've opened a pull request for documenting the new component on docs.cloudfoundry.com.  Feel free to watch and/or comment on the pull request.

Cheers,
Tammer Saleh

Matt Stine

unread,
Jun 25, 2013, 4:41:21 PM6/25/13
to vcap...@cloudfoundry.org
Looks groovy. :-)
--
Matt Stine | Community Engineer, Cloud Foundry | Pivotal
901-493-5546 | mst...@goPivotal.com

Mike Youngstrom

unread,
Jun 25, 2013, 4:57:00 PM6/25/13
to vcap...@cloudfoundry.org
Awesome!  I've been hoping for something like this.

Mike

Dr Nic Williams

unread,
Jun 25, 2013, 7:31:56 PM6/25/13
to vcap...@cloudfoundry.org
Tammer & co, this will be great. Love the diagram. I look forward to a life size Loggerator in the Pivotal offices one day!

Nic
--
Dr Nic Williams
Stark & Wayne LLC - consultancy for Cloud Foundry users
twitter @drnic

Manikandan Lakshminarayaan

unread,
Jun 26, 2013, 12:03:23 AM6/26/13
to vcap...@cloudfoundry.org
Hi,

Can we not use logstash for aggregating logs instead of a new project or how different is loggregator different from logstash

 - Mani

James Bayer

unread,
Jun 26, 2013, 3:19:12 AM6/26/13
to vcap...@cloudfoundry.org
This was one of Tammer's last activities before being out of the office a few days, so let me just jump in and say we did a review of the many of the other solutions in the logging space, there are many. The Cloud Foundry requirements are somewhat unique in that we require a multi-tenant solution that is dynamically configurable as new tenants are created and start sending logs immediately. We have not taken the decision to write something new lightly. If you look at the design documents, I think you'll find that the design is straight-forward. We believe our solution will be as well.
--
Thank you,

James Bayer

Kristian Øllegaard

unread,
Jun 26, 2013, 10:39:42 AM6/26/13
to vcap...@cloudfoundry.org
Hi there,

Did you also take https://github.com/heroku/logplex into consideration? They are more or less working with the same issues as Cloud Foundry.

Kind regards,

Kristian

James Bayer

unread,
Jun 26, 2013, 11:18:22 AM6/26/13
to vcap...@cloudfoundry.org
Yes, we had a team that spent some hands-on time with LogPlex for several weeks. It's obviously a very capable project used at scale for Heroku and successfully implemented there.

Ultimately we have decided that while the goals are aligned, the team was not able to be very productive with LogPlex. Some of which was due to lack of Erlang familiarity on the team. Some of it was due to difficulty understand it from the at-the-time documentation and test coverage (which I think may have improved some since the first effort started). Some of it was due to perception that it was closely tied to some of Heroku's architecture, which was not as adaptable for our needs. Whether others feel those are accurate or not, these were our realized experiences after digging in on it, hence where we are now. 

Mike Youngstrom

unread,
Jun 26, 2013, 1:41:20 PM6/26/13
to vcap...@cloudfoundry.org
Is there an approximate time table for integration?


On Tue, Jun 25, 2013 at 2:35 PM, Tammer Saleh <m...@tammersaleh.com> wrote:

Tammer Saleh

unread,
Jun 27, 2013, 1:34:36 PM6/27/13
to vcap...@cloudfoundry.org
We don't have a public timetable, but I can say that this is actively worked on and that we're focused on getting something into production as soon as possible.

Cheers,
Tammer Saleh

Matt Stine

unread,
Jun 27, 2013, 1:40:10 PM6/27/13
to vcap...@cloudfoundry.org
Correct me if I'm wrong, but couldn't the Loggregator Drainer hook up to logstash should you desire to use it?


On Tue, Jun 25, 2013 at 9:03 PM, Manikandan Lakshminarayaan <man...@gmail.com> wrote:

James Bayer

unread,
Jun 28, 2013, 9:54:38 AM6/28/13
to vcap...@cloudfoundry.org
Yes, we're trying to build a flexible building block that meets the needs of CF to aggregate logs into a single stream from multiple related sources in a dynamic way that has overload protection. That's the first part. The 2nd part is to provide a very generic drain API and protocol that can be used to retrieve this aggregated log that can then be fed to other places...something like LogStash if you like.
Reply all
Reply to author
Forward
0 new messages