[ANN] Logger - distributed logging system

[Tim Dickinson]

Logster is a distributed logging system. have you ever wanted to collect logs from multiple sources and view them in one place? Well Logster is the answer.

https://github.com/MangoRaft/Logger

[Aria Stewart]

On Mar 10, 2015, at 11:17 AM, Tim Dickinson <price...@gmail.com> wrote:

Logster is a distributed logging system. have you ever wanted to collect logs from multiple sources and view them in one place? Well Logster is the answer.

Neat!

How do you handle dropped log packets?

[Tim Dickinson]

You don't. It was designed to collect hundreds if not thousands of log lines per second. The UDP server is made to scale. So to prevent packet lose just scale up the UDP server.

[Aria Stewart]

> On Mar 10, 2015, at 7:36 PM, Tim Dickinson <price...@gmail.com> wrote:
>
> You don't. It was designed to collect hundreds if not thousands of log lines per second. The UDP server is made to scale. So to prevent packet lose just scale up the UDP server.

Aw, sad. It's easier to scale without things if you have a protocol that handles retries like TCP. This way you just have to overprovision like 10x and hope you still don't miss any due to bursting. :(

["Aurélio A. Heckert"]

Tim, UDP is mandatory? Is it the only protocol implemented or we can
configure it to use TCP?
If it's UDP only today, is there a plan to support "other ways"?

[Jeremy Darling]

Well, you could use something like (the soon to be released) Precis (https://github.com/precis-logging/precis) project.  Built on top of a centralized event message bus (by default it will use MongoDB capped collections) and already being used for 100,000+ messages a second.

The event bus used by Precis can be switched out to fit your needs so you could use RabbitMQ, Redis, etc instead of Mongo's capped collections.

I'm busy right now removing all of our custom code and open sourcing the entire thing.  Goal is to have the first GA in the wild by end of March to middle of April.

Disadvantage to this scale of logging is your talking about a full SOA with many services to maintain.  Advantage is that scaling just requires deploying new processing nodes.

Honestly Logger looks pretty snazzy and I've already added it to my list of projects to look at as a plugin in place of or along side FluentD in Precis.

Of course there is always ELK (ick), Splunk, Octopussy, Graylog, or you could just build it yourself if you don't like the features of what is already available.

 - Jeremy

--
Job board: http://jobs.nodejs.org/
New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md
Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
To post to this group, send email to nod...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/B39400AF-9039-4339-9093-60C9A828BA8E%40nbtsc.org.
For more options, visit https://groups.google.com/d/optout.

[Tim Dickinson]

If anyone knows of a TCP reconnect module please let me know.

[Tim Dickinson]

It would be really easy to implement the TCP protocol with logster. If its something people would like to see I would be more than happy to implement it.

[Adil Hasan]

Hello,
it sounds quite interesting. Perhaps Tim you allow the log information
to be buffered at the source to minimisee loss?
Best wishes,
adil

[Tim Kuijsten]

It's not that unusual to run syslogd over udp. It's actually the only
option if you run the version that ships with OpenBSD base.

Aria Stewart schreef op 11-03-15 om 03:47:

[Karl Tiedt]

That sounds like a bit of false logic... "scaling the UDP server" in no way makes up for the lack of retries in TCP... that statement addresses only one potential reason for a UDP connection to fail... If your only solution is "scale it up" you still fail to account for the primary reason why a packet might get lost.... network issues.

clearly these logs will never be important.

-Karl Tiedt

--
Job board: http://jobs.nodejs.org/
New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md
Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
To post to this group, send email to nod...@googlegroups.com.

[Tim Dickinson]

>  Perhaps Tim you allow the log information to be buffered at the source to minimisee loss?

There is a buffer that clears on a timer or the buffer length. So it will send a batch of logs in one message.

>clearly these logs will never be important.

Never is a strong word. Like I said a TCP or even a HTTP server would not be hard to implement. As people seem to be interested in it I will be implementing it as an option.

>If it's UDP only today, is there a plan to support "other ways"? 

Yes coming soon. :)

[Tim Kuijsten]

Maybe a bit late, but just read a blog post of the rsyslog author from
2008 where he states some interesting things about syslog over tcp.

"So just let me repeat: plain tcp syslog is *not* a reliable solution.
It works well as long as everything is working well, but it screws up
when the network or the server breaks... And, yes, you don't really need
to care about that until you try to make the rest of the system fully
reliable."
http://blog.gerhards.net/2008/04/on-unreliability-of-plain-tcp-syslog.html

And a month later:
"This is the ultimate evidence that you can not build a reliable syslog
infrastructure just on top of TCP (without app-layer acks)."
http://blog.gerhards.net/2008/05/why-you-cant-build-reliable-tcp.html

Just fyi.

-Tim

Tim Dickinson schreef op 11-03-15 om 18:37:

> --
> Job board: http://jobs.nodejs.org/
> New group rules:
> https://gist.github.com/othiym23/9886289#file-moderation-policy-md
> Old group rules:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> ---
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to nodejs+un...@googlegroups.com

> <mailto:nodejs+un...@googlegroups.com>.

> To post to this group, send email to nod...@googlegroups.com

> <mailto:nod...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/nodejs/c7e0e20a-f3b6-4445-a274-4034d1221aae%40googlegroups.com
> <https://groups.google.com/d/msgid/nodejs/c7e0e20a-f3b6-4445-a274-4034d1221aae%40googlegroups.com?utm_medium=email&utm_source=footer>.