Facebook Scribe Best Practices

[TomTom]

Hey Fellows,
Would anyone at FB, be able to explain their Scribe design. For
example how many "master" scribe servers do you aggregate the 10
Billion daily log messages to. How do you handle high availability? Do
you load balance to a hot scribe master standby, and use shared
storage?

Thanks again!
-Tom

[anthony]

Tom,

You might be interested in this Scribe Tech Talk at Facebook earlier
this year:
http://www.facebook.com/video/video.php?v=650882334523&ref=mf

To support high availability we have a tier of Scribe servers that
load balances traffic between themselves. And instead of having to
worry about having hot standbys, we are moving most of our logging
into hadoop using Scribe's Hdfs support. So if any of the machines in
our Scribe tier or Hadoop tier go down, the other machines in that
tier will be able to handle the extra traffic.

In addition, we run Scribe locally on all of our web servers and other
machines. So if an unexpected failure causes the entire Scribe tier
or Hadoop tier to go down or become unreachable, data will be safely
buffered locally until the problem is resolved.

And we currently log more than 20TB/day through Scribe in over 100
Billion messages.

-Anthony