compression and/or secure network transfer?

[jwieland]

I have 3 geo located web clusters (8 web servers per cluster) each
cluster generates about 150 GB of logs per day. I use scribe to
combine all the web servers logs in each cluster to a single server
residing in the same cluster.

I would like to start aggregating those three log files into a central
system with the intention to import the data into hadoop. I'm
wondering if it would make sense for scribe to forward the data
outside my network into central system and have scribe directly write
directly into hdfs. The transfer would need to be secure and
compression would be a bonus, not sure if scribe supports this or is
it a tool that you should run inside your controlled network?

If not, I suppose back to trusty scp to transfer the files.

Thanks for your help,

Jason

[tsuna]

On Wed, Jan 20, 2010 at 5:55 PM, jwieland <jwie...@gmail.com> wrote:
> The transfer would need to be secure and
> compression would be a bonus, not sure if scribe supports this or is
> it a tool that you should run inside your controlled network?

Hi Jason,
Scribe doesn't officially support compression but there's a version
out there that adds LZO support: http://github.com/emaland/scribe

AFAIK Scribe doesn't support secure transfers but you can always
tunnel it through an SSH tunnel or some other kind of VPN between your
sites.

--
Benoit Sigoure aka Tsuna
Software Engineer @ www.StumbleUpon.com

[Jason Wieland]

tsuna wrote:

On Wed, Jan 20, 2010 at 5:55 PM, jwieland <jwie...@gmail.com> wrote:
  

The transfer would need to be secure and
compression would be a bonus, not sure if scribe supports this or is
it a tool that you should run inside your controlled network?
    

Hi Jason,
Scribe doesn't officially support compression but there's a version
out there that adds LZO support: http://github.com/emaland/scribe

AFAIK Scribe doesn't support secure transfers but you can always
tunnel it through an SSH tunnel or some other kind of VPN between your
sites.
  

Thanks for heads up with LZO, I noticed you (tsuna) cut a branch from emaland and patched up with most of the master changes.  Would I be better off cloning yours?  Is there any plans on merging LZO comp in the master?

I agree, VPN looks like the best option.

Also I'll be creating RPM's .spec files for centos 5.x (rhel) for scribe and the thrift framework.  Would anyone be interested in them?

Jason

[tsuna]

On Thu, Jan 21, 2010 at 10:19 AM, Jason Wieland <jwie...@gmail.com> wrote:
> Thanks for heads up with LZO, I noticed you (tsuna) cut a branch from
> emaland and patched up with most of the master changes.  Would I be better
> off cloning yours?  Is there any plans on merging LZO comp in the master?

I believe Eric is working with Anthony to merge his changes in
facebook's master.
I suggest you clone Eric's branch, mine has a couple of bugs I need to fix.

> I agree, VPN looks like the best option.

SSH tunnels are really simple :)