Thrift 0.8.0 and EMR follow up

[Jeroen van Dijk]

A while ago I asked about the combination of Thrift 0.8.0 and newer EMR AMI versions (> 2.0.5)  (see https://groups.google.com/forum/#!msg/cascalog-user/9kBdt-NNFPA/ac362wIP1hIJ). Some of you were, but were using an older AMI version (2.0.5).

Back then I decided to use a newer AMI version (2.1.3) that required an older Thrift version. This is due to fact that the Hadoop distribution of the AMI comes with HBase, which comes with Thrift 0.7.0.

Now I wonder, 9 months later, if some of you who are running 0.8.0 Thrift are using a newer AMI version?

I ask this now, because I want to upgrade Elephantdb which requires Thrift 0.8.0.

Thanks,

Jeroen

[Soren Macbeth]

I run thrift 0.8.0 on EMR with the latest AMI 2.3.5, however I am using MapR M3 edition, not the standard apache/amazon hadoop distro. That said, I have no issues with thrift 0.8.0 on MapR/EMR.

--
You received this message because you are subscribed to the Google Groups "cascalog-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascalog-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
http://about.me/soren

[Jeroen van Dijk]

Thanks for the reply Soren. I'll give the MapR distribution a try as well. 

Out of interest since the MapR AMI's are roughly 25% more expensive, what are the extra features that you use that make this worth it? And I also couldn't find what version of Hadoop the MapR AMI's are comparable with, is it 1.0X or more 0.20?

[Mason]

Worth double-checking, but I believe the MapR M3 distro has no additional cost and comes with Hadoop 1.0.X.

[Soren Macbeth]

That's correct. That's what we use. 

[Jeroen van Dijk]

I think I've finally solved this problem. Yesterday I ran into this issue again. After some searching I came accross the following configuration option for Hadoop  0.20.205 [1]

-c mapreduce.user.classpath.first=true -m mapreduce.user.classpath.first=true

This makes sure your own libraries are  loaded before the Hadoop libraries. In my case Thrift 0.8.0 before Thrift 0.7.0... Handy isn't it ;-)

I'm guessing the defaults are a bit different for newer versions of Hadoop. Time for an upgrade as well.

[1] http://stackoverflow.com/questions/11685949/overriding-default-hadoop-jars-in-class-path

[Sam Ritchie]

Ah! Awesome, that makes sense.

Jeroen, would you please add this information to the Cascalog wiki? Perhaps under an FAQ page? Totally fine if this is the only question for now... I just don't want to lose the information.

Thanks,
Sam

Jeroen van Dijk

June 13, 2013 9:01 AM

I think I've finally solved this problem. Yesterday I ran into this issue again. After some searching I came accross the following configuration option for Hadoop  0.20.205 [1]

-c mapreduce.user.classpath.first=true -m mapreduce.user.classpath.first=true

This makes sure your own libraries are  loaded before the Hadoop libraries. In my case Thrift 0.8.0 before Thrift 0.7.0... Handy isn't it ;-)

I'm guessing the defaults are a bit different for newer versions of Hadoop. Time for an upgrade as well.

[1] http://stackoverflow.com/questions/11685949/overriding-default-hadoop-jars-in-class-path

--
You received this message because you are subscribed to the Google Groups "cascalog-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascalog-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Soren Macbeth

May 29, 2013 11:52 AM

That's correct. That's what we use. 

--
http://about.me/soren

--
You received this message because you are subscribed to the Google Groups "cascalog-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascalog-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Mason

May 29, 2013 11:51 AM

Worth double-checking, but I believe the MapR M3 distro has no additional cost and comes with Hadoop 1.0.X.

On 5/14/13 01:58 AM, Jeroen van Dijk wrote:

--
You received this message because you are subscribed to the Google Groups "cascalog-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascalog-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Jeroen van Dijk

May 14, 2013 1:58 AM

Thanks for the reply Soren. I'll give the MapR distribution a try as well. 

Out of interest since the MapR AMI's are roughly 25% more expensive, what are the extra features that you use that make this worth it? And I also couldn't find what version of Hadoop the MapR AMI's are comparable with, is it 1.0X or more 0.20?

--
You received this message because you are subscribed to the Google Groups "cascalog-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascalog-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Soren Macbeth

May 13, 2013 7:32 AM

I run thrift 0.8.0 on EMR with the latest AMI 2.3.5, however I am using MapR M3 edition, not the standard apache/amazon hadoop distro. That said, I have no issues with thrift 0.8.0 on MapR/EMR.

--
http://about.me/soren

--
You received this message because you are subscribed to the Google Groups "cascalog-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascalog-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Sam Ritchie, Twitter Inc
703.662.1337
@sritchie

[Jeroen van Dijk]

Hi Sam,

I think a FAQ is a good idea. I started one here https://github.com/nathanmarz/cascalog/wiki/FAQ

Is this what you had in mind?

[Sam Ritchie]

This is fantastic!

Quantisan, what's the best way to keep work on the wiki in sync with the new cascalog.org?

Jeroen van Dijk

June 14, 2013 3:39 AM

Hi Sam,

I think a FAQ is a good idea. I started one here https://github.com/nathanmarz/cascalog/wiki/FAQ

Is this what you had in mind?

--
You received this message because you are subscribed to the Google Groups "cascalog-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascalog-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Sam Ritchie

June 13, 2013 1:39 PM