mapr experience
b...@benmabey.com
I'm trying to run a pigpen generated script on a mapr cluster (hadoop 2.5.1, pig 0.14.0-mapr-1504) and am I'm running into dependency issues. Specifically, it looks like mapr has a different version of snappy which is causing problems for nippy[1] and hence the pigpen runtime.
I noticed on the wiki an example of a project.clj file for cloudera's distribution. I'm going to try to create a similar one for mapr to see if that helps, although I am doubtful since the snappy version will still be conflicting.
Has anyone else ran pigpen on a mapr cluster with success?
Thanks,
Ben
1. https://gist.github.com/bmabey/45d5b1aac2ab1a104f6dcf54d5fb8a53
b...@benmabey.com
> Hi all,
> I'm trying to run a pigpen generated script on a mapr cluster (hadoop 2.5.1, pig 0.14.0-mapr-1504) and am I'm running into dependency issues. Specifically, it looks like mapr has a different version of snappy which is causing problems for nippy[1] and hence the pigpen runtime.
>
> I noticed on the wiki an example of a project.clj file for cloudera's distribution. I'm going to try to create a similar one for mapr to see if that helps, although I am doubtful since the snappy version will still be conflicting.
Yeah, I just tried using mapr deps but as expected this did not solve the problem. I'm thinking the only way around this will be to modify the nippy lib to use the provided snappy lib or package up the needed snappy dep into a different namespace. *sigh*
b...@benmabey.com
> On Tuesday, June 14, 2016 at 7:40:22 PM UTC-6, b...@benmabey.com wrote:
> > Hi all,
> > I'm trying to run a pigpen generated script on a mapr cluster (hadoop 2.5.1, pig 0.14.0-mapr-1504) and am I'm running into dependency issues. Specifically, it looks like mapr has a different version of snappy which is causing problems for nippy[1] and hence the pigpen runtime.
> >
> > I noticed on the wiki an example of a project.clj file for cloudera's distribution. I'm going to try to create a similar one for mapr to see if that helps, although I am doubtful since the snappy version will still be conflicting.
>
> Yeah, I just tried using mapr deps but as expected this did not solve the problem. I'm thinking the only way around this will be to modify the nippy lib to use the provided snappy lib or package up the needed snappy dep into a different namespace. *sigh*
It looks like pig packages snappy version 0.2 (https://github.com/apache/pig/blob/7069e77781a91a3537a9b8ef6b74f60986b2a2f3/ivy/libraries.properties#L98, https://github.com/mapr/pig/blob/release-0.14.0/ivy.xml#L421) for piggybank purposes while nippy uses 0.4.
What I don't understand is how pigpen works with any pig distributions using this older snappy dep. In other words, I don't think this is a mapr problem but a general pig dep issue and I'm surprised no one else has ran into this.
(Sorry for the spam updates!)
Matt Bossenbroek
--
You received this message because you are subscribed to the Google Groups "PigPen Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pigpen-suppor...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Ben Mabey
The exact error was:
https://gist.github.com/bmabey/45d5b1aac2ab1a104f6dcf54d5fb8a53
I have fixed the issue. By default pig appears to rely on snappy 0.2 which is getting on the classpath while nippy relies on snappy 0.4. It was easy to adjust nippy to use snappy 0.2: https://github.com/bmabey/nippy/commit/c974489c9359fbfd917eb435ca674d8866f6d20c
I then published my fork nippy jar to clojars and updated my project.clj as so:
[com.netflix.pigpen/pigpen "0.3.1"
:exclusions [com.taoensso/nippy]]
[bmabey/nippy-snappy-0.2 "2.10.0"]
Hope this helps someone else later.
-BenYou received this message because you are subscribed to a topic in the Google Groups "PigPen Support" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pigpen-support/UyJwlUOCG_U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pigpen-suppor...@googlegroups.com.