Re: hadoop pig issues

[Paul DeCoursey]

PIG 0.7.0 gives me an even better error:

Caused by: java.lang.IllegalArgumentException: Wrong FS: mongodb://127.0.0.1/demo.excitelog, expected: file:///

This makes me think that it isn't even trying to use the MongoLoader.  But if I remove the mongodb:// I get this:

Caused by: java.lang.IllegalArgumentException: Invalid URI Format.  URIs must begin with a mongodb:// protocol string.

What is going on here?  My love for Mongo has been fading in recent days.

On Monday, September 17, 2012 10:33:21 AM UTC-5, Paul DeCoursey wrote:

I'm trying out the examples and I'm getting

org.apache.pig.backend.executionengine.ExecException: ERROR 0: java.io.IOException: No FileSystem for scheme: mongodb

I also see this in the log as well:

java.lang.IncompatibleClassChangeError: Found class org.apache.hadoop.mapreduce.TaskAttemptContext, but interface was expected

at com.mongodb.hadoop.pig.MongoStorage.putNext(MongoStorage.java:86)

at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139)

at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:98)

at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:639)

at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)

at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)

at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:269)

at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:262)

at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)

at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)

at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)

What version of PIG does the mongo-hadoop connector support?  I've tried 0.10 and 0.9.2 and both give me these errors.

[Mike O'Brien]

Paul - 

Let me look into this a bit, I will try to figure out what's going on and get back to you on this as soon as I can.

-Mike

[Paul DeCoursey]

It appears to be related to "java.lang.IncompatibleClassChangeError: Found class org.apache.hadoop.mapreduce.TaskAttemptContext, but interface was expected"  From what I can tell after that happens it fails the task.  I thought maybe it was because I was running in local mode, but even on a pseudo-cluster I get the same issue.  It won't run at all on EMR, but it's failing when trying to register the jars, I must be doing something wrong there.

[Paul DeCoursey]

Full log, if it helps.

Backend error message

---------------------

Error: Found class org.apache.hadoop.mapreduce.TaskAttemptContext, but interface was expected

Pig Stack Trace

---------------

ERROR 1002: Unable to store alias raw

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to store alias raw

at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1595)

at org.apache.pig.PigServer.registerQuery(PigServer.java:583)

at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:966)

at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)

at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)

at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)

at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)

at org.apache.pig.Main.run(Main.java:500)

at org.apache.pig.Main.main(Main.java:114)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 0: java.io.IOException: No FileSystem for scheme: mongodb

at org.apache.pig.PigServer.launchPlan(PigServer.java:1338)

at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1298)

at org.apache.pig.PigServer.execute(PigServer.java:1288)

at org.apache.pig.PigServer.access$400(PigServer.java:125)

at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1590)

... 13 more

Caused by: java.io.IOException: No FileSystem for scheme: mongodb

at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1324)

at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:65)

at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1345)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:244)

at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)

at org.apache.pig.StoreFunc.cleanupOnFailureImpl(StoreFunc.java:171)

at org.apache.pig.StoreFunc.cleanupOnFailure(StoreFunc.java:158)

at org.apache.pig.PigServer.launchPlan(PigServer.java:1334)

... 17 more

================================================================================

[Paul DeCoursey]

Has anyone else seen this issue?  We have been struggling to figure it out for some time now and have hit the wall.

[Mike O'Brien]

Paul - 

Are you testing this on a single node? I have seen this error with pig, it was resolved by using the correct execution type - try running with "pig -x local"