"Can not create a Path from an empty string" with oddjob.MultipleValueOutputFormat

[lance bradley]

Hi,

I get this problem in a majority of runs of my job, although sometimes it does succeed. I am attempting to use MultipleValueOutputFormat from oddjob to control the path of the emitted files.

Here is my code. You will see in the reducer i log the path (to splunk) and I never see that occur with and empty string. Any help would be greatly appreciated!

https://gist.github.com/lancepantz/1c564ff6095a64a6518a

thanks!

-lance

[lance bradley]

Here's the expanded error:

INFO:centipede.logger:path: ABTestAssignment

INFO:centipede.logger:path: GemsTurn

INFO:centipede.logger:path: Install

INFO:centipede.logger:path: InternalPurchase

INFO:centipede.logger:path: Invite

close failed in file object destructor:

Error in sys.excepthook:

Original exception was:

java.lang.RuntimeException: java.lang.IllegalArgumentException: Can not create a Path from an empty string

at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:386)

at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:586)

at org.apache.hadoop.streaming.PipeReducer.reduce(PipeReducer.java:97)

at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:528)

at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:429)

at org.apache.hadoop.mapred.Child$4.run(Child.java:255)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)

at org.apache.hadoop.mapred.Child.main(Child.java:249)

Caused by: java.lang.IllegalArgumentException: Can not create a Path from an empty string

at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)

at org.apache.hadoop.fs.Path.<init>(Path.java:90)

at org.apache.hadoop.fs.Path.<init>(Path.java:45)

at sun.reflect.GeneratedConstructorAccessor23.newInstance(Unknown Source)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

at clojure.lang.Reflector.invokeConstructor(Reflector.java:180)

at oddjob.MultipleValueOutputFormat$_generateFileNameForKeyValue.invoke(MultipleValueOutputFormat.clj:18)

at oddjob.MultipleValueOutputFormat.generateFileNameForKeyValue(Unknown Source)

at org.apache.hadoop.mapred.lib.MultipleOutputFormat$1.write(MultipleOutputFormat.java:87)

at org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.write(ReduceTask.java:466)

at org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:506)

at org.apache.hadoop.streaming.PipeMapRed$MROutputThread.run(PipeMapRed.java:431)

[Adrian Maceiras]

I ran into this issue... Here is how I fixed it:

https://github.com/jblomo/oddjob/pull/5