How to pass parameters to mapper or reducer ?

[Jeff Zhang]

Hi all,

I'd like to pass parameters to mapper or reducers,  I wonder how can I get the parameter in the mapper or reducer function.  Is there anything like Configuration object that I can use in mapper or reducer function ? Thanks

[Eric Torreborre]

Hi Jeff,

I need to think about it and do some testing (what's your typical use case?). My first idea is that any value you're using in functions passed to the map or reduce operation on a DList should be serialized and thus be accessible when executed on the cluster.

If that's not the case you might want to look at the DistCache class which serialize / deserialize objects and put them in the distributed cache of Hadoop.

E.

[Jeff Zhang]

Hi Eric,

It looks like a serialization issue.  I have a Configuration file which need to passed to mapper and reducer function. 

Here's the main view of my Configuration Class:

class DataConf(file: String) {

  val log = Logger.getLogger(classOf[DataConf])

  var contextFields: Seq[ContextField] = _

  var eventFields: Seq[EventField] = _

  var splitterClass: String = _

  var parserClass: String = _

And here's the error message:

Exception in thread "main" com.thoughtworks.xstream.io.StreamException:  : Invalid null character in text to output

at com.thoughtworks.xstream.io.xml.StaxWriter.setValue(StaxWriter.java:160)

at com.thoughtworks.xstream.io.WriterWrapper.setValue(WriterWrapper.java:45)

at com.thoughtworks.xstream.converters.collections.CharArrayConverter.marshal(CharArrayConverter.java:34)

at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)

at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)

at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)

at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.marshallField(AbstractReflectionConverter.java:229)

at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter$2.writeField(AbstractReflectionConverter.java:208)

at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter$2.<init>(AbstractReflectionConverter.java:171)

at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doMarshal(AbstractReflectionConverter.java:116)

at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.marshal(AbstractReflectionConverter.java:72)

at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)

at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)

....................

在 2013年3月5日星期二UTC+8下午7时04分34秒，Eric Torreborre写道：

[Jeff Zhang]

This issue is fixed in 0.7.0-SNAPSHOT which use the binary serialization instead of xml

BTW, what is the estimated release date of 0.7.0 ?

在 2013年3月6日星期三UTC+8上午9时29分33秒，Jeff Zhang写道：

[Eric Torreborre]

>BTW, what is the estimated release date of 0.7.0 ?

I would say between one month and 2. We have quite a few features planned for 0.7.0 but the main one is the new Persist API (and the internal refactoring).

Two new features have been added since: a REPL and a "checkpoint" functionality. It's not been officially announced yet because we are just testing how they work on our own projects.

I'm all for doing a release sooner anyway if the main users on this list can at least validate that 0.7.0 works fine for them in production.

E.