[scalding] java.io.NotSerializableException: org.apache.hadoop.conf.Configuration

[Chris Heller]

I want to execute an HDFS operation at the successful completion of my Job. To set this up I capture the Configuration in my Job and then add a onCompleted flow listener which closes over the Configuration variable.

It seems that this isn't allowed, and I get the error seen in the subject line.

Now why one cannot serialize a hadoop Configuration is beyond me, given that one of its methods gives you a way to serialize the configuration to XML.

So my question is, before I go and manually write out the Configuration to a buffer as XML and close over that buffer... is there a better approach I could be taking to accomplish this task?

-Chris

[Koert Kuipers]

a flow gets serialized, and yes a hadoop Configuration is not serializable. instead maybe use the config (Job.config) which is a Map that also contains all the settings from the hadoop Configuration.

or just override run in your job to do what you need to after the job finishes. no flowListener needed so no serialization needed.

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/21546612-ebc4-440d-b6e7-bc71c8e97e82%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Heller]

Good suggestion. I think overriding 'run' might be the simpler approach.

I'll see how that works. Thanks.

-Chris