calling web service from cascading

[una...@gmail.com]

Hello everyone, I could use some assistance from the experts here.

What's the best way of calling a web service from a Cascading operation?

Am querying an internal API and parsing the output to populate TupleEntry values.

My code ends up with an exception. Obviously it's not able to serialize the object. 

My broader question is: what's the best practice around doing something like this?

If this is the way to go (or not) what can I do to solve this problem or avoid this anti-pattern?

Thanks in advance -

Caused by: java.io.NotSerializableException: org.jboss.resteasy.client.jaxrs.ResteasyClient
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
at java.util.HashMap.writeObject(HashMap.java:1129)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1495)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
at cascading.flow.hadoop.util.JavaObjectSerializer.serialize(JavaObjectSerializer.java:57)
at cascading.flow.hadoop.util.HadoopUtil.serializeBase64(HadoopUtil.java:282)
at cascading.flow.hadoop.HadoopFlowStep.pack(HadoopFlowStep.java:193) 

 

[Ken Krugler]

Often the classes used to query web services aren’t serializable.

So what you want to save with your function is the set of values required to instantiate the client. Typically those are serializable (e.g. rest endpoint, username, various other settings)

The actual client field in the function class is flagged as transient, so that it isn’t serialized.

Then in your function’s open() method, you use these settings to create the client.

And in the close() method you close it, of course.

— Ken

-- 
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at https://groups.google.com/group/cascading-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/23b207c0-067d-46b7-bdea-497226d35b27%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--------------------------

Ken Krugler

+1 530-210-6378

http://www.scaleunlimited.com

custom big data solutions & training

Hadoop, Cascading, Cassandra & Solr

[una...@gmail.com]

Great thanks for the spot-on advice! this works as expected now. 

The more I use Cascading the more I fall in love with it :)

[Chris K Wensel]

fwiw, the ‘context’ object in a Function allows you to initialize resources that are not thread safe

see the bottom of this section

http://docs.cascading.org/cascading/3.1/userguide/ch13-using-operations.html#functions 

ckw

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/e5405abb-5a8f-4141-a223-dbe48840ab84%40googlegroups.com.

[una...@gmail.com]

Cool, thanks for tip Chris. Using the 'context' seems like the way to go, to avoid creating unnecessary objects & reusing when possible.