Serializable functions?

[Michael Jaaka]

Hi,

Since in Clojure 1.2 all data structures are serializable and Clojure
is homoiconicity language. Why don't make functions serializable? I'm
trying to pass function with RMI and execute in on the second REPL.
Right now, the only way to get it working is to pass string and
evaluate it on REPL. However it would be nice if I could use Clojure
in a way like this:


(.process unit { :input-process (fn[] .... ) :output-porces
(fn[] .... ) })


where unit is a RMI stub with process method in its interface and
hashmap passed as a parameter to the method is a detailed work to do
by the unit.

The purpose of all is something like this: http://java.sun.com/docs/books/tutorial/rmi/client.html

BTW. Java allows to set policy on the remote unit, so it is able to
load (by rmi classloader) missing classes (for both - client and
server side).

Any thoughts?
Thanks!

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

[Konrad Hinsen]

On 11.05.2010, at 11:42, Michael Jaaka wrote:

> Since in Clojure 1.2 all data structures are serializable and Clojure
> is homoiconicity language. Why don't make functions serializable? I'm
> trying to pass function with RMI and execute in on the second REPL.

I didn't try this myself yet, but my understanding is that is possible to serialize functions with Clojure 1.2. However, this requires that the JVM that serializes and the JVM that deserializes use the same class files. In practice, this means you have to AOT compile your code and use the same class (or jar) files everywhere.

Konrad.

[Michael Jaaka]

Well, this is obvious, but I would like to eliminate AOT and
distribution between units. This would give advantage over non-
functional languages like C++/Java since functions in Clojure should
be treated just like any other value. Currently they are handicapped.

Right now I can imagine that implementation would look like this. Each
function has it own raw input form (string which is parsed by clj
reader). On serialization, that raw form would be serialized and
received on the other unit. With first call, such deserialized
function would check if it is compiled, if not then it compiles its
raw form then executes it.

There is still one issue - dependences from modules used by serialized/
deserialized function. The solution for that is fact that each
function before serialization is compiled. During this compilation
process the list of class dependences would be included with such
function. Now during deserialization process RMI classloader would
automatically import all dependences. To decrease overhead due to raw
form and list of dependences each definition of such serializable
function had to be somehow marked just like all values are marked with
"implements Serializable" in Java.

[Konrad Hinsen]

On 11.05.2010, at 13:07, Michael Jaaka wrote:

> Right now I can imagine that implementation would look like this. Each
> function has it own raw input form (string which is parsed by clj
> reader). On serialization, that raw form would be serialized and
> received on the other unit. With first call, such deserialized
> function would check if it is compiled, if not then it compiles its
> raw form then executes it.

That's not sufficient because you have to serialize closures, not plain functions. You need more than just the source code.

[Kevin Downey]

http://github.com/technomancy/serializable-fn/

--
And what is good, Phaedrus,
And what is not good—
Need we ask anyone to tell us these things?

[verd]

Hi Michael,

Some time ago I came along Harbor: a server which can serialize Java
objects (and classes) over the wire
See http://coolharbor.100free.com/index.htm

Not open source but free. Hope you get some inspiration.

verd

[Brian Hurt]

Personally, I think this idea is a bad one, that will come back to bite people hard- but I doubt anyone is going to listen to me.

If you serialize the code (either in lisp s-expression form, or java byte code), you can easily end up serializing most of the whole program.  See, serialization isn't a bi-directional protocol, there's no way for the recipient to say "OK, I've got that code, you don't have to send it".  So to serialize a function, you have to serialize all the functions it depends upon, and then all the functions the depended-up functions depend upon, and so on.

Depending upon the code already being in the recipient's program simply creates a very specific dependency.  This serialized file or stream can only be read with this precise version of the code.  Make a small change, and whomp- old files can't be read, and you can't communicate with older version of the code- because they want you to call function foobar__57, and you don't have foobar__57, you have foobar__59.

And in clojure, it's very easy for closures to sneak into your data structures.  Got a lazy list?  It's got a closure.  Got a tree map or set?  It's got a closure.  And so on.

Now, get off my lawn.

Brian

[Phil Hagelberg]

On Tue, May 11, 2010 at 6:18 AM, Konrad Hinsen

<konrad...@fastmail.net> wrote:
>> Right now I can imagine that implementation would look like this. Each
>> function has it own raw input form (string which is parsed by clj
>> reader). On serialization, that raw form would be serialized and
>> received on the other unit. With first call, such deserialized
>> function would check if it is compiled, if not then it compiles its
>> raw form then executes it.
>
> That's not sufficient because you have to serialize closures, not plain functions. You need more than just the source code.

Actually in Clojure 1.2 the closure's lexical environment is exposed
to macros, so this is doable. The serializable-fn tool that Kevin
linked to demonstraes a primitive form of this, though I haven't tried
closing over too many different types yet.

(defn- save-env [env form]
(if env
`(let ~(vec (apply concat (for [[name local] env]
[name (.eval (.init local))])))
(~@form))
form))

(defmacro fn [& sigs]
`(with-meta (clojure.core/fn ~@sigs)
{:type ::serializable-fn
::source (quote ~(save-env &env &form))}))

The problem is that the .eval method isn't supported on all expression
types; it needs a more generalized way of getting the value of any
compiler expression. I haven't had a chance to really dig into the
compiler to see if there's a single way to do that, but I wouldn't be
surprised if there were a simpler solution.

-Phil

[Michael Jaaka]

Kevin and Phil nice work.

Now if only object returned by fn had implemented Serializable and
custom serialization functions which were doing:

private void writeObject(ObjectOutputStream out) throws IOException
{
(with-out-str
(print (pr-str this)))
}

private void readObject(ObjectInputStream in)
throws IOException, ClassNotFoundException {
(eval (read-string in))
}

some simple expressions could be already transmited.

Also fn could be named somehow more declarative like sfn to highlight
this overhead.

Almost there!!!

On 11 Maj, 20:12, Phil Hagelberg <p...@hagelb.org> wrote:
> On Tue, May 11, 2010 at 6:18 AM, Konrad Hinsen
>

[Michael Jaaka]

Hi! I have found two issues which stops me from using function
serializations.



(ns code.serialfn
(:refer-clojure :exclude (fn)))

(defn- save-env [env form]
(if env
`(let ~(vec (apply concat (for [[name local] env]
[name (.eval (.init local))])))
(~@form))
form))

(defmacro fn [& sigs]
`(with-meta (clojure.core/fn ~@sigs)
{:type ::serializable-fn
::source (quote ~(save-env &env &form))}))

(defmethod print-method ::serializable-fn [o ^Writer w]
(print-method (::source (meta o)) w))




; here are some examples

; gives null pointer

(prn-str { :kozak (fn [z]
[z (fn ([] (println "abc")) ([a] (println a))) ]) })

; gives can't eval locals - unsupported operation

(defn pipe
([] (pipe java.util.Collections/EMPTY_LIST))
([col]
(let [q (java.util.concurrent.LinkedBlockingQueue. col)]
[(take-while #(not (= % q)) (repeatedly #(.take q) ))
(fn ([e] (.put q e))
([] (.put q q)))])))


any help?

thanks