Clojure and NetKernel

[Stuart Sierra]

On Feb 14, 12:51 pm, Steve Harris <steveO...@gmail.com> wrote:
> I'm currently dealing with a web-services oriented framework
> (NetKernel) that allots units of work to Java methods on threads of
> the frameworks own choosing, from a pool.

Hi Steve,

I got interested in NetKernel recently, I was wondering if you have
tried using Clojure with it, and if you have any experiences to share.

Thanks,
Stuart

[Steve Harris]

Hi Stuart,

It's not much, but I did do a little NetKernel module for Clojure
support that defines an "active:clojure" URI scheme. I found clojure
was really easy to integrate. There was almost nothing to do in fact,
because there's no interpreter to setup for each request. That had
been a little painful (and slow) with SISC Scheme which I had been
using with NK previously. I have a hunch that the Clojure accessors
are also generally quicker to start and execute than those written in
beanshell and the other supported scripting languages in NetKernel,
but I haven't benchmarked anything yet.

In fact I haven't had a chance to use it much at all after I got the
module working. I've been busy working on a Clojure port of sxpath to
use together with it, for slicing and dicing all the xml data in NK.
But I'd be happy to email you the little clojure module that I have so
far if you'd like (it's about 400k).

I've also got some scripts for automatically registering with NK my
modules which live outside of the NK directory. They're all Scheme +
SXPath, but I'm planning to redo them in Clojure soon. I hate redoing
the NK setup of my modules over and over on different machines, or
when I download a new distribution of NetKernel...

[Steve Harris]

Stuart,
I uploaded the Clojure NetKernel module to the files section (it
turned out to only by 4K once I removed the clojure jar :)
If you do try it and have any problems (with the module or NK in
general) just let me know. There's a README within it for getting
started.

Steve

[Stuart Sierra]

Thanks, Steve! I'll check it out.
-Stuart

[Randy Kahle]

Steve,

I took a quick look at the NetKernel module you posted. Interesting! I
am going to fire it up later today and try it.

I would like to mention this in my blog about NetKernel (http://www.
1060.org/blogxter/publish/8). You don't indicate in the module your
copyright, would you mind clarifying that before I proceed?

Thank you for this work!

Randy


On Mar 18, 11:06 am, Steve Harris <steveO...@gmail.com> wrote:
> Stuart,
> I uploaded the ClojureNetKernelmodule to the files section (it

[Steve Harris]

Hi Randy,
great to see you taking an interest in Clojure, I think it complements
NK nicely. NK strikes me as being very functional in nature and
encourages a very functional or expression-oriented style for apps/
services, so it's really nice to get to use a functional language with
it. I've got a lot to learn from both projects I think.

The module in the Files section here is public domain so feel free to
do with at as you care to, or to relicense your copy however you want.
I'm going to rehost it in the next day or so together with some xml
processing stuff I've been doing, and I'll put up a more formal
license (and lack of warranty yada yada) then if that helps. I'm also
intending to get together some simple examples of xml processing with
Clojure and NetKernel, for NK users that haven't seen Clojure.
Hopefully next week...

Cheers, STeve

[Steve Harris]

I put the NetKernel Clojure module up at
http://github.com/scharris/nk-mod-clojure/

It's the same thing posted here except I changed the module URN.

[Randy Kahle]

Steve,

I agree with you about NetKernel and Clojure being an interesting
pair.

Both have the fundamental notion that values should be (Clojure) or
are (NetKernel) immutable. Also the Ref in Clojure (I need to study it
more...) seems to be similar to the notion of have everything in
NetKernel identified by a logical URI address within an address space.

Thank you for the note about the copyright. There will be some news
about moving this forward soon.

What I am going to think about next is the nature of the Clojure
immutable datatypes and whether they should be returned within a
NetKernel Aspect. For example, would it make sense to have
IAspectPersistentMap of IAspecISeq?

Randy

[Randy Kahle]

Steve,

For the XML processing that you are doing in Clojure for NetKernel,
how are you representing the XML in memory? Are you using one of
NetKernel supported forms (e.g. DOM, XDA, JDOM, etc) or are you using
the Clojure data structures (Map, Vector, etc) in some combination?

Randy

[Steve Harris]

Right now the data structures I'm using for xml are pure Clojure. I
wanted to make it as natural as possible to use built-in Clojure
functions, so the data is very "vanilla". Earlier on another thread
Rich had some good comments about why he thinks a plain-data strategy
like that is good, and I guess I tend to agree.

So elements are either just Clojure sequences (usually lists) or else
vectors. Tags are either symbols or keywords, or [uri sym|kwd] pairs
if namespace qualified. The processing library doesn't care about the
exact form within these bounds, which Clojure makes pretty easy, and
the parser can produce any variant depending on how it's called. I
think the sequence representation would be preferred when lazy
consumption is helpful, and in other situations the vector
implementation might give better performance (not tested, yet).

Here's an example document two different ways:

; as a Clojure list:
'(*TOP*
(*PI* "myapp" "a processing instruction")
(account {title "Savings 1" created "5/5/2008"}
(owner "12398")
(balance {currency "USD"} "3212.12")
(descr-html "Main " (b "short term savings") " account.")
(report-separator " ")
(*PI* "myapp" "another processing instruction"))))

; as a Clojure vector and with keyword tags
'[*TOP*
[*PI* "myapp" "a processing instruction"]
[:account {:created "5/5/2008", :title "Savings 1"}
[:ownerid "12398"]
[:balance {:currency "USD"} "3212.12"]
[:descr-html "Main " [:b "short term savings"] " account."]
[:report-separator " "]
[*PI* "myapp" "another processing instruction"]]]

I get your point about it being nonstandard in the Java world. I
wouldn't mind doing a transreptor though to the more common forms (I
might need some help though, haven't done one yet ...)

If you want to see example path expressions in Clojure and their
outputs, take a look at:
http://github.com/scharris/cxpath/tree/master/tutorial.clj
That tutorial is a work in progress. It doesn't yet show how to deal
with namespaces or how to integrate regular Clojure functions into
path expressions (which is probably the biggest advantage vs. the W3C
XPath). The syntax will probably (hopefully) change somewhat too (I
don't like *at* for attributes, wish I could have @ but that's not
allowed in symbols or keywords).

A couple of notes:
- This whole approach and some of the code derives from SXML and
SXPath, on which there's a lot more info at http://okmij.org/ftp/Scheme/xml.html.
- There are other approaches to doing xml in Clojure, esp. Rich's
map-based xml.clj in the Clojure distribution. I chose this different
approach because I've been using SXPath in Scheme (and like it).

- STeve

[Randy Kahle]

Steve,

Most interesting. Thank you.

I have a lot to learn about Clojure and this is helpful.

Before I delve more deeply into the question of in-memory structures
for XML and implications I am going to learn a bit more about the
standard Clojure data types. I am going to do that by helping Tom
Hicks with his mod_clonk NetKernel module.

I have written some aspects and transreptors and handed them back to
Tom as a template. Once we figure out how best to transfer Clojure
data through NetKernel I can then look at the question of transreption
of XML in-memory forms.

- Randy

> SXPath, on which there's a lot more info athttp://okmij.org/ftp/Scheme/xml.html.

[Tom Hicks]

Steve,

I'm don't mean to rock the boat but this question just popped into my
mind....
Have you assured yourself that the set of valid XML
identifiers maps to the set of valid Clojure symbols/keywords?
That is, can an XML file have a tag-name or attribute-name which
isn't legal as a Clojure keyword or symbol?
-tom


On Apr 17, 1:05 pm, Steve Harris <steveO...@gmail.com> wrote:
> ....

> So elements are either just Clojure sequences (usually lists) or else
> vectors. Tags are either symbols or keywords, or [uri sym|kwd] pairs
> if namespace qualified.

> ...
>...
> - STeve

[Steve Harris]

Yeah I think there's no problem there, xml names are pretty limited.
(See http://www.w3.org/TR/REC-xml/#NT-Name We're interested in the
"Name" production).

If we accept the XML namespaces rec in addition to xml (note: that
does NOT mean our xml has to have namespaces), then we won't have :'s
embedded arbitrarily in our names, ie. aside from the namespace
separator, and there's no problem that I can see. I know the xml spec
itself has to continue to allow for namespace-unaware processors and
support xml created prior to the namespace spec, but I'm really not
interested in supporting it.

Also assuming Rich didn't go out of his way to exclude high unicode
values, but I think he's too busy for that... :)

[Tom Hicks]

Sorry, I didn't follow all of this but I think we're talking about the
same thing...the Clojure use of colon for keywords and that being a
problem
if the keyword name becomes an XML tag name. By default, the
'toString' method on a keyword 'abc' returns the string ':abc'. When
I try to use ':abc' as a tag in DOM, I get this exception:

org.w3c.dom.DOMException: NAMESPACE_ERR: An attempt is made to create
or change an object in a way which is incorrect with regard to
namespaces.

So, how do you avoid that problem in the vector representation
you showed above?
regards,
-tom

On Apr 18, 7:06 pm, Steve Harris <steveO...@gmail.com> wrote:
> Yeah I think there's no problem there, xml names are pretty limited.

> (Seehttp://www.w3.org/TR/REC-xml/#NT-Name We're interested in the

[Rich Hickey]

On Apr 20, 11:30 pm, Tom Hicks <hickstoh...@gmail.com> wrote:
> Sorry, I didn't follow all of this but I think we're talking about the
> same thing...the Clojure use of colon for keywords and that being a
> problem
> if the keyword name becomes an XML tag name. By default, the
> 'toString' method on a keyword 'abc' returns the string ':abc'. When
> I try to use ':abc' as a tag in DOM, I get this exception:
>
> org.w3c.dom.DOMException: NAMESPACE_ERR: An attempt is made to create
> or change an object in a way which is incorrect with regard to
> namespaces.
>
> So, how do you avoid that problem in the vector representation
> you showed above?

I'm not sure what Steve does, but there is a name function that
returns the name part of a keyword as a string:

(str :fred)
-> ":fred"

(name :fred)
-> "fred"

The inability to embed colons in names has come up before as an issue
for representing XML and I'd like to encourage reports on any other
issues so that I might address them. I am definitely interested in
being able to represent XML/HTML in Clojure data structures directly.

Rich

[Steve Harris]

Tom,
pay no attention to the colon in the Clojure keyword, it's just
marking it as a keyword so don't think of it as part of the name.
The :my-element tag would correspond to an xml tag of "my-element",
both for reading from and writing to xml. When you use a namespace
aware xml parser, the parser will give you the namespace uri's and
local names already separated, no :'s to deal with.

Not sure if that answers your concerns or not, but if you want to see
the complete parser with some examples at the bottom, it's at:
http://github.com/scharris/cxpath/tree/master/cxml.clj

Steve

[Steve Harris]

A couple of things I miss:

- a variant of syntax quote that doesn't do namespace qualifying of
symbols.
I think symbols look a little nicer as xml tags and require less
explanation to people who know xml/xpath but not clojure. But more
especially symbols often look a lot clearer than keywords as syntax
symbols of embedded languages, where adding an initial colon on an
already symbolic name can be confusing (e.g | is clear, while :| is
kindof confusing visually to me).

- Allowing @ in symbols would be very nice, not just for XPath but
in a lot of other cases where it's general meaning ("at") would be
useful.

[jim]

I read this blog post:

http://www.defmacro.org/ramblings/lisp.html

when I was first starting lisp. This quote stuck in my head:

"Everything we've learned about Lisp so far can be summarized by a
single statement: Lisp is executable XML with a friendlier syntax."

So when I had to do a little XML processing recently, I wrote this
code:

(defn xzip-to-list [xzip]
(cond
(not xzip) nil
(not (node xzip)) nil
(branch? xzip) (concat (list (first (read-string (name (:tag (node
xzip)))))
(:attrs (node xzip)))
(loop [result nil, child (down xzip)]
(cond
(not child) (reverse result)
(not (branch? child)) (reverse (conj result (node child)))
true (recur (conj result (xzip-to-list child)) (right
child)))))
true (node xzip)))

(defn xml-to-list [xml]
(xzip-to-list (xml-zip xml)))

I know the formatting is screwy, but it converts an XML tree to a
Clojure list with a tags attributes as maps. Much like csxml did.

Then I defined macros for each of the tag names in the XML.

So I run the XML through the func to convert it to a list, then I
'eval' the list to process it.

All that to say that xml-zip is all I've needed thus far to do what
I've needed to do with XML. If I do more XML work, I'll probably
write some tools to automate the creation of the macros from an XML
file that I want to process.

jim

> > > (Seehttp://www.w3.org/TR/REC-xml/#NT-NameWe're interested in the

[Tom Hicks]

On Apr 21, 6:44 am, Rich Hickey <richhic...@gmail.com> wrote:
> I'm not sure what Steve does, but there is a name function that
> returns the name part of a keyword as a string:
>
> (str :fred)
> -> ":fred"
>
> (name :fred)
> -> "fred"
>
> The inability to embed colons in names has come up before as an issue
> for representing XML and I'd like to encourage reports on any other
> issues so that I might address them. I am definitely interested in
> being able to represent XML/HTML in Clojure data structures directly.
>
> Rich

Thanks for that useful solution. It's too bad that these functions are
not reversed for keywords. If they were, one could turn strings,
characters, symbols, AND
keywords into strings with the same fn call. Now, one has to check the
data type first and, if it's a keyword, use a different function to
convert to string.

Examples:

(str 'a) --> "a"
(str "a") --> "a"
(str \a) --> "a"
but
(str :a) --> ":a"
(name :a) --> "a"

cheers,
-tom

[Chouser]

On Mon, Apr 21, 2008 at 9:56 PM, Tom Hicks <hicks...@gmail.com> wrote:

Examples:

(str 'a)  --> "a"
(str "a") --> "a"
(str \a) --> "a"
but
(str :a) --> ":a"
(name :a) --> "a"

Or allow 'name' to work for strings and chars, which wouldn't change any currently working behavior.

--Chouser