Re: trying out netty 4 in clojure

[Stuart Sierra]

I recently came across this issue too.

The solution is to cast the number to an Integer, as in:

    (.option bootstrap ChannelOption/SO_BACKLOG (int 1024))

Netty expects an Integer value for that option, and Clojure defaults to Long for literals.

I think this is an example of a bad interaction between Clojure's reflective compiler and Netty's use of generics in ChannelOption.

-S



On Thursday, March 14, 2013 12:09:10 PM UTC-4, shlomi...@gmail.com wrote:

It seemed to be a problem with setting (.option b ChannelOption/SO_BACKLOG 100). if i remove that, it works fine..

On Tuesday, March 12, 2013 3:42:15 PM UTC+2, shlomi...@gmail.com wrote:

I am trying to run the echo example (rewritten in clojure) and i get "channel not registered to an event loop".  

https://gist.github.com/vadali/5142879

this is the bootstrap object printed out:

#<ServerBootstrap ServerBootstrap(group: NioEventLoopGroup, channelFactory: NioServerSocketChannel.class, options: {SO_BACKLOG=100}, childGroup: NioEventLoopGroup, childHandler: index_server.netty.proxy$io.netty.channel.ChannelInitializer$0@6094cae2)>

here is the stack trace:

Exception in thread "main" java.lang.IllegalStateException: channel not registered to an event loop

at io.netty.channel.AbstractChannel.eventLoop(AbstractChannel.java:159)

at io.netty.channel.nio.AbstractNioChannel.eventLoop(AbstractNioChannel.java:105)

at io.netty.channel.nio.AbstractNioChannel.eventLoop(AbstractNioChannel.java:41)

at io.netty.channel.DefaultChannelHandlerContext.executor(DefaultChannelHandlerContext.java:492)

at io.netty.channel.DefaultChannelHandlerContext.newPromise(DefaultChannelHandlerContext.java:1652)

at io.netty.channel.DefaultChannelHandlerContext.close(DefaultChannelHandlerContext.java:1100)

at io.netty.channel.DefaultChannelPipeline.close(DefaultChannelPipeline.java:1005)

at io.netty.channel.AbstractChannel.close(AbstractChannel.java:230)

at io.netty.bootstrap.ServerBootstrap.doBind(ServerBootstrap.java:152)

at io.netty.bootstrap.AbstractBootstrap.bind(AbstractBootstrap.java:262)

at io.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:46)

at io.netty.bootstrap.AbstractBootstrap.bind(AbstractBootstrap.java:237)

at io.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:46)

at test.netty$_main.invoke(netty.clj:42)

at clojure.lang.Var.invoke(Var.java:411)

...

What silly thing am i missing?

[Norman Maurer]

@Stuart do you know if there is anything we can do about to give Clojure a correct hint ?

---

Norman Maurer

nma...@redhat.com

JBoss, by Red Hat

--
 
---
You received this message because you are subscribed to the Google Groups "Netty discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netty+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Stuart Sierra]

What I showed: use the `int` function as a type cast.
-S

[Norman Maurer]

I was more looking for something that we could do in ChannelOption to let clojure handle it correctly.

---

Norman Maurer

nma...@redhat.com

JBoss, by Red Hat

[th.van...@gmail.com]

Hi, 

Has anyone succeeded in using a recent (4.0.12) version of Netty in combination with Clojure? I tried the suggestions mentioned above, but no luck.

Thanks in advance,

Thomas

[Norman Maurer]

what error you see?

--

[th.van...@gmail.com]

I tried a combination of [1] and [2] and this is what I have at the moment:

(ns netty.core

  (:import   [io.netty.bootstrap AbstractBootstrap ServerBootstrap]

             [io.netty.channel ChannelFuture ChannelInitializer ChannelOption

                               ChannelHandlerContext ChannelInboundHandlerAdapter ChannelHandler]

             [io.netty.handler.logging LogLevel LoggingHandler]

             io.netty.buffer.ByteBuf

             io.netty.channel.socket.SocketChannel

             io.netty.channel.nio.NioEventLoopGroup

             io.netty.channel.socket.nio.NioServerSocketChannel

             java.util.concurrent.atomic.AtomicInteger)

  (:use [clojure.stacktrace])

  (:gen-class))

(defn echo-server-handler []

  (proxy [ChannelInboundHandlerAdapter] []

    (channelRead [this ctx msg]

                 (.write ctx msg))

    (channelReadComplete [this ctx]

                         (.flush ctx))

    (exceptionCaught [this ctx cause]

      (print-stack-trace cause)

      (.close ctx))))

(defn -main []

  (let [b (ServerBootstrap.)]

    (try

      (->  b

          (.group (NioEventLoopGroup.) (NioEventLoopGroup.))

          (.channel io.netty.channel.socket.nio.NioServerSocketChannel)

          (.option  ChannelOption/SO_BACKLOG (int 100))

          (.childHandler

           (proxy [ChannelInitializer] []

             (initChannel [ch]

               (-> ch   (.pipeline) (.addLast (.addLast (aset (make-array ChannelInboundHandlerAdapter 1) 0 echo-server-handler))))))))

      (println b) ; print the server bootstrap object

      (let [f (-> b

                  (.bind 2424)

                  (.sync))]

        (-> f

            (.channel)

            (.closeFuture)

            (.sync)))

      (finally (.shutdown b)))))

and when I start the EchoClient [3] (in Java) I get the following error:

 (-main)

#<ServerBootstrap ServerBootstrap(group: NioEventLoopGroup, channelFactory: NioServerSocketChannel.class, options: {SO_BACKLOG=100}, childGroup: NioEventLoopGroup, childHandler: netty.core.proxy$io.netty.channel.ChannelInitializer$0@89912783)>

Nov 14, 2013 9:04:49 PM io.netty.channel.ChannelInitializer channelRegistered

WARNING: Failed to initialize a channel. Closing: [id: 0x12a4ed6d, /127.0.0.1:54328 => /127.0.0.1:2424]

java.lang.ArrayStoreException

at clojure.lang.RT.aset(RT.java:2242)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)

at java.lang.reflect.Method.invoke(Method.java:613)

at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)

at clojure.lang.Reflector.invokeStaticMethod(Reflector.java:207)

at netty.core$_main$fn__1193.invoke(form-init8384742756774487331.clj:12)

So for some reason it doesn't seem to like the type I try to put in the array (I think), but not sure how to get around it.

TIA,

Thomas

[1] https://gist.github.com/vadali/5142879

[2] http://blog.quiptiq.com/2013/05/20/porting-the-netty-discardserver-example-to-clojure/

[3] https://github.com/netty/netty/tree/master/example/src/main/java/io/netty/example/echo

["이희승 (Trustin Lee)"]

I wonder what Clojure folks thinks about this issue. Looks like a bug
in Clojure roughly guessing from the stack trace?

>> <http://gmail.com> wrote:
>>
>> I am trying to run the echo example (rewritten in clojure)
>> and i get "channel not registered to an event loop".
>>
>> https://gist.github.com/vadali/5142879
>> <https://gist.github.com/vadali/5142879>
>>
>> this is the bootstrap object printed out:
>> #<ServerBootstrap ServerBootstrap(group: NioEventLoopGroup,
>> channelFactory: NioServerSocketChannel.class, options:
>> {SO_BACKLOG=100}, childGroup: NioEventLoopGroup,
>> childHandler:
>> index_server.netty.proxy$io.netty.channel.ChannelInitializer$0@6094cae2)>
>>
>> --
>>
>> ---
>> You received this message because you are subscribed to the
>> Google Groups "Netty discussions" group.
>> To unsubscribe from this group and stop receiving emails from it,

>> send an email to netty+un...@googlegroups.com <javascript:>.

>> For more options, visit https://groups.google.com/groups/opt_out

>> <https://groups.google.com/groups/opt_out>.

>
> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "Netty discussions" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to netty+un...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

--
https://twitter.com/trustin
https://twitter.com/trustin_ko
https://twitter.com/netty_project

[shlomi...@gmail.com]

Hey,

there were a few problems with your attempt:

1. echo-server-handler was a function rather than a var. you passed that function into the java vector which caused the invalid type - causing the aset problem.

2. you used an extra addLast, instead, you needed to return the original changed vector (aset return the last set val), like this:

     (let [a (make-array ChannelHandler 1)]

         (aset a 0 echo-server-handler)

         a)

I used (into-array ... ) instead. Here is a working version based on your attempt:

(ns netty4.core

  (:import   [io.netty.bootstrap AbstractBootstrap ServerBootstrap]

             [io.netty.channel ChannelFuture ChannelInitializer ChannelOption

                               ChannelHandlerContext ChannelInboundHandlerAdapter ChannelHandler]

             [io.netty.handler.logging LogLevel LoggingHandler]

             io.netty.buffer.ByteBuf

             io.netty.channel.socket.SocketChannel

             io.netty.channel.nio.NioEventLoopGroup

             io.netty.channel.socket.nio.NioServerSocketChannel

             java.util.concurrent.atomic.AtomicInteger)

  (:use [clojure.stacktrace])

  (:gen-class))

(def echo-server-handler

  (proxy [ChannelInboundHandlerAdapter] []

    (channelRead [this ctx msg]          (.write ctx msg))

    (channelReadComplete [this ctx]   (.flush ctx))

    (exceptionCaught [this ctx cause]

      (print-stack-trace cause)

      (.close ctx))))

(defn -main []

  (let [b (ServerBootstrap.)]

    (try

      (->  b

          (.group (NioEventLoopGroup.) (NioEventLoopGroup.))

          (.channel io.netty.channel.socket.nio.NioServerSocketChannel)

          (.option  ChannelOption/SO_BACKLOG (int 100))

          (.childHandler

           (proxy [ChannelInitializer] []

             (initChannel [ch]

               (-> ch  (.pipeline)

                   (.addLast (into-array ChannelHandler [echo-server-handler])))))))

[th.van...@gmail.com]

Hi,

Thank you for your suggestions, they have certainly improved things. Unfortunately when I now run it I get this error:

(Which is rather strange I have to admit)

netty.core=> (-main)

#<ServerBootstrap ServerBootstrap(group: NioEventLoopGroup, channelFactory: NioServerSocketChannel.class, options: {SO_BACKLOG=100}, childGroup: NioEventLoopGroup, childHandler: netty.core.proxy$io.netty.channel.ChannelInitializer$0@8ea69a49)>

Nov 15, 2013 5:30:28 PM io.netty.channel.DefaultChannelHandlerContext invokeExceptionCaught

WARNING: An exception was thrown by a user handler's exceptionCaught() method while handling the following exception:

java.lang.IllegalArgumentException: Can't call public method of non-public class: public io.netty.channel.ChannelFuture io.netty.channel.DefaultChannelHandlerContext.write(java.lang.Object)

at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:88)

at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)

at netty.core$fn__1185$fn__1190.invoke(form-init8990654338957554067.clj:3)

at netty.core.proxy$io.netty.channel.ChannelInboundHandlerAdapter$0.channelRead(Unknown Source)

at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)

at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324)

...

...

I am running Clojure 1.5.1 and Netty 4.0.12Final.

Thomas

[Shlomi Vaknin]

Hey,

I didnt try to run it, but from the code it seems to be correct - DefaultChannelHandlerContext is a private class, and I guess that this is the class of ctx in (channelRead [this ctx msg]  (.write ctx msg))

Ill try to run it later when I get home, and see if I get this error.

--
 
---
You received this message because you are subscribed to a topic in the Google Groups "Netty discussions" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/netty/mcGswAZ5NeQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to netty+un...@googlegroups.com.

[shlomi...@gmail.com]

hey,

I confirmed this is the case on my end too, here is the latest code: https://gist.github.com/vadali/5142879#file-4-0-12-final-clj .

This takes me back to an old bug I opened to the clojure project some time ago about some bad public/private handling, it doesnt seem to be exactly the same, but probably stems from a similar problem. 

Netty seems to be making heavy use of public/private classes, and it seems to get in the way. In this example, both ChannelOutboundInvoker and DefaultChannelHandlerContext are marked as private (the first is the origin of the .write function, and the second is the default implementation).

Lee - is this a must? I mean, why does a default implementation of an interface has to be private? is there an engineering reason, or is it something you could consider making public? the other options would be to make java wrappers on these private classes/interfaces to bypass this limitation (as someone started doing)

[Norman Maurer]

I would prefer to keep them private to be honest as the are not meant to be instanced by users. Or we maybe could do public with package private constructor

--
 
---
You received this message because you are subscribed to the Google Groups "Netty discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netty+un...@googlegroups.com.

[shlomi...@gmail.com]

Hey

Ok, I spent most of the last two days trying to figure out who does what wrong.. here are my findings:

It seems that clojure is trying to resolve which method should actually get called. To do this, it traverses the dependency tree of the object, in our case its ctx, which is of type ChannelHandlerContext. 

Now, what actually gets passed around is DefaultChannelHandlerContext which is a private class that has a public write method. now, since the class itself is public, clojure cant make a reflective call to it directly (java's reflection fails because its private). I am not sure how this mechanism works in plain java, maybe this is a "feature" where if you have a direct call (not reflective) this is not checked. i dont know. what i do know is that clojure is starting to look upwards to find a non-private base class that it could execute write against. to do so, it invokes ChannelHandlerContext.getClass().getMethods(), which brings back the following results (among many others..):

   #<Method public abstract io.netty.channel.ChannelFuture io.netty.channel.ChannelOutboundInvoker.write(java.lang.Object)>,

   #<Method public abstract io.netty.channel.ChannelFuture io.netty.channel.ChannelOutboundInvoker.write(java.lang.Object,io.netty.channel.ChannelPromise)>,

so, it only finds the next best candidate to be ChannelOutboundInvoker, which actually defines the write method - but it is also private! there is nothing else clojure could look for here.

So I am seeing that there's no way for clojure to find a non-private base-class defining that method, simply because it does not exist. 

I tried to look at why clojure ever go the reflective way even though we have types annotations, but the places i found in the code didnt seem to be correct.

Stuart Sierra - I would be very happy if you could try and point me in a right direction for this (would help solving that bug I filed)

On Saturday, November 16, 2013 9:15:34 AM UTC+2, Norman Maurer wrote:

I would prefer to keep them private to be honest as the are not meant to be instanced by users. Or we maybe could do public with package private constructor

 I think turning them into public classes with package private constructor could solve this issue, but thats a road i am yet to take.. ill try that soon.. 

Thanks

[shlomi...@gmail.com]

Hey - an 

I would prefer to keep them private to be honest as the are not meant to be instanced by users. Or we maybe could do public with package private constructor

 

I think turning them into public classes with package private constructor could solve this issue, but thats a road i am yet to take.. ill try that soon.. 

This solution works perfectly - made DefaultChannelHandlerContext public and its constructor package-private, and it all works great. Is this something we could expect future versions of netty to have? 

Thanks!

[Norman Maurer]

Why this can't be fixed in clojure? I think it's clearly a bug in there

[shlomi...@gmail.com]

This seems more like a reflection issue on the jdk - i couldnt find a way to invoke this method using reflection (aside implicitly marking setAccesible(true) on that class anyway ) - if you can help me figure one out ill be happy.

also, dont take this the wrong way, but even despite clojure having a different bug (still using reflection when it shouldnt), i dont think passing around private implementations to the user is the best design.. :) 

if your reason is not allowing users to instantiate - then you should keep the constructor invisible, if you dont intend your users to even get an instance, then make the thing private, but you have a mish-mash of private/public that doesnt make too much sense to me. (but that just me)

I dont work on the clojure language, i am just a clojure programmer very interested in netty, and i'd love to get these two to work together. I think having a private constructor is a good compromise (hey, you suggested it! :D)

[Stuart Sierra]

On Sun, Nov 17, 2013 at 10:08 AM, Norman Maurer <norman...@googlemail.com> wrote:

Why this can't be fixed in clojure?

I did some deep digging into this issue with Clojure at
http://dev.clojure.org/jira/browse/CLJ-1243

The problem appears to be the Java reflection API, which the Clojure compiler uses.

Java reflection has long-standing bugs in access rules; see
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4283544

It is theoretically possible to make the Clojure compiler examine bytecode to resolve Java methods instead of reflection. This would be a major change to the compiler and is not likely to happen in the near future.

Until that time, it seems like Clojure programs will not be able to use Netty 4.

This problem may also affect other JVM languages which use reflection to resolve methods. (Groovy? JRuby?)

-S

[Norman Maurer]

--
 

So to make it clear we can not use package private classes and expose them via interfaces ?

Bye,

Norman

[shlomi...@gmail.com]

Hey,

It seems that each method must have at least one public base class that defines it, so in our case, its enough to make the interface ChannelOutboundInvoker public for this to be solved (at least for the write method).

[Norman Maurer]

Alright...

@trustin: what you think about to chsnge the interface to be public?

--

["이희승 (Trustin Lee)"]

Alternatively, we could remove those interfaces and move their methods to its subtypes, because they are not really same in terms of semantics.  For example, Channel.fireChannelActive() and ChannelHandlerContext.fireChannelActive() are completely different when it comes to where the event starts from.  Same applies for the most methods in ChannelOutboundInvoker and ChannelPropertyAccess.

[Norman Maurer]

I'm fine with both.... Just thought make them public is the easiest solution

["이희승 (Trustin Lee)"]

It's easiest but the documentation will look bad in my opinion.

[Norman Maurer]

True... Go for it