Empty console for OSGi based application, but no errors reported during startup

[Mads Ingstrup]

Hi,

I am trying to enable the console for monitoring an OSGi based akka
application, but no data shows up in the web console.

I started with the sample application which works as expected with the
same versions of akka that are used in my application.

The jars used for the console were re-packaged as an OSGi bundle with the
manifest headers necessary to enable the weaving with aspectj (equinox
aspects).

The weaving is done for the akka bundle, and the atmos logic starts
without reporting any errors. At least I assume it does, because I can see
that it does establish a connection to the atmos process on the default 
28667 port. The application is launched with -javaagent:atmos-libs/weaver/aspectjweaver.jar
and -Djava.library.path=atmos-libs/sigar/


My application uses akka-clustering, but I am only starting a single
instance. I had to change the port for akka remote so that my
application uses port 2553 while
atmos uses the default 2552; otherwise I get the following error
messages in the log:

DEBUG 16:51:17.189 sending message RemoteMessage:
[SelectChildName(system,SelectChildName(cluster,SelectChildName(core,SelectChildName(daemon,InitJoin))))]
to
[Actor[akka.tcp://MySe...@10.1.1.127:2552/]]<+[akka.tcp://MySe...@10.1.1.127:2552/]
from
[Actor[akka://atmos/system/cluster/core/daemon/joinSeedNodeProcess#4232237]]
[akka.remote.EndpointWriter, atmos-akka.actor.default-dispatcher-18]
DEBUG 16:51:17.192 received remote-destined message RemoteMessage:
[SelectChildName(system,SelectChildName(cluster,SelectChildName(core,SelectChildName(daemon,InitJoin))))]
to
[Actor[akka.tcp://MySe...@10.1.1.127:2552/]]<+[akka.tcp://MySe...@10.1.1.127:2552/]
from
[Actor[akka://atmos/system/cluster/core/daemon/joinSeedNodeProcess#4232237]]
[akka.remote.EndpointWriter, atmos-akka.actor.default-dispatcher-18]
ERROR 16:51:17.193 dropping message [class akka.actor.SelectChildName]
for non-local recipient [Actor[akka.tcp://MySe...@10.1.1.127:2552/]]
arriving at [akka.tcp://MySe...@10.1.1.127:2552] inbound addresses are
[akka.tcp://at...@10.1.1.127:2552] [akka.remote.EndpointWriter,
atmos-akka.actor.default-dispatcher-18]

The configuration for atmos (obtained by using the render method on the
actual config object passed to atmos) is:

trace {
    buffer {
        local-limit=20
        size-limit=100
        time-limit="500 milliseconds"
    }
    deadlock-watch-frequency="60 seconds"
    defaults {
        sampling=1000
        system-traceable {
            "/"=off
            "/system"=off
            "/system/*"=off
            "/system/log*-Slf4jTraceContextEventHandler"=on
            "/system/log*-Slf4jTraceContextLogger"=on
        }
        traceable=off
    }
    dispatcher-poll-interval="3 seconds"
    enabled=true
    events {
        actions=on
        actors=on
        event-stream=on
        futures=off
        iteratees=off
        netty=on
        remoting=on
        runnables=on
        scheduler=on
        system-messages=on
        temp-actors=on
    }
    futures=off
    internal-system {
        akka {
            actor {
                debug {
                    event-stream=on
                    lifecycle=on
                }
            }
            log-config-on-start=on
            loglevel=DEBUG
            remote {
                log-received-messages=on
                log-sent-messages=on
            }
        }
        atmos-trace-dispatcher {
            executor=fork-join-executor
            fork-join-executor {
                parallelism-max=2
                parallelism-min=2
            }
        }
    }
    iteratees=off
    max-length-long-message=200
    max-length-short-message=30
    max-stack-trace-size=20
    node=MyServer
    play {
        defaults {
            sampling=1
            traceable=on
        }
        sampling {
            "*"=1
        }
        traceable {
            "*"=on
        }
    }
    remote-life-cycle=true
    sampling {
        "*"=1
    }
    send {
        capacity=100
        daemonic=off
        global {
            daemonic=on
        }
        port=28667
        retry=on
        warn=on
    }
    shutdown-timeout="5 seconds"
    system-metrics-poll-interval="3 seconds"
    tags {}
    traceable {
        "*"=on
        "/system/log*"=on
    }
    use-dispatcher-monitor=on
    use-system-metrics-monitor=on
}

[Jim Powers]

Mads,

Sorry for the delay.

We have looked into getting console working with OSGi containers and so far no luck.  That said, the fact that you are using Equinox instead of Apache Felix might give you a chance.  I think that the problem is related to where the weaving agent kicks in and what classloader it works with.  The last time I looked at this it was in the context of Apache Felix and I had no luck getting the weaving instrumentation to work, however, Equinox does have support for AspectJ weaving: http://eclipse.org/equinox/weaving/.  I think that you need to have a weaving hook as part of your bundle that wires-in the AspectJ weaver.  Then it should pick up on the instrumented code in the trace jars.

Also of note: the console trace jars are not OSGi compliant.  In my experiments I had to rebundle the console trace jars using some maven trickery.  I've attached the pom.xml I used, you may need to update the versions of things for the version of Akka you are using.

--
Jim Powers
Senior Engineer @ Typesafe
Twitter: @corruptmemory