Yet Another Non-Functioning Console (Akka)

[Randall Schulz]

Hi,

I've read all the messages here (and several in Stack Overflow) about how to get the Typesafe Console to actually display information for my Akka application without any luck.

In the initial dump of the Akka configuration, this set of lines appears:

    # SAF-akka.conf: 49

    "atmos" : {

        # SAF-akka.conf: 50

        "trace" : {

            # SAF-akka.conf: 71

            "buffer" : {

                # SAF-akka.conf: 76

                #  Maximum time before flushing

                "time-limit" : "1000 milliseconds",

                # SAF-akka.conf: 73

                #  Maximum number of events stored before flushing

                "size-limit" : 1000

            },

            # SAF-akka.conf: 51

            "enabled" : true,

            # SAF-akka.conf: 52

            "node" : "saf-task-runner",

            # SAF-akka.conf: 65

            #  Trace futures outside of actor traces

            "futures" : "on",

            # SAF-akka.conf: 59

            "sampling" : {

                # SAF-akka.conf: 61

                "*" : 1000,

                # SAF-akka.conf: 60

                "/user/saf-agent/*" : 1

            },

            # SAF-akka.conf: 54

            "traceable" : {

                # SAF-akka.conf: 56

                "*" : "off",

                # SAF-akka.conf: 55

                "/user/saf-agent/*" : "on"

            }

        }

    },

This makes me think my config is in the right place. I ran the application doing actual work for several seconds(that's a lot of work and a whole lot of message sends and receives) and the Console display remains all 0 and N/A.

I tried adding event-handlers = ["com.typesafe.atmos.trace.Slf4jTraceContextEventHandler"] within the akka { ... } block and that resulted in a ClassDefNotFoundException:

akka.ConfigurationException:

 Could not start Event Handler due to

  [akka.ConfigurationException:

   Logger specified in config can't be loaded

    [com.typesafe.atmos.trace.Slf4jTraceContextEventHandler]

   due to

    [java.lang.ClassNotFoundException: com.typesafe.atmos.trace.Slf4jTraceContextEventHandler]]

(Line breaks and indents are mine.)

The relevant SBT dependencies are:

"com.typesafe.akka"  %% "akka-actor"       % "2.2.3",

"com.typesafe.akka"  %% "akka-slf4j"       % "2.2.3",

"com.typesafe.akka"  %% "akka-remote"      % "2.2.3",

"com.typesafe.atmos" %% "trace-akka-2.2.1" % "1.3.1",

I presume that trace-akka-2.2.1 is compatible with Akka 2.2.3.

There are no occurrences of No trace receiver in my log file.

The only occurrences of "atmos" (case-insensitively) anywhere in the log file are:

    "atmos" : {

            "com.typesafe.atmos.trace.Slf4jTraceContextLogger"

The Atmos agent and Console UI logs are almost entirely empty:

% tail -n 1000 -F *-log

==> atmos-log <==

2013-12-17 09:10:05,816 INFO  [akka.event.slf4j.Slf4jEventHandler] [] [] : Slf4jEventHandler started

2013-12-17 09:10:05,860 INFO  [U] [ActorSystem(query)] [query-akka.actor.default-dispatcher-4] : *** No license available. Defaulting to in-memory persistence. ***

2013-12-17 09:10:06,077 INFO  [akka.event.slf4j.Slf4jEventHandler] [] [] : Slf4jEventHandler started

2013-12-17 09:10:06,121 INFO  [akka.event.slf4j.Slf4jEventHandler] [] [] : Slf4jEventHandler started

2013-12-17 09:10:06,794 INFO  [akka://query/user/IO-HTTP/listener-0] [akka://query/user/IO-HTTP/listener-0] [query-akka.actor.default-dispatcher-3] : Bound to localhost/127.0.0.1:8660

==> ui-log <==

Play server process ID is 26647

2013-12-17 09:10:22,222 INFO  [play] [] [] : Application started (b)

2013-12-17 09:10:22,314 INFO  [play] [] [] : Listening for HTTP on /0:0:0:0:0:0:0:0:9900

2013-12-17 09:14:02,819 INFO  [akka.event.slf4j.Slf4jEventHandler] [] [] : Slf4jEventHandler started

I am entirely at a loss how to make the Console work.

Randall Schulz

[Peter Vlugter]

Hi Randall,

There are three things needed to run Console:

- aspectj weaving enabled (java agent or weaving classloader)
- dependencies for the trace aspects and trace recording
- config to enable tracing and select actors for tracing

The dependencies and config you list look they should work.

You don't mention the aspectj weaver - how are you enabling this?

You can enable the aspectj verbose logging to confirm that classes are in fact being instrumented. Or a simple check is to run your application with weaving but without the atmos trace collector running, in which case you'll get the "no trace receiver" message when it tries to send the data.

Alternatively, and if you're using sbt, you could try the sbt-atmos plugin for an easy way to get started:

https://github.com/sbt/sbt-atmos

Peter

> --
> You received this message because you are subscribed to the Google Groups "Typesafe Console User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to typesafe-conso...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Randall Schulz]

I launch the JVM with the AspectJ agent option in its command line: java -javaagent:lib/weaver/aspectjweaver.jar

I was missing the sbt-atmos SBT plug-in and its atmosSettings.

However, it now seems (based on a conversation in #akka) that I have to run from SBT? If so, I need an alternative, 'cause that's a non-starter. We deploy OneJAR executables within an ecosystem that cannot deploy SBT-launched services.

What are my alternatives?

[Peter Vlugter]

On 18/12/2013, at 10:55 am, Randall Schulz <rsc...@sonic.net> wrote:

> I launch the JVM with the AspectJ agent option in its command line: java -javaagent:lib/weaver/aspectjweaver.jar

Okay. If you run this without atmos, do you get the "no trace receiver" warning?

Are you trying this with a onejar? There are aop.xml files that enable the weaving on akka and scala classes. For a fat jar we need to make sure these are available.

> I was missing the sbt-atmos SBT plug-in and its atmosSettings.
>
> However, it now seems (based on a conversation in #akka) that I have to run from SBT?

Yes. The sbt-atmos plugin is for running Console for development only.

> If so, I need an alternative, 'cause that's a non-starter. We deploy OneJAR executables within an ecosystem that cannot deploy SBT-launched services.
>
> What are my alternatives?

Getting your deployment setup working with Console should certainly be possible. It could be the aop configuration, if you're running from a single jar. I think the aop.xml files will need to be combined for everything to work. Or if you'd like to send us a sample project with your setup, we can confirm what the missing piece is.

Note that Typesafe Console is focused on development. We're recommending other solutions for production monitoring, and we're partnering with companies in the production monitoring space. The console description page has more information on this: http://typesafe.com/platform/runtime/console

Peter

[Randall Schulz]

OK, I made some actual progress this time. The "ecosystem" I referred to includes Puppet resetting a customized version of our service launch script that I use to get that -javaagent option in. I forgot that my suspension of Puppet on the target machine had expired and it had effectively removed my -javaagent option.

With that reinstated, I got a real live Typesafe Console display. For about a minute, then it all stopped. In Atmos agent log file (std out+err of typesafe-console atmos) I find things like:

Several of these:

java.util.concurrent.TimeoutException: Futures timed out after [5 seconds]

... each of which is followed by what looks like a compressed stack backtraace.

That goes on for a while then this is logged:

2013-12-17 14:03:03,900 ERROR [U] [ActorSystem(query)] [query-akka.actor.default-dispatcher-19] : Uncaught error from thread [query-akka.actor.default-dispatcher-19] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled

java.lang.OutOfMemoryError: GC overhead limit exceeded

Which would seem like the end of the road, but this pattern (the TimeoutException, compact stack trace and, OOME / "shutting down JVM" repeats several times over.

That Console application remains running through this.

Anyway, I suppose the upshot is that the Atmos agent needs more RAM. I see from the launcher script that it passes on -Xmx, -Xms and other JVM opts.

What's a good heap limit for the Atmos agent?

Randall Schulz

[Peter Vlugter]

Cool. Good that instrumentation is working now.

Yes, it sounds like Atmos needs more memory.

The default limit when running with sbt-atmos is 1GB, and apps with heavy load run out of memory with this setting. The default configuration, however, traces all actors with a sampling rate of 1.

I would try 2GB, and see how it goes. You can also reduce the amount of data being processed with sampling rates or reducing the number of actors that are traceable.

Let us know what you find works. There are also some configuration options for the types of analysis that are performed. What we don't have yet are easily tweakable memory limits within Atmos itself, but I believe that will be worked on for the new development version.

Cheers,
Peter

[Randall Schulz]

Hi,


On Tuesday, 17 December 2013 15:11:33 UTC-8, Peter Vlugter wrote:

…

Yes, it sounds like Atmos needs more memory.

The default limit when running with sbt-atmos is 1GB, and apps with heavy load run out of memory with this setting. The default configuration, however, traces all actors with a sampling rate of 1.

I'm using the BASH script to launch Atmos agent and the UI. In the current version of the script (the 1.3.1 distribution), the defaults for initial and maximum heap are 512 MB:

[[ -z "$optXms" && -z "$optXmx" ]] && optXms="-Xms512m" && optXmx="-Xmx512m"

I upped it to 1 GB via the command line.


Randy