Errors running Faban benchmark

[A S]

Hey Guys,

I am working on a SPECJ setup (www.spec.org) which uses Faban to drive benchmark tests on the appserver.

I am seeing errors running the benchmark. It is not clear what exactly is failing...

java.lang.Exception: Driver failed to complete benchmark run

Stack Trace:

Class	Method	Line
com.sun.faban.harness.DefaultFabanBenchmark2	end	400
sun.reflect.NativeMethodAccessorImpl	invoke0
sun.reflect.NativeMethodAccessorImpl	invoke	57
sun.reflect.DelegatingMethodAccessorImpl	invoke	43
java.lang.reflect.Method	invoke	606
com.sun.faban.harness.util.Invoker	invoke	118
com.sun.faban.harness.engine.AnnotationBenchmarkWrapper	end	168
com.sun.faban.harness.engine.GenericBenchmark	start	301
com.sun.faban.harness.engine.RunDaemon	run	338
java.lang.Thread

Also looks like fenxi didnt finish processing results after the benchmark failure

Generating html pages ... 0.16s
17:40:46		SEVERE exception	FenXi process command [fenxi, process, /usr/specj2010/faban/harness/faban/output/specjdriverharness.4U/, /usr/specj2010/faban/harness/faban/output/specjdriverharness.4U//post/, specjdriverharness.4U] failed.

I see this and the links for  "Summary Result" and "Detailed Results" are not live. How do I go about figuring out what went wrong?

[Shanti Subramanyam (gmail)]

Not sure which version of Faban you are running because we no longer use Fenxi...

Shanti

--
You received this message because you are subscribed to the Google Groups "faban-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to faban-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[A S]

I am using this benchmark application https://www.spec.org/jEnterprise2010/ (v1.03) which uses faban. So the harness has a copyright information indicating 2005-2009. So I am guessing some version in the time frame 2009-2010.

When I run the benchmark, I do see statistics displayed, but it abruptly terminates with what I indicated earlier. Not sure why or how I would go about solving this problem?

What is not clear to me is...if the program is terminating for some reason and then Fenxi fails to process the results (because they are not there) or is the program terminating because Fenxi is misconfigured (say couldn't process the data and write it to a DB) [which means that benchmark did run to completion] and I should look at other things?

Do you have any tips on how I would trouble shoot the problem?

[A S]

So it turns out this is an inconsistent problem. Sometimes the FABAN runs to completion, so it is not Fenxi or any other result processing thing. However, I sometimes see abrupt termination of Faban benchmark with this message.. (any hints? usually contains some error message indicating socket refused etc)

Master terminated with exit value 143

java.rmi.ConnectException: Connection refused to host: 172.16.41.37; nested exception is: 
java.net.ConnectException: Connection refused

----------------------------------------------------------------------------------------------------

Also, I see this problem often just before I get to "Steady State Completed" ... The specj uses Faban version 1.0.1 I think.

Also, where can I find documentation on how to distribute faban bechmark (load) on multiple clients for version 1.0.1?

[Shanti Subramanyam (gmail)]

Try running the latest version of Faban.

[A S]

I dont think its a Fenxi problem because as far as I understand it is only involved in post processing of results. The master exiting phenomenon is before it gets to that Fenxi stage. It usually is during one of the these states that the master exits (RAMP UP, STEADY STATE) because it sees a kill from the AgentBootStrap.java process. I am using Faban 1.0, because I am using a benchmark product that was developed using Faban 1.0 

https://www.spec.org/jEnterprise2010/


I must say that if I choose shorter runs, I can run the benchmark to completion, so this is an inconsistent problem.

[A S]

Hello Guys,

I eventually resolved the problem. There is a very good chance this is a bug with OPEN JDK implementation and how it handles sub process (If you google this you will see some bugs resolved in 1.8, I am using 1.7). Swapping out OPEN JDK runtime to Hotspot runtime  ran the benchmark without even a hint of a problem.

A S

[Jeremy Whiting]

Hi AS,
 Do you mean this Hotspot runtime ?

http://openjdk.java.net/groups/hotspot/

Jeremy

[A S]

Hi Guys,

Once I swapped the OPEN JDK runtime (Iced Tea) with Oracle runtime (Hotspot), I didnt see any problem.

Sorry, earlier I said Hotspot instead of Iced Tea. I wanted to share this in the forum for other users who may or may not see the issue.

Ashwin

[Jeremy Whiting]

Hi Ashwin,
 Thanks for making the correction. It is necessary to get these things correct or it become folk law and unfounded reasons for not using OpenJDK.

 I've come across this same "Master terminated with 143" error code.
 By switching to the Oracle JDK you probably have got round it because I think that JDK bundles tzdata information with the JDK. Making is self contained.

 Whereas OpenJDK is relying on the tzdata package on the system. In addition to tzdata-java. For some reason tzdata is having an effect on Faban driver clients. Causing the benchmark run to fail.
 When using OpenJDK with certain version of tzdata package installed I experience the same Master terminated with 143 error. tzdata version 2016d in that situation. Switching back to 2015a made the problem stop.

 I am following it up with the tzdata package maintainers to narrow down exactly which version of tzdata the issue started to appear.

Regards,
Jeremy

[Brian C]

Hello all,
From information I have gathered second hand from others who have spent a lot of hours on this issue, the error 143 issue occurs in some version of Java greater than 8u60. I have verified that it does not occur with Oracle JDK8u60, but does occur with Oracle JDK8u131. Also, there is a workaround for Faban (also gained from the hard work of others). By adding the following option to the java command line in faban/harness/faban/bin/agent (or agent.cmd for Windows):

-Djava.rmi.dgc.leaseValue=144000000

This should allow Faban to run for 40 hours.

Thanks,
- Brian Chesebro

[Shanti Subramanyam (gmail)]

Thanks Brian. It would be great if you can file a ticket on github.com/akara/faban.

--
You received this message because you are subscribed to the Google Groups "faban-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to faban-users+unsubscribe@googlegroups.com.