:WARNING:Retry connecting to CmdAgent

[Zubair Sediqi]

Hello,

 

After running the benchmark I see error "Can not contact hadoop-6-11(also when change it to IP 192.168.100.13 the same problem). The issue was solved by editing the file $FABAN_HOME/bin/Linux removing the last part and of the file and adding only interface name(eth0 in my case).But now the error message is  ":WARNING:Retry connecting to CmdAgent@hadoop-6-11, count X".Here is full scenario. I use three ubuntu based Virtual  Machines(VM) as Faban Master, Backend Database and Frontend server with following components.  

 

Client Machine (Hadoop-6-10, 192.168.100.3)

       - Faban Master 

      

Backend Machine (hadoop-6-13, 192.168.100.15)

        - MySQL

        - Tomcat

        - Geocoder Emulator

Frontend Machine( hadoop-6-11, 192.168.100.13)

       - Olio Implementatioin in PHP(web application)

       - Nginx

       - APC

       - PHP 

    All the installation and configuration is successfully done according to   http://parsa.epfl.ch/cloudsuite/web.html

    I have to run the Web Serving Benchmark from command line , attached is my run.xml file. The java path is the same in all machines. SSH is enabled between machines without prompting for password. I queried your mail archive but could not solve the problem yet. 

Here is the problem I get in Client machine when I run the Faban.

*************************************************

Client Machine

ubuntu@hadoop-6-10:~/web-release/apache-olio-php-src-0.2/workload/php/trunk/deploy$ $FABAN_HOME/bin/fabancli submit OlioDriver default run.xml

OlioDriver.2R

ubuntu@hadoop-6-10:~/web-release/apache-olio-php-src-0.2/workload/php/trunk/deploy$ $FABAN_HOME/bin/fabancli showlogs OlioDriver.2R -ft

14:02:26:INFO:Starting OlioDriver run using /home/ubuntu/web-release/faban/output/OlioDriver.2R/

14:02:26:INFO:START TIME : Mon May 27 14:02:26 UTC 2013

14:02:26:INFO:Starting Registry.

14:02:26:INFO:Registry started.

14:02:56:INFO:CmdService: Configured FileAgent@hadoop-6-10 on server hadoop-6-10

14:03:06:WARNING:Retry connecting to CmdAgent@hadoop-6-11, count 1.

14:03:16:WARNING:Retry connecting to CmdAgent@hadoop-6-11, count 2.

14:03:26:WARNING:Retry connecting to CmdAgent@hadoop-6-11, count 3.

14:03:36:WARNING:Retry connecting to CmdAgent@hadoop-6-11, count 4.

14:03:46:WARNING:Retry connecting to CmdAgent@hadoop-6-11, count 5.

14:03:56:WARNING:Retry connecting to CmdAgent@hadoop-6-11, count 6.

14:04:06:WARNING:Retry connecting to CmdAgent@hadoop-6-11, count 7.

14:04:16:WARNING:Retry connecting to CmdAgent@hadoop-6-11, count 8.

14:04:26:WARNING:Retry connecting to CmdAgent@hadoop-6-11, count 9.

14:04:36:WARNING:Retry connecting to CmdAgent@hadoop-6-11, count 10.

14:04:36:SEVERE:Could not connect to CmdAgent@hadoop-6-11

14:04:36:SEVERE:CmdService setup failed. Exiting

14:04:44:INFO:[/home/ubuntu/web-release/faban/bin/fenxi, process, /home/ubuntu/web-release/faban/output/OlioDriver.2R/, /home/ubuntu/web-release/faban/output/OlioDriver.2R//post/, OlioDriver.2R]

stdout:

Creating database ...                            2.34s

Parsing raw files ...                            0.01s

Processing profile default_profile               ...OK

Opening Database ...                             1.28s

Executing queries ...                            0.07s

Generating html pages ...                        0.21s

14:04:44:INFO:Killing benchmark

14:04:44:INFO:killing CmdAgent@hadoop-6-10

14:04:45:INFO:OlioDriver Completed/Terminated

ubuntu@hadoop-6-10:~/web-release/apache-olio-php-src-0.2/workload/php/trunk/deploy$ 

********************************************************************** 

Frontend Machine(for your information)

ubuntu@hadoop-6-11:~$ sudo netstat -pan | grep ":9000"

tcp        0      0 127.0.0.1:9000          0.0.0.0:*               LISTEN      12013/php-fpm.conf)

ubuntu@hadoop-6-11:~$ sudo netstat -pan | grep ":80"

tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      7090/nginx      

ubuntu@hadoop-6-11:~$

***********************************************************************

Backend Machine(for your information)

ubuntu@hadoop-6-13:~$ $CATALINA_HOME/bin/startup.sh

Using CATALINA_BASE:   /home/ubuntu/web-release/apache-tomcat-6.0.35

Using CATALINA_HOME:   /home/ubuntu/web-release/apache-tomcat-6.0.35

Using CATALINA_TMPDIR: /home/ubuntu/web-release/apache-tomcat-6.0.35/temp

Using JRE_HOME:        /home/ubuntu/jdk1.7.0_17

Using CLASSPATH:       /home/ubuntu/web-release/apache-tomcat-6.0.35/bin/bootstrap.jar

ubuntu@hadoop-6-13:~$ jps

19834 Bootstrap

26908 Jps

ubuntu@hadoop-6-13:~$ 

***************************************************************************

All the setup is done on Ubuntu VMs over 3 different physical machines. 

Any Idea how to solve the issue ? 

Regards

Zubair

[Shanti Subramanyam (gmail)]

Did you see the FAQ at http://faban.org/1.1/docs/FAQ.html

Zubair

--
You received this message because you are subscribed to the Google Groups "faban-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to faban-users...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[jsubirat]

Hi Shanti,

I looked at the FAQ that you mention but I couldn't come with any solution. I'm facing the exact same problem as Zubair. Could you help us further with this issue? I looked at the agent logs in the Backend machine and it prints:

Agent(Daemon) starting agent with options: [10.0.0.101, 10.0.0.101, 10.0.0.100, /usr/lib/jvm/java-6-openjdk-armhf, -Dfaban.home=/home/ubuntu/web-release/faban/, -Djava.security.policy=/home/ubuntu/web-release/faban/config/faban.policy, -Djava.util.logging.config.file=/home/ubuntu/web-release/faban/config/logging.properties, -Dfaban.registry.port=9998, -Dfaban.logging.port=9999, -Xmx3g, -Xms256m, -XX:+DisableExplicitGC, faban.benchmarkName=OlioDriver, faban.download=http://10.0.0.101:9980/, /, :]

java.rmi.ConnectException: Connection refused to host: 10.0.0.101; nested exception is: 

        java.net.ConnectException: Connection refused

        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)

        at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)

        at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)

        at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:340)

        at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)

        at com.sun.faban.common.RegistryLocator.getRegistry(RegistryLocator.java:137)

        at com.sun.faban.harness.agent.AgentBootstrap.startAgents(AgentBootstrap.java:300)

        at com.sun.faban.harness.agent.AgentBootstrap.startDaemon(AgentBootstrap.java:151)

        at com.sun.faban.harness.agent.AgentBootstrap.main(AgentBootstrap.java:94)

Caused by: java.net.ConnectException: Connection refused

        at java.net.PlainSocketImpl.socketConnect(Native Method)

        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)

        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)

        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)

        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:385)

        at java.net.Socket.connect(Socket.java:546)

        at java.net.Socket.connect(Socket.java:495)

        at java.net.Socket.<init>(Socket.java:392)

        at java.net.Socket.<init>(Socket.java:206)

        at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)

        at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:146)

        at com.sun.faban.harness.agent.AgentSocketFactory.createSocket(AgentSocketFactory.java:97)

        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)

Do you have any clue of what can the problem be? Many thanks!

[Cheng Cheng]

Hi Jsubirat,

I am also facing the same issue as yours when configuring on EC2. Have you solved the problem?

Thanks!

Cheng

[Shanti Subramanyam (gmail)]

The big problem on EC2 is that the /etc/hosts file needs to be edited. There is a line that has localhost with 127.0.1.1. This needs to be replaced with the actual ip of the local host.

You can get the localIP as follows:
localIP=`ifconfig |grep Bcast | cut -d':' -f2 | sed 's/Bcast.*//'`

Without this address, the agent/master communication doesn't work.

Shanti

For more options, visit https://groups.google.com/d/optout.

[Ubiratam de Paula]

Hi Shanti,

I am also facing the same issue as Jsubirat when configuring two local machines with Ubuntu 14.04.
Can you help me?

P.S.: I can make ssh without password between both machines.

Thanks!

Ubiratam

[Shanti Subramanyam (gmail)]

Please send your /etc/hosts file on the master and agent machines as well as the errors you are seeing.

Shanti

--

[shashank shekhar]

Hi,

I am also facing the same issue and tried all the suggested options in the FAQ, did you find a solution for this problem?

Regards,

Shashank

[Shanti Subramanyam (gmail)]

I am responding to this old thread so as to document the solution to this problem.

I recently ran into the exact same issue when trying to run on EC2 instances.

The fix is to remove the Faban bin/Linux/interface script. If this script exists, Faban tries to use it to find the 'interface' to use when connecting to the agent. If the script works correctly, it should return the ip address of the Master. But on EC2, it returns the ip address of the agent host.

So when the Agent gets started, it tries to connect to the Faban RMI registry on its own host which doesn't exist.

At least for now, simply removing the 'interface' script has solved the problem for me.

Shanti

To unsubscribe from this group and stop receiving emails from it, send an email to faban-users+unsubscribe@googlegroups.com.

[Mohan Murthy]

Hi All,

Sorry to hijack this thread. I am trying to run Olio benchmark in Faban (cloudsuite web server benchmark). I am able to schedule the runs, but the problem is the Run Daemon is not running (please refer screenshot), faban log looks fine. I manually tried to start Run Daemon by adding the code " RunQ.getHandle().startRunDaemon();" in schedule-run.jsp but no luck. How to resolve this issue? any idea?

I use three VMs (ubuntu) as Faban client, Backend and Frontend server.  

Thanks

Mohan