Web Serving Benchmark Failure

[kara...@namal.edu.pk]

Dear members,

       Hope you are fine. We are working on web serving benchmark. We have installed faban as workload driver against olio. The benchmark that we have deployed is not running successfully. We are facing some errors and tried to search out these errors but no useful help found. We have followed the tutorials given in links given below, but none of them is running successfully.

http://doctorsnail.net/98/ 

  

http://parsa.epfl.ch/cloudsuite/web.html 

The configuration of our web serving environment is:

Front End and Back End Machine (On 1 VM)

Client-Faban (On other VM)

The error which we are facing is attached.Your response is eagerly awaited.

Thank you and have a great day.

Regards,

Karam

[Vincenzo Ferme]

Dear Karam,

your issue might be related with the following discussion:

• https://groups.google.com/forum/#!searchin/faban-users/localhost|sort:relevance/faban-users/Y_GKZt3mHvw/icdBEJyAdxgJ
  

Regards

[Muhammad Karam Shehzad]

We have tried all these options before sending you an email. SSH is working in all VMs, firewall is also blocked, host file is also edited. But, we don't know why still we are facing connection refused problem. Your help in this regard would be very helpful.  Thanks,  --- Best Regards ---------------- Muhammad Karam Shehzad Department of Electrical and Electronic Engineering Student (Year-4) Namal College, Mianwali, Pakistan e. karam....@hotmail.com m. +923155758422

--
You received this message because you are subscribed to a topic in the Google Groups "faban-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/faban-users/t2puQ-Jx868/unsubscribe.
To unsubscribe from this group and all its topics, send an email to faban-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Vincenzo Ferme]

Can you try to enable FINEST Logging level to get all the details about what Faban is doing?

[Muhammad Karam Shehzad]

PFA.

--- Best Regards ---------------- Muhammad Karam Shehzad Department of Electrical and Electronic Engineering Student (Year-4) Namal College, Mianwali, Pakistan e. karam....@hotmail.com m. +923155758422

[Muhammad Karam Shehzad]

Dear Ferme,

On your request, we have enabled FINEST Logging level to get all the details about Faban and we came out with some details which can be seen in attached folder. If you could spare some time to look into these files as our benchmark is not running successfully ? We shall be very thankful.

Configuration:

Front-End Machine & Back-End Machine are installed on one VM (IP: 192.168.40.43)

Client Machine is installed on another VM (IP: 192.168.40.45) 

Your response is eagerly awaited. Thank you and have a great day. 

---

Best Regards

----------------

Muhammad Karam Shehzad

Department of Electrical and Electronic Engineering

Namal College, Mianwali, Pakistan

e. karam....@hotmail.com

m. +923155758422 

To unsubscribe from this group and all its topics, send an email to faban-users+unsubscribe@googlegroups.com.

[Vincenzo Ferme]

Dear Karam,

from what I can see from the following configuration in the run.xml file:

<fa:hostConfig>

            <fa:host>192.168.40.43</fa:host>

            <fh:enabled>true</fh:enabled>

            <fh:cpus>0</fh:cpus>

            <fh:tools>vmstat 10</fh:tools>

            <fh:userCommands/>

</fa:hostConfig>

you are setting as host for the driver (that simulates the client interaction), the IP of the Front-End Machine & Back-End Machine are installed on one VM and not the IP of the Client Machine is installed on another VM (IP: 192.168.40.45) 

Then according to the log, Faban is not able to find the CmdAgent on that server (Could not connect to CmdA...@192.168.40.43), because of course it is not there. Try to use 192.168.40.45 and let me know if the situation changes.

[Muhammad Karam Shehzad]

Dear Ferme,

Thank you once again. We will let you know after trying it.  

---

Best Regards

----------------

Muhammad Karam Shehzad

Department of Electrical and Electronic Engineering

Student (Year-4)

Namal College, Mianwali, Pakistan

e. karam....@hotmail.com

m. +923155758422 

To unsubscribe from this group and all its topics, send an email to faban-users+unsubscribe@googlegroups.com.

[Vincenzo Ferme]

Dear Saqib,

unfortunately I don’t see anything suspicious.

The only things I can think about are: 1) security issues from the JVM running Faban (try to run it with super user access, if safe on the machine you are running Faban), 2) version of the JVM, I would try also with the Oracle 1.7.

Let me know if you overcome the issue.

Regards,

Vincenzo Ferme

Skype: vincenzoferme
Web: http://www.vincenzoferme.it
Email: in...@vincenzoferme.it

On 18 Sep 2016, at 16:06, Saqib Ilyas <saqib...@namal.edu.pk> wrote:

Dear Vincenzo

We had a week long break here. During that time, while at home, I cloned the setup from http://parsa.epfl.ch/cloudsuite/web.html on a different machine, to the best of my abilities. These are three VMs running on the same physical PC.

VM1: Client machine: 192.168.122.91

VM2: Backend machine: 192.168.122.130

VM3: Frontend machine: 192.168.122.145

I can ssh into VM2 and VM3 from VM1 without being prompted for a password. When scheduling the run, I have set the driver machine as 192.168.122.91 as you had suggested. Now, I am getting the following errors:

23:48:39   FINER Executable map:
fenxi.cmd : /home/faban/Downloads/web-release/faban/bin/fenxi.cmd
iostat : /home/faban/Downloads/web-release/faban/bin/Linux/iostat
agent.cmd : /home/faban/Downloads/web-release/faban/bin/agent.cmd
psef : /home/faban/Downloads/web-release/faban/bin/Linux/psef
fenxi : /home/faban/Downloads/web-release/faban/bin/fenxi
interface : /home/faban/Downloads/web-release/faban/bin/Linux/interface
rsh : /usr/bin/ssh -n
agent : /home/faban/Downloads/web-release/faban/bin/agent
messages : /home/faban/Downloads/web-release/faban/bin/Linux/messages
fabancli.cmd : /home/faban/Downloads/web-release/faban/bin/fabancli.cmd
mcstat : /home/faban/Downloads/web-release/faban/bin/mcstat
fhb : /home/faban/Downloads/web-release/faban/bin/fhb
pushagents : /home/faban/Downloads/web-release/faban/bin/pushagents
fhb.cmd : /home/faban/Downloads/web-release/faban/bin/fhb.cmd
rootinstall.sh : /home/faban/Downloads/web-release/faban/bin/rootinstall.sh
makeagent : /home/faban/Downloads/web-release/faban/bin/makeagent
sysinfo : /home/faban/Downloads/web-release/faban/bin/Linux/sysinfo
fabancli : /home/faban/Downloads/web-release/faban/bin/fabancli 23:48:39   FINE Executing [/home/faban/Downloads/web-release/faban/bin/agent, 192.168.122.91, 192.168.122.91, 192.168.122.91, /home/faban/Downloads/jdk1.6.0, -Dfaban.home=/home/faban/Downloads/web-release/faban/, -Djava.security.policy=/home/faban/Downloads/web-release/faban/config/faban.policy, -Djava.util.logging.config.file=/home/faban/Downloads/web-release/faban/config/logging.properties, -Dfaban.registry.port=9998, -Dfaban.logging.port=9999, -Xmx3g, -Xms256m, -XX:+DisableExplicitGC, faban.benchmarkName=OlioDriver] 23:48:39   FINE Executing: [/home/faban/Downloads/web-release/faban/bin/agent, 192.168.122.91, 192.168.122.91, 192.168.122.91, /home/faban/Downloads/jdk1.6.0, -Dfaban.home=/home/faban/Downloads/web-release/faban/, -Djava.security.policy=/home/faban/Downloads/web-release/faban/config/faban.policy, -Djava.util.logging.config.file=/home/faban/Downloads/web-release/faban/config/logging.properties, -Dfaban.registry.port=9998, -Dfaban.logging.port=9999, -Xmx3g, -Xms256m, -XX:+DisableExplicitGC, faban.benchmarkName=OlioDriver] 23:48:39   SEVERE
exception Could not execute [/home/faban/Downloads/web-release/faban/bin/agent]on machine 192.168.122.91 23:48:39   SEVERE CmdService setup failed. Exiting 23:48:39   SEVERE
exception FenXi process command [fenxi, process, /home/faban/Downloads/web-release/faban/output/OlioDriver.1I/, /home/faban/Downloads/web-release/faban/output/OlioDriver.1I//post/, OlioDriver.1I] failed.

I can't see why the process is unable to ssh and run agent on 192.168.122.91. I tried copy pasting the command into a terminal window and it worked just fine, agent was spawned.

Can you find anything suspicious from the above?

Thanks and best regards

Muhammad Saqib Ilyas

[Muhammad Karam Shehzad]

Dear Vincenzo, We had installed all three machines (client, frontend, backend) on different PCs. There were some errors which we faced while we ran benchmark. Details of configuration and errors are below:    Faban master/client: 192.168.40.11 backend: 192.168.40.12 Olio frontend:192.168.40.13 We started the run setting "Driver" as 192.168.40.11. Faban master was showing that it is unable to connect toCmdA...@192.168.40.13. We saw on wireshark that a packet was sent from master to frontend with destination port 9981 with SYN flag, but was immediately replied to with RST + ACK flags.  We made sure that faban agent was running on the frontend. If it wasn't, we ran faban/bin/agent 9981. Then, netstat -an | grep 9981 showed that 0.0.0.0:9981 was listening on the frontend. But, we still got the same situation from the master. From the master machine, when we manually connected: telnet 192.168.40.13 9981, we got connected. When we press ctrl+c to exit Telnet, the agent stopped on the frontend machine, too. That is, the port 9981 was no longer listening. Please find the attachments for more details.  The question is, why is it that faban master isn't able to connect to the agent daemon running on frontend, whereas we are able to do so manually? Your response is eagerly awaited. Thank you!  Regards, Muhammad Karam Shehzad On Tue, Sep 20, 2016 5:51 PM, Vincenzo Ferme fermev...@gmail.com wrote: Perfect, thank you.

Vincenzo Ferme Skype: vincenzoferme Web: http://www.vincenzoferme.it Email: in...@vincenzoferme.it

On 19 Sep 2016, at 20:21, Saqib Ilyas <saqib...@namal.edu.pk> wrote:

Dear Vincenzo Thank you very much. I will do one last test before we post the conclusion our experience to the Google group. I will disconnect the Ethernet and put the machine back on Wifi. If the problem returns, then we can be confident that the problem is due to wifi. Otherwise, it could have been something else. If bringing the machine back to Ethernet makes the problem go away, then we can be even more confident of the conclusion.

Thanks and best regards Muhammad Saqib Ilyas

On 19 September 2016 at 19:46, Vincenzo Ferme <fermev...@gmail.com> wrote: Dear Saqib, I reply inline. Can you please update the Google Group discussion with all the information we shared privately, so other users can access the knowledge.  Thank you.

Vincenzo Ferme Skype: vincenzoferme Web: http://www.vincenzoferme.it Email: in...@vincenzoferme.it

On 19 Sep 2016, at 03:47, Saqib Ilyas <saqib...@namal.edu.pk> wrote:

Dear Vincenzo This morning, I brought the system to my office, turned off wifi, hooked the system up to a small Ethernet switch. I booted up the VMs. I stress that the Ethernet switch was only for the physical machine. The VMs are obviously connected to the virtual switch. I started the faban master on the client machine, the database server on the backend machine and Nginx/PHP-FPM on the frontend machine. I started a run with 5 concurrent users and it ran to completion and has given some results. It has been reported that virtual bridging on Wifi adapters has some issues (https://wiki.xen.org/wiki/Xen_in_WiFi_networks). Perhaps it is related. Happy you solved the problem, I did not know about this particular issue. Sorry for that. I am still getting some exceptions in the results (Service thread read 0 bytes). I guess this is due to excessive load on the frontend machine during the test (3 VMs are running on a Core2Duo machine and share the same physical hard disk even if it is an SSD). Am I right? If they are exception reported by logging level lower than INFO, then it is just Faban login, it does not impact the correctness of the execution. At least this is my understanding after using Faban for some years now. I also got some "Encoutering EOF after reading 0 bytes" errors that follow read -1 bytes from channel java.nio lines. The same as before.  I also got exceptions reading /var/mysql/data/192.168.122.130.err messages.  A total of 7 tests seemed to have failed, others passed.  This is something I never saw before. Do you think we have the rig set up right by now and just need better resources to get testing done right? Yes, the setup seems correct and usually if you can go end to end with the benchmark, then with Faban you can be sure what you are doing is the right thing.

[Vincenzo Ferme]

Hi, do you have the log with FINER setting?

Moreover, does the frontend driver show some logs?

Regards

[Muhammad Karam Shehzad]

Dear Vincenzo,

Attached is the 'faban output file' with FINEST logging level enabled and Log file of front-end machine.

Thanks!

---

Best Regards

----------------

Muhammad Karam Shehzad

Department of Electrical and Electronic Engineering

Student (Year-4)

Namal College, Mianwali, Pakistan

e. karam....@hotmail.com

m. +923155758422 

--

[Muhammad Karam Shehzad]

Faban-Log files are also attached.

---

Best Regards

----------------

Muhammad Karam Shehzad

Department of Electrical and Electronic Engineering

Student (Year-4)

Namal College, Mianwali, Pakistan

e. karam....@hotmail.com

m. +923155758422