New to Zorka & Zico - Setup & configuration issues

[Ricardo Lopes]

Hello everyone,

 

I’ve been using Zabbix since 1.8 version to monitor our systems.

Currently I will need to monitor LDAP, Websphere 8, Jboss, REST. In addition to that I’ve started searching for Zabbix Templates and that’s how I’ve discovered Zorka and Zico.

There is a big enthusiast about this because we believe it will help to improve a lot the performance issues we’ve been having.

 

Lets put the hands on in the issues I would like to raise.

So, we’ve been following the installation guide but for some reason we don’t see any plot.

 

 

######

    1

######

-        Agent zip file was transferred and unzip to the WAS home: /opt/WebSphere/AppServer/profiles/l075v20/zorka

 

-        We did a vim to the file: /opt/WebSphere/AppServer/profiles/l075v20/config/cells/SOL/nodes/l075v20/servers/server1/server.xml

o   The we did the change mentioned:

o   <jvmEntries xmi:id="JavaVirtualMachine_1183122130078" verboseModeClass="false" verboseModeGarbageCollection="false" verboseModeJNI="false" maximumHeapSize="6144" runHProf="false" hprofArguments="" debugMode="false" debugArgs="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=7777" genericJvmArguments="-javaagent:/opt/WebSphere/AppServer/profiles/l075v20/zorka/zorka.jar=/opt/WebSphere/AppServer/profiles/l075v20/zorka -Xjit:disableInterpreterProfiling -Dfile.encoding=UTF-8 -Dcom.ibm.ws.pm.pmcache.ignoreDeletedEntry=true -Xoptionsfile=/opt/WebSphere/AppServer/profiles/l075v20/properties/jksProps.txt -Djavax.xml.transform.TransformerFactory=org.apache.xalan.processor.TransformerFactoryImpl -Dsun.reflect.inflationThreshold=0 -Xgc:preferredHeapBase=0x100000000">

 

-        Once again we did the change mentioned for the server policy. We added the line:

o   // Allow ZORKA agent
grant codeBase "file: =/opt/WebSphere/AppServer/profiles/l075v20/zorka/-" {
permission java.security.AllPermission; };

 

 

We did stopServer.sh server1

And we did startServer.sh server1

 

We activated the traces and we have a zabbix agent installed in this machine. The scripts are inside the jar, so I believe we don’t need to unzip the jar and have the scripts folder correct? The server starts without errors …..

However we don’t see any logs in the zorka folder, or any info in zabbix. Server starts with success.

 

 

Any ideas what we are doing wrong?

We also have some exceptions with Zico.

We have JDK1.7 in the system.

 

Any kind soul would be able to give me a 10min help over here, skype or gmail? J Appreciated!

[Ricardo Lopes]

And BTW, about Zico,

Any idea why I have this error?
I have this in multiple envs :( I cant seem to put it working :(

[Ricardo Lopes]

2014-07-03 16:25:48.599:WARN:oejw.WebAppContext:main: Failed startup of context o.e.j.w.WebAppContext@18e8d651{/,file:/opt/zico/tmp/webapp/,null}{file:/opt/zico/zico.war}
java.net.ConnectException: Connection timed out
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at java.net.Socket.connect(Socket.java:528)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
        at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
        at sun.net.www.http.HttpClient.New(HttpClient.java:308)
        at sun.net.www.http.HttpClient.New(HttpClient.java:326)
        at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932)
        at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1300)
        at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(XMLEntityManager.java:637)
        at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startEntity(XMLEntityManager.java:1290)
        at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startDTDEntity(XMLEntityManager.java:1257)
        at com.sun.org.apache.xerces.internal.impl.XMLDTDScannerImpl.setInputSource(XMLDTDScannerImpl.java:263)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.dispatch(XMLDocumentScannerImpl.java:1164)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.next(XMLDocumentScannerImpl.java:1050)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:964)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
        at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
        at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:649)
        at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:333)
        at org.eclipse.jetty.xml.XmlParser.parse(XmlParser.java:246)
        at org.eclipse.jetty.webapp.Descriptor.parse(Descriptor.java:67)
        at org.eclipse.jetty.webapp.WebDescriptor.parse(WebDescriptor.java:204)
        at org.eclipse.jetty.webapp.MetaData.setWebXml(MetaData.java:192)
        at org.eclipse.jetty.webapp.WebXmlConfiguration.preConfigure(WebXmlConfiguration.java:60)
        at org.eclipse.jetty.webapp.WebAppContext.preConfigure(WebAppContext.java:460)
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:496)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:125)
        at org.eclipse.jetty.server.Server.start(Server.java:358)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:107)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:60)
        at org.eclipse.jetty.server.Server.doStart(Server.java:325)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at com.jitlogic.zico.main.ZicoMain.run(ZicoMain.java:56)
        at com.jitlogic.zico.main.ZicoMain.main(ZicoMain.java:46)
2014-07-03 16:25:48.706:INFO:oejs.ServerConnector:main: Started ServerConnector@7a45682c{HTTP/1.1}{0.0.0.0:8642}
2014-07-03 16:25:48.708:INFO:oejs.Server:main: Started @66456ms

[Rafal Lewczuk]

Hi,

If there are no logs appearing in $ZORKA_HOME/log directory, then either path to $ZORKA_HOME is incorrect (that thing added after '=' in -javaagent option) or agent has no rights to write into log/ directory. If websphere server works as some other user than root (it actually should work with privileges of ordinary user), check if zorka log directory is writable to that user. Zorka log file should always appear.

Regards,
rle

[Rafal Lewczuk]

This zico error is strange, I haven't seen it (yet).  This is propably xerces trying to fetch DTD for web.xml file. On Internet connected servers this is just works. For non-connected environments it (potentially) can cause problems. Try editing web.xml inside zico.war and remove <!DOCTYPE...> tag (first three lines).

Let me know if it works.

[Ricardo Lopes]

Hi Rafal, it must be that. I will try.
These are Cloud Performance labs. They are in a LAB without any internet access (I can get internet access only if I make a tunnel :-) hehehe )
So... do you mean that if I try it at home in a VM it will work if that vm has internet access?
Meanwhile I will do that change into zico.war

 

I will provide you feedback. Thank you so much!

[Ricardo Lopes]

I think it is something else I am missing... maybe something in the java env. We have jdk 1.7, but I've downloaded version 1.8 as required for 1.04 version .
I will check that for my zorka issue.

 

 

[Ricardo Lopes]

Rafal,

 

About Zico issue!

You're the man! I've used WinRAR to take the web.xml file. Removed the lines you've mentioned and put the xml back in the .war file.
I don't have that blow anymore :-)

[root@l075v25 log]# tail -100f zico.log
2014-07-03 21:57:23,657 INFO [main] o.j.r.p.g.GuiceResteasyBootstrapServletContextListener [Slf4jLogger.java:92] found module: com.jitlogic.zico.core.inject.ProdZicoModule
2014-07-03 21:57:24,515 INFO [main] c.j.z.c.UserManager [UserManager.java:123] SU mode enabled or user database is empty. Adding default 'admin' user with 'zico' password.
2014-07-03 21:57:24,671 INFO [main] c.j.z.c.TraceTemplateManager [TraceTemplateManager.java:84] Templates DB is empty but JSON dump was found. Importing...
2014-07-03 21:57:24,770 INFO [main] c.j.z.c.TraceTemplateManager [TraceTemplateManager.java:95] Template DB import finished succesfully.
2014-07-03 21:57:24,987 INFO [main] c.j.z.c.ZicoService [ZicoService.java:109] Starting ZICO service at /10.79.41.145:8640
2014-07-03 21:57:24,988 INFO [main] c.j.z.c.ZicoService [ZicoService.java:116] ZICO service started.
2014-07-03 21:57:24,992 INFO [main] o.j.r.p.g.ModuleProcessor [Slf4jLogger.java:92] registering factory for com.jitlogic.zico.core.services.HostService
2014-07-03 21:57:25,044 INFO [main] o.j.r.p.g.ModuleProcessor [Slf4jLogger.java:92] registering factory for com.jitlogic.zico.core.services.SystemService
2014-07-03 21:57:25,047 INFO [main] o.j.r.p.g.ModuleProcessor [Slf4jLogger.java:92] registering factory for com.jitlogic.zico.core.services.TraceTemplateService
2014-07-03 21:57:25,051 INFO [main] o.j.r.p.g.ModuleProcessor [Slf4jLogger.java:92] registering factory for com.jitlogic.zico.core.services.TraceDataService
2014-07-03 21:57:25,055 INFO [main] o.j.r.p.g.ModuleProcessor [Slf4jLogger.java:92] registering factory for com.jitlogic.zico.core.services.UserService

 

And finally the IP is there.

Maybe you should take the opportunity either to remote this for 1.0.5 or to document this. I don't know.. but.. I would guess this is used by a lot of people in lab envs :-) I cant wait to test this

Meanwhile I will check zorca and come back to you.
BTW, about zorca, I already have my zabbux 2.2.4 agent running. What should I do ? Because it looks like zorka / zico already have on zabbix agent integrated correct?

[Rafal Lewczuk]

Ricardo,

This web.xml bug is fixed now and will appear in 1.0.5. It slipped into latest zorka release where I've upgraded Jetty to newer version (and it started doing stupid things, like fetching DTD in order to verify web.xml).

Regarding zabbix, there is a bunch of ready to use templates distributed along with agent. If you have zabbix integration enabled, you can start by importing interesting templates and experimenting with them. (zabbix integ is enabled by default, yet you may need to enter proper zabbix server address into zorka.properties).

Regards,
rle

[Shawn]

Hi there,

sorry for hijacking this post.

Up until recently, I have been monitoring using jconsole and log file from cronjob to monitor my web app server, and wanting to have some sort of 'centralised solution' to monitor the health of the server.

I have installed zabbix 2.2 and added those parameters in your installation sections into zarko.properties, but all i got back was out-of-box data (OS level data) from zabbix's agent, nothing from zarko's agent.

zarbbix server 10.8.8.242, web server 10.8.8.219

This is what I had put into zarko.properties

zabbix.enabled = yes

zabbix.listen.addr = 127.0.0.1

zabbix.listen.port = 10055

zabbix.server.addr = 127.0.0.1,10.8.8.242

I had even added additional agent with 10.8.8.219:10055 into to host configuration, but still nothing.

is there something I'm doing wrong here? or is it that zarko cannot run on the same server as zabbix's agent?

sorry for the newbie questions.

Shawn

[Ricardo Lopes]

Hi Rafal,

 

Awasome work!

Everything is working! :) Thanks! BTW, what do you mean by JDK5 bytecode format ?

I have 2x versions of WAS. 8 and 6.1, but you've mentioned that it requires the above conversion.

 

 

 

Shawn, I have it working, but instead of creating and additional host... I've stopped zabbix agent and changed zorka port from 10055 to 10050

[Shawn]

Thanks Ricardo, I will give that a try.

There are another 2 java processes running on the server, which I only need to ensure the processes are running. There is also a database on there, which I need to ensure it is running, and there is enough disk space.

I have got the above checks working into zabbix agent's user parameters.

That's why im trying to get zorka agent working along side with zabbix agent; or is it not possible?

Sorry, for all these questions, im still new to zabbix and learning how to use it.

Shawn

[Rafal Lewczuk]

Shawn,

If you see OS level data, zabbix is querying native Zabbix agent, not Zorka. Zorka agent does not provide anything OS-related, only JVM-related things.

Try checking if zorka port (10055) is properly configured in zabbix console. Do not use JMX port, zorka uses native zabbix protocol, so change standard port (10050) to what you have configured in zorka agent. Also, you might need to configure separate host object in zabbix for every JVM instance on your server (distinct from host object monitoring OS).

Regards,
rle

[Jey]

Hi Rafel,

I have installed zorka in our environment recently, i followed the above mentioned reply as well. I am facing issue when i try to add "zorka JVM template" with below error message. I have sucessfully started zorka agent in my websphere app server and port 10055 is in listning state.

log from zorka:-
" 2014-07-15 18:18:42 INFO ZabbixAgent ZORKA-zabbix core is listening at /0.0.0.0:10055.
2014-07-15 18:18:42 INFO AgentInstance Enabling Syslog subsystem ....
2014-07-15 18:18:42 INFO AgentInstance Enabling SNMP subsystem ...
2014-07-15 18:18:42 INFO SpyClassTransformer Adding JAVAX_JMS_LISTENER spy definition."

Error message in zabbix GUI (when adding as new host using the zorka agent port 10055) :-
Cannot find host interface on "SG D Homepage" for item key "zorka.jmx["java", "java.lang:type=OperatingSystem", "OpenFileDescriptorCount"]".

I have added only was/ibm.bsh in startup of zorka.properties file , Do i need to add other bsh files such as jvm.bsh so on... ? please advise where i am making mistake, i am also new to zabbix :-) ...

I would like to appreciate your work , its amazing ...
Thanks
Jey

[Rafal Lewczuk]

Hi Jey,

Is your application server working on Windows ?

If so, then this particular item won't work because it is available only on Unix/Linux systems. You can find it in zabbix template and safely remove it.

Regards,
rle

[Manish Kumar]

Hi Rafal,

Sorry jumping on this thread!!

I'm new to zorka, can you please let me know how i can test my java application deployed on windows 8 machine in tomcat application server.

I'm trying to read the documents as well but no luck to start this from my side.

Can you please guid me how i can start monitering any application deployed in apache tomcat in windows 7/8 machine?

Thanks a lot in advance!!

[MKumar]

Hi Ricardo,

 

Can you please help me to setup zorka agent for my java application deployed in tomcat7 app server  in linux machine?

 

As I'm new to zorka,I was following zorka web page but unfortunately there is no luck.

I was following below steps after that also i could not able to see any log file under zorka\log directory.

Steps which i'm following:

1. install the tomcat in Linux machine

2. copy the zorka agent under tomcat home directory

3. edit the zorka.property file with below details:

# Example: Apache Tomcat configuration with CAS server

scripts = jvm.bsh, zabbix.bsh, /home/ec2-user/bin/tomcat8/zorka/scripts/apache/tomcat.bsh, apps/cas.bsh

zorka.spy.compute.frames = yes

tracer = yes

4.copy the tomcat.bsh (bsh file copied from zorka github link) file into zorka/script/apache folder

5. added CATALINA_OPTS="$CATALINA_OPTS -javaagent:/home/ec2-user/bin/tomcat8/zorka/zorka.jar=/home/ec2-user/bin/tomcat8/zorka" into tomcat\bin\catalina.sh file(in the end of file).

on zorka site, its showing to add this line into <server-home>/bin/Catalina.conf but unfortunately catalina.conf is not available anywhere in tomcat directory.

7. deployed existing web project into tomcat webapps folder

6. started the tomcat with startup.sh command

7. tomcat started

 

not able to see any activity logged in zorka\log directory.

 

Please help me to implement zorka in our applications.

Thanks a ton tin advance.