Installation and usage

[Fernando Tadao ito]

Hello!

My name is Fernando, and I'm a student in UFSCar (Universidade Federal de São Carlos). I'm trying to install and run Hogzilla, but the tutorial on the site is using older versions of Hadoop and HBase that are not available anymore. Can this be used in recent versions?

And what tools are absolutely necessary for running the program? I'm not interested in installing the Snorby GUI, for example. Does the program run in the same machine that runs Snort? Can it use the Snort logs directly, instead of running through a MYSQL database?

Sorry for the long list of questions, and thanks for the attention!

Fernando Tadao Ito

[Paulo Angelo]

Hi Fernando,

    You can use updated versions of Hadoop and HBase. The procedures should be the same.

     MySQL is needed just for Snorby. If you don't need a GUI, you will not need MySQL neither PigTail. The other parts are necessary.

     You can run everything in one box. However, if you have a relatively large volume of data, you probably will need at least 2 boxes. You can begin with one and see what happens.

     Hogzilla access the data flows in HBase, which is scalable. Currently, it doesn't support direct access to Snort logs.

 Welcome! Feel free to ask and use.

Regards,

PA

[Fernando Tadao ito]

Heya!

In the install tutorial, you refer to http://ids-hogzilla.org/downloads/Hogzilla-v0.5.1-alpha.jar, but there is nothing in this link... Is there another link to the .jar file?

[Fernando Tadao ito]

If there is no ready-made .jars around, no problem. How can I compile the Scala source code? Tried sbt > run, but it didn't work...

[Paulo Angelo]

Hi Fernando,

    We are developing the support for SFlows and the "jar" was unavailable for a while. However, I uploaded a version, in the same URL, which you can use.

    I recommend you to install the Scala-IDE (http://scala-ide.org/) to change and compile the code. You probably will want to implement some customization. By other side, scala is a very easy and interesting language. :)

    In a few days we are launching the SFlow support with some nice new approaches. These new methods are performing well in our labs.

    Let us know if you could install and run in your environment.

Best regards,

Paulo Angelo

[Fernando Tadao ito]

Almost there!

The whole setup is complete, and the only exception I get is when I create a Task in Spark to execute Hogzilla itself.

16/06/28 17:41:12 INFO yarn.Client: Application report for application_1467145730267_0004 (state: FINISHED)
16/06/28 17:41:12 INFO yarn.Client:
     client token: N/A
     diagnostics: User class threw exception: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:
Tue Jun 28 17:41:11 BRT 2016, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68197: row 'hogzilla_flows,,00000000000000' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=idstest.dc.ufscar.br,16201,1467145816984, seqNum=0

     ApplicationMaster host: 192.168.88.216
     ApplicationMaster RPC port: 0
     queue: default
     start time: 1467146358528
     final status: FAILED
     tracking URL: http://IDSTest.dc.ufscar.br:8088/proxy/application_1467145730267_0004/
     user: hogzilla
Exception in thread "main" org.apache.spark.SparkException: Application application_1467145730267_0004 finished with failed status
    at org.apache.spark.deploy.yarn.Client.run(Client.scala:1029)
    at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1076)
    at org.apache.spark.deploy.yarn.Client.main(Client.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
16/06/28 17:41:12 INFO util.ShutdownHookManager: Shutdown hook called
 

I think this is a saving issue, Hogzilla cannot communicate with the database and hogzilla_events in the HBase. Do you know where should I look first to solve this error?

Thanks, and sorry for all these questions!

[Paulo Angelo]

Hi Fernando,

   Is HBase accessible by hbase shell? Can you list the tables below in the hbase prompt?

hogzilla_flows
hogzilla_sflows
hogzilla_events
hogzilla_sensor
hogzilla_signatures
hogzilla_mynets
hogzilla_reputation
hogzilla_histograms

    If you cannot connect via "hbase shell", check that HBase is running. If you don't have all tables created, try to create them using the following commands.

create 'hogzilla_flows','flow','event'
create 'hogzilla_sflows','flow'
create 'hogzilla_events','event'
create 'hogzilla_sensor','sensor'
create 'hogzilla_signatures','signature'
create 'hogzilla_mynets','net'
create 'hogzilla_reputation','rep'
create 'hogzilla_histograms','info','values'

    Let us know if could or could not solve the question.

Best regards,

Paulo Angelo

Em quarta-feira, 22 de junho de 2016 14:24:55 UTC-3, Fernando Tadao ito escreveu:

[Fernando Tadao ito]

Yes, it is accessible. And the tables were created successfully. This was an error of timeout values, and it has been corrected. Now, the problem lies in memory: the hogzilla.sh script described in the tutorial hangs the computer and reboots it.

[Nada Essaouini]

Hi Fernando,

I have the same error. Could you tel me please how you change the value of the timeout?

Thank you in advance

Nada