Finished Large OpenSOC Deployment - Path Forward

[david...@gmail.com]

We just finished a large (1PB+) enterprise OpenSOC deployment using HDP 2.2.  Really excited about where this could head, and we [B23] have some unique ideas related of automated ingest, updated visualizations, and machine learning.

One customer concern was a transparent roadmap for OpenSOC releases.  We have a pull request pending for some basic items.  Any updates?

[jaygo...@gmail.com]

Hi, would it be possible to share the deployment guide?

My email is jay.g...@gmail.com.

Thanks in advance.

Best regards,
Jay

[david...@gmail.com]

Hi Jay,

Since we forked the OpenSOC repo a few weeks ago (due to the fact that there were no pull requests being accepted for a while) we have decided to keep commits to that branch to a minimum.  This is so our fork does not diverge too far from the main branch.

We have documented a lot of the PCAP customizations we've made including better code to capture raw packets at scale, changes to the Storm PCAP topology, as well as better visualizations in Kibana, but we are going to wait to commit that until a more robust repository emerges that accepts pull requests.

We have a lot of experience installing and deploying OpenSOC and can basically do it in our sleep now, so if there's a specific question we can help with, please let me know.

[Melih Göksal]

Hi Dave,

I got go ahead until  writing  "fab vagrant quickstart" in command-line.  Actually you can see exaclty where i got stuck with this link.
Any help would be appreciated.

Best.
Melih

22 Ekim 2015 Perşembe 05:21:15 UTC+3 tarihinde Dave Hirko yazdı:

[Dave Hirko]

Let me take a look and see if I can replicate that problem problem using the vagrant repo.  I've heard that the vagrant repo is not that robust.  I also haven't used a windows machine in some time, so I won't be able to fully replicate the exact environment you are using.

When we test, we test the actual opensoc-streaming and opensoc-ui repo's on Amazon Web Services (AWS) EC2 service (running CentOS 6.x).  Again, not using the vagrant setup.

I'll let you know what I find and hopefully it will help...

[Melih Göksal]

Actually i haven't used windows either. I am using ubuntu 14.04. I had a look streaming repo when i heard about OpenSOC, but couldn't find any guide that explain how-to's. That's why i think vagrant repo might be better to start with OpenSOC. So vagrant repo is just like a stepping-stone for now.

On the other hand, i need to deploy streaming and ui repo's after when i am done with vagrant repo. Could you lead to show where i can begin deploying streaming repo ?

Thanks for your  reply.

24 Ekim 2015 Cumartesi 01:48:27 UTC+3 tarihinde Dave Hirko yazdı:

[Dave Hirko]

I replicated the problem with the vagrant repo that was in the link you sent in your prior post.  In the link you provided it describes the Win7 environment, so I assumed you were working with that too.  Either way, it looks like this issue has been replicated now in Windows, Ubuntu, and now Mac OS X.  Working on a resolution now.

[node1] Executing task 'format_namenode'

[node1] sudo: /opt/hadoop/bin/hdfs namenode -format vagrant -nonInteractive

[node1] out: /opt/hadoop/bin/hdfs: line 276: /usr/java/default/bin/java: No such file or directory

[node1] out: /opt/hadoop/bin/hdfs: line 276: exec: /usr/java/default/bin/java: cannot execute: No such file or directory

[node1] out:

Warning: sudo() received nonzero return code 126 while executing '/opt/hadoop/bin/hdfs namenode -format vagrant -nonInteractive'!

[node1] Executing task 'supervisorctl_start'

[node1] sudo: supervisorctl start namenode

[node1] out: Traceback (most recent call last):

[node1] out:   File "/usr/bin/supervisorctl", line 5, in <module>

[node1] out:     from pkg_resources import load_entry_point

[node1] out:   File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2655, in <module>

[node1] out:     working_set.require(__requires__)

[node1] out:   File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 648, in require

[node1] out:     needed = self.resolve(parse_requirements(requirements))

[node1] out:   File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 546, in resolve

[node1] out:     raise DistributionNotFound(req)

[node1] out: pkg_resources.DistributionNotFound: meld3>=0.6.5

[node1] out:

Fatal error: sudo() received nonzero return code 1 while executing!

[Dave Hirko]

There are a couple of issues I've experienced troubleshooting the vagrant setup.

In terms of the error you described in the link.  That particular error is due to a version compatibility issue related to supervisor (http://supervisord.org/) and meld3.  The workaround is documented here (https://github.com/puphpet/puphpet/issues/1492)

On all four vagrant nodes, I first removed the installed version of meld3:

sudo pip uninstall meld3

Then I installed a specific version of meld3 that is compatible with the supervisord module:

sudo pip install 'meld3 == 1.0.1'

This seemed to fix the issue you documented in the link - the same one I also experienced.

Once I fixed that issue, I continued to experience additional issues formatting the namenode with the vagrant repo.

Since the vagrant repo is no longer accepting pull requests to address on-going issues like this, again, we think a better bet is to try and replicate the opensoc-streaming code outside of vagrant unless you were willing to hack your way through some of these issues.  

If you are interested in some of the deployment methods we are working on, please feel free to reach out directly to us:

ope...@b23.io 

[Mark Bittmann]

Hi Jay,

I'll put together and post a deployment guide for opensoc-streaming. It will have to assume proficiency in setting up the underlying tools (Kafka, Storm, Elasticsearch), but I think I can improve upon the existing docs. Look for something early next week. 

Mark

On Wednesday, October 21, 2015 at 8:35:07 PM UTC-4, jaygo...@gmail.com wrote:

[alizade...@gmail.com]

Hi Dave,
I'm Kind of confused on OpenSOC.
this is where I am: I have 4 vagrant nodes up. the opensoc-vagrant branch I used is master.(and nothing else.)
Now I really don't know what is my next step, and I actually doubt about installing vagrant being the first thing to do.

So can you just give me a tip on the steps that should be taken? something like a roadmap.
no need to be so detail.

Thanks.

[alizade...@gmail.com]

Hi Mark,
You've said something about deployment guide, Is it done already?
any way that I can have it?

[Mark Bittmann]

Hi,

Most of the information in this forum and on the OpenSOC github is out of date. Due to a lack of activity/community in the OpenSOC codebase, we have migrated it to an incubating Apache project called Metron under new stewardship. The Metron is community is still growing but has a healthy amount of ongoing activity. Sorry for any confusion around the existing codebase. I would recommend starting with the Metron code. Please feel free to reach out to the dev list with any issues you have with deployment. 

website: https://metron.incubator.apache.org

wiki: https://cwiki.apache.org/confluence/display/METRON/Metron+Wiki

codebase: https://github.com/apache/incubator-metron

dev list: https://mail-archives.apache.org/mod_mbox/incubator-metron-dev/