Warning on OpenSOC

[John Omernik]

All - 

I stumbled across OpenSOC and thought "ooh an open project where I could work with peers,share ideas, and collaborate on a real issue facing many orgs".  I was honestly a bit hesitant when I saw it was sponsored by some big names as that can go one of two ways, really well, or really poorly.  I dove in and hoped for the best. 

Well I have my answer... 

I received an email today, from an "Account Manager" at Hortonworks looking to schedule some time to talk about OpenSoc and share with me more information on Hortonworks.  He even linked to my introduction post on this forum as to where he got my name.  This concerns me on a number of fronts.  First of all, OpenSOC as it is now, is a great concept that is VERY hard to get off the ground due to complexity.  This is not a bad thing, as data is complex, but there are ways to smooth this process.   There is a support forum (this forum) where there could be the foundation for the solving of this complexity problem, but as is, there is very little support!  Lots of questions, and few answers, or even activity at all.  If the "sponsors" of this project were really focused on the community, they would be facilitating active discussion, helping with problems, identifying which problems keep coming up and trying to help with a wiki or doc site that helped people get going. 

My experience has been: try to engage in community and receiving no responses for a month. Then, after a month get a call from a sales guy from one of the sponsors looking to have a conversation on OpenSoc as well as talk about their product.  (Note my original email even stated I am using MapR, so why push the other distribution?)  This seems wrong to me. This does not encourage an open community discussion.  I will not assume the absolute worst here: That the design of what is released and what is not released is actually setup to get these high value sales calls.  I am hoping that neither Cisco or Hortonworks would stoop that low, however, some in the field that I work in, when I shared this experience, were quick to take that approach. One company, when they approached Cisco with questions on OpenSOC were offered to have a rack shipped to their org to "try it".  They said no thanks and walked away.   I hope this isn't just high pressure sales, but if it is this is disturbing. 

I think the concepts that this project puts forth are right on the money, however, be warned in posting here, you may get sales calls you didn't expect. 

Also of note and as a sort of postscript:  My post is not meant to imply that the engineers who are working on this are bad or suspect. I am sure many of them are good people trying to do some great work.   My comments should not be seen as a reflection of these people, instead, I just wanted to outline the issues I have in the sale approach and how it can absolutely kill the sense of community and make some frankly walk away from what is a great concept. 

Disappointed in Hortonworks and Cisco, 

John

[friz...@gmail.com]

Oh wow, thanks John.  I was looking in to this and thought it had a lot of potential and really cool but if the community is dead and this is nothing more than a thinly veiled sales honeypot then I'm going to look elsewhere.  I'd rather have an adequate product with a vibrant community than a great product with no community and I don't have time to fend off sales vultures.  In their defense though, it can be very hard to be owned/funded by a large for-profit multi-national company like Cisco and still maintain a viable, healthy community.  Just look at Sourcefire/Snort.  Money can be a powerful motivator and often the good of the community is sacrificed by the greed of the few.  But I can't blame them too much. If you want something, make it yourself and/or get involved in a real, symbiotic community.

Cheers,

*Friz*

[Shannon Francis]

That's really unfortunate to hear, John and Friz, I was hoping to use this stack as well, it seems like a pretty forward-looking view of where enterprise security is heading.

Maybe we could cut our teeth and trade notes and get more content in the wiki/forum?

-SJF

[John Omernik]

That's what I had hoped to do. (See my post about getting access to the wiki) My issue at this point is trust in what Hortonworks and Cisco are doing.  Even if the community did rise up to work on stuff, will be always having to warn new people to watch out for sales calls? At this point, for me, and others I have spoken to trust is lost. I am not sure the appropriate approach going forward. 

John

[james....@gmail.com]

Hi John,

I am one of the maintainers of the project.  The project is currently alive and well.  We have over 10 developers working on it right now and they are contributing new features on by-weekly basis.  We hold 2-week sprints.  The use cases and features that we work on are primarily driven by our business cases, but anyone is welcome to contribute.  OpenSOC has been adopted by multiple organizations.  Some of them run it as is, some of them add their own value-add components and re-sell it, and some of them just use it for research purposes.  We generally hear positive feedback from the adopters so your post took us a bit by surprise.  Let me try to address some of your concerns.  

- This is our honest attempt to build a grassroots effort behind big data security analytics.  It took a lot of engineering hours to put this together and this is not intended as a sales honeypot.  We wanted to contribute back to the open source community because so many tools we use on OpenSOC are free and open source.  It seemed like the right thing to do.  Our upsell is a managed service called Managed Threat Defense (MTD) that uses OpenSOC as one of its components + advanced analytics that are built on top of the streaming framework we have here.  We don't sell OpenSOC as a standalone framework and have resisted multiple requests from our org to do so.  

- We are not a giant conglomerate.  We are a small team in Cisco dedicated to applying big data frameworks to security analytics.  Our team size is very modest and we have to work lean and get by on very little funding.  Hortonworks are not a part of our team.  We did a POC with them about a year ago, but have since moved on to a different code baseline.  I am not sure why Hortonworks is trolling these boards trying to sell OpenSOC because (a) it's free, (b) it has evolved considerably since they last worked with us, and (c) the intended community for this project are highly advanced Hadoop professionals who probably don't need consulting anyhow.  We'll put in a call to Hortonworks tomorrow and find out.

- As I mentioned in my previous bullet point, the intended audience for this project are advanced big data professionals who want to dabble in security analytics.  We would love to bring the bar down and come up with automated installs, VMs, demo videos, etc.  Unfortunately, every time we try to do that we get a critical feature request and we have to re-allocate our efforts elsewhere.  I do agree with you that the barrier to entry to OpenSOC is quite high.  I also do agree that we need to do more to lower it.  This criticism is fair.  We'll have a series of meetings this week to see if we can do something in the next few weeks to address some of your concerns.  

Thanks,

James

[pkbha...@gmail.com]

hi John,

I totally agree with you.The Cisco team have given you just a framework there is no real content in it.I have run all the examples of their Topologies and 

even used Elasticsearch and their UI along with LDAP. I have also configured HBase and have successfully run their total setup with no issues, but found nothing useful.They have mentioned DPI Topology but you cannot see that in their code.They have talked about predictive analysis and that is also not there. They have just given you "Parser" for different log types and connected ES and Hbase with Storm and nothing more than that. The real analysis is missing like the DPI part and the predictive analysis part. It seems as if it is honeytrap . I have run all their examples and have found nothing useful. It's just central logging thing with little bit of enrichment and no "Analysis" part which was most required.

regards

Prateek Bhati 

[John Omernik]

James - 

Thank you for your response.  I've also included Prateek's email below, as I'd like to to talk to that as well.  I do appreciate the work that has been done on OpenSoC. Let me say that what drew me to this was the presentation and the concept.  The idea that something like this has been looked at, and funded was intriguing to me, despite my already mentioned hesitancy.  

To your first point about intentions.  I think even in my initial email I made a point to separate the folks working on the project (such as yourself) from the actions of the overall company or the individuals who are making the sales pushes.  We know there are great people working large companies, and that's what I liked about this project.  I would advocate that in creating a grassroots community, an org like Cisco could do well to look at some of the other small communities in the data space. Spark, Storm, Mesos, Kafka etc all seem to have vibrant communities where the startups companies are plugged in as part of their jobs. Not just to develop, but to interact .Have team members take turns responding to questions, set guidelines for how people respond to try to stay vendor neutral . Try to identify contributors and work to facilitate their work even outside of your dev team.  I asked for Wiki access a while ago to no response. I am not sure how community driven a password protected wiki is (even to read!).  You WANT others who have implemented things starting to help those who are trying to implement, however, if NO ONE outside of your can implement without your orgs help, then it feels less about community and more about Cisco/Hortonworks.  

Another point here, the presentation on the website talks about all the tuning that went it into getting things setup and humming along, but I could not find examples or explanations on what was tweaked to get that performance. What changes were made? How can we configure our tools to do this type of stuff better. Once again, this seems odd to me, teasing these performance tweaks, but not actually publishing them.  

Second Point: I am sorry to disagree with you on this, but you are Cisco. Perceptions are reality to many people, and I can appreciate your point of view, but consider this. If you have to explain what you did in your post to every person who questions Cisco's involvement, then you in a sense proving the perception issue.  To break that perception, and help people think more openly about Cisco, there needs to be that community involvement, let people see for themselves that their perception is incorrect.  Unfortunately, for every loud-mouthed poster like myself, there is likely 10-20 others who just say "meh, Cisco sales" say nothing and walk away, thus you never get a chance to respond to their perceptions.  Given that you work at Cisco, that lies on your shoulders. In addition, on the Hortonworks stuff, they didn't try to "sell" me OpenSOC, instead it was a reaching out to talk about how they support OpenSOC and wanted to help sell that support and the Hortonworks platform. To me, it was still trolling the boards trying to sell me something, but I am not sure if it makes a difference in how you approach.  I will follow-up with you privately to provide more details

Third Point: This is where helping the community and being proactive would help you. Getting some folks involved in your direction, to help drive things. Breaking up parts of the OpenSoc so you can "install" these pieces and see it in a basic form and add more as needed, I'd love to talk to you about your data model and taking the Kafka related stuff in looking to integrate pieces from the Confluent platform for Schema Management. Provide you information how others (like my org) have done a Security Data platform, and how to make it great platform for Security, while still handling other information well. But as of now, people struggle just to get see something of the project working.  It has all the pieces, but is missing the community to make it great. 

I also wanted to take time to address Prateek's email.  While I appreciate the support, I do feel Prateek's Analysis comments could use some thoughts: Basically, I see OpenSOC as a frame work of tools that one can use to see better into your network traffic, alerts, events etc.  The analysis will always need to be done at a human level. OpenSOC gives your analysts at your fingertips data, without having to dig around all over the place, that is awesome. I love what I see in the potential of OpenSOC doing, but one thing I DON'T expect out of OpenSOC is to do the analysis part for me. 

I hope that helps to clarify my thoughts to both posts. Like I said, work needs to be done on the community side, perhaps that can repair things, I can be hopeful.

John 

Prateek's email 

I totally agree with you.The Cisco team have given you just a framework there is no real content in it.I have run all the examples of their Topologies and 

even used Elasticsearch and their UI along with LDAP. I have also configured HBase and have successfully run their total setup with no issues, but found nothing useful.They have mentioned DPI Topology but you cannot see that in their code.They have talked about predictive analysis and that is also not there. They have just given you "Parser" for different log types and connected ES and Hbase with Storm and nothing more than that. The real analysis is missing like the DPI part and the predictive analysis part. It seems as if it is honeytrap . I have run all their examples and have found nothing useful. It's just central logging thing with little bit of enrichment and no "Analysis" part which was most required.

--
You received this message because you are subscribed to a topic in the Google Groups "OpenSOC Support" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/opensoc-support/05e1kh8Af1Q/unsubscribe.
To unsubscribe from this group and all its topics, send an email to opensoc-suppo...@googlegroups.com.
To post to this group, send email to opensoc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/opensoc-support/bde50e2b-b5e8-41f2-99ce-fff76e8c26b4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[raman...@gmail.com]

Hi good to see that u can install and comment on features. Pl can u take pains to write a doc and post, so that we too can see what exactly it is
We have been struggling and just have no clue how to install
Poor documentation, never expected such thing from Cisco. I do understand a small team is working, but unless the doc is good, it's difficult to follow
Too many puts on design are available and no proper doc at all

Request those who succeeded to post a full doc on their installation

Thank u
Radha

[james....@gmail.com]

Hi,

I posted a video on creating topologies a few months ago.  Let me know if that's enough

https://medium.com/@JamesSirota/adding-a-topology-to-opensoc-648a1c50331b

https://www.youtube.com/watch?v=3BgyEvJxDjY

Thanks