NxFilter and Graylog, how to separate logging/reporting from NxFilter.

[Jinhee]

I have built a content pack for Graylog. When you import it into Graylog, it creates a dashboard for NxFilter log data.

Now we can separate logging/reporting from NxFilter and we can scale up further.

  1. Download 'nxfilter-content-pack.json' file attached.

  2. Import it on Graylog GUI.

    - System > Content Packs > Import content pack

  3. After import it, you can select 'NxFilter' content packs.

    - Select 'Dashboard for the last 2 hours' and apply it.

  4. It uses UDP/1514 port.

  5. On NxFilter GUI, Go to 'Config > Setup > Syslog', change 'Syslog Port' to 1514.

    - And change 'Syslog Host' as well.

  6. Restart NxFilter and you will see your dashboard getting populated.

    - Select 'NxFilter 2 hours' dashboard on Graylog GUI.

* This is only a simple example. If you want to go further read Graylog document.

* I tested it with Graylog v2.2.0-1 and it was their OVA file.

After you build your own logging/reporting on Graylog then you can bypass logging/reporting on NxFilter traffic DB. To bypass it,

add 'syslog_only = 1' into /nxfilter/conf/cfg.properties file. We might add this option on GUI into the next version though. 

[Jinhee]

This is the capture image of the dashboard on my Graylog installation.

[Rob Asher]

Great job Jinhee!  I'm running an older version of Graylog, v2.0.3, and I'll test importing your content pack on it and let you know.  I'm also inputting on 5514 so I'll have to make a few changes.

Rob

[sup...@kernel.inf.br]

Very good! We have one in use but will import your content pack.

[sup...@kernel.inf.br]

[sup...@kernel.inf.br]

I changed the port, like Rob Asher said and changed the filter to use Grok.

{

  "extractors": [

    {

      "title": "NxFilter ",

      "extractor_type": "grok",

      "converters": [],

      "order": 0,

      "cursor_strategy": "copy",

      "source_field": "message",

      "target_field": "",

      "extractor_config": {

        "grok_pattern": "%{HOSTNAME:srv} %{WORD:sys}\\|%{TIMESTAMP_ISO8601:Time}\\|%{WORD:Block}\\|%{HOSTNAME:Domain}\\|%{GREEDYDATA:User}\\|%{IP:ClientIP}\\|%{GREEDYDATA:Policy}\\|%{GREEDYDATA:Category}\\|%{GREEDYDATA:Reason}\\|%{INT:Type}\\|%{GREEDYDATA:Group}",

        "named_captures_only": true

      },

      "condition_type": "regex",

      "condition_value": "^*NxFilter\\|*"

    }

  ],

  "version": "2.2.0-SNAPSHOT"

}

[Jinhee]

So you know it better than me. I don't know what 'Grok' is even. I just read the document several days ago and made that content pack. Actually I found there's a thing like 'Content Pack' yesterday. Maybe you can write your own content pack then. I will keep mine as the minimal example. I didn't know it would be this much easy. Though there's some limit when I think about applying it on NxCloud but if you are good enough at Java, you can hack it on back ground level.

Anyway, now there's no limit for logging and reporting. If this one is that much good at performance we can focus on the other things. Today, I made NxFilter bypassing almost everything on its logging part when you set 'syslog_only' parameter. And you still can have all the logging and reporting on Graylog.

You can build and upload your content pack here. Or maybe you can distribute it from your site.

[Rob Asher]

I wasn't able to upload the content pack into v2.0.3 so I went ahead and created my own.  I also use a grok extractor very similar to one posted as I'm importing syslog from several different systems besides NxFilter. 

Condition

• Will only attempt to run if the message includes the string NxFilter

Configuration

• grok_pattern: %{IPORHOST:hostname} (NxFilter\||)%{TIMESTAMP_ISO8601:time}\|%{DATA:blocked}\|%{URIHOST:domain}\|%{DATA:IPGroup}\|%{IP:sourceIP}\|%{DATA:policy}\|%{DATA:category}\|%{DATA:reason}\|%{INT:type}\|%{GREEDYDATA:group}
• named_captures_only:

[Jinhee]

Hi Rob,

Yours seems good. Can you upload it here. I can test it on my system to see if it's working. If it's working I want to use it as the example one.

Jinhee

[Jinhee]

Here's several things we need to think about,

1. Using Grok instead of split.

  The reason I prefer using split is that it might be better for performance. This pattern capturing might take some CPU load. But Rob has more than several thousands users. And Bruno who's the moderator of our Portuguese forum also has more than 500 users so I guess it's OK. We go with Grok.

2. 'NXFILTER' prefix.

  There's one problem with this. It was supposed to be uppercase but I found a bug recently and it shows up as 'NxFilter' it will be fixed on v4.0.4 so don't make it anything official before you test it on v4.0.4.

3. Naming and category.

  When you build a content pack. You need to select a category to which your content pack belongs. At first I was thinking of 'DNS Filter' but I changed it to 'NxFilter'. It was temporary and I didn't fix it yet. I was going to link it from my tutorial as an example so I didn't care that much. But now Bruno wants to upload it into Graylog 

  market place and we need to think about which is better or if we can use 'NxFilter' there.

  And the other thing is that being a native English speaker Rob is right about naming. On my dashboard,

    Top domain

  

  But on Rob's,

    Top Domains

  Actually it was a typo on mine but 'Top Domains' or 'Top Categories' looks better. But what about 'Top Client IP's'? Shouldn't it be 'Top Client IPs'? I am going to change these things on NxFilter GUI on v4.0.4 as well.

4. List of log data or raw data.

  When you click something on the dashboard you get the list of raw data that are related to the chart. That's fine. Very nice for finding out correlations. That's why we use this kind of system. But shouldn't we see it as a formatted list? Like on NxFilter GUI? For me, it's just a list of raw data and when I select these fields on the left pane then I get a formatted list. Anybody knows how to make it appearing as a formatted list always?

5. Using GELF.

  They are talking about a new protocol for Graylog. It's GELF. At first, I was thinking of introducing it but I don't want to make things too much complicated on NxFilter GUI. So I am going to watch it for a while. If the current Syslog one is OK we will go with that.

6. Performance enhancnig by using Graylog.

  On v4.0.4, I made it possible to set 'Log Retention Days' to be 0. When you set it to 0, it will bypass writing log data into its own traffic DB and there will be almost no access to its traffic DB. Means now we can be free from the heavy load on traffic DB. We now can focus on filtering, resolving, caching. NxFilter will be even faster in future and I hope we can say that NxFilter can handle several 10s of thousands users easily. At the moment, we can say 'several thousands'. It's still big but not enough for some users.

7. Using Graylog on NxCloud.

  Maybe you are not interested on this subject as you are all NxFilter users. But only the difference between NxFilter log data and NxCloud log data is that NxCloud sends 'Operator Name' instead of 'Group Name'. So we can build Graylog dashboard for NxCloud admin easily.

  However, NxCloud is a multi-tenant software. And each operator needs to have his/her own GUI. So we should be able to link it from NxCloud GUI using this kind of URL,

    http://graylog.mynxcloud.com/operator-dashboard.jsp?opname=jinhee

  And on Graylog GUI, each operator should be restricted to view his/her own data only. If we can solve this problem maybe we can go for cloud filtering business by ourselves. I thought about that but the obstacle was logging/reporting. Now we may find some easy solution.

8. Last 24 hours, last 7 days.

  If you want to build or release your own content pack it is OK. You can upload it into Graylog market or you can release it from your site. But I guess you might need to add 

  dashboards for last 24 hours and last 7 days. Or maybe for yesterday and the last 7 days as we do it on NxFilter GUI.

[sup...@kernel.inf.br]

I'm using the filter too, because I receive logs from other systems on the same input.

[sup...@kernel.inf.br]

1. Ok.

2 and 3. Better uses the system name 'NxFilter' and not 'DNS Filter', the application name is much more assertive.

4. I agree with your idea about a formatted list.

5. Using this NxFilter will works only with Graylog? I don't think that is a good way, using Syslog the user can uses other tools like ELK.

6. Yes, the I/O will reduce.

7. Like an old iframe?

8. Good idea. Derivating on the Dashboard of 2 hours, extending time.

[Triet Nguyen]

NXFilter supports Syslog output.

[Rob Asher]

Hi Jinhee,

I'll upload my content pack but it has some things like the source hostname that would be specific to our environment that would have to be changed.  There are probably a few others as well that would need changed but maybe it would be helpful to get someone started. I also upgraded my Graylog server to v2.2.0 so I should be caught up with everyone else.  As to the questions you proposed

1. Grok was pretty simple to parse out the string from NxFilter but split is probably just as easy and would be more user friendly to edit or add to later if the syslog message is changed.
2. Maybe a case insensitive regex for nxfilter would be better to filter on.  Something like /^NXFILTER/ig 
3. We can work on naming however you like.  I think you're right about the "Top Client IPs".
4. I haven't looked for a way to get a formatted list but I like that idea.  
5. I agree that syslog is pretty much the standard.  No need to limit what systems can be used by switching to a different protocol.
6. Even the built in logging of NxFilter works fine.  The ability to get everything in one location is helpful for us though plus easier to customize how we view the information.
7. I haven't looked into the role based portion of graylog much but it may be possible to do something like what you describe with it.  Especially if operator name can be used in a grok pattern match to limit what shows on the users dashboard.
8. I think if someone is using graylog with NxFilter, they can probably create their own dashboard from the supplied extractors.  At least have a good starting point to display the information that they want instead of trying to create all the different dashboards for them.  Just my $0.02 on that.
   

Rob

[Jinhee]

I was talking about GELF as an add-on. We will keep Syslog as well. And we are not going to change its format. However, I have to tell you that there was a someone talking about data loss when he was using NxFilter's Syslog exportation to Graylog. He said that it stopped completely for several hours and resumed in my memory. I think there's something wrong with his system but since I couldn't verify it I was thinking of adding one more way of log exportation.

Actually naming is trivial matter. Especially if it's on an example one. I am not going to write anything other than an example one on Graylog myself. At least for now. I have other things to do.

That role based admin on Graylog might be a way to go. Maybe you could just link 'Report Server' and give users an admin account for their dashboard. Especially makes sense if you use a shared NxFilter instead of NxCloud for your cloud filtering service. There is actually people are using NxFilter that way. I guess for most people setting up policy is not that important. For them fixed policy is enough. But still they might want to view their own report.

They should be able to build their own dashboard if they want to use Graylog. That seems like making sense. But in reality, many people try to use NxFilter without previous system admin experience. It will happen to this one most likely. Though nobody has to help with these people. Any I guess there maybe someone like Bruno wanting to build a ready-made dashboard for NxFilter users and distributing it. In that case, you would need to think about 24 hours and 7 days option.

The reason I was talking about NxCloud is that it's one missing thing for us if we want to scale it up. If you want to scale it up using NxCloud, one thing you need is a separated logging/reporting and the other one is Anycast. I think Anycast is a pure network design problem but 'logging/reporting' is different. If anyone knows how to build per-operator GUI on Graylog and link it from NxCloud I can recommend him to these cloud business providers when they ask me about their business. These people want too much out of their small investment, they want to scale it up infinitely before they even start anything. Maybe someone can make some money by building a logging/reporting server for them.

[Jinhee]

Bruno, I already answered most of your questions in another posting. But several things more.

Use whatever name you like, I am not going to distribute anything other than a simple example. And don't need to be iframe. Whatever works would be OK.

[Jinhee]

I ended up with using Grok. I think it's interesting having field names with regex pattern.

  http://nxfilter.org/tutorial.html#using_graylog

[JWalker_PE]

I think that is good put the version of Graylog