Nxlog Modules Download
Vonnie Halcon
Files and sockets are added to the core by the various modules, and the coredelegates events when necessary. Modules also dispatch log events to the core,which passes each one to the appropriate module. In this way, the core cancentrally control all events and the order of their execution makingprioritized processing possible. Each event belonging to the same moduleinstance is executed in sequential order, not concurrently. This ensures thatmessage order is kept and allows modules to be written without concern forconcurrency. Yet because the modules and routes run concurrently, the globallog processing flow remains parallelized.
Output modules provide functionality for sending log data toa local or remote destination. An output module instance is a sink,destination, or consumer. It is responsible for consuming eventrecords produced by one or more input module instances.
The NXLog language can be extended with extensionmodules. Extension module instances do not process log datadirectly. Instead, they provide features (usually functions andprocedures) that can be used from other parts of the processingpipeline. Many extension module instances require no directivesother than the Module directive.
I get same errors on two 64 bit machines (Windows 10 and Windows Server 2012R2)
Tried with processmonitor.exe to see if nxlog-processor.exe fail to load the dll module but I do not see any load attempt.
I suspect that I must missing a macroscopic thing but am unable to spot it.
Spotted the problem: while copying the code snippet from the pdf manual to the .conf file, I copied an invisible character (0xA0) present in the beginning of the 'module' directive.That character was cocking silently the NXLog parser.Duplicating that row to use it with other modules spreaded the error.
I am deploying nxlog on a Linux server (Red Hat Enterprise release 6.6). I am currently trying to remove a file after nxlog has finished processing. From the nxlog documentation, I am using file_remove, which is not working. Here is my config that does not throw any syntax errors when starting nxlog. As seen, I used a time interval of every 1 minute as I do not know the best way to have the file deleted after processing. In the debug log, I do not see an attempt to try and match files in the directory for removing:
I'm trying to get the DNS logging going with the im_etw input module with no luck. I get this error on my log, ERROR Failed to load module from C:\Program Files (x86)\nxlog\modules\input\im_etw.dll, The specified module could not be found. ; The specified module could not be found. I have a enterprise version of nxlog running. Not sure how to install that module.
My thought was that even though the nxlog log didn't show an error, that this port was causing issues as is may be in use. Hence, I tried to open some more tcp ports (5000-5010) as shown in the picture. However, when I try any of these ports (5000-5010) in the nxlog config file, I get this error:
NXLog is not sending log files for specific windows files unless it is labeled TEST. If it is labeled anything else I receive this error searching for it in NagiosLogserver: ERROR module 'test2' is not declared at C:\Program Files (x86)\nxlog\conf\nxlog.conf:73
Once installed, open the NXLog configuration file located at: C:\Program Files (x86)\nxlog\conf\nxlog.conf and paste this sample text into the file, adjusting the values for your account as necessary:
NXLog does not drop any log messages unless instructed to. It can process input sources in a prioritized order, meaning that a higher priority source will be always processed before others. This can further help avoiding UDP message loss for example. In case of network congestion or other log transmission problems, NXLog can buffer messages on the disk or in memory. Using loadable modules it supports different input sources and log formats, not only limited to syslog but windows event log, audit logs, and custom binary application logs.
With NXLog it is possible to use custom loadable modules similarly to the Apache Web server. In addition to the online log processing mode, it can be used to process logs in batch mode in an offline fashion. NXLog's configuration language, with an Apache style configuration file syntax, enables it to rewrite logs, send alerts or execute any external script based on the specified criteria.
NXLog has a special field, $raw_event. This field is handled by the transport (UDP, TCP, File, etc) modules to read input into and write output from it. This field is also used later to parse the log message into further fields by various functions, procedures and modules.
By utilizing loadable modules, the plugin architecture of NXLog allows it to read data from any kind of input, parse and convert the format of the messages, and then send it to any kind of output. Different input, processor and output modules can be used at the same time to cover all the requirements of the logging environment. The following figure illustrates the flow of log messages using this architecture.
The core of NXLog is responsible for parsing the configuration file, monitoring files and sockets, and managing internal events. It has an event based architecture, all modules can dispatch events to the core. The NXLog core will take care of the event and will optionally pass it to a module for processing. NXLog is a multi-threaded application, the main thread is responsible for monitoring files and sockets. These are added to the core by the different input and output modules. There is a dedicated thread handling internal events. It sleeps until the next event is to be processed then wakes up and dispatches the event to a worker thread. NXLog implements a worker thread-pool model. Worker threads receive an event which must be processed immediately. This way the NXLog core can centrally control all events and the order of their execution making prioritized processing possible. Modules which handle sockets or files are written to use non-blocking I/O in order to ensure that the worker threads never block. The files and sockets monitored by the main thread also dispatch events which are then delegated to the workers. Each event belonging to the same module is executed in sequential order, not concurrently. This ensures that message order is kept and prevents concurrency issues in modules. Yet the modules (worker threads) run concurrently, thus the global log processing flow is greatly parallelized.
When an input module receives data, it creates an internal representation of the log message which is basically a structure containing the raw event data and any optional fields. This log message is then pushed to the queue of the next module in the route and an internal event is generated to signal the availability of the data. The next module after the input module in a route, can be either a processor module or an output module. Actually an input or output module can also process data through built-in code or using the NXLog language execution framework. The only difference is that processor modules are run in another worker thread, thus parallelizing log processing even more. Considering that processor modules can also be chained, this can efficiently distribute work among multiple CPUs or CPU cores in the system.
The basic steps shown in the graphic below highlight how to download and install NXLog, modify the default nxlog.conf file to ingest the raw logs and output in Rapid7 UEF, then simply send and verify successful ingestion into InsightIDR.
I installed NXLog CE to the default location on the same system that has the logs I need to collect, C:\Program Files (x86)\nxlog. This folder contains a subfolder called conf that has the NXLog configuration file, nxlog.conf. Although the NXLog install created a default nxlog.conf, I now need to edit it to manipulate my logs. I start by making a copy of the nxlog.conf as a backup.
When NXLog starts, it will create a diagnostic log for itself called nxlog.log. I chose to write the nxlog.log file to C:\Program Files (x86)\nxlog\data, its typical location (this is a setting in nxlog.conf).
NXLog uses nxlog.conf to understand how to read in logs, manipulate them, and forward them to a receiving device. We need to edit nxlog.conf to input and convert logs into UEF, and forward them to the InsightIDR collector.
Before we test, we need to add directives by defining all the NXLog extensions being used in nxlog.conf. So far, our nxlog.conf defines one extension for syslog, which allows up to output logs as syslog. We will need two extensions in the nxlog.json file: one for syslog, and one for JSON. As our final logs should be in JSON, we should add in a for outputting using syslog, xm_json. I am not yet using the xm_json extension, but I know I must as my final logs are required to be in JSON format.
We need to add in an extension for kvp and modify my section to run the kvp module. After some testing to determine which options work with my input, I discover that my logs have tabs separating the key value pairs. Now, nxlog.conf looks like this:
Do you see the kvp2 module being used in the else statement? I test this and verify that indeed, I have the input being written out in kvp format. Now I can start deleting fields for real! I add in the delete ($field) procedure and test it. Here is nxlog.conf now:
Now that the logs appear to be in the proper UEF format, I will add it into InsightIDR. Earlier, we chose forwarding the logs from NXLog via syslog over UDP port 10010, and that is defined in nxlog.conf. Now I just need to log in to InsightIDR and add this event source.
Unfortunately, this does not take Daylight Savings Time into consideration. Programmatically resolving this issue is beyond this introductory document. Therefore, I can either live with this or manually adjust nxlog.conf twice per year.
The other option would be, if possible, to use nxlog on that DHCP server where you would not only be sending the DHCP log data to IDR, but could also be sending it to whichever syslog server you need to. Here is the link for nxlog:
df19127ead