Wazuh App for Splunk Api error

587 views
Skip to first unread message

Federico Serra

unread,
Sep 25, 2018, 8:30:32 AM9/25/18
to wa...@googlegroups.com
Hi all, 

 Today I have updated my wazuh managers and wazuh api to release 3.6.1. I am using Splunk as a backend to store logs, and I have the following error when I try to configure Wazuh API to connect from Splunk: 

Cannot add API without connection 

What does it mean this? Splunk server can connect to Wazuh's API port 55000 without problems and user and pass are correct... Any idea? 
Splunk is installa on Mac OSx high sierra

cat /opt/splunk/etc/apps/SplunkAppForWazuh/default/package.conf

sh-3.2# cat /Applications/Splunk/etc/apps/SplunkAppForWazuh/default/package.conf
[app]
version = 3.6.1
revision = 15

[wazuh]
version = 3.6.1

[splunk]
version = 7.1.3

{
   "error": 0,
   "data": {
      "msg": "Welcome to Wazuh HIDS API",
      "api_version": "v3.6.2",
      "hostname": “*****",
      "timestamp": "Tue Sep 25 2018 14:24:57 GMT+0200 (CEST)"
   }
}


manuel....@wazuh.com

unread,
Sep 25, 2018, 12:36:19 PM9/25/18
to Wazuh mailing list

Hi Federico,

I think that the app upgrade process went wrong, It seems that you’re still having content of previous versions in your app files. I recommend to you the following commands as root in order to upgrade the app properly:

  1. Remove the previous Splunk app for Wazuh version:

    $SPLUNK_HOME/bin/splunk remove app SplunkAppForWazuh
    
  2. Download the latest version of the app from here.

  3. Install the new package:

    $SPLUNK_HOME/bin/splunk install app v3.6.1_7.1.3.tar.gz
    
  4. Restart the Splunk service:
    $SPLUNK_HOME/bin/splunk restart
    

Also, I noticed that you’re using the v3.6.2 Wazuh API. This version wasn’t released yet, and it’s not compatible with the latest version of the Splunk app. I’d recommend to you to install the latest Wazuh v3.6.1 from packages, you can do it by following the official documentation, don’t hesitate to write here for any doubt you may have.
I’ll be waiting for your feedback.

Best regards

Manuel Jiménez

unread,
Sep 28, 2018, 1:05:00 AM9/28/18
to chic...@gmail.com, Wazuh mailing list

Hi Federico,

I think that the app upgrade process went wrong, It seems that you’re still having content of previous versions in your app files. I recommend to you the following commands as root in order to upgrade the app properly:

  1. Remove the previous Splunk app for Wazuh version:

    $SPLUNK_HOME/bin/splunk remove app SplunkAppForWazuh
    
  2. Download the latest version of the app from here.

  3. Install the new package:

    $SPLUNK_HOME/bin/splunk install app v3.6.1_7.1.3.tar.gz
    
  4. Restart the Splunk service:
    $SPLUNK_HOME/bin/splunk restart
    

Also, I noticed that you’re using the v3.6.2 Wazuh API. This version wasn’t released yet, and it’s not compatible with the latest version of the Splunk app. I’d recommend to you to install the latest Wazuh v3.6.1 from packages, you can do it by following the official documentation, don’t hesitate to write here for any doubt you may have.
I’ll be waiting for your feedback.

Best regards


--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/DA47840D-3D9D-4B5A-82B8-4C0DC5F433C0%40gmail.com.
For more options, visit https://groups.google.com/d/optout.


--

Husnain Bustam

unread,
Oct 29, 2018, 11:49:56 PM10/29/18
to Wazuh mailing list
Hi Manuel,
I have experienced a similar issue reported by Federico. I have reinstalled the Wazuh apps as per your instructions. That worked for me and I was able to register Wazuh API to the Wazuh app in the Splunk. Please note I am using the Splunk free edition. I was able to see some data in the UI. But after a while, I saw the API connection to Wazuh Splunk app is lost. When I tried to reconnect to the API I saw the following error message. Can you please share your thoughts on this?

This behavior is happening with Splunk only. I have a separate ELK stack running. In the Kibana I have not faced any issues so far. I have used same Wazuh managers and agents to connect to both Wazuh app in Kibana and Splunk. Thank you in advance for your help. Regards
Husnain Bustam

Wazuh version: 
wazuh-manager-3.6.1-1.x86_644
wazuh-api-3.6.1-1.x86_64

Error message from the Splunk server: /opt/splunk/var/log/splunk/SplunkAppForWazuh.log
{ "date": "2018-10-29 23:30:18,964" , "level": "ERROR" , "message": "Error making API request: database instance has no attribute 'logger'" }
{ "date": "2018-10-29 23:30:23,289" , "level": "ERROR" , "message": "Error in get_apis endpoint: database instance has no attribute 'logger'" }
{ "date": "2018-10-29 23:30:23,395" , "level": "ERROR" , "message": "Error making API request: database instance has no attribute 'logger'" }
{ "date": "2018-10-29 23:30:24,727" , "level": "ERROR" , "message": "Error in get_apis endpoint: database instance has no attribute 'logger'" }
{ "date": "2018-10-29 23:30:24,837" , "level": "ERROR" , "message": "Error making API request: database instance has no attribute 'logger'" }
{ "date": "2018-10-29 23:30:35,334" , "level": "ERROR" , "message": "Error in get_apis endpoint: database instance has no attribute 'logger'" }
{ "date": "2018-10-29 23:30:35,492" , "level": "ERROR" , "message": "Error making API request: database instance has no attribute 'logger'" }
{ "date": "2018-10-29 23:30:35,607" , "level": "ERROR" , "message": "{"error": "database instance has no attribute 'logger'"}" }
{ "date": "2018-10-29 23:31:21,467" , "level": "ERROR" , "message": "{'error': "database instance has no attribute 'logger'"}" }
{ "date": "2018-10-29 23:31:24,783" , "level": "ERROR" , "message": "{'error': "database instance has no attribute 'logger'"}" }

Manuel Jiménez

unread,
Oct 30, 2018, 6:30:13 AM10/30/18
to hab...@gmail.com, wa...@googlegroups.com

Hello Husnain,

It seems that the app you’ve just installed have missed some files, maybe for some incompatibilities with previous versions of the app. In order to check it, please try to reinstall by following these steps:

  1. Completely remove the Splunk app, this command will not interfere with any index and also any data will be compromised.
rm -rf /opt/splunk/etc/apps/SplunkAppForWazuh/
  1. Download the app from here and install it:
$SPLUNK_HOME/bin/splunk install app v3.6.1_7.1.3.tar.gz
  1. Restart your Splunk instance:
$SPLUNK_HOME/bin/splunk restart

Apologies for the inconvenience, I’ll be waiting for your feedback.

Best regards


--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

For more options, visit https://groups.google.com/d/optout.

Husnain Bustam

unread,
Oct 30, 2018, 9:25:51 AM10/30/18
to Wazuh mailing list
Manuel,
After reinstalling it is stable for the last two hours. I will keep monitoring the Splunk Wazuh app and Wazuh API connectivity stability. The only thing I have done differently this time along with the instruction provided by you is, deleted the following dir,
/opt/splunk/etc/apps/wazuh-splunk-master/
I do not know how this is created. I have deleted this folder and after re-installing the app I do not see this folder has been created. I will keep yopu posted on the stability. Thank you so much for your speedy response. Appreciated. 

# pwd

/opt/splunk/etc/apps

# ls

alert_logevent introspection_generator_addon  search   splunk_httpinput

alert_webhook launcher       SplunkAppForWazuh  splunk_instrumentation

appsbrowser learned       splunk_archiver   SplunkLightForwarder

framework legacy       SplunkForwarder   splunk_monitoring_console

gettingstarted sample_app       splunk_gdi   user-prefs

Manuel Jiménez

unread,
Oct 30, 2018, 9:39:10 AM10/30/18
to Personal Email ID, wa...@googlegroups.com

Hello Husnain,

It seems that you cloned the master branch directly from the GitHub repository, which is fine, but the folder that contains the app is located inside `wazuh-splunk-master`  and is called SplunkAppForWazuh. I'd recommend you to download the wanted release by navigating to releases section in the Splunk App repository:
image.png

Then, select the version that you want to download and download the compressed file:
image.png
That .tar.gz file is already prepared for being installed on Splunk with the command:

/opt/splunk/bin/splunk install app <file.tar.gz>

Which will result in a new folder called SplunkAppForWazuh in the /opt/splunk/etc/apps/ folder.

I hope this helps. Please do not hesitate to write here again whenever you may need it.

Best regards,
Manuel


--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

For more options, visit https://groups.google.com/d/optout.

Husnain Bustam

unread,
Oct 30, 2018, 7:09:24 PM10/30/18
to Wazuh mailing list
Hi Manuel, 
Thank you so much for the detailed explanation. Right after I re-installed the app it is stable for the last 12 hours. I am keeping an eye on it. I will keep you posted if I find any unstable behavior. Thanks again. 
Regards
Husnain Bustam
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/67c81c71-d4ba-4bb2-a9bb-05580b8ea2f8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Husnain Bustam

unread,
Oct 31, 2018, 1:09:10 PM10/31/18
to Wazuh mailing list
Hi Manuel, 
I am getting Wazuh App and API connection error again. The connection was stable for about 22 hours. But this got disconnected while I was navigating between different tabs in the Wazuh Splunk app. From the "/opt/splunk/var/log/splunk/SplunkAppForWazuh.log" I can see the following messages.  Today after this issue happened I have re-installed the app and so far it is working now for the last 30 minutes. So the summary is, after re-installing the connection is stable for a while. If the connection is lost, to re-establish the connection between Wazuh app and API, I need to reinstall the Splunk Wazuh apps. Can you please share your thoughts on it? Please let me know if I can provide any more information. Thank you in advance. 
Regards
HB

{ "date": "2018-10-31 09:42:03,754" , "level": "ERROR" , "message": "Error making API request: database instance has no attribute 'logger'" }

{ "date": "2018-10-31 11:25:53,666" , "level": "ERROR" , "message": "Error making API request: database instance has no attribute 'logger'" }

{ "date": "2018-10-31 11:25:53,669" , "level": "ERROR" , "message": "Error making API request: list index out of range" }

{ "date": "2018-10-31 11:25:57,024" , "level": "ERROR" , "message": "Error making API request: database instance has no attribute 'logger'" }

{ "date": "2018-10-31 11:27:05,560" , "level": "ERROR" , "message": "Error making API request: database instance has no attribute 'logger'" }

{ "date": "2018-10-31 11:27:05,569" , "level": "ERROR" , "message": "Error making API request: database instance has no attribute 'logger'" }

{ "date": "2018-10-31 11:27:05,597" , "level": "ERROR" , "message": "Error making API request: database instance has no attribute 'logger'" }

{ "date": "2018-10-31 11:27:05,602" , "level": "ERROR" , "message": "Error making API request: database instance has no attribute 'logger'" }

{ "date": "2018-10-31 11:27:05,616" , "level": "ERROR" , "message": "Error making API request: database instance has no attribute 'logger'" }

{ "date": "2018-10-31 11:27:05,896" , "level": "ERROR" , "message": "Error in get_apis endpoint: database instance has no attribute 'logger'" }

{ "date": "2018-10-31 11:27:05,976" , "level": "ERROR" , "message": "Error making API request: database instance has no attribute 'logger'" }

{ "date": "2018-10-31 11:27:06,071" , "level": "ERROR" , "message": "Error in get_apis endpoint: database instance has no attribute 'logger'" }

{ "date": "2018-10-31 11:27:06,154" , "level": "ERROR" , "message": "Error making API request: database instance has no attribute 'logger'" }

{ "date": "2018-10-31 11:27:06,240" , "level": "ERROR" , "message": "{"error": "database instance has no attribute 'logger'"}" }

{ "date": "2018-10-31 11:27:09,903" , "level": "ERROR" , "message": "Error in get_apis endpoint: database instance has no attribute 'logger'" }

{ "date": "2018-10-31 11:27:09,980" , "level": "ERROR" , "message": "Error making API request: database instance has no attribute 'logger'" }

{ "date": "2018-10-31 11:28:05,362" , "level": "ERROR" , "message": "{'error': "database instance has no attribute 'logger'"}" }

{ "date": "2018-10-31 12:15:34,865" , "level": "ERROR" , "message": "{"error": "database instance has no attribute 'logger'"}" }

{ "date": "2018-10-31 12:16:38,216" , "level": "ERROR" , "message": "Error in agent module constructor: Extra data: line 1 column 13 - line 1 column 297 (char 12 - 296)" }

{ "date": "2018-10-31 12:16:38,219" , "level": "ERROR" , "message": "Error in API module constructor: Extra data: line 1 column 13 - line 1 column 297 (char 12 - 296)" }

{ "date": "2018-10-31 12:16:38,227" , "level": "ERROR" , "message": "Error in manager module constructor: Extra data: line 1 column 13 - line 1 column 297 (char 12 - 296)" }

{ "date": "2018-10-31 12:18:39,920" , "level": "ERROR" , "message": "Error in agent module constructor: Extra data: line 1 column 13 - line 1 column 297 (char 12 - 296)" }

{ "date": "2018-10-31 12:18:39,923" , "level": "ERROR" , "message": "Error in API module constructor: Extra data: line 1 column 13 - line 1 column 297 (char 12 - 296)" }

{ "date": "2018-10-31 12:18:39,932" , "level": "ERROR" , "message": "Error in manager module constructor: Extra data: line 1 column 13 - line 1 column 297 (char 12 - 296)" }

{ "date": "2018-10-31 12:22:09,215" , "level": "ERROR" , "message": "{"error": "'manager' object has no attribute 'db'"}" }

{ "date": "2018-10-31 12:22:53,946" , "level": "ERROR" , "message": "Cannot connect to API : 'manager' object has no attribute 'session'" }

Manuel Jiménez

unread,
Oct 31, 2018, 3:23:04 PM10/31/18
to Personal Email ID, wa...@googlegroups.com

Hello Husnain,

I’m sorry you’re still facing this issue. Have you verified the connection between the Splunk instance and Wazuh API machines when the connection in the app was lost? You can try a curl to the API from the Splunk machine like this:

curl -u foo:bar http(s)://wazuh-api-ip:55000/

Change foo:bar by your API username and password,55000 by your porty and also set https if SSL was enabled. If the API does not respond, then the problem is located on the API side.
On the other hand, when the API connection is lost, the app redirects you to the settings section by itself. Please, check that the API list is still correct, and also press the check connection button:
image.png
I’d be helpful if you could check and paste here what message is returned.
Also, have you tried to restart the Splunk service when the API connection is lost? You can try this on your Splunk instance:

/opt/splunk/bin/splunk restart

Thanks for your patience.

Best regards,
Manuel


--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

For more options, visit https://groups.google.com/d/optout.

Husnain Bustam

unread,
Nov 6, 2018, 12:22:14 PM11/6/18
to Wazuh mailing list
Hi Manuel, 
Sorry for the late response. 
I have re-installed the Wazuh Splunk app. After every fresh installation (uninstall and re-install), I can connect to the API from App. While navigating the app, the connection gets disconnected. After that, I did curl from Splunk server to the Wazuh API. 

root@***-wsplunk-***:~$ curl -u  ***********:*********** http://172.29.19.18:55000/?pretty

{

   "error": 0,

   "data": {

      "msg": "Welcome to Wazuh HIDS API",

      "api_version": "v3.6.1",

      "hostname": "******************",

      "timestamp": "Tue Nov 06 2018 11:57:02 GMT-0500 (EST)"

   }

}


After that, I have checked the API configurations from the Splunk Wazuh app. 

Screen Shot 2018-11-06 at 12.02.40 PM.png


Then I checked the /opt/splunk/var/log/splunk/SplunkAppForWazuh.log on the Splunk server. 
{ "date": "2018-11-06 11:24:15,736" , "level": "ERROR" , "message": "Error making API request: database instance has no attribute 'logger'" }

{ "date": "2018-11-06 11:24:42,800" , "level": "ERROR" , "message": "Error making API request: database instance has no attribute 'logger'" }

{ "date": "2018-11-06 11:24:42,800" , "level": "ERROR" , "message": "Error making API request: database instance has no attribute 'logger'" }

{ "date": "2018-11-06 11:25:13,844" , "level": "ERROR" , "message": "Error making API request: database instance has no attribute 'logger'" }

{ "date": "2018-11-06 11:25:13,845" , "level": "ERROR" , "message": "Error making API request: database instance has no attribute 'logger'" }

{ "date": "2018-11-06 11:29:53,313" , "level": "ERROR" , "message": "Error making API request: database instance has no attribute 'logger'" }

{ "date": "2018-11-06 11:29:53,314" , "level": "ERROR" , "message": "Error making API request: list index out of range" }

{ "date": "2018-11-06 11:29:53,317" , "level": "ERROR" , "message": "Error making API request: database instance has no attribute 'logger'" }

{ "date": "2018-11-06 11:29:53,317" , "level": "ERROR" , "message": "Error making API request: database instance has no attribute 'logger'" }

{ "date": "2018-11-06 11:29:54,951" , "level": "ERROR" , "message": "Error making API request: database instance has no attribute 'logger'" }

{ "date": "2018-11-06 11:30:09,248" , "level": "ERROR" , "message": "Error in get_apis endpoint: database instance has no attribute 'logger'" }

{ "date": "2018-11-06 11:30:09,333" , "level": "ERROR" , "message": "Error making API request: database instance has no attribute 'logger'" }

{ "date": "2018-11-06 11:30:09,439" , "level": "ERROR" , "message": "Error in get_apis endpoint: database instance has no attribute 'logger'" }

{ "date": "2018-11-06 11:30:09,520" , "level": "ERROR" , "message": "Error making API request: database instance has no attribute 'logger'" }

{ "date": "2018-11-06 11:30:09,615" , "level": "ERROR" , "message": "{"error": "database instance has no attribute 'logger'"}" } 

I can see the above error message in the logs. After that, I have restarted the Splunk daemon. 

/opt/splunk/bin/splunk restart

Stopping splunkd...

Shutting down.  Please wait, as this may take a few minutes.

.....................................                      [  OK  ]

Stopping splunk helpers...

                                                           [  OK  ]

Done.


Splunk> All batbelt. No tights.


Checking prerequisites...

Checking http port [8000]: open

Checking mgmt port [8089]: open

Checking appserver port [127.0.0.1:8065]: open

Checking kvstore port [8191]: open

Checking configuration...  Done.

Checking critical directories... Done

Checking indexes...

Validated: _audit _internal _introspection _telemetry _thefishbucket history main summary wazuh wazuh-monitoring-3x

Done

Checking filesystem compatibility...  Done

Checking conf files for problems...

Done

Checking default conf files for edits...

Validating installed files against hashes from '/opt/splunk/splunk-7.2.0-8c86330ac18-linux-2.6-x86_64-manifest'

All installed files intact.

Done

All preliminary checks passed.


Starting splunk server daemon (splunkd)...  

Done

                                                           [  OK  ]


Waiting for web server at http://127.0.0.1:8000 to be available... Done



If you get stuck, we're here to help.  

Look for answers here: http://docs.splunk.com


The Splunk web interface is at http://**********************:8000


After the restart, I tried to connect to the API from the App. And this is not working. The only way it works, if I uninstall and re-install the app. But that thing lasts for a short period of time. Can you please share your thoughts on it. Please let me know if you need any more information to troubleshoot this issue. 
Thanks. 
Regards
Husnain Bustam


Manuel Jiménez

unread,
Nov 6, 2018, 2:04:52 PM11/6/18
to Personal Email ID, wa...@googlegroups.com

Hello Husnain,

This error looks pretty strange to me. Just for make sure, please execute the following command when the API is still connecting:

cat /opt/splunk/etc/apps/SplunkAppForWazuh/bin/apilist.json

Check that the content of the file is the already inserted API list and the values are correct. After that, please check it again once the API gets disconnected.
I will also try to reproduce your case and will let you know about any solution as soon as possible, so any information about your system or infrastructure will be appreciated. Thanks for your patience.

Kind regards,
Manuel


--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

For more options, visit https://groups.google.com/d/optout.

Husnain Bustam

unread,
Nov 6, 2018, 4:21:55 PM11/6/18
to Wazuh mailing list
Manuel, 
Thank you so much for helping out on this. 

Following is the behavior I can see. In two independent trial, I have seen the following behavior. 
  • In the clean state (after uninstalling and re-installing the app), after connecting to the Wazuh API to the Wazuh Splunk app, the ```/opt/splunk/etc/apps/SplunkAppForWazuh/bin/apilist.json``` file contains the right info. And I have validated the json file and looks good to me. 
  • Then I started navigating the Wazuh app from the Web UI
  • Then the /opt/splunk/etc/apps/SplunkAppForWazuh/bin/apilist.json file content changed. The change caused the issue. 
    • In the log file, I can see the similar logs reported earlier. 
System Info:
Manager and worker VM (Clustered) || OS: CentOS 7 || vCPU: 2 || Memory: 16 || LB: No
No of agents: 12 || Agent OS: CentOS-7

# In the normal state - immediate state after registering API
# cat /opt/splunk/etc/apps/SplunkAppForWazuh/bin/apilist.json  | jq
{
  "apis": {
    "1": {
      "filterType": "cluster.name",
      "managerName": "***-ossecm-d01.example.com",
      "portapi": "55000",
      "userapi": "************",
      "passapi": "************",
      "filterName": "wazuh-cluster",
      "url": "http://172.22.19.158",
      "id": "89f7aec5-d727-4603-b0bb-2883112335d7"
    }
  },
  "_default": {}
}


# File content changed. 
# While navigating the Apps, the entries in the file get changed. 

# cat /opt/splunk/etc/apps/SplunkAppForWazuh/bin/apilist.json  | jq
{
  "apis": {}
}
": {"
parse error: Invalid literal at line 1, column 28

# tail -f /opt/splunk/etc/apps/SplunkAppForWazuh/bin/apilist.json 
{"apis": {}}": {"filterType": "cluster.name", "managerName": "***-ossecm-d01.example.com", "portapi": "55000", "userapi": "*********", "passapi": "*************", "filterName": "wazuh-cluster", "url": "http://172.22.19.158", "id": "89f7aec5-d727-4603-b0bb-2883112335d7"}}, "_default": {}}
Please let me know if I can provide any information to troubleshoot this issue. Please let me know if you have any questions. Thanks. 
Regards
Husnain Bustam

Manuel Jiménez

unread,
Nov 7, 2018, 4:37:15 AM11/7/18
to Personal Email ID, wa...@googlegroups.com

Hi Husnain,

First I’d like to thank you for your detailed report. I tried to reproduce your case without success, everything worked fine in my lab so we have to dig into the reason why that file is being modified in order to take rid of it. I’d suggest to perform a few actions so that we can find the certain point where the problem is occurring. If you are on a lab environment and don’t have any relevant data to lose, I think it would be a good idea to completely remove and reinstall the full Splunk installation and the Wazuh app in order to discard a Splunk problem due to a bad installation. In the case you’re on a production environment, you want to keep your index data, or you’re using another app that you don’t want to remove, just ignore that. Also it be helpful if you could deploy a new installation in a lab or virtual environment and check if the problem is still happening on it. The actions I’d suggest you to do are the following:

  1. Let’s monitor the apilist.json file. Please execute these commands as root when the Splunk service is stopped and keep checking the results after starting it and while you’re navigating to the app, so you can notice the exact point the file is being modified. In one terminal:
    # This will cat the content of the file on real time.
    watch -n0 'cat /opt/splunk/etc/apps/SplunkAppForWazuh/bin/apilist.json'
    
    In another terminal:
    # This will monitor which processes are accessing to the file
    watch -n0 'lsof /opt/splunk/etc/apps/SplunkAppForWazuh/bin/apilist.json'

Sorry for the inconveniences.

Regards,
Manuel


--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

For more options, visit https://groups.google.com/d/optout.

Manuel Jiménez

unread,
Nov 7, 2018, 5:42:18 AM11/7/18
to Personal Email ID, wa...@googlegroups.com

Hi again Husnain,

I forgot to mention that we found a bug in our log module, that’s why we couldn’t read certain information about the error and got "Error making API request: database instance has no attribute 'logger' instead. Despite that bug was not causing the issue you’re currently facing, now we can get a better track of it by logging the actual error message. Please, update the app by downloading the patched .tar.gz file from here and following the updating process described in above messages.

I’d also like to thank you about your collaboration, it’s being much helpful for us to improve the product so we can provide the best quality software to the community.

Kind regards,
Manuel

Husnain Bustam

unread,
Nov 7, 2018, 11:34:54 AM11/7/18
to Wazuh mailing list
Manuel, 
Thank you so much for sending detailed troubleshooting steps. Appreciated.

All of the VMs are development VMs. I have tried the following two setups to find the root cause of the issue. I have tried three times to reproduce the issue in every setup and every time I was able to reproduce the issue. Sometime it will be quick while navigating the app from the Web UI and sometimes it takes a while. 
Setup-1:
Splunk Instance - New OS and Splunk installation, Wazuh App installed here (latest version with the bug fix for logger) || 2vCPU, 4GB RAM, 100GB /
Wazuh manager and agents: Existing instances. 

Setup-2:
Wazuh Manager - New OS and Wazuh installation, Splunk forwarder installed on this node. || 2vCPU, 4GB RAM, 100GB /
Wazuh Agent - New OS and Wazuh installation || 2vCPU, 4GB RAM, 100GB /
Splunk Instance - New OS and Splunk installation, Wazuh App installed here (latest version with the bug fix for logger) || 2vCPU, 4GB RAM, 100GB /

When the issue occurred: 
  • Wazuh App --> Management --> Status. When I clicked here I was able to see the following messages for the first time after fresh installation of Splunk and app. 
{ "date": "2018-11-07 11:04:04,947" , "level": "ERROR" , "message": "Error at get document DB: No JSON object could be decoded" }
{ "date": "2018-11-07 11:04:04,951" , "level": "ERROR" , "message": "Error making API request: No JSON object could be decoded" }
{ "date": "2018-11-07 11:04:04,953" , "level": "ERROR" , "message": "Error at get document DB: No JSON object could be decoded" }
{ "date": "2018-11-07 11:04:04,955" , "level": "ERROR" , "message": "Error making API request: No JSON object could be decoded" }
  • Wazuh App --> Agents --> Agent 1 (002 e.g.) --> {Surfing all the options e.g. Policy monitoring }
Mostly when surfing agents section I saw the API was getting disconnected. Please check the following output. 
  • Wazuh App --> Agents --> Agent 1 (002 e.g.) --> Inventory 
While navigating this, I saw a new message as follows. After this, I was not able to see data under the packages section. 

Screen Shot 2018-11-07 at 11.07.29 AM.png



# After the issue happens
[root@site-wsplunk-d02 ~]# cat /opt/splunk/etc/apps/SplunkAppForWazuh/bin/apilist.json | ./jq-linux64 
{
  "apis": {}
}
{}
parse error: Expected value before ',' at line 1, column 16
[root@site-wsplunk-d02 ~]# cat /opt/splunk/etc/apps/SplunkAppForWazuh/bin/apilist.json 
{"apis": {}} {}, "apis": {"1": {"id": "0f97be28-ad78-484b-9a35-ea0573e5e587", "passapi": "6Kp2qt1TGMQPvGGPxOLnaQ==", "filterType": "cluster.name", "url": "http://172.22.19.158", "managerName": "site-ossecm-d01.example.com", "filterName": "pcln-cluster", "portapi": "55000", "userapi": "pclnusr"}


[root@site-wsplunk-d02 ]# lsof /opt/splunk/etc/apps/SplunkAppForWazuh/bin/apilist.json
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF      NODE NAME
python  11478 root    6u   REG  253,0      296 201470892 /opt/splunk/etc/apps/SplunkAppForWazuh/bin/apilist.json
python  11478 root    7u   REG  253,0      296 201470892 /opt/splunk/etc/apps/SplunkAppForWazuh/bin/apilist.json
python  11478 root    8u   REG  253,0      296 201470892 /opt/splunk/etc/apps/SplunkAppForWazuh/bin/apilist.json

[root@site-wsplunk-d02 ]# ps -elf | grep 11478
0 S root     11478 11325  1  80   0 - 522777 poll_s 08:54 ?       00:00:15 /opt/splunk/bin/python -O /opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/root.py --proxied=127.0.0.1,8065,8000

[root@site-wsplunk-d02 splunk]# tail -f SplunkAppForWazuh.log 
{ "date": "2018-11-07 09:10:32,873" , "level": "ERROR" , "message": "Error at get document DB: Extra data: line 1 column 4 - line 1 column 285 (char 3 - 284)" }
{ "date": "2018-11-07 09:10:32,875" , "level": "ERROR" , "message": "Error making API request: Extra data: line 1 column 4 - line 1 column 285 (char 3 - 284)" }
{ "date": "2018-11-07 09:10:32,875" , "level": "ERROR" , "message": "Error at get document DB: No JSON object could be decoded" }
{ "date": "2018-11-07 09:10:32,874" , "level": "ERROR" , "message": "Error making API request: list index out of range" }
{ "date": "2018-11-07 09:10:32,878" , "level": "ERROR" , "message": "Error making API request: No JSON object could be decoded" }
{ "date": "2018-11-07 09:10:32,897" , "level": "ERROR" , "message": "Error at get document DB: Extra data: line 1 column 14 - line 1 column 297 (char 13 - 296)" }
{ "date": "2018-11-07 09:10:32,898" , "level": "ERROR" , "message": "Error making API request: Extra data: line 1 column 14 - line 1 column 297 (char 13 - 296)" }
{ "date": "2018-11-07 09:10:34,393" , "level": "ERROR" , "message": "Error at get document DB: Extra data: line 1 column 14 - line 1 column 297 (char 13 - 296)" }
{ "date": "2018-11-07 09:10:34,393" , "level": "ERROR" , "message": "Error making API request: Extra data: line 1 column 14 - line 1 column 297 (char 13 - 296)" }
{ "date": "2018-11-07 09:10:36,941" , "level": "ERROR" , "message": "Error at get document DB: Extra data: line 1 column 14 - line 1 column 297 (char 13 - 296)" }
{ "date": "2018-11-07 09:10:36,942" , "level": "ERROR" , "message": "Error in get_apis endpoint: Extra data: line 1 column 14 - line 1 column 297 (char 13 - 296)" }
{ "date": "2018-11-07 09:10:37,030" , "level": "ERROR" , "message": "Error at get document DB: Extra data: line 1 column 14 - line 1 column 297 (char 13 - 296)" }
{ "date": "2018-11-07 09:10:37,031" , "level": "ERROR" , "message": "Error making API request: Extra data: line 1 column 14 - line 1 column 297 (char 13 - 296)" }
{ "date": "2018-11-07 09:10:37,142" , "level": "ERROR" , "message": "Error at get document DB: Extra data: line 1 column 14 - line 1 column 297 (char 13 - 296)" }
{ "date": "2018-11-07 09:10:37,142" , "level": "ERROR" , "message": "Error in get_apis endpoint: Extra data: line 1 column 14 - line 1 column 297 (char 13 - 296)" }
{ "date": "2018-11-07 09:10:37,219" , "level": "ERROR" , "message": "Error at get document DB: Extra data: line 1 column 14 - line 1 column 297 (char 13 - 296)" }
{ "date": "2018-11-07 09:10:37,219" , "level": "ERROR" , "message": "Error making API request: Extra data: line 1 column 14 - line 1 column 297 (char 13 - 296)" }
{ "date": "2018-11-07 09:10:37,330" , "level": "ERROR" , "message": "Error at get all documents DB module: Extra data: line 1 column 14 - line 1 column 297 (char 13 - 296)" }
{ "date": "2018-11-07 09:10:37,330" , "level": "ERROR" , "message": "{"error": "Extra data: line 1 column 14 - line 1 column 297 (char 13 - 296)"}" }


# /opt/splunk/bin/splunk cmd python get_current_version.py 

ERROR

"Error : Traceback: Traceback (most recent call last):

  File ""get_current_version.py"", line 20, in <module>

    app = cli.getConfStanza('version','app')

  File ""/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli_common.py"", line 357, in getConfStanza

    raise ParsingError, ""no '%s' stanza exists in %s.conf.  Your configuration may be corrupt or may require a restart."" % (stanza, confName)

ParsingError: (""no 'app' stanza exists in version.conf.  Your configuration may be corrupt or may require a restart."",)

"


Please let me know if I can provide any more information to troubleshoot this issue. Thank you again for your help. 
Regards
Husnain Bustam


SplunkAppForWazuh.log

Manuel Jiménez

unread,
Nov 13, 2018, 7:16:05 AM11/13/18
to Personal Email ID, wa...@googlegroups.com
Hello Husnain,

Apologies for the late reply. In order to completely reproduce the issue with your very same environment, please indicate your Splunk version here and also the operating system you're using in your lab. After several attempts I still haven't been able to reproduce this case, so I'm very interested about find it and give you any solution.

Thanks,

Best regards,
Manuel
--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

For more options, visit https://groups.google.com/d/optout.

Manuel Jiménez

unread,
Dec 5, 2018, 6:19:45 AM12/5/18
to Personal Email ID, Wazuh mailing list
Hello Husnain,

Was this problem resolved? I'm still interested in resolving this issue. I'm concerned about this is happening in other environments, but I'm not still able to reproduce it.

Best regards,
Manuel
Reply all
Reply to author
Forward
0 new messages