Fix for unassigned shard in SELKS 4 on a standalone install?

[bug...@gmail.com]

Hi,

I just did a new SELKS 4.0 install as a standalone VM, everything is working fine but I am getting the "unassigned shard" yellow warning.

I remember a similar issue in SELKS 2.0, which could be fixed by add following the instruction there:

https://github.com/StamusNetworks/SELKS/issues/17

basically editing the elasticsearch.yml

This doesn't work anymore :(

Is there a new way to fix this issue?

This is what I get from Scirius GUI:

• Relocating shards: 0
• Initializing shards: 0
• Unassigned shards: 1

I did see a thread here:

https://groups.google.com/forum/#!searchin/selks/shards%7Csort:relevance/selks/pWZykflnAVg/7yN0nDwWBAAJ

But I can't really see a solution appart the mention of using a template (not sure where and how!)

no need to worry about it? (appart from the fact I'd like everything to be green! ;)

Cheers,

B.

[Peter Manev]

On Thu, Oct 12, 2017 at 12:35 PM, <bug...@gmail.com> wrote:
> Hi,
>
> I just did a new SELKS 4.0 install as a standalone VM, everything is working
> fine but I am getting the "unassigned shard" yellow warning.
> I remember a similar issue in SELKS 2.0, which could be fixed by add
> following the instruction there:
> https://github.com/StamusNetworks/SELKS/issues/17
> basically editing the elasticsearch.yml
>
> This doesn't work anymore :(
>
> Is there a new way to fix this issue?
> This is what I get from Scirius GUI:
>
> Relocating shards: 0
> Initializing shards: 0
> Unassigned shards: 1
>

You can try -

curl -XPUT 'localhost:9200/_settings' -d '{
"index" : {
"number_of_replicas" : 0
}
}'

> I did see a thread here:
> https://groups.google.com/forum/#!searchin/selks/shards%7Csort:relevance/selks/pWZykflnAVg/7yN0nDwWBAAJ
>
> But I can't really see a solution appart the mention of using a template
> (not sure where and how!)
> no need to worry about it? (appart from the fact I'd like everything to be
> green! ;)
>
> Cheers,
> B.
>

> --
> IRC: Let's talk about SELKS on Freenode IRC on the #SELKS channel
> Wiki: https://github.com/StamusNetworks/SELKS/wiki
> GitHub: https://github.com/StamusNetworks/SELKS
> Blog: https://www.stamus-networks.com/theblog/
> Twitter: @StamusN
> g+: Stamus Networks
> ---
> You received this message because you are subscribed to the Google Groups
> "SELKS" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to selks+un...@googlegroups.com.
> To post to this group, send email to se...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Regards,
Peter Manev

[bug...@gmail.com]

Thanks it works!

However, since I modified the /etc/elasticsearch.yml to add the 2 old lines trying to fix the issue (and restarting elasticsearch) I have the following error in my kibana dashboard:

Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"script_exception","reason":"link error","script_stack":["doc['alert.signature_id'].value"," ^---- HERE"],"script":"doc['alert.signature_id'].value","lang":"expression"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"logstash-tls-2017.10.12","node":"oaPcRbG2QDWKrYScFsjkqg","reason":{"type":"script_exception","reason":"link error","script_stack":["doc['alert.signature_id'].value"," ^---- HERE"],"script":"doc['alert.signature_id'].value","lang":"expression","caused_by":{"type":"parse_exception","reason":"Field [alert.signature_id] does not exist in mappings"}}}]},"status":500} at https://192.168.1.183/bundles/kibana.bundle.js?v=15554:29:12529 at Function.Promise.try (https://192.168.1.183/bundles/commons.bundle.js?v=15554:82:22246) at https://192.168.1.183/bundles/commons.bundle.js?v=15554:82:21602 at Array.map (native) at Function.Promise.map (https://192.168.1.183/bundles/commons.bundle.js?v=15554:82:21557) at callResponseHandlers (https://192.168.1.183/bundles/kibana.bundle.js?v=15554:29:12145) at https://192.168.1.183/bundles/kibana.bundle.js?v=15554:29:400 at processQueue (https://192.168.1.183/bundles/commons.bundle.js?v=15554:38:23621) at https://192.168.1.183/bundles/commons.bundle.js?v=15554:38:23888 at Scope.$eval (https://192.168.1.183/bundles/commons.bundle.js?v=15554:39:4619)

I have rebooted the box

I reset the logs and stats as per your wiki

a status on elasticsearch shows:

root@SELKS:/opt/selks/Scripts/Configs/SELKS4# systemctl status elasticsearch

● elasticsearch.service - Elasticsearch

   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)

   Active: active (running) since Fri 2017-10-13 04:16:45 BST; 13min ago

     Docs: http://www.elastic.co

  Process: 725 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)

 Main PID: 740 (java)

    Tasks: 70 (limit: 4915)

   CGroup: /system.slice/elasticsearch.service

           └─740 /usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOn

Oct 13 04:16:44 SELKS systemd[1]: Starting Elasticsearch...

Oct 13 04:16:45 SELKS systemd[1]: Started Elasticsearch.

Anything I can do, besides reinstalling, to fix this issue?

Thanks,

B.

[bug...@gmail.com]

Also, just noticed something, in the elasticsearch it references "logstash-tls-2017.10.12".

I looked in /var/log/elasticsearch and I can only see:

logstash-plain-2017.10.11

So "plain" instead of "tls" and a day earlier.

Now that I have done my test again, the error references 2017.10.13 and in the /var/log there is only 2710.10.12 (also as plain and not tls)

Not sure if that's normal and if there is another location for those logs?!

Cheers,

B.

[Peter Manev]

On 13 Oct 2017, at 05:33, bug...@gmail.com wrote:

Thanks it works!

However, since I modified the /etc/elasticsearch.yml to add the 2 old lines trying to fix the issue (and restarting elasticsearch) I have the following error in my kibana dashboard:

What two old lines are these? What are you referring to?

Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"script_exception","reason":"link error","script_stack":["doc['alert.signature_id'].value"," ^---- HERE"],"script":"doc['alert.signature_id'].value","lang":"expression"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"logstash-tls-2017.10.12","node":"oaPcRbG2QDWKrYScFsjkqg","reason":{"type":"script_exception","reason":"link error","script_stack":["doc['alert.signature_id'].value"," ^---- HERE"],"script":"doc['alert.signature_id'].value","lang":"expression","caused_by":{"type":"parse_exception","reason":"Field [alert.signature_id] does not exist in mappings"}}}]},"status":500} at https://192.168.1.183/bundles/kibana.bundle.js?v=15554:29:12529 at Function.Promise.try (https://192.168.1.183/bundles/commons.bundle.js?v=15554:82:22246) at https://192.168.1.183/bundles/commons.bundle.js?v=15554:82:21602 at Array.map (native) at Function.Promise.map (https://192.168.1.183/bundles/commons.bundle.js?v=15554:82:21557) at callResponseHandlers (https://192.168.1.183/bundles/kibana.bundle.js?v=15554:29:12145) at https://192.168.1.183/bundles/kibana.bundle.js?v=15554:29:400 at processQueue (https://192.168.1.183/bundles/commons.bundle.js?v=15554:38:23621) at https://192.168.1.183/bundles/commons.bundle.js?v=15554:38:23888 at Scope.$eval (https://192.168.1.183/bundles/commons.bundle.js?v=15554:39:4619)

I have rebooted the box

I reset the logs and stats as per your wiki

What are you trying to achieve?

[Peter Manev]

On 13 Oct 2017, at 05:50, bug...@gmail.com wrote:

Also, just noticed something, in the elasticsearch it references "logstash-tls-2017.10.12".

I looked in /var/log/elasticsearch and I can only see:

logstash-plain-2017.10.11

So "plain" instead of "tls" and a day earlier.

Now that I have done my test again, the error references 2017.10.13 and in the /var/log there is only 2710.10.12 (also as plain and not tls)

What test is that - that you have done ?

[bug...@gmail.com]

Hi Pevma,

This is exactly what I did when trying to fix the unassigned warning:

- Edited /etc/elasticsearch/elasticsearch.yml

- Added the following two lines at the end:

index.number_of_shards: 1
index.number_of_replicas: 0

- Restarted elasticsearch

- I got reds everywhere in Scirius

- so I removed those extra lines

- restarted elasticsearch

- In scirius I got the yellow unassigned warnings back

Meaning back to square one, so this is when I originally posted this post.

In the mean time, I also noticed that my Kibana dashboards did not display any data anymore!

And saw the error at the top (which I have included the details in my previous comment)

I did run the command you gave me to clear the unassigned shard warnings. It works, everything is green in Scirius, I can also see alerts being generated.

But I still have the issue with my kibana dashboards not displaying any data and with that error on the top.

I tried to reboot the box

Stop and start elasticsearch

run the selks upgrade script a few times (this is where I noticed another error and posted a different thread)

I also purged my stats and logs following instructions from your wiki

I tried to understand a bit more what the error meant, and in doing so tried to locate that logstash logs referenced in the error.

Now, I am stuch and not sure what is wrong as I am not familiar with the inside of elasticsearch and why it fails...

It does reference something about an alert it cannot find... maybe a problem with its index?!

All this happens after I edited the elasticsearch.yml, added those two lines, restarted elasticsearch, removed those two lines, restarted elasticsearch... as if I "corrupted" my elasticsearch config by doing so...

B.

[bug...@gmail.com]

Another piece of information:

When trying to load KIBANA dashboards I do not always get that error!

So the following dashboards generates no error and display data:

-> ERROR and no data

SN ALERTS

-> ERROR but displaying some data

SN IDS

-> OK:

SN ALL

SN DNS

SN File Transaction

SN Flow

SN HTTP

SN OVERVIEW

SN STATS

SN TLS

-> No Error and NO DATA: (but I think this is normal)

SN SMTP 

SN SSH

SN VLAN

The error I get is:

Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"script_exception","reason":"link error","script_stack":["doc['alert.signature_id'].value"," ^---- HERE"],"script":"doc['alert.signature_id'].value","lang":"expression"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"logstash-tls-2017.10.13","node":"oaPcRbG2QDWKrYScFsjkqg","reason":{"type":"script_exception","reason":"link error","script_stack":["doc['alert.signature_id'].value"," ^---- HERE"],"script":"doc['alert.signature_id'].value","lang":"expression","caused_by":{"type":"parse_exception","reason":"Field [alert.signature_id] does not exist in mappings"}}}]},"status":500} at https://192.168.1.183/bundles/kibana.bundle.js?v=15554:29:12529 at Function.Promise.try (https://192.168.1.183/bundles/commons.bundle.js?v=15554:82:22246) at https://192.168.1.183/bundles/commons.bundle.js?v=15554:82:21602 at Array.map (native) at Function.Promise.map (https://192.168.1.183/bundles/commons.bundle.js?v=15554:82:21557) at callResponseHandlers (https://192.168.1.183/bundles/kibana.bundle.js?v=15554:29:12145) at https://192.168.1.183/bundles/kibana.bundle.js?v=15554:29:400 at processQueue (https://192.168.1.183/bundles/commons.bundle.js?v=15554:38:23621) at https://192.168.1.183/bundles/commons.bundle.js?v=15554:38:23888 at Scope.$eval (https://192.168.1.183/bundles/commons.bundle.js?v=15554:39:4619)

[bug...@gmail.com]

Looking a bit more in the error, if you look at the following:

["doc['alert.signature_id'].value"," ^---- HERE"]

This would seem to indicate there is an alert signature with the value of " ^---- HERE"

Is that the problem? or am I miss-interpreting that line.

I don't know how to navigate the elasticsearch database to check if thats the case and fix that issue (again, could it have been caused because the change to elasticsearch.yml I did was wrong and somehow corrupted elasticsearch database!?)

Cheers,

B.

[Peter Manev]

On 13 Oct 2017, at 07:20, bug...@gmail.com wrote:

Hi Pevma,

This is exactly what I did when trying to fix the unassigned warning:

- Edited /etc/elasticsearch/elasticsearch.yml

- Added the following two lines at the end:

index.number_of_shards: 1
index.number_of_replicas: 0

Yes - those are for ES2.x , they don’t work in ES5 

- Restarted elasticsearch

- I got reds everywhere in Scirius

- so I removed those extra lines

- restarted elasticsearch

- In scirius I got the yellow unassigned warnings back

Meaning back to square one, so this is when I originally posted this post.

In the mean time, I also noticed that my Kibana dashboards did not display any data anymore!

And saw the error at the top (which I have included the details in my previous comment)

I did run the command you gave me to clear the unassigned shard warnings. It works, everything is green in Scirius, I can also see alerts being generated.

But I still have the issue with my kibana dashboards not displaying any data and with that error on the top.

Try to reset the dashboards from Scirius.