Would like to separate out SG logs from ES logs.

[ch s]

I guess this is more of a log4j2 question than anything else, but we have a requirement to let another team have access to our elasticsearch logs (And by this, I mean the log file generated by the elasticsearch process controlled by log4j2.properties, not the documents that are inside the elasticsearch cluster itself). The only problem is that the logs for the searchguard process are combined with the regular elasticsearch process logs. Is there a way to separate the two so that we can put the searchguard logs into a different file than the one elasticsearch is using for its own logs?

I did experiment with the auditlog, but that still doesn't capture the actual searchguard process logs. It works great for audit events though!

I'm guessing this would require some changes to log4j2.properties, but understanding that file is completely foreign to me.

Thanks!

* Search Guard and Elasticsearch version

SG 5.6.7, ES 5.6.7

* Installed and used enterprise modules, if any

Auditlog

[ch s]

Anyone? If this isn't possible, that's fine, I'd just like to know.

[SG]

of course you can do this. but as you said this is more log4j than SG related.
All loggers for searchguard start with com.floragunn or sg_action_trace (for auditlogs you can configure the loggername yourself)
So just route them into another file than the rest of the logs.

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/341eaa26-3c07-4255-8f08-bfd60f78793c%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[CHS]

You don't have any examples of integrating this with the existing elasticsearch log4j2 file?

[Jochen Kressin]

For an example you can refer to the Search Guard docs:

https://docs.search-guard.com/latest/troubleshooting-tls

So basically you create a new logger and a new appender in your log4j2.properties file.The appender specifies the file name and location amongst other settings, and is connected to the actual logger via an appenderRef. For example, have a look at how the deprecation logger is defined. Just follow this pattern, and replace the "deprecation" with, for example, "searchguard". The base packe for all SG relates classes is "com.floragunn".

logger.deprecation.name = org.elasticsearch.deprecation

logger.deprecation.level = warn

logger.deprecation.appenderRef.deprecation_rolling.ref = deprecation_rolling

logger.deprecation.additivity = false

appender.deprecation_rolling.type = RollingFile

appender.deprecation_rolling.name = deprecation_rolling

appender.deprecation_rolling.fileName = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}_deprecation.log

appender.deprecation_rolling.layout.type = PatternLayout

appender.deprecation_rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] %marker%.-10000m%n

appender.deprecation_rolling.filePattern = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}_deprecation-%i.log.gz

appender.deprecation_rolling.policies.type = Policies

appender.deprecation_rolling.policies.size.type = SizeBasedTriggeringPolicy

appender.deprecation_rolling.policies.size.size = 1GB

appender.deprecation_rolling.strategy.type = DefaultRolloverStrategy

appender.deprecation_rolling.strategy.max = 4

On Thursday, April 19, 2018 at 8:48:32 AM UTC-7, ch s wrote:

You don't have any examples of integrating this with the existing elasticsearch log4j2 file?

On April 19, 2018 10:45:12 AM CDT, SG <in...@search-guard.com> wrote:

of course you can do this. but as you said this is more log4j than SG related.
All loggers for searchguard start with com.floragunn or sg_action_trace (for auditlogs you can configure the loggername yourself)
So just route them into another file than the rest of the logs.

 Am 19.04.2018 um 17:14 schrieb ch s <chs...@gmail.com>:
 
 Anyone? If this isn't possible, that's fine, I'd just like to know.
 
 On Tuesday, April 17, 2018 at 11:09:56 AM UTC-5, ch s wrote:
 I guess this is more of a log4j2 question than anything else, but we have a requirement to let another team have access to our elasticsearch logs (And by this, I mean the log file generated by the elasticsearch process controlled by log4j2.properties, not the documents that are inside the elasticsearch cluster itself). The only problem is that the logs for the searchguard process are combined with the regular elasticsearch process logs. Is there a way to separate the two so that we can put the searchguard logs into a different file than the one elasticsearch is using for its own logs?
 
 I did experiment with the auditlog, but that still doesn't capture the actual searchguard process logs. It works great for audit events though!
 
 I'm guessing this would require some changes to log4j2.properties, but understanding that file is completely foreign to me.
 
 Thanks!
 
 
 * Search Guard and Elasticsearch version
 SG 5.6.7, ES 5.6.7
 
 * Installed and used enterprise modules, if any
 Auditlog
 
 
 
 -- 
 You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.

 To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

[CHS]

Thanks for the tips. Looks like I've got it mostly working. Logs are now going to a separate logfile, but some logs are still being captured in the ES log file.

For example (lines trimmed):

[2018-04-19T21:42:20,491][WARN ][c.f.d.a.l.b.LDAPAuthorizationBackend] Unable to connect to ldapserver due to[trimmed]

[2018-04-19T21:42:20,492][ERROR][c.f.s.a.BackendRegistry  ] Problems retrieving roles for User [name=admin, roles=[]] [trimmed]

[2018-04-19T21:51:19,555][WARN ][c.f.s.h.SearchGuardHttpServerTransport] [hostname] Someone (/127.0.0.1:50448) speaks http plaintext [trimmed]

Here's what I've got added to the log4j2.properties:

logger.searchguard.name = com.floragunn

logger.searchguard.level = warn

logger.searchguard.appenderRef.deprecation_rolling.ref = searchguard_rolling

logger.searchguard.additivity = false

appender.searchguard_rolling.type = RollingFile

appender.searchguard_rolling.name = searchguard_rolling

appender.searchguard_rolling.fileName = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}_searchguard.log

appender.searchguard_rolling.layout.type = PatternLayout

appender.searchguard_rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] %marker%.-10000m%n

appender.searchguard_rolling.filePattern = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}_searchguard-%i.log.gz

appender.searchguard_rolling.policies.type = Policies

appender.searchguard_rolling.policies.size.type = SizeBasedTriggeringPolicy

appender.searchguard_rolling.policies.size.size = 1GB

appender.searchguard_rolling.strategy.type = DefaultRolloverStrategy

appender.searchguard_rolling.strategy.max = 4

--
You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/sJ9J3cci1Qw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/54e4bf58-d749-4d91-98d2-e90644af962b%40googlegroups.com.

[SG]

i guess you need to set logger.searchguard.level = ALL here

> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/341eaa26-3c07-4255-8f08-bfd60f78793c%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/sJ9J3cci1Qw/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to search-guard...@googlegroups.com.

> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/54e4bf58-d749-4d91-98d2-e90644af962b%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

> To post to this group, send email to search...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/CALY-9dJR01e79dJgHeUdV1MJekg_bUY5XK%2BDq%2BFeRSbTLOcySQ%40mail.gmail.com.