Wazuh Indexer 4.5.1 error: No user wzhmonitor found

[Carlos Lopez]

Hi all,

I have upgraded my wazuh infrastructure from 4.4.5 to 4.5.1. All it is working ok except on wazuh indexers. I am using three wazuh indexers hosts with LDAP authentication. On these nodes the following error appears:

[2023-09-04T10:11:51,727][ERROR][o.o.s.a.BackendRegistry ] [rhelose01.domain.com] Cannot retrieve roles for User [name=wzhmonitor, backend_roles=[admin], requestedTenant=null] from ldap due to OpenSearchSecurityException[OpenSearchSecurityException[No user wzhmonitor found]]; nested: OpenSearchSecurityException[No user wzhmonitor found];
org.opensearch.OpenSearchSecurityException: OpenSearchSecurityException[No user wzhmonitor found]
at com.amazon.dlic.auth.ldap.backend.LDAPAuthorizationBackend.fillRoles(LDAPAuthorizationBackend.java:946) ~[opensearch-security-2.6.0.0.jar:2.6.0.0]
at org.opensearch.security.auth.BackendRegistry.authz(BackendRegistry.java:456) [opensearch-security-2.6.0.0.jar:2.6.0.0]
at org.opensearch.security.auth.BackendRegistry$5.call(BackendRegistry.java:493) [opensearch-security-2.6.0.0.jar:2.6.0.0]
at org.opensearch.security.auth.BackendRegistry$5.call(BackendRegistry.java:486) [opensearch-security-2.6.0.0.jar:2.6.0.0]
at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4853) [guava-30.0-jre.jar:?]
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529) [guava-30.0-jre.jar:?]
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278) [guava-30.0-jre.jar:?]
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155) [guava-30.0-jre.jar:?]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045) [guava-30.0-jre.jar:?]
at com.google.common.cache.LocalCache.get(LocalCache.java:3951) [guava-30.0-jre.jar:?]
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4848) [guava-30.0-jre.jar:?]
at org.opensearch.security.auth.BackendRegistry.authcz(BackendRegistry.java:486) [opensearch-security-2.6.0.0.jar:2.6.0.0]
at org.opensearch.security.auth.BackendRegistry.authenticate(BackendRegistry.java:295) [opensearch-security-2.6.0.0.jar:2.6.0.0]
at org.opensearch.security.filter.SecurityRestFilter.checkAndAuthenticateRequest(SecurityRestFilter.java:191) [opensearch-security-2.6.0.0.jar:2.6.0.0]
at org.opensearch.security.filter.SecurityRestFilter$1.handleRequest(SecurityRestFilter.java:124) [opensearch-security-2.6.0.0.jar:2.6.0.0]
at org.opensearch.rest.RestController.dispatchRequest(RestController.java:312) [opensearch-2.6.0.jar:2.6.0]
at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:398) [opensearch-2.6.0.jar:2.6.0]
at org.opensearch.rest.RestController.dispatchRequest(RestController.java:241) [opensearch-2.6.0.jar:2.6.0]
at org.opensearch.security.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [opensearch-security-2.6.0.0.jar:2.6.0.0]
at org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:366) [opensearch-2.6.0.jar:2.6.0]
at org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:445) [opensearch-2.6.0.jar:2.6.0]
at org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:356) [opensearch-2.6.0.jar:2.6.0]
at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:55) [transport-netty4-client-2.6.0.jar:2.6.0]
at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:41) [transport-netty4-client-2.6.0.jar:2.6.0]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:71) [transport-netty4-client-2.6.0.jar:2.6.0]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.87.Final.jar:4.1.87.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1373) [netty-handler-4.1.87.Final.jar:4.1.87.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) [netty-handler-4.1.87.Final.jar:4.1.87.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) [netty-handler-4.1.87.Final.jar:4.1.87.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) [netty-codec-4.1.87.Final.jar:4.1.87.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) [netty-codec-4.1.87.Final.jar:4.1.87.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) [netty-codec-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.87.Final.jar:4.1.87.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.87.Final.jar:4.1.87.Final]
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.opensearch.OpenSearchSecurityException: No user wzhmonitor found

wzhmonitor user appears under Security/Internal Users .... but this user does not exist in my LDAP servers, obviously. How can I fix this error? Do I need to setup wzhmonitor user in my LDAP servers? And do I need to assign a password?

Best regards,
C. L. Martinez

[David José Iglesias Lopez]

Hello Carlos,

Based on a quick research about your error, and after reading https://forum.opensearch.org/t/opensearchsecurityexception-no-user-admin-found/8500, https://forum.opensearch.org/t/ldap-backend-roles-authz-not-working/13982/10 and https://forum.opensearch.org/t/users-from-internal-db-checked-over-ldap/1112, it looks like the solution is to add the internal user to the skip_users section:

You have to add internal user list under the autnz/config in the following form:
skip_users: - <wzhmonitor>

Please let me know if this helps!

[Carlos Lopez]

Thanks david but it doesn't seem to work .... I have added wzhmonitor to skip_users, but nothing. Same error:

skip_users:
# - 'cn=Michael Jackson,ou*people,o=TEST'
# - '/\S*/'
- admin
- kibanaserver
- kibanaro
- logstash
- readall
- snapshotrestore
- wzhmonitor

[David José Iglesias Lopez]

I am sorry to hear that Carlos,

I will ask the team and keep working on it to try and help you further. I would suggest you also ask in the OpenSearch community since the error looks very specific to that.