Re: [rabbitmq-users] RabbitMQ 4.1.0 {badmatch,{error,eacces}}

67 views
Skip to first unread message

Michal Kuratczyk

unread,
Jun 10, 2025, 5:45:17 AMJun 10
to rabbitm...@googlegroups.com
We've reproduced this issue and merged a "fix" that should be included in the next release.

As far I can tell, this is caused by a strange Windows behaviour (bug?) where
closing a file doesn't immediately release the file handle, so an attempt to rename/delete
a file/folder immediately after closing the file might fail. Going forward, we'll just retry once after 20ms.
In my test env, it was enough to get rid of the error, but it still might happen, just much less
frequently.

For those who find this discussion when looking for solutions to the "eaccess" problem: by far the most
common reasons for such errors are incorrect filesystem permission settings and Windows security
features / additional security software installed.

Best,

On Tue, May 20, 2025 at 4:27 PM Michael Klishin <michael....@gmail.com> wrote:
"eacces" means the node cannot access one of the files it needs to use. On Windows, this almost certainly means that a security tool of software prevents RabbitMQ from doing it.
This is not "some log noise that can be ignored", this is a serious issue NOT in RabbitMQ but rather in your environment.

Windows is a supported OS but we have never recommended it for production, in part due to such obscure effects from anti-virus and other security tools,
and because monitoring and deployment automation on Linux are much more mature and widely used.
You can use the WSL and containers to run RabbitMQ without any of the Windows-specific complications [2].

Khepri is listed as a stable feature flag in the docs and for RabbitMQ 4.2, it will be (already is in the `main` branch) the default metadata store. That suggests are certain
level of confidence from the core team. That does not, however, suggest that we support or recommend running RabbitMQ with Khepri in environments where nodes would not be able to reliably access their local files
because of security tools, which is an extremely common thing to see amongst the Windows users.

Finally, here is your official statement straight from the MPLv2 license you use RabbitMQ under [1]: the software is provided as is, without a warranty of
any kind, you assume all the risks and complications associated with using free, open source software.
On Tuesday, May 20, 2025 at 6:58:56 AM UTC-4 drahos...@gmail.com wrote:
Hello,
I removed handle.exe, restarted cluster, nothing changed.

I would like to ask you for an official statement, if version 4.1 with khepri_db is supported on windows and is ready for production.
I'm asking because with version 3.12.6 we had serious cluster problems (sporadic and impossible to simulate), so I see use of current version with khepri_db as the only way.

Thanks,
Peter 

On Friday, 16 May 2025 at 19:14:50 UTC+2 Vilius Šumskas wrote:

Hi,

 

have you tried removing handle.exe utility from the system completely to see if this fixed the problem?

 

--

    Vilius

 

From: rabbitm...@googlegroups.com <rabbitm...@googlegroups.com> On Behalf Of Peter Drahoš
Sent: Thursday, May 15, 2025 2:35 PM
To: rabbitmq-users <rabbitm...@googlegroups.com>
Subject: Re: [rabbitmq-users] RabbitMQ 4.1.0 {badmatch,{error,eacces}}

 

Hello,

we removed symantec from 2 nodes, set permissions to db folder to everyone (with replace all child...). So security sw and permission problem can be excluded.

Problem is still there.

I traced snapshot folders for queue name which appeared in error log.

Error happend at 2025-05-15 12:38:28, snapshot folder with file has timestamp  2025-05-15 12:53. So move succeeded at later time. Is there some retry?

It seems to me like some moves are successfull and some not, so it seems like application problem (concurrency, ...).

 

One thing just for sure:

Before the symantec was removed, on one node in event viewer there were many logs like this:

Event: Tamper Protection Detection
Security risk detected: C:\WINDOWS\TEMP\HANDLE64.EXE

 

But there are no logs now and problem persists, so I guess it's not related to this problem. 

Does rabbitmq use it? 

 

Thanks,

Peter

On Thursday, 15 May 2025 at 09:30:39 UTC+2 Michal Kuratczyk wrote:

There were changes to the checkpointing/snapshotting logic, and indeed 4.1 like promotes snapshots more often than 4.0.7 did,

so that explains why the frequency of these errors changed. As for the error itself, there's really nothing we can do here:

eaccess means that RabbitMQ/Erlang tried to perform a filesystem operation and it was refused by the operating system.

We have a checkpoint file and we want to promote it to a snapshot status. On the filesystem level, it's just a file

rename/move operation: in the RabbitMQ's data folder, there is a folder for a given queue and it has "checkpoints"

and "snapshots" subfolders. This operation tries to move "checkpoints/SOME_FOLDER" to become "snapshots/SOME_FOLDER".

For some reason, your system doesn't allow that.

 

 

On Thu, May 15, 2025 at 9:00 AM Peter Drahoš <drahos...@gmail.com> wrote:

Hello,

it's default installation, so the windows service runs as Local System account. Data folder permission are correct and these folders are the same for years.

We have Symantec Endpoint Protection on servers, we looked into every log and there are no info about blocking, rejecting at all.

And the files in db folders like snapshot.dat have curret modified timestamp.

First I tried to upgrade existing node from 4.0.7 to 4.1.0 and this caused frequency of error occurrence from 2-3 in month to tens in day. So the rabbitmq upgrade was the only change on the server.

 

Thanks,

Peter

 

On Wednesday, 14 May 2025 at 15:31:00 UTC+2 Michal Kuratczyk wrote:

`eaccess` just means filesystem permissions don't allow some operations. RabbitMQ expects to have full access to its data folder.

Make sure your filesystem permissions are set correctly. Also, it's common on Windows for security software to interfere with

the software that's running (rejecting "suspicious" operations such as file access...), so it could be that.

 

 

On Wed, May 14, 2025 at 3:14 PM Peter Drahoš <drahos...@gmail.com> wrote:

Hello,

we have on premise cluster with 4 nodes on windows server 2019 (VMware). 

Versions: RabbitMQ 4.1.0 (khepri_db), Erlang 27.3.4

All queues are of type quorum.

Avg Message rates: <5 /s

.Net 8 apps with easynetq

 

I did clean install, after few hours these errors started to appear on some nodes (tens of occurrences in 24h). Applications are not affected, just the errors in logs.

I noticed the same errors also with rabbitmq 4.0.7 but very sporadic, also without affect on apps.

 

What does it mean?

It's safe for production environment?

 

2025-05-13 15:20:03.868000+02:00 [warning] <0.28945.0> queue 'BaseNotification_WhitelistQueue_Worker' in vhost 'DEV': Snapshot write process <0.226939.0> exited with {{badmatch,{error,eacces}},[{ra_snapshot,'-promote_checkpoint/2-fun-0-',5,[{file,[115,114,99,47,114,97,95,115,110,97,112,115,104,111,116,46,101,114,108]},{line,420}]}]}
2025-05-13 15:20:03.868000+02:00 [error] <0.226939.0> Error in process <0.226939.0> on node 'rabbit@vs-pro-odapl02' with exit value:
2025-05-13 15:20:03.868000+02:00 [error] <0.226939.0> {{badmatch,{error,eacces}},
2025-05-13 15:20:03.868000+02:00 [error] <0.226939.0>  [{ra_snapshot,'-promote_checkpoint/2-fun-0-',5,
2025-05-13 15:20:03.868000+02:00 [error] <0.226939.0>                [{file,"src/ra_snapshot.erl"},{line,420}]}]}

 

Thank you,

Peter

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/rabbitmq-users/d759b58e-fb08-42ce-8f70-dd7dd38aea87n%40googlegroups.com.


 

--

Michal

RabbitMQ Team


This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.


 

--

Michal

RabbitMQ Team


This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/rabbitmq-users/1c3a668e-d598-4580-8945-ab524bf5fa43n%40googlegroups.com.


--
Michal
RabbitMQ Team

This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.
Reply all
Reply to author
Forward
0 new messages