Huge elasticsearch log files causing server to run out of space

48 views
Skip to first unread message

John Coquitlam

unread,
Jun 3, 2022, 2:30:10 PMJun 3
to AtoM Users
Hi,
We are having problems with elasticsearch log files growing to close to and sometimes over 1GB every day.
Because of this, there have been a few times where our server ran out of space and bad gateway displayed on the browser.
Those files are in /var/log/elasticsearch. For the time being, we have compress those files but are there some issue with elasticsearch?
Our setup is as follows:

atom 2.6.1
elasticsearch 5.6.16
ubuntu 18.04.6
php 7.2.24
mysql 8.0.28
nginx 1.14.0

I have attached two screen captures showing an error message in elasticsearch.log and folder view.

Thanks,
John
elasticsearch_log.png
elasticsearch_error.png

Dan Gillean

unread,
Jun 3, 2022, 3:36:33 PMJun 3
to ICA-AtoM Users
Hi John, 

I found this page in the ES 5.6 documentation, which includes an example of how to configure automated rollover of logs, as well as how to configure logging levels. See: 
The times that our system administrators have encountered this kind of issue for hosted clients, configuring automated rollover and default logging levels is how they have addressed it. 

It's an edge case, but one of our developers also mentioned that if you've accidentally left Debug mode enabled in production, this could generate a lot of logs as well. See: 
Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056
@accesstomemory
he / him


--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/53d90498-7808-48a1-8b12-5152caecdad0n%40googlegroups.com.

Jim Adamson

unread,
Jun 6, 2022, 11:15:34 AMJun 6
to ica-ato...@googlegroups.com
Hi John,

Further to Dan's reply, you could take a look at a post of mine from last year, in which I include our customisations to the ES log4j2.properties file, designed to reduce disk space usage:


I hope you find it useful.

Thanks, Jim

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/53d90498-7808-48a1-8b12-5152caecdad0n%40googlegroups.com.


--
Jim Adamson
Systems Administrator/Developer
Facilities Management Systems
IT Services
LFA/023 | Harry Fairhurst building | University of York | Heslington | York | YO10 5DD

John Coquitlam

unread,
Jun 7, 2022, 11:34:30 AMJun 7
to AtoM Users
Hi Jim,
Thanks for the info.
I updated our log4j2 file to set action level to warn and appended gz to the appender.rolling.filePattern yesterday.
Checking this morning it did not appear to be working as there were no gz file created and the log file created was still huge.
Does the server need to be rebooted for this to take effect?

Thanks,
John

Jim Adamson

unread,
Jun 7, 2022, 12:22:11 PMJun 7
to ica-ato...@googlegroups.com
Hi John,

You shouldn't need a server reboot; just a service restart should suffice. On Ubuntu it's systemctl restart elasticsearch

Thanks, Jim

John Coquitlam

unread,
Jun 10, 2022, 12:00:49 PMJun 10
to AtoM Users
Hi Jim,
After restarting the service on June 7, then on June 8 and 9 we notice no changes in the log folder content.
Yesterday morning (June 9), we rebooted the server and immediately a gz file was created but it was for June 7.
This morning we notice still no new log file has been created.
Could this be because we changed the action.level to warn?
Maybe I made a mistake in the properties file but I checked again.
I have attached screen captures of the log folder content and our changes in the properties file.
Thanks for your help.

Thanks,
John
elasticsearch_properties.PNG
elasticsearch_log_jun10.PNG

Jim Adamson

unread,
Jun 22, 2022, 6:58:05 AM (7 days ago) Jun 22
to ica-ato...@googlegroups.com
Hi John,

Sorry for the late reply. Did you add the appender.rolling.strategy ... lines? Your screenshots don't show any. I attach a screenshot of a diff between the default file (left) and our customised one (right).

Thanks, Jim

2022-06-22 11_53_02-Comparison_of_default_and_customised_log4j2_properties.png
Reply all
Reply to author
Forward
0 new messages