Huge elasticsearch log files causing server to run out of space

[John Coquitlam]

Hi,

We are having problems with elasticsearch log files growing to close to and sometimes over 1GB every day.
Because of this, there have been a few times where our server ran out of space and bad gateway displayed on the browser.
Those files are in /var/log/elasticsearch. For the time being, we have compress those files but are there some issue with elasticsearch?
Our setup is as follows:

atom 2.6.1
elasticsearch 5.6.16
ubuntu 18.04.6
php 7.2.24
mysql 8.0.28
nginx 1.14.0

I have attached two screen captures showing an error message in elasticsearch.log and folder view.

Thanks,

John

[Dan Gillean]

Hi John, 

I found this page in the ES 5.6 documentation, which includes an example of how to configure automated rollover of logs, as well as how to configure logging levels. See: 

• https://www.elastic.co/guide/en/elasticsearch/reference/5.6/settings.html#logging
  

The times that our system administrators have encountered this kind of issue for hosted clients, configuring automated rollover and default logging levels is how they have addressed it. 

It's an edge case, but one of our developers also mentioned that if you've accidentally left Debug mode enabled in production, this could generate a lot of logs as well. See: 

• https://www.accesstomemory.org/docs/latest/admin-manual/maintenance/debug/
  

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/53d90498-7808-48a1-8b12-5152caecdad0n%40googlegroups.com.

[Jim Adamson]

Hi John,

Further to Dan's reply, you could take a look at a post of mine from last year, in which I include our customisations to the ES log4j2.properties file, designed to reduce disk space usage:

https://groups.google.com/g/ica-atom-users/c/meLEMsn0Xqk/m/KU7OCW0yCwAJ

I hope you find it useful.

Thanks, Jim

--

You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/53d90498-7808-48a1-8b12-5152caecdad0n%40googlegroups.com.

--

Jim Adamson

Systems Administrator/Developer
Facilities Management Systems
IT Services

LFA/023 | Harry Fairhurst building | University of York | Heslington | York | YO10 5DD

+44 (0)1904 323859

Email disclaimer: https://www.york.ac.uk/docs/disclaimer/email.htm

[John Coquitlam]

Hi Jim,

Thanks for the info.

I updated our log4j2 file to set action level to warn and appended gz to the appender.rolling.filePattern yesterday.

Checking this morning it did not appear to be working as there were no gz file created and the log file created was still huge.

Does the server need to be rebooted for this to take effect?

Thanks,

John

[Jim Adamson]

Hi John,

You shouldn't need a server reboot; just a service restart should suffice. On Ubuntu it's systemctl restart elasticsearch

Thanks, Jim

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/19dbdc7e-6e29-443c-9a34-0c0168908ee2n%40googlegroups.com.

[John Coquitlam]

Hi Jim,

After restarting the service on June 7, then on June 8 and 9 we notice no changes in the log folder content.

Yesterday morning (June 9), we rebooted the server and immediately a gz file was created but it was for June 7.

This morning we notice still no new log file has been created.

Could this be because we changed the action.level to warn?

Maybe I made a mistake in the properties file but I checked again.

I have attached screen captures of the log folder content and our changes in the properties file.

Thanks for your help.

Thanks,

John

[Jim Adamson]

Hi John,

Sorry for the late reply. Did you add the appender.rolling.strategy ... lines? Your screenshots don't show any. I attach a screenshot of a diff between the default file (left) and our customised one (right).

Thanks, Jim

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/1419211f-8861-4ca3-a011-f51d8fa17382n%40googlegroups.com.