CVE-2021-44228 - remote code execution - Elasticsearch 5.6.16-1

[Kevin Bowrin]

Hello all,

I'm sure CVE-2021-44228 is causing some headaches this morning. It's a critical remote code execution vulnerability in log4j, a logging library built into most Java applications.

Is elasticsearch 5.6.16-1 (installed from the https://artifacts.elastic.co/packages/5.x package repository per https://www.accesstomemory.org/en/docs/2.6/admin-manual/installation/linux/ubuntu-bionic/) vulnerable to CVE-2021-44228? That vulnerability might be mitigated by the fact that most AtoM users don't interact with elasticsearch directly.

Is an AtoM installation (latest 2.6.4) which uses elasticsearch 5.6.16 vulnerable?

Thanks everyone,

Kevin Bowrin
Carleton University Library

[Kevin Bowrin]

One mitigation strategy: On our AtoM installations, elasticsearch is started using systemd. The file `/etc/sysconfig/elasticsearch` can be edited to change the options passed to Java when elasticsearch is starting.

Change the file so that the line:

#ES_JAVA_OPTS

has this option:

ES_JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true"

then restart the service.

As far as I can tell, elasticsearch 5.6.16 is using log4j 2.11.1. This version is newer than 2.10.0, so we can use this mitigation strategy. (https://www.lunasec.io/docs/blog/log4j-zero-day/)

[Brandon Uhlman]

Hi, Kevin.

Another possible mitigation strategy: I downloaded the log4j 2.15.0 binary release (which is supposed to be patched for this CVE), stopped ElasticSearch, swapped out the "offending" log4j JARs at /usr/share/elasticsearch/lib [1] with the equivalents from 2.15.0 in all three of our AtoM instances, and restarted. ElasticSearch still seems to be logging and searching fine -- admittedly not a highly scientific testing strategy.

Brandon

[1] That's log4j-1.2-api-2.11.1.jar, log4j-api-2.11.1.jar, log4j-core-2.11.1.jar for anyone playing the home game.

[Karl Goetz]

Hi Kevin,

I don't have a canonical answer, I will just note that the doco (https://www.accesstomemory.org/en/docs/2.6/admin-manual/maintenance/populate-search-index/) says "AtoM maintains an Elasticsearch search index to provide fast, full-text search results with faceting.".

In that scenario your attack surface is likely to hinge on how everything is configured. If you:

• Have elasticsearch
• Index user supplied data (like the full text being indexed)
• Log everything being indexed

Then you would be at risk. I do not believe that last point holds true for a standard AtoM install, so in a sense you'll have to "opt in" to being attacked this way using AtoM.

Karl.

---

From: ica-ato...@googlegroups.com <ica-ato...@googlegroups.com> on behalf of Kevin Bowrin <kjbo...@gmail.com>
Sent: Saturday, 11 December 2021 2:16 AM
To: AtoM Users <ica-ato...@googlegroups.com>
Subject: [atom-users] CVE-2021-44228 - remote code execution - Elasticsearch 5.6.16-1

 

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/09e8acea-645d-4a15-ba1f-a7ed013b5cb7n%40googlegroups.com.

This email is confidential, and is for the intended recipient only. Access, disclosure, copying, distribution, or reliance on any of it by anyone outside the intended recipient organisation is prohibited and may be a criminal offence. Please delete if obtained in error and email confirmation to the sender. The views expressed in this email are not necessarily the views of the University of Tasmania, unless clearly intended otherwise.

[Jim Adamson]

Given the urgency of this, my advice would be to mitigate or upgrade as soon as possible. The version of log4j2 (2.11.1) that comes with elasticsearch 5.6.16 for the current release of AtoM (2.6) is definitely vulnerable. Whether it's possible to exploit this against the default AtoM setup is an unanswered question, but not worth hanging around for an answer!

I'm sure Dan will post something soon (he's aware of the vulnerability), but if folks are worried and eager to ensure their system is not compromised, follow either of the mitigation strategies posted in the previous posts by Kevin and Brandon.

Thanks, Jim

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/ME2PR01MB42751C4AA3A545B1757D6A3FD0739%40ME2PR01MB4275.ausprd01.prod.outlook.com.

--

Jim Adamson

Systems Administrator/Developer
Facilities Management Systems
IT Services

LFA/023 | Harry Fairhurst building | University of York | Heslington | York | YO10 5DD

+44 (0)1904 323859

Email disclaimer: https://www.york.ac.uk/docs/disclaimer/email.htm

[r.alex....@gmail.com]

FYI, it does appear possible to execute this against AtoM 2.6.4. :(

I've implemented both mitigation strategies and security scans suggest that ours is no longer vulnerable.

Alex

[Dan Gillean]

Hi everyone, 

Thank you for all the proactive discussion and workarounds shared here. 

Elastic is putting out two new ES releases today, but they are patches for ES 6 and 7 versions. Unfortunately at this time, AtoM 2.6 still uses ES 5.x, and 2.7 is close enough to release that we will likely not be able to upgrade ES in that version either. 

However, the workaround suggested earlier in this thread by Kevin is also endorsed by the Elastic team - for further details, please see: 

• https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476
  

Affected Versions:
Elasticsearch versions 5.0.0+ contain a vulnerable version of Log4j. We’ve confirmed that the Security Manager mitigates the remote code execution attack in Elasticsearch 6 and 7; investigation is still underway for Elasticsearch 5.

Solutions and Mitigations:
Set the JVM option -Dlog4j2.formatMsgNoLookups=true

For now, even if you are not using Log4j, we recommend that AtoM users implement this fix to ensure that their installation is secure. We will continue investigating additional options and impacts, and will post further information as it becomes relevant.  

I think Brandon's suggestion of manually upgrading your Log4j version is also a good idea, but I would also suggest implementing the JVM option described above for peace of mind. 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/d0d8d36f-08df-4a29-a66c-609a9d105116n%40googlegroups.com.

[BrzI Channel]

Hi Kevin,

I cannot find /etc/sysconfig/elasticsearch

which of these should I edit  ?

[Service]
Environment=ES_HOME=/usr/share/elasticsearch
Environment=CONF_DIR=/etc/elasticsearch
Environment=DATA_DIR=/var/lib/elasticsearch
Environment=LOG_DIR=/var/log/elasticsearch
Environment=PID_DIR=/var/run/elasticsearch
EnvironmentFile=-/etc/default/elasticsearch

It looks like I nee dot change the last one EnvironmentFile=-/etc/default/elasticsearch. Can somebody please confirm ?

Thanks

[Dan Gillean]

Hi all, 

Our team has now put together a brief FAQ on this issue, with a recommended fix. I wanted to share it in this thread since it is different than what's been recommended here so far. Please see: 

• https://wiki.accesstomemory.org/Development/Security/Log4j-2021-12
  

I've made a separate post in the forum with this link as well. As noted in that announcement, the recommendation on our wiki is not the only way to patch the issue - it's simply the easiest and least disruptive method our team has found and tested for production installations. 

For older versions of AtoM: 

• This post includes information to help you discover what version of Log4j you have installed: https://stackoverflow.com/a/70324648
• This Elasticsearch documentation should help you find the relevant configuration file: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/settings.html

While older (1.x) versions of Log4j may not be vulnerable to this particular issue, there are other known issues with these Log4j releases, and ideally, you should update them as soon as possible. 

Regards, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

On Mon, Dec 13, 2021 at 5:17 PM BrzI Channel <brzic...@gmail.com> wrote:

Hi,

can you tell me where this file is in AtoM 2.5.3 - 172. I cannot find it in that location.

Thanks

On Friday, December 10, 2021 at 8:04:03 AM UTC-8 Kevin Bowrin wrote:

--

You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/fe41c334-f039-427b-9029-0d9cb64e75f4n%40googlegroups.com.

[Steve Lapommeray]

Hi Dan,

I don't thnk the zip option works for all 2.x versions of log4j as is stated in the security announcement. I have log4j core 2.11.1 and the zip command returns a "Nothing to do" message.

Thanks,

Steve

[Jim Adamson]

Hi Steve,

Are you certain you're in the correct working directory? (/usr/share/elasticsearch/lib). And can you list the directory (ls -l) to check that log4j-core-2.11.1.jar does indeed exist there? The zip error: Nothing to do! message indicates that it has not found a file that matches the glob pattern log4j-core-*.jar included in the zip command line.

Thanks, Jim

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/0990fd4b-555e-4a5d-bf3d-4abf109d5c26n%40googlegroups.com.

[Steve Lapommeray]

Hi Jim,

I double-checked after your comment and it had to do with the way I was running the command via Ansible. Running it manually on the command line worked fine. Thanks!

Steve