Need clear documentation on how to make slf4j use the ESAPI html encoder automatically

[David Karr]

We use slf4j, and we're trying to use the html encoder in ESAPI.  Initially, we were using version 2.1.0.1 of ESAPI. I asked about the slf4j integration on StackOverflow, and I was told that version 2.2.0.0 has direct support for slf4j.  The problem is, except for the github issue mentioning this, I can't find any documentation that shows how to actually do this.  From some implications in the issue thread, I imagine it might be setting the ESAPI.Logger to Slf4JLogFactory, but that's just a guess.

What am I missing?

[Kevin W. Wall]

David,

I'm going to assume that:

1. You already have SLF4J configured to work separately by your application, but just don't have it configured for ESAPI to use it.
2. You already have ESAPI logging to log correctly in some other way (i.e., it's already logging with either java.util.logging or log4j 1.x).

In that case, I think that all you should need to do find the "ESAPI.Logger" property in your ESAPI.properties file and change it to:

ESAPI.Logger=org.owasp.esapi.logging.slf4j.Slf4JLogFactory

which is documented in ESAPI.properties file itself. (By the way, make sure you pull the ESAPI.properties file from GitHub from the 'configuration/esapi/ESAPI.properties' file and the test one from 'src/test/resources/esapi' which is prominently marked 'TEST VERS' at the top. Unfortunately, we haven't figured out how to get a zip file of that and other files under 'configuration/' directory uploaded to Maven Central even though we've tried many different ways to get that done._

Otherwise, it should be used just like you normally use ESAPI logging just as you normally would if you were using it with JUL or log4j 1.x.

@Jeremiah: Did I leave anything out?

-kevin

On Sun, Oct 13, 2019 at 2:19 AM David Karr <davidmic...@gmail.com> wrote:

We use slf4j, and we're trying to use the html encoder in ESAPI.  Initially, we were using version 2.1.0.1 of ESAPI. I asked about the slf4j integration on StackOverflow, and I was told that version 2.2.0.0 has direct support for slf4j.  The problem is, except for the github issue mentioning this, I can't find any documentation that shows how to actually do this.  From some implications in the issue thread, I imagine it might be setting the ESAPI.Logger to Slf4JLogFactory, but that's just a guess.

What am I missing?

--
You received this message because you are subscribed to the Google Groups "ESAPI Project Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to esapi-project-u...@owasp.org.
To view this discussion on the web visit https://groups.google.com/a/owasp.org/d/msgid/esapi-project-users/cf1a9130-610a-4183-bfe8-9350d0b42ae0%40owasp.org.

--

Blog: http://off-the-wall-security.blogspot.com/    | Twitter: @KevinWWall
NSA: All your crypto bit are belong to us.

[David Karr]

On Sunday, October 13, 2019 at 9:46:48 AM UTC-7, Kevin W. Wall wrote:

David,

I'm going to assume that:

1. You already have SLF4J configured to work separately by your application, but just don't have it configured for ESAPI to use it.

Yes, we get Loggers from the slf4j LoggerFactory class. We have ESAPI available, but all we're doing with it so far is calling the encoders directly, like "ESAPI.encoder().encodeForHTML(...)".

1. You already have ESAPI logging to log correctly in some other way (i.e., it's already logging with either java.util.logging or log4j 1.x).

I assume you're listing these as "either this one or that one", not both. I don't believe we have the latter.

In that case, I think that all you should need to do find the "ESAPI.Logger" property in your ESAPI.properties file and change it to:

ESAPI.Logger=org.owasp.esapi.logging.slf4j.Slf4JLogFactory

We do have a ESAPI.properties file that sets a handful of ESAPI properties.  I've added this line to that file.

which is documented in ESAPI.properties file itself. (By the way, make sure you pull the ESAPI.properties file from GitHub from the 'configuration/esapi/ESAPI.properties' file and the test one from 'src/test/resources/esapi' which is prominently marked 'TEST VERS' at the top. Unfortunately, we haven't figured out how to get a zip file of that and other files under 'configuration/' directory uploaded to Maven Central even though we've tried many different ways to get that done._

Otherwise, it should be used just like you normally use ESAPI logging just as you normally would if you were using it with JUL or log4j 1.x.

@Jeremiah: Did I leave anything out?

-kevin

After I made that change, I executed a line like

    log.debug("stuff {}", object)

where I had deliberately set one of the properties of the object to have html text in it, and the result had the unmodified html text in it.  I then instead executing a line like

    log.debug( "stuff {}", ESAPI.encoder().encodeForHTML(object.toString())

And the result had the html text properly encoded.

I also set breakpoints in the ESAPI code, and the first execution never hit any of them.  The latter did hit several. Note that our ESAPI.properties file has

    Encoder.DefaultCodecList=HTMLEntityCodec,PercentCodec,JavaScriptCodec

Which I would guess means that it should do the html encoding implicitly.

So, either I'm misunderstanding and misinterpreting what this ESAPI integration is supposed to do, or I'm missing a step.

On Sun, Oct 13, 2019 at 2:19 AM David Karr <davidmic...@gmail.com> wrote:

We use slf4j, and we're trying to use the html encoder in ESAPI.  Initially, we were using version 2.1.0.1 of ESAPI. I asked about the slf4j integration on StackOverflow, and I was told that version 2.2.0.0 has direct support for slf4j.  The problem is, except for the github issue mentioning this, I can't find any documentation that shows how to actually do this.  From some implications in the issue thread, I imagine it might be setting the ESAPI.Logger to Slf4JLogFactory, but that's just a guess.

What am I missing?

--
You received this message because you are subscribed to the Google Groups "ESAPI Project Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to esapi-project-users+unsub...@owasp.org.

To view this discussion on the web visit https://groups.google.com/a/owasp.org/d/msgid/esapi-project-users/cf1a9130-610a-4183-bfe8-9350d0b42ae0%40owasp.org.

[David Karr]

On Sunday, October 13, 2019 at 10:20:12 AM UTC-7, David Karr wrote:

On Sunday, October 13, 2019 at 9:46:48 AM UTC-7, Kevin W. Wall wrote:

David,

I'm going to assume that:

1. You already have SLF4J configured to work separately by your application, but just don't have it configured for ESAPI to use it.

Yes, we get Loggers from the slf4j LoggerFactory class. We have ESAPI available, but all we're doing with it so far is calling the encoders directly, like "ESAPI.encoder().encodeForHTML(...)".

1. You already have ESAPI logging to log correctly in some other way (i.e., it's already logging with either java.util.logging or log4j 1.x).

I assume you're listing these as "either this one or that one", not both. I don't believe we have the latter.

In that case, I think that all you should need to do find the "ESAPI.Logger" property in your ESAPI.properties file and change it to:

ESAPI.Logger=org.owasp.esapi.logging.slf4j.Slf4JLogFactory

We do have a ESAPI.properties file that sets a handful of ESAPI properties.  I've added this line to that file.

which is documented in ESAPI.properties file itself. (By the way, make sure you pull the ESAPI.properties file from GitHub from the 'configuration/esapi/ESAPI.properties' file and the test one from 'src/test/resources/esapi' which is prominently marked 'TEST VERS' at the top. Unfortunately, we haven't figured out how to get a zip file of that and other files under 'configuration/' directory uploaded to Maven Central even though we've tried many different ways to get that done._

Otherwise, it should be used just like you normally use ESAPI logging just as you normally would if you were using it with JUL or log4j 1.x.

@Jeremiah: Did I leave anything out?

-kevin

After I made that change, I executed a line like

    log.debug("stuff {}", object)

where I had deliberately set one of the properties of the object to have html text in it, and the result had the unmodified html text in it.  I then instead executing a line like

    log.debug( "stuff {}", ESAPI.encoder().encodeForHTML(object.toString())

And the result had the html text properly encoded.

I also set breakpoints in the ESAPI code, and the first execution never hit any of them.  The latter did hit several. Note that our ESAPI.properties file has

    Encoder.DefaultCodecList=HTMLEntityCodec,PercentCodec,JavaScriptCodec

Which I would guess means that it should do the html encoding implicitly.

So, either I'm misunderstanding and misinterpreting what this ESAPI integration is supposed to do, or I'm missing a step.

And none of the breakpoints I set in Slf4JLogFactory are getting hit, so I'm obviously missing something important here.

[Jeremiah Stacey]

In order for the HTMLCodec to be applied, the property 'Logger.LogEncodingRequired' must also be set to true in ESAPI.properties.

Without that value, only newlines are replaced.

[Kevin W. Wall]

David,

That property Jeremiah mentioned is disabled by default as most don't use HTML viewers for examining ESAPI log entries.

The property isn't new / unique to the SLF4J logger support though. That property has been there since at least 2016-01-16, so the 2.1.0.1 release which was released on 2016-02-05. It might even predate that.

-kevin

--
Blog: http://off-the-wall-security.blogspot.com/  |  Twitter:  @KevinWWall
NSA: All your crypto bit are belong to us.

To unsubscribe from this group and stop receiving emails from it, send an email to esapi-project-u...@owasp.org.

To view this discussion on the web visit https://groups.google.com/a/owasp.org/d/msgid/esapi-project-users/cf1a9130-610a-4183-bfe8-9350d0b42ae0%40owasp.org.

--

Blog: http://off-the-wall-security.blogspot.com/    | Twitter: @KevinWWall
NSA: All your crypto bit are belong to us.

--

You received this message because you are subscribed to the Google Groups "ESAPI Project Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to esapi-project-u...@owasp.org.
To view this discussion on the web visit https://groups.google.com/a/owasp.org/d/msgid/esapi-project-users/0e25b64d-382c-0427-4e9b-793fa970868d%40gmail.com.

[David Karr]

On Sunday, October 13, 2019 at 3:51:50 PM UTC-7, Jeremiah Stacey wrote:

In order for the HTMLCodec to be applied, the property 'Logger.LogEncodingRequired' must also be set to true in ESAPI.properties.

Without that value, only newlines are replaced.

Thanks, but there's obviously something else going on here.  I set that property, but it had no effect. It also didn't encode newlines.

The fact that I mentioned earlier, that none of the breakpoints in Slf4JLogFactory are being hit, is pretty critical. There's a missing element that is resulting in slf4j not using ESAPI at all.

[Jeremiah Stacey]

Reading back through this, I would like to offer a clarification.

• ESAPI logging delegates to SLF4J

• As with all of the ESAPI LogFactory implementations, you must use the ESAPI Logging API in your application to gain the benefits.

The way the subject line reads and from the logging snippets you've offered I get the impression you're using the SLF4J logging API in your application and want to have ESAPI receive the log message and escape it?

The behavior is actually inverse from that.  Your application must use the ESAPI Logging API, and the ESAPI Logger can delegate to a configured SLF4J configuration. 

The escaping happens before SLF4J, not after.

Please let me know if that changes the issue at all.  I will do my best to continue debugging this evening.

To unsubscribe from this group and stop receiving emails from it, send an email to esapi-project-u...@owasp.org.
To view this discussion on the web visit https://groups.google.com/a/owasp.org/d/msgid/esapi-project-users/e2b32ca3-aafd-4125-978c-6afcd1025b18%40owasp.org.

[David Karr]

On Monday, October 14, 2019 at 3:56:53 AM UTC-7, Jeremiah Stacey wrote:

Reading back through this, I would like to offer a clarification.

• ESAPI logging delegates to SLF4J

• As with all of the ESAPI LogFactory implementations, you must use the ESAPI Logging API in your application to gain the benefits.

The way the subject line reads and from the logging snippets you've offered I get the impression you're using the SLF4J logging API in your application and want to have ESAPI receive the log message and escape it?

The behavior is actually inverse from that.  Your application must use the ESAPI Logging API, and the ESAPI Logger can delegate to a configured SLF4J configuration. 

The escaping happens before SLF4J, not after.

Please let me know if that changes the issue at all.  I will do my best to continue debugging this evening.

I see.  Yes, that completely changes the issue. I was under the impression I could get this done without any code changes, but the result is exactly the opposite. It might have been reasonable if the ESAPI logging interface was functionally identical to the slf4j interface, but it is not. Besides changing the package for Logger and LogFactory, I will have to change every single log call that takes parameters into the string, which is almost all of them. We extensively use the slf4j feature that allows "late evaluation" of the format parameters, which saves some performance for things like the "debug" logging level, when "info" level is typically used by default.

My alternative, which is what I started doing, is changing every parameter to manually call the ESAPI encoder. I don't know which option is worse.

To view this discussion on the web visit https://groups.google.com/a/owasp.org/d/msgid/esapi-project-users/e2b32ca3-aafd-4125-978c-6afcd1025b18%40owasp.org.

[Kevin W. Wall]

David,

Comments inline, below.

On Mon, Oct 14, 2019 at 9:32 AM David Karr <davidmic...@gmail.com> wrote:

On Monday, October 14, 2019 at 3:56:53 AM UTC-7, Jeremiah Stacey wrote:

Reading back through this, I would like to offer a clarification.

• ESAPI logging delegates to SLF4J

• As with all of the ESAPI LogFactory implementations, you must use the ESAPI Logging API in your application to gain the benefits.

The way the subject line reads and from the logging snippets you've offered I get the impression you're using the SLF4J logging API in your application and want to have ESAPI receive the log message and escape it?

The behavior is actually inverse from that.  Your application must use the ESAPI Logging API, and the ESAPI Logger can delegate to a configured SLF4J configuration. 

The escaping happens before SLF4J, not after.

Please let me know if that changes the issue at all.  I will do my best to continue debugging this evening.

I see.  Yes, that completely changes the issue. I was under the impression I could get this done without any code changes, but the result is exactly the opposite. It might have been reasonable if the ESAPI logging interface was functionally identical to the slf4j interface, but it is not.

That was never an option. We had still had to support the previously supported loggers (JUL and log4j 1.x). In some future incarnation of ESAPI 3.0, we are going to be discarding backward compatibility and at that time may decide to just use something like SLF4J directly, but that is still a long way off. And rather than trying to make ESAPI 3.x backward compatible with 2.x, we probably will take a similar tactic to Apache Struts and just rename the packages from org.owasp.esapi to org.owasp.esapi3 or something similar. That way they can co-exist while developers migrate from 2.x to 3.x. But again, that's still a ways off.

Besides changing the package for Logger and LogFactory, I will have to change every single log call that takes parameters into the string, which is almost all of them.

The difficulty in output encoding is knowing a priori which context (between HTML tags, in HTML attributes, CSS, JavaScript, or URLs) is going to be used, since those have different concerns and/or ways of output encoding. (That's why we tell people not to do output encoding and then shove the encoded output into a DB because someone may extract that encoded data from the DB and use it in a different context from where you intended it.) But maybe, for the logging that you want to do, there is only a single context (e.g., between HTML tags) and your fixed (static) input contains no HTML markup???

In that case, maybe the best approach would be to just write a custom log appender or a customized subclass of java.io.OutputStream and get SLF4J to use it instead whatever it is using by default.

We extensively use the slf4j feature that allows "late evaluation" of the format parameters, which saves some performance for things like the "debug" logging level, when "info" level is typically used by default.

In most production environments that I've seen, debug logging is disabled and the log level is set to 'warn' or maybe 'info' and if they way to enable debug logging, they require a Change Request or similar ticket to enable debugging. (In part, because oftentimes, debug logging will log things like confidential consumer data, etc. that should not normally be exposed in the logs.) In those situations, it is customary to use explicit checks before doing the logging as in:

if ( logger.isDebug() ) {

logger.debug(" ... some expensive debug logging construct here ... ");

}

and if debug logging is not enabled, the JIT optimizes out this code rather quickly. But even if you did that, it involves a code rewrite (although, with some thought, you probably could automate it). But instead, it looks as though you have decided to marry yourself to SLF4J's "late evaluation" of formatting parameters. (Makes me wish for the days of macros like C/C++'s pre-processor or M4.)

One last approach that may help you if you could maybe use a aspect-oriented programming approach for logging (like Spring AOP and/or AspectJ). For example, something like is described here: https://www.yegor256.com/2014/06/01/aop-aspectj-java-method-logging.html. That likely involves a bigger effort, but in the long term may simplify things.

Hope that some of these ideas at least are a bit helpful.

Best regards,

-kevin

[Jeremiah Stacey]

An additional option is to use the log4j2 implementation for SLF4J. 

https://logging.apache.org/log4j/2.x/log4j-slf4j-impl/index.html

In your log4j2 configuration,you can use the `%enc{%m}` syntax to encode your message body which will prevent certain characters from rendering in your logs.

https://logging.apache.org/log4j/2.x/manual/layouts.html

Search for 'encode{pattern}' on the page for more information on the specific characters.

Hope that some of these ideas at least are a bit helpful.

Best regards,

-kevin

--
You received this message because you are subscribed to the Google Groups "ESAPI Project Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to esapi-project-u...@owasp.org.
To view this discussion on the web visit https://groups.google.com/a/owasp.org/d/msgid/esapi-project-users/CAOPE6PhLzKsX4PR%3Dd5Ah5_uWGN1nrso%2BOWk91gzAc9TmYK2NQQ%40mail.gmail.com.