jboss cusom json logging format. MDC values in parent keys

[Baitanik Talukder]

Hi , 
i need a way to print the log in json format 

1. stack trace should be in single line.. should not create new lines

2. MDC values should come in the json parent keys . 

I have tried with :

/subsystem=logging/json-formatter=json_format:add(exception-output-type=formatted, key-overrides=[timestamp="time"], meta-data=[@version=1]) 

/subsystem=logging/console-handler=CONSOLE:write-attribute(name="named-formatter", value="json_format")
 

which achieves the 1st criteria but fails to do the 2nd one.. 

I have also tried with 

/subsystem=logging/console-handler=CONSOLE:write-attribute(name="formatter", value="{\"time\": \"%d{yyyy-MM-dd HH:mm:ss,SSS}\", \"level\": \"%-5p\", \"tenant_id\": \"%X{tenant}\", \"x-request-id\": \"%X{x-request-id}\", \"user-uuid\" : \"%X{user_id}\", \"client_id\": \"%X{client_id}\", \"loggerName\": \"%c\", \"threadName\": \"%t\", \"message\": \"%s%E\"}%n")

which achieves the 2nd criteria , but fails to do the 1st one..  

https://stackoverflow.com/questions/66887372/jboss-cusom-json-logging-format-mdc-values-in-parent-keys

Could someone please help.. how do I configure the subsystem to achieve both ? 

[James Perkins]

There is not currently a way to do this short of writing a custom formatter. This seems like a reasonable feature request however to have a flat key/value pair for MDC data. You could file an RFE here https://issues.redhat.com/browse/LOGMGR. This may take a while to expose in WildFly itself however.

[Baitanik Talukder]

okay.. yeah sure.. will raise a request for the MDC one.. 

Also is there any way to have stack trace in single line like in spring boot logback we have %replace(%xException){'\n','\u2028'}%nopex ... which basically replaces the newline char with some other char... so that it comes in one line, which in turns in Kibana dashboard also comes in one line... 

That would also work for me.. 

There is something filter-spec -> substituteAll .. but that gets applied to only message.. not the stack trace ? 

[James Perkins]

The exception-output-type=formatted should be on one line. It should escape the new line with "\n".

[Baitanik Talukder]

Sorry for confusion.

exception-output-type=formatted ->this I think applicable only for json formatter. Not pattern formatter

As you have mentioned MDC to flat key /value pair is not supported in json formatter.

I was trying to pattern formatter to by pass that.

/subsystem=logging/console-handler=CONSOLE:write-attribute(name="formatter", value="{\"time\": \"%d{yyyy-MM-dd HH:mm:ss,SSS}\", \"level\": \"%-5p\", \"tenant_id\": \"%X{tenant}\", \"x-request-id\": \"%X{x-request-id}\", \"user-uuid\" : \"%X{user_id}\", \"client_id\": \"%X{client_id}\", \"loggerName\": \"%c\", \"threadName\": \"%t\", \"message\": \"%s%E\"}%n")

In this pattern formatter how we can escape the exception trace new line chars?

Thanks in advance...

[James Perkins]

Ah, there is no way in the pattern formatter. That really shouldn't be used to output JSON. It won't properly escape any messages that could include invalid JSON text.

[Baitanik Talukder]

Yeah... as of now i had to write a custom json formatter class... 

class will extend the jboss jsonformatter ... 

and it is overriding @before.. 

there I am adding those fields to parent json... 

Only thing jboss does not allow to remove fields.. to now I have duplicates mdc values.. on in parent key/value... anothe in mdc key value.. which is fine for now..... 

But writing this class... I had to configure with 'custom-formatter' ... 

now "exception-output-type=formatted, key-overrides=[timestamp="time"]" --> which was available in /subsystem=logging/json-formatter, those i can not configure.. I guess...

 

because /subsystem=logging/custom-formatter does not allow exception-output-type or  key-overrides keys.... only module name , class name is there..

public class CustomJsonFormatter extends JsonFormatter {

    public CustomJsonFormatter() {

        super();

    }

    @Override

    public void before(final Generator generator, final ExtLogRecord record) throws Exception {

        generator.add("x-request-id", record.getMdcCopy().get("x-request-id"));

        generator.add("tenant_id", record.getMdcCopy().get("tenant_id"));

        generator.add("user_id", record.getMdcCopy().get("user_id"));

        generator.add("client_id", record.getMdcCopy().get("client_id"));

    }

}

[James Perkins]

Yes a custom-formatter is the only way to do this.

One minor note for performance. You don't need to use the record.getMdcCopy(). You can just use record.getMdc("x-request-id") for example. Using the getMdcCopy() method creates a new map with each invocation.

[Baitanik Talukder]

One thing... 
in jboss json formatter we have this option.. ("exception-output-type=formatted")

/subsystem=logging/json-formatter=json_format:add(exception-output-type=formatted, key-overrides=[timestamp="time"], meta-data=[@version=1])

/subsystem=logging/console-handler=CONSOLE:write-attribute(name="named-formatter", value="json_format")

when using the cusomt-json formatter.. 

/subsystem=logging/custom-formatter=custom-json-formatter:add(class=com.xyz.logging.CustomJsonFormatter, module=com.xyz.logger)

/subsystem=logging/console-handler=CONSOLE:write-attribute(name=named-formatter, value=custom-json-formatter)

import org.jboss.logmanager.formatters.JsonFormatter;

public class CustomJsonFormatter extends JsonFormatter {

    public CustomJsonFormatter() {

        super();

    }

    public CustomJsonFormatter(final String keyOverrides) {

        super(keyOverrides);

    }

    public CustomJsonFormatter(final Map<Key, String> keyOverrides) {

        super(keyOverrides);

    }

    @Override

    public void before(final Generator generator, final ExtLogRecord record) throws Exception {

        Map<String, String> recordMdc = record.getMdcCopy();

        generator.add("x-request-id", recordMdc.get("x-request-id"));

        generator.add("user_id", recordMdc.get("user_id"));

        generator.add("client_id", recordMdc.get("client_id"));

        

    }

Here how do I use these options like exception-output-type=formatted or key-overrides=[timestamp="time"] , in the custom formatter subsystem

If i directly add here custom-json-formatter:add(class=com.xyz.logging.CustomJsonFormatter, module=com.xyz.logger,  key-overrides=[timestamp="time"]) .. it tells invalid argument .. 

could you please help.. 

[James Perkins]

On Wednesday, April 21, 2021 at 7:32:30 AM UTC-7 baitanik...@gmail.com wrote:

One thing... 
in jboss json formatter we have this option.. ("exception-output-type=formatted")

/subsystem=logging/json-formatter=json_format:add(exception-output-type=formatted, key-overrides=[timestamp="time"], meta-data=[@version=1])

/subsystem=logging/console-handler=CONSOLE:write-attribute(name="named-formatter", value="json_format")

when using the cusomt-json formatter.. 

/subsystem=logging/custom-formatter=custom-json-formatter:add(class=com.xyz.logging.CustomJsonFormatter, module=com.xyz.logger)

/subsystem=logging/console-handler=CONSOLE:write-attribute(name=named-formatter, value=custom-json-formatter)

import org.jboss.logmanager.formatters.JsonFormatter;

public class CustomJsonFormatter extends JsonFormatter {

    public CustomJsonFormatter() {

        super();

    }

    public CustomJsonFormatter(final String keyOverrides) {

        super(keyOverrides);

    }

    public CustomJsonFormatter(final Map<Key, String> keyOverrides) {

        super(keyOverrides);

    }

    @Override

    public void before(final Generator generator, final ExtLogRecord record) throws Exception {

        Map<String, String> recordMdc = record.getMdcCopy();

        generator.add("x-request-id", recordMdc.get("x-request-id"));

        generator.add("user_id", recordMdc.get("user_id"));

        generator.add("client_id", recordMdc.get("client_id"));

        

    }

Here how do I use these options like exception-output-type=formatted or key-overrides=[timestamp="time"] , in the custom formatter subsystem

For the exception-output-type you'd need to set it as a property. Something like:

/subsystem=logging/custom-formatter=custom-json-formatter:add(class=com.xyz.logging.CustomJsonFormatter, module=com.xyz.logger, properties={exceptionOutputType=FORMATTED})

OR

/subsystem=logging/custom-formatter=custom-json-formatter:write-attribute(name=properties, value={exceptionOutputType=FORMATTED})

They key-overrides will have to work differently because there is no mutable property for it. You'd need to override the getKey() method.

[Baitanik Talukder]

thannks .. properties={exceptionOutputType=FORMATTED} worked...