How to set druid's logging properly

40 views
Skip to first unread message

张文彬

unread,
Mar 21, 2022, 9:45:28 PMMar 21
to Druid User
Hi Druid,

Druid version:0.22.1

"apache-druid-0.22.1/conf/druid/cluster/_common/log4j2.xml" and "common.runtime.properties" has been configed, but the log file exists in both directory, "apache-druid-0.22.1/logs" and "apache-druid-0.22.1/var/sv"

How to set logging properly?  Objective:The log file can be divided into several small files, and the history file can be deleted automatically. Thank you!

MiddleManagerConfig:

1、apache-druid-0.22.1/conf/druid/cluster/_common/log4j2.xml

<Configuration status="WARN" monitorInterval="300">
    <properties>
        <!--log files dir -->
        <property name="LOG_HOME">/usr/local/apache-druid-0.22.1/logs</property>
        <!--log file names -->
        <property name="COORDINATOR-OVERLORD">coordinator-overlord</property>
        <property name="BROKER">broker</property>
        <property name="ROUTER">router</property>
        <property name="HISTORICAL">historical</property>
        <property name="MIDDLEMANAGER">middleManager</property>
    </properties>
    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{ISO8601} %p [%t] %c - %m%n"/>
        </Console>

        <RollingRandomAccessFile name="rollingRandomAccessFile_COORDINATOR-OVERLORD"
                                 fileName="${LOG_HOME}/${COORDINATOR-OVERLORD}.log"
                                 filePattern="${LOG_HOME}/${COORDINATOR-OVERLORD}-%d{yyyy-MM-dd:HH-mm}-%i.log">
            <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n" />
            <Policies>
                <SizeBasedTriggeringPolicy size="64 MB" />
            </Policies>
            <DefaultRolloverStrategy max="20" />
        </RollingRandomAccessFile>

        <RollingRandomAccessFile name="rollingRandomAccessFile_BROKER"
                                 fileName="${LOG_HOME}/${BROKER}.log"
                                 filePattern="${LOG_HOME}/${BROKER}-%d{yyyy-MM-dd:HH-mm}-%i.log">
            <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n" />
            <Policies>
              <!--  <TimeBasedTriggeringPolicy interval="1" /> -->
                <SizeBasedTriggeringPolicy size="64 MB" />
            </Policies>
            <DefaultRolloverStrategy max="20" />
        </RollingRandomAccessFile>

        <RollingRandomAccessFile name="rollingRandomAccessFile_ROUTER"
                                 fileName="${LOG_HOME}/${ROUTER}.log"
                                 filePattern="${LOG_HOME}/${ROUTER}-%d{yyyy-MM-dd:HH-mm}-%i.log">
            <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n" />
            <Policies>
                <SizeBasedTriggeringPolicy size="64 MB" />
            </Policies>
            <DefaultRolloverStrategy max="20" />
        </RollingRandomAccessFile>

        <RollingRandomAccessFile name="rollingRandomAccessFile_HISTORICAL"
                                 fileName="${LOG_HOME}/${HISTORICAL}.log"
                                 filePattern="${LOG_HOME}/${HISTORICAL}-%d{yyyy-MM-dd:HH-mm}-%i.log">
            <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n" />
            <Policies>
                <SizeBasedTriggeringPolicy size="64 MB" />
            </Policies>
            <DefaultRolloverStrategy max="20" />
        </RollingRandomAccessFile>

        <RollingRandomAccessFile name="rollingRandomAccessFile_MIDDLEMANAGER"
                                 fileName="${LOG_HOME}/${MIDDLEMANAGER}.log"
                                 filePattern="${LOG_HOME}/${MIDDLEMANAGER}-%d{yyyy-MM-dd:HH-mm}-%i.log">
            <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %-5level %logger{36}-%msg%n" />
            <Policies>
                <SizeBasedTriggeringPolicy size="64 MB" />
            </Policies>
            <DefaultRolloverStrategy max="20" />
        </RollingRandomAccessFile>
    </Appenders>

    <Loggers>
        <Root level="info">
            <AppenderRef ref="Console"/>
        </Root>

        <!-- Set level="debug" to see stack traces for query errors -->
        <Logger name="org.apache.druid.server.QueryResource" level="info" additivity="false">
            <Appender-ref ref="rollingRandomAccessFile_BROKER"/>
        </Logger>
        <Logger name="org.apache.druid.server.QueryLifecycle" level="info" additivity="false">
            <Appender-ref ref="rollingRandomAccessFile_BROKER"/>
        </Logger>

        <!-- Set level="debug" or "trace" to see more Coordinator details (segment balancing, load/drop rules, etc) -->
        <Logger name="org.apache.druid.server.coordinator" level="info" additivity="false">
            <Appender-ref ref="rollingRandomAccessFile_COORDINATOR-OVERLORD"/>
        </Logger>

        <!-- Set level="debug" to see low-level details about segments and ingestion -->
        <Logger name="org.apache.druid.segment" level="info" additivity="false">
            <Appender-ref ref="rollingRandomAccessFile_MIDDLEMANAGER"/>
        </Logger>

        <!-- Set level="debug" to see more information about extension initialization -->
        <Logger name="org.apache.druid.initialization" level="info" additivity="false">
            <Appender-ref ref="rollingRandomAccessFile_MIDDLEMANAGER"/>
        </Logger>

        <!-- Quieter logging at startup -->
        <Logger name="org.skife.config" level="warn" additivity="false">
            <Appender-ref ref="rollingRandomAccessFile_MIDDLEMANAGER"/>
        </Logger>
        <Logger name="com.sun.jersey.guice" level="warn" additivity="false">
            <Appender-ref ref="rollingRandomAccessFile_MIDDLEMANAGER"/>
        </Logger>
        <Logger name="org.apache.druid.server" level="warn" additivity="false">
            <Appender-ref ref="rollingRandomAccessFile_HISTORICAL"/>
        </Logger>
    </Loggers>
</Configuration>

2、apache-druid-0.22.1/conf/druid/cluster/_common/common.runtime.properties

Extensions
druid.extensions.loadList=["druid-datasketches", "druid-lookups-cached-global","mysql-metadata-storage","druid-hdfs-storage","druid-histogram","druid-kafka-indexing-service"]

# Logging
druid.startup.logging.logProperties=true

# Zookeeper
druid.zk.service.host=hd1:2181,hd2:2181,hd3:2181
druid.zk.paths.base=/druid

# For MySQL
druid.metadata.storage.type=mysql
druid.metadata.storage.connector.connectURI=jdbc:mysql://mysqlnode:3306/druid?characterEncoding=UTF-8
druid.metadata.storage.connector.user=druid
druid.metadata.storage.connector.password=druid

# Deep storage - HDFS
druid.storage.type=hdfs
druid.storage.storageDirectory=hdfs://myha01/druid/segments

# Indexing service logs
druid.indexer.logs.type=hdfs
druid.indexer.logs.directory=hdfs://myha01/druid/indexing-logs
#druid.indexer.task.restoreTasksOnRestart=true
#druid.indexer.task.gracefulShutdownTimeout=PT1M

# Monitoring
druid.monitoring.monitors=["org.apache.druid.java.util.metrics.JvmMonitor"]
druid.emitter=logging
druid.emitter.logging.logLevel=trace

# Storage type of double columns
druid.indexing.doubleStorage=double

druid.host=dd2


Mark Herrera

unread,
Mar 22, 2022, 4:00:16 PMMar 22
to Druid User
Hi,

You can set a Log Retention Policy to enable deletion of old task logs.

For my own clarification, may I ask what you mean by dividing the logs into several small files?

Best,

Mark

Reply all
Reply to author
Forward
0 new messages