log files are not compressed after upgrading to 2.14.3

[Jungho Ahn]

Hello,

I noticed that log files are not compressed after I upgraded to 2.14.3.

Was there any change in log configuration?

Thanks,

[Hugo Arès]

Hi,

We have 2.14.3 on our dev server and logs are compressed once per day.

If you execute the show-queue command, is the log compressor task there?

ssh -p 29418 localhost gerrit show-queue

Task     State        StartTime         Command
------------------------------------------------------------------------------
2fcf4e69 07:27:10.534 10:26:00.599      change cleanup runner
99ce3f23 10:43:08.250 10:43:07.250      [high-availability] Clean up expired file based websessions
efedb6d1 Jun-12 13:25 10:25:56.120      Log File Compressor
------------------------------------------------------------------------------
  3 tasks

[Jungho Ahn]

Hi Hugo,

The log compressor task seems to be start at Sep-16, but after that all logs are not compressed:

Task     State        StartTime         Command

------------------------------------------------------------------------------

e01a8890 03:00:03.342 Sep-16 00:14      GC runner

a011f074 Jun-02 06:13 Sep-16 00:13      Log File Compressor

------------------------------------------------------------------------------

  2 tasks

-r--r--r-- 1 gerrit2 gerrit2  20557 Sep 16 00:13 error_log.2017-09-15.gz

-rw-r--r-- 1 gerrit2 gerrit2  10540 Sep 16 23:47 error_log.2017-09-16

-rw-r--r-- 1 gerrit2 gerrit2   3170 Sep 17 23:02 error_log.2017-09-17

-rw-r--r-- 1 gerrit2 gerrit2 182220 Sep 18 23:41 error_log.2017-09-18

-rw-r--r-- 1 gerrit2 gerrit2  41930 Sep 19 23:20 error_log.2017-09-19

-rw-r--r-- 1 gerrit2 gerrit2  31437 Sep 20 23:14 error_log.2017-09-20

-rw-r--r-- 1 gerrit2 gerrit2  20778 Sep 21 23:54 error_log.2017-09-21

Thanks,

--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to a topic in the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/repo-discuss/xDQ_D8O4m_o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to repo-discuss+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Jungho

[Bassem Rabil]

The log compressor seems working, it started on Sept 16th and it compressed the past day log Sept 15th at midnight 00:13 as shown in timestamp.

[Jungho Ahn]

I see, then why does it not compress log files after that? All logs since Sept 16th are not compressed.

--

--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to a topic in the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/repo-discuss/xDQ_D8O4m_o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to repo-discuss+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Jungho

[Hugo Arès]

Log compressor task is executed in the default thread pool which only have one thread per default[1].

Take a thread dump (jstack -l <gerrit_pid>) and look for a thread named 'WorkQueue-1', maybe it's blocked/waiting preventing the log compressor task from executing.

[1]https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#execution.defaultThreadPoolSize

On Friday, September 22, 2017 at 2:08:24 PM UTC-4, Jungho Ahn wrote:

I see, then why does it not compress log files after that? All logs since Sept 16th are not compressed.

On Fri, Sep 22, 2017 at 10:56 AM, Bassem Rabil <bassem.ra...@ericsson.com> wrote:

The log compressor seems working, it started on Sept 16th and it compressed the past day log Sept 15th at midnight 00:13 as shown in timestamp.

On Friday, September 22, 2017 at 1:43:30 PM UTC-4, Jungho Ahn wrote:

-r--r--r-- 1 gerrit2 gerrit2  20557 Sep 16 00:13 error_log.2017-09-15.gz

-rw-r--r-- 1 gerrit2 gerrit2  10540 Sep 16 23:47 error_log.2017-09-16

--

--
To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to a topic in the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/repo-discuss/xDQ_D8O4m_o/unsubscribe.

To unsubscribe from this group and all its topics, send an email to repo-discuss...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Jungho

[Jungho Ahn]

jstack showed:

"WorkQueue-1" #53 prio=5 os_prio=0 tid=0x00007f5dd6068000 nid=0x9b0 waiting on condition [0x00007f43dfdfe000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00007f45c6ebcbb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:

        - None

Does this mean it's just waiting a task?

I restarted the server and then it compressed all the existing logs. I'm not sure what happened before.

BTW, is there a way to set the maximum number of logs to keep?

--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to a topic in the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/repo-discuss/xDQ_D8O4m_o/unsubscribe.

To unsubscribe from this group and all its topics, send an email to repo-discuss+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Jungho

[Jungho Ahn]

I forgot the mention that I increased the #threads to 2.

Now the jstack shows:

"WorkQueue-1" #53 prio=5 os_prio=0 tid=0x00007f7dd5fd2800 nid=0x6b1f waiting on condition [0x00007f6550d65000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00007f65c70853d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"WorkQueue-2" #54 prio=5 os_prio=0 tid=0x00007f7dd5fc1000 nid=0x6b20 waiting on condition [0x00007f6550c64000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00007f65c70853d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

--

Jungho

[Hugo Arès]

This stack trace means the executor thread was waiting for a task so the thread was not blocked.

To only thing left that could explain a task no longer being executed is if it threw an exception during execution, subsequent executions would have been suppressed.

Check the review_site/logs/error_log file for a stack trace related to LogFileCompressor.

Jungho

[Jungho Ahn]

I see. Weird. I just checked all error_logs*, but I couldn't find any logs related to log file compression.

--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to a topic in the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/repo-discuss/xDQ_D8O4m_o/unsubscribe.

To unsubscribe from this group and all its topics, send an email to repo-discuss+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Jungho

[Makson Lee]

we have similar issue, our version is 2.14.4.

$ ls

delete_log            gerrit.pid            replication_log.2017-09-23

error_log             gerrit.run            sshd_log

error_log.2017-09-23  httpd_log             sshd_log.2017-09-23

error_log.2017-09-24  httpd_log.2017-09-23  sshd_log.2017-09-24

gc_log                httpd_log.2017-09-24

gc_log.2017-09-23     replication_log

$ ssh -p 29418 user@localhost gerrit show-queue

--wide

Task     State        StartTime         Command

------------------------------------------------------------------------------

77d112ee 00:00:07.533 Sep-23 08:23      change cleanup runner

37ecfab4 Jun-15 23:23 Sep-23 08:23      Log File Compressor

------------------------------------------------------------------------------

  2 tasks

[Hugo Arès]

This change should give us more info why log compression is no longer working:

https://gerrit-review.googlesource.com/#/c/gerrit/+/129490/

[Hugo Arès]

This change should fix the issue: https://gerrit-review.googlesource.com/c/gerrit/+/133550