What is doing 'Index-Interactive'?

[vista...@gmail.com]

Hi,all:

     Aftrer take the cmd 'ssh -p 29418 xxx@yyyygerrit show-queue -w -q '

    I found a lots of 'Index-Interactive'  that 'status:merged' 

Queue: Index-Interactive

2eacc804              11:57:30.503      status:merged

8e713494              11:57:30.655      status:merged

ae72f897              11:57:30.809      status:merged

0e45442b              11:57:30.818      status:merged

2e3e08b1              11:57:30.924      status:merged

ae2438ab              11:57:31.358      status:merged

0e17843f              11:57:31.425      status:merged

4e181c77              11:57:31.618      status:merged

    What was doing?

Thanks!  

    

[vista...@gmail.com]

pls help!! :( 

Thanks.

[vista...@gmail.com]

Tha 'jstack' show that:

"Index-Interactive-15" prio=10 tid=0x00007f686c011000 nid=0x25b8 waiting on condition [0x00007f6b9afee000]

   java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

- parking to wait for  <0x00007f6bf9289160> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)

at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

"Index-Interactive-14" prio=10 tid=0x00007f686c00f000 nid=0x25b6 waiting on condition [0x00007f6b9b1f0000]

   java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

- parking to wait for  <0x00007f6bf9289160> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)

at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

"Index-Interactive-13" prio=10 tid=0x00007f686c010000 nid=0x25b5 runnable [0x00007f6b9b2f1000]

   java.lang.Thread.State: RUNNABLE

at org.apache.lucene.util.PriorityQueue.downHeap(PriorityQueue.java:284)

at org.apache.lucene.util.PriorityQueue.updateTop(PriorityQueue.java:211)

at org.apache.lucene.search.TopFieldCollector.updateBottom(TopFieldCollector.java:523)

at org.apache.lucene.search.TopFieldCollector$SimpleFieldCollector$2.collect(TopFieldCollector.java:243)

at org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:221)

at org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)

at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)

at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:821)

at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)

at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:744)

at org.apache.lucene.search.IndexSearcher.searchAfter(IndexSearcher.java:729)

at org.apache.lucene.search.IndexSearcher.searchAfter(IndexSearcher.java:671)

at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:577)

at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:627)

at com.google.gerrit.lucene.LuceneChangeIndex$QuerySource.doRead(LuceneChangeIndex.java:345)

at com.google.gerrit.lucene.LuceneChangeIndex$QuerySource.access$300(LuceneChangeIndex.java:279)

at com.google.gerrit.lucene.LuceneChangeIndex$QuerySource$1.call(LuceneChangeIndex.java:325)

at com.google.gerrit.lucene.LuceneChangeIndex$QuerySource$1.call(LuceneChangeIndex.java:322)

at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)

at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)

at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

On Wednesday, November 28, 2018 at 11:58:36 AM UTC+8, vista...@gmail.com wrote:

[Luca Milanesio]

Can you share your Gerrit version and gerrit.config?

    

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[vista...@gmail.com]

Hi,lucamilanesio.

My gerrit version is 2.13.7.

some contents of 'gerrit.config':

[container]

        heapLimit = 60g

        user = gerrit2

        javaHome = /usr/lib/jvm/java-7-openjdk-amd64/jre

        startupTimeout = 120

[sshd]

        threads = 140

        batchThreads = 2

        maxAuthTries = 12

        streamThreads = 50

        listenAddress = *:29418

        tcpKeepAlive = false

        idleTimeout = 40m

        rekeyTimeLimit = 0

        rekeyBytesLimit = 107374182400

        maxConnectionsPerUser = 80

[core]

        packedGitLimit = 2g

        packedGitWindowSize = 32k

        packedGitOpenFiles = 512

[index]

        type = LUCENE

        onlineUpgrade = false

        threads = 15

        maxTerms = 2048

    

--
--
To unsubscribe, email repo-disc...@googlegroups.com

[vista...@gmail.com]

The server is '32 CPUs and 80G RAM'

[Luca Milanesio]

You have 15 threads dedicated to reindexing (the "Index-Interactive-NN" ones you saw")

If you see some queueing, it is because you're generating more reindexing events than the ones you can write to disk to Lucene.

Luca.

To unsubscribe, email repo-discuss...@googlegroups.com

[vista...@gmail.com]

Thank you,lucamilanesio.

When I found lots of  'Index-Interactive' with 'status:merged', the cpu was used more than 100%.    :(

To unsubscribe, email repo-disc...@googlegroups.com

[Luca Milanesio]

On 4 Dec 2018, at 15:31, vista...@gmail.com wrote:

Thank you,lucamilanesio.

When I found lots of  'Index-Interactive' with 'status:merged', the cpu was used more than 100%.    :(

How many?

Did you have any massive operation on changes?

Did anyone possibly pushed hundreds of changes all at once?

Luca.

To unsubscribe, email repo-disc...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[vista...@gmail.com]

Maybe it is not the problem of ‘Index-Interactive’ and the real problem is that ‘clone repo from gerrit’

To unsubscribe, email repo-disc...@googlegroups.com

[vista...@gmail.com]

I got the ‘show-cacche’ that time.

To unsubscribe, email repo-disc...@googlegroups.com

[Saša Živkov]

On Tue, Dec 4, 2018 at 5:01 PM <vista...@gmail.com> wrote:

Maybe it is not the problem of ‘Index-Interactive’ and the real problem is that ‘clone repo from gerrit’

A more reliable way to find out which threads are using the CPU is to execute:

$ top -H

and take several thread ids from the top of the list.

At the same time create a few thread dumps of the Gerrit process and then check what the

top threads from the "top -H" command are doing.

 

To unsubscribe, email repo-discuss...@googlegroups.com

[vista...@gmail.com]

losts of jstack log which ‘Index-Interactive’ like that:

"Index-Interactive-15" prio=10 tid=0x00007fccb8095000 nid=0xb4ed runnable [0x00007fcf2bbfa000]

   java.lang.Thread.State: RUNNABLE

at org.apache.lucene.util.PriorityQueue.downHeap(PriorityQueue.java:284)

at org.apache.lucene.util.PriorityQueue.updateTop(PriorityQueue.java:211)

at org.apache.lucene.search.TopFieldCollector.updateBottom(TopFieldCollector.java:523)

at org.apache.lucene.search.TopFieldCollector$SimpleFieldCollector$2.collect(TopFieldCollector.java:243)

at org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:221)

--

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

[Leandro Fonseca]

Hi Luca,

Sorry for ressucting this very old thread, but regarding your question to Z: "Did anyone possibly pushed hundreds of changes all at once?"

I do have this same problem that Z reported, and I just experienced a scenario where I received around 1000 pushes in a 10 minutes interval, and that got me one of these index-interactive threads, that uses 100% of the CPU and is stuck on the queue.

Should I expect having one of these index-interactive threads created everytime I receive a high number of pushes in a small interval?

Is there something we can do to avoid having these stuck threads (besides maybe asking the users to throttle/reduce their parallel pushes)?

Thanks a lot,

Leandro.

To unsubscribe, email rep...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-d...@googlegroups.com.

[Luca Milanesio]

On 9 Dec 2019, at 13:13, Leandro Fonseca <lean...@motorola.com> wrote:

Hi Luca,

Sorry for ressucting this very old thread, but regarding your question to Z: "Did anyone possibly pushed hundreds of changes all at once?"

I do have this same problem that Z reported, and I just experienced a scenario where I received around 1000 pushes in a 10 minutes interval, and that got me one of these index-interactive threads, that uses 100% of the CPU and is stuck on the queue.

Should I expect having one of these index-interactive threads created everytime I receive a high number of pushes in a small interval?

Is there something we can do to avoid having these stuck threads (besides maybe asking the users to throttle/reduce their parallel pushes)?

You could throttle incoming operations using the quota plugin:

HTH

Luca.

[1] https://gerrit.googlesource.com/plugins/quota

To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/b1267878-35a3-48a2-b367-bb5b19f2cb23%40googlegroups.com.

[Leandro Fonseca]

Hey Luca,

Thanks for your answer.

Unfortunately these 'status:merged' tasks (Index-Interactive threads on jstack) are not only initiated when a massive number of pushes are done, I also see them started at some other random times when no pushes are being done to the Gerrit server.

I read your comment at another related thread (link below) where you suggest it could also happen in an environment with a high number of open changes. Based on that I have also taken action and considerably reduced my number of open changes (by enabling the auto abandon feature as you recommended), however the index-interactive threads are still showing up and getting stuck. These are the number of changes I have on each status:

# of changes   change status

           250190    A

        350    d 

    1135004    M

       2462    n

I'm now checking how many merged changes I have for each project&branch combination. My top 5 looks like that:

# of merged changes    project & branch

   2994                project5 branch5

   3993                project4 branch4

   4564                project3 branch3

   4593                project2 branch2

   5866                project1 branch1

Some of these projects above are very big and have a lot of binnaries, others are just full of thousands of regular commits on multiple branches. 

The numbers above, of course, are for the project&branch combination. Each one of the these listed projects have other thousands of changes at other branches.

So, now come a couple of questions I hope you have some time to answer:

- Are my numbers of merged changes by project&branch too high? Would that justify the reason why I get these stuck index-interactive threads?

- In a index point of view, a project with thousands of changes on multiple branches will always have a bad performace and be a problem, even if the branches with lots of changes are hidden (READ blocked to all users including the Gerrit user), and the only active branches have a relatively low number of changes?

- Any idea of what else could initiating these hanging index-interactive tasks that gets stuck on the queue using 100% of the allocated CPU thread?

- Is there any Gerrit system setting, administrative routine (such as a gerrit-gc with some specific option, JavaGC, etc) OR project.git/config configuration that could be help aliviating/fixing this problem (I have already followed all the recommendations on the Performace Tunning slides, presented at some of the Gerrit Summits, link also below)

- I'm currently using Lucene Index. Do you think Elastic Search would also have this problem?

Thanks again,

Leandro.

Links:

https://groups.google.com/forum/#!searchin/repo-discuss/status$3Amerged%7Csort:date/repo-discuss/W_gfS7kZmQM/outA7jKHAwAJ

https://pt.slideshare.net/JohannesNicolai1/gerrit-performance-tuning-talk-at-google-summit

[Luca Milanesio]

On 18 Dec 2019, at 02:47, Leandro Fonseca <lean...@motorola.com> wrote:

Hey Luca,

Thanks for your answer.

Unfortunately these 'status:merged' tasks (Index-Interactive threads on jstack) are not only initiated when a massive number of pushes are done, I also see them started at some other random times when no pushes are being done to the Gerrit server.

Which version are you running? There was a bug (or feature?) that was keeping reindexing the same changes forever.

I read your comment at another related thread (link below) where you suggest it could also happen in an environment with a high number of open changes. Based on that I have also taken action and considerably reduced my number of open changes (by enabling the auto abandon feature as you recommended), however the index-interactive threads are still showing up and getting stuck. These are the number of changes I have on each status:

# of changes   change status

           250190    A

        350    d 

    1135004    M

       2462    n

I'm now checking how many merged changes I have for each project&branch combination. My top 5 looks like that:

# of merged changes    project & branch

   2994                project5 branch5

   3993                project4 branch4

   4564                project3 branch3

   4593                project2 branch2

   5866                project1 branch1

Some of these projects above are very big and have a lot of binnaries, others are just full of thousands of regular commits on multiple branches. 

The numbers above, of course, are for the project&branch combination. Each one of the these listed projects have other thousands of changes at other branches.

So, now come a couple of questions I hope you have some time to answer:

- Are my numbers of merged changes by project&branch too high? Would that justify the reason why I get these stuck index-interactive threads?

With 1.1M merged changes, you may have repos with lots of refs. Have you checked them with git-sizer?

- In a index point of view, a project with thousands of changes on multiple branches will always have a bad performace and be a problem, even if the branches with lots of changes are hidden (READ blocked to all users including the Gerrit user),

It doesn't matter if the changes are visible or not: as long as they are open, they will keep on being reindexed.

and the only active branches have a relatively low number of changes?

Gerrit doesn't have the knowledge or concept of active branch: what do you mean exactly?

- Any idea of what else could initiating these hanging index-interactive tasks that gets stuck on the queue using 100% of the allocated CPU thread?

Two things:

- modifications on other changes for the same project targeting the same branch

- advance in the target branch

- Is there any Gerrit system setting, administrative routine (such as a gerrit-gc with some specific option, JavaGC, etc) OR project.git/config configuration that could be help aliviating/fixing this problem (I have already followed all the recommendations on the Performace Tunning slides, presented at some of the Gerrit Summits, link also below)

change cleanup and auto-abandon are your friend to get rid of the inactive open changes.

- I'm currently using Lucene Index. Do you think Elastic Search would also have this problem?

This is NOT a problem about storing the index but calculating its value. Lucene vs. ES wouldn't make any difference in this case.

HTH

Luca.

To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/d70ea85b-bc27-433c-a33f-b6d03cac6471%40googlegroups.com.

[Z]

Hi,Leandro:

     Was the issue resolved by you?

Thank you.