VariableThroughputTimer + ConcurrencyThreadGroup = Thread Interruptions

[Roger Abelenda]

Hello,

I am the main maintainer for jmeter-java-dsl (https://abstracta.github.io/jmeter-java-dsl/) which uses several of published JMeter Plugins heavily and simplify their usage from java code. Thank you very much for the excellent plugins you have included.

In particular we have created the concept of rpsThreadGroup (https://abstracta.github.io/jmeter-java-dsl/guide/#throughput-based-thread-group) which in fact uses VariableThroughputTimer in combination with ConcurrencyThreadGroup.

Recently I user detected some issues (https://github.com/abstracta/jmeter-java-dsl/issues/257) while using this component which we traced to the logic of VariableThroughputTimer while stopping threads and how ConcurrencyThreadGroup handles them. In particular JVM threads being interrupted which may cause all type of exceptions to raise in test plan depending the location where the thread is executing. Some examples are Jexl Cancellation exceptions, IO & Socket exceptions, etc.

We tried changing the logic of VariableThroughputTimer like this (which avoids threads interrupts):

```
public static class NonInterruptingVariableThroughputTimer extends VariableThroughputTimer {

    @Override
    protected void stopTest() {
      JMeterContextService.getContext().getThreadGroup().tellThreadsToStop();
    }

  }
```

And the issues are no longer present.

Do you see any potential problem to implement such logic? Do you think it would be worth to add some option to existing VariableThroughputTimer avoid interrupting, or even making this behavior the default one and only interrupt threads when is strictly necessary?

Regards, and thanks again for the great work.

[Andrey Pokhilko]

Hi,

I don't understand why should we override stopTest, it already contains graceful calls to JMeter engine.

The only place with native thread interripting I see is this block: https://github.com/undera/jmeter-plugins/blob/master/plugins/tst/src/main/java/kg/apc/jmeter/timers/VariableThroughputTimer.java#L106

That line was contributed by Philippe in https://github.com/undera/jmeter-plugins/pull/193

I guess we'll need his explanation what was the motivation for additional thread interruption.

[Roger Abelenda]

Hello, thank you for your prompt response.

As far as I have seen this is the logic of VariableThroughputTimer.stopTest method:

```
        if (stopTries > 30) {
            throw new IllegalStateException("More than 30 retries - stopping with exception");
        }

        if (lastStopTry == time) {
            return;
        }
        log.info("No further RPS schedule, asking threads to stop...");
        lastStopTry = time;
        stopTries++;
        if (stopTries > 10) {
            log.info("Tries more than 10, stopping engine NOW!");
            StandardJMeterEngine.stopEngineNow();
        } else if (stopTries > 5) {
            log.info("Tries more than 5, shutting down engine!");
            StandardJMeterEngine.stopEngine();
        } else if (stopTries > 3) {
            AbstractThreadGroup threadGroup = JMeterContextService.getContext().getThreadGroup();
            log.info("Tries more than 3, hard stopping threads of Thread Group : {}", threadGroup.getName());
            threadGroup.tellThreadsToStop();
        } else {
            AbstractThreadGroup threadGroup = JMeterContextService.getContext().getThreadGroup();
            log.info("Stopping gracefuly threads of Thread Group : {}", threadGroup.getName());
            threadGroup.stop();
        }
```

And since, we are using ConcurrencyThreadGroup this is the logic of AbstractDynamicThreadGroup.stop():

```
    public void stop() {
        running = false;
        threadStarter.interrupt();
        for (DynamicThread thread : threads) {
            thread.interrupt();
            thread.interruptOSThread();
        }
    }
```

Here we can see that is actually sending a interruptOSThread() invocation to the thread.

Maybe the solution is not changing stopTest in VariableThroughputTimer (which was the approach I took since was the one with less changes and less impact), but to change stop() in ConcurrencyThreadGroup, or maybe there is some other solution. I find though, the logic in VariableThroughputTimer.stop() to be more complex than initially expected, and it's final behavior to user a little unpredictable depending on the amount of threads you have, how much they take to stop, etc. I guess this logic has been implemented to solve known and experienced issues, but I wonder if we may come up with a simpler and more predictable one. 

In any case, I am currently more concerned in avoiding the thread interruptions than to change too much existing logic. I am more than willing to help in whatever I can.

Thank you again, and I am eager to hear from you again.

[Andrei Pokhilko]

The need in interrupting OS thread comes from the fact that there might be some operation running that last unpredictably long, or hang forever. A test that obeys the "stop" command with some exceptions printed is better than test that does not stop at all.

I don't really see how can we change timer or thread group to suit your use-case. If you have ideas that will not break existing capabilities - let's discuss those.

--
Andrei

--
You received this message because you are subscribed to a topic in the Google Groups "jmeter-plugins" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jmeter-plugins/drzb62Cw1fg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jmeter-plugin...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jmeter-plugins/a884f84c-5f14-443e-ae3f-9cfcb92f4dafn%40googlegroups.com.

[Roger Abelenda]

I think it would be nice to be consistent with Jmeter default thread group which does not interrupt threads when the specified thread duration ends.

Additionally, I think we could implement in ConcurrencyThreadGroup some option for users to select to stop threads "gracefully" or interrupt them. The first option would avoid any unexpected exceptions, which are hard to understand to users, or some time even hard to discern if an Exception is due to a thread interruption or some other reason (like server closing the connection). But this option, if not careful, could lead to never ending threads/test plans or not as precise throughput matching (do you know common cases when this may happen?). The second option would allow for stopping threads as soon as the criteria for stopping them has been met, but could generate unexpected exceptions in test plans, which users would need to ignore. We could even implement a third option (if we want to give users even more control), to try stopping threads greacefully, and if they don't stop after X time, then interrupt them.

I think in general this could avoid surprises, and could simplify the logic of stopping threads. Maybe in the end is more logic, but initially sounds simpler than current logic. Current logic has worked well for so long and I don't know again the story and all potential scenarios that may be solving nowadays as it is, but without further understanding, looks like many ifs with arbitrary values for using different strategies for stopping the threads, which makes the behavior unpredictable to users.

What do you think?

Side note: Nowadays JMeter also provides Open Model Thread Group, which also provides an alternative for RPS thread groups. Another option, could be to use that, but I would like to still leave the option open to use the most well known and used option, and the one that lets create additional communities on top of JMeter, in the DSL for some time, which is the solution provided by VariableThroughputTimer + ConcurrencyThreadGroup.

To view this discussion on the web visit https://groups.google.com/d/msgid/jmeter-plugins/d42a5380-07b0-4bdd-affa-ad0e17224127%40gmail.com.

[Andrei Pokhilko]

In Jmeter standard ThreadGroup, the jvmThread also gets interrupted: https://github.com/apache/jmeter/blob/master/src/core/src/main/java/org/apache/jmeter/threads/ThreadGroup.java#L349 . That happens if "stop threads now" is called.

I'm ok with the logic to interrupt JVM thread only if "now" flag is set, the same it happens in standard ThreadGroup.

However, I don't have much time for coding this, my role has to be limited to reviewer.

--
Andrei

To view this discussion on the web visit https://groups.google.com/d/msgid/jmeter-plugins/CAE_L69a_ujCeh0fi9hRSu5YdC3ihDD2jNcH4siecCX5y8Wjpew%40mail.gmail.com.