options timeout does not work with my custom step

[Nikhil Bhoski]

Hi,

I have developed my own step named "runMATLABCommand" which runs MATLAB commands in the pipeline. when i use this step with options block with 6 minutes pause  as shown below.my step does not seem to have any effect of this overriden timeout value . It always gets terminated after 5 minutes no matter what value i set in options block. if run i through sh step then it works fine. am i missing something in my step implementation which would abide the overridden values.

pipeline {

    agent {

        label 'master'

    }

    

     environment { 

        PATH = "${PATH}:/usr/local/MATLAB/R2019b/bin"

    }

  options {

        timeout(time: 15, unit: "MINUTES")

    }

    stages{

        

        stage('Run command ') {

          

            

            steps

            {

                runMATLABCommand "disp('************TIMEOUT STARTED********');disp(datetime('now'));pause(6*60);disp('************TIMEOUT ENDED********');disp(datetime('now'))"    

                     

   

            } 

        }              

    }

}

Thanks & Regards

Nikhil

[Jesse Glick]

On Thu, Jul 23, 2020 at 3:47 AM Nikhil Bhoski <nikhil...@gmail.com> wrote:
> when i use this step with options block with 6 minutes pause as shown below.my step does not seem to have any effect of this overriden timeout value . It always gets terminated after 5 minutes

You are most likely blocking the whole CPS VM thread, which is illegal. See:

https://github.com/jenkinsci/workflow-step-api-plugin/blob/master/README.md#creating-an-asynchronous-step

[Nikhil Bhoski]

Thanks Jesse,

 I had referred the same doc and i guess i am following as per recommendation. Here is my step execution class 

https://github.com/jenkinsci/matlab-plugin/blob/master/src/main/java/com/mathworks/ci/MatlabCommandStepExecution.java 

Which will be called in 

https://github.com/jenkinsci/matlab-plugin/blob/master/src/main/java/com/mathworks/ci/RunMatlabCommandStep.java

Thanks & Regards

Nikhil

[Gavin Mogan]

Its been a long time since i wrote plugins using these APIs but how come your on stop isn't doing anything? Shouldn't it kill the process?

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/12357eb4-d90e-4494-8c6f-619c22a718a4o%40googlegroups.com.

[Nikhil Bhoski]

Hi Gavin ,

The stop is implemented as per the doc which Jesse shared. it reads like below

 But i get your point now and i guess i should kill the process in stop. any ideas on how could i get access to the process in stop () ? 

You should also implement stop to terminate the step. It could simply read

getContext().onFailure(cause);

On Thursday, 23 July 2020 21:22:12 UTC+5:30, Gavin Mogan wrote:

Its been a long time since i wrote plugins using these APIs but how come your on stop isn't doing anything? Shouldn't it kill the process?

On Thu., Jul. 23, 2020, 8:48 a.m. Nikhil Bhoski, <nikhil...@gmail.com> wrote:

Thanks Jesse,

 I had referred the same doc and i guess i am following as per recommendation. Here is my step execution class 

https://github.com/jenkinsci/matlab-plugin/blob/master/src/main/java/com/mathworks/ci/MatlabCommandStepExecution.java 

Which will be called in 

https://github.com/jenkinsci/matlab-plugin/blob/master/src/main/java/com/mathworks/ci/RunMatlabCommandStep.java

Thanks & Regards

Nikhil

On Thursday, 23 July 2020 19:09:30 UTC+5:30, Jesse Glick wrote:

On Thu, Jul 23, 2020 at 3:47 AM Nikhil Bhoski <nikhil...@gmail.com> wrote:
> when i use this step with options block with 6 minutes pause  as shown below.my step does not seem to have any effect of this overriden timeout value . It always gets terminated after 5 minutes

You are most likely blocking the whole CPS VM thread, which is illegal. See:

https://github.com/jenkinsci/workflow-step-api-plugin/blob/master/README.md#creating-an-asynchronous-step

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.

[Devin Nusbaum]

Although you tried to make your step asynchronous, it is not actually asynchronous. `execMatlabCommand` runs during your StepExecution’s `start` method and internally calls `Process.join` to wait for the process to complete, so your `start` method doesn’t return until the process completed. Another indication that your step is not asynchronous is that you call `getContext().onSuccess(true);` before you `return false` from the step, meaning that your step has already completed.

You need to change things so that your step starts the command and then sets up a background process to check its status periodically to know if it is completed. Doing this for external processes is somewhat complex, but you could look at the `sh` and `bat` steps for an example.

Stepping back a little bit, I would try to investigate to see if you could implement this plugin’s functionality in a different way (maybe a block-scoped step that sets up environment variables, etc. as needed) and just have users use the standard `sh` and `bat` steps to launch the Matlab command so that you do not have to reimplement that functionality.

> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/47dea698-df6c-485d-9df3-9230e305c480o%40googlegroups.com.

[Nikhil Bhoski]

Thanks Alot Davin for such detailed explanation. I will check sh and bat and try understand its implementation .  However i was curious to know if this is the reason that possibly causing issue of not abiding timeout block .

Thanks & Regards

Nikhil

> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/47dea698-df6c-485d-9df3-9230e305c480o%40googlegroups.com.

[Devin Nusbaum]

Yes, it is like Jesse said. The CPS VM thread automatically interrupts any synchronous code/step that takes more than 5 minutes to execute. Once you fix the step so it executes asynchronously (or redesign it so the actual work is done by the `sh` step) then the CPS VM thread will stop interrupting it and the timeout set by the `timeout` step will take effect.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/d948f10c-eb14-42f4-aba5-ee765a1daf19o%40googlegroups.com.

[Nikhil Bhoski]

Ahhh Thanks a ton Devin. appreciate your help on this. Also i am trying to learn the sh implemenattion as you suggested and i am looking into these classes here i hope i am at rite location.

https://github.com/jenkinsci/workflow-durable-task-step-plugin/blob/e7c2d46eee43966560a3b9274aab3745a8e8f841/src/main/java/org/jenkinsci/plugins/workflow/steps/durable_task/DurableTaskStep.java#L99

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/d948f10c-eb14-42f4-aba5-ee765a1daf19o%40googlegroups.com.

[Jesse Glick]

On Thu, Jul 23, 2020 at 1:02 PM Nikhil Bhoski <nikhil...@gmail.com> wrote:
> i am looking into these classes here i hope i am at rite location.
>
> https://github.com/jenkinsci/workflow-durable-task-step-plugin/blob/e7c2d46eee43966560a3b9274aab3745a8e8f841/src/main/java/org/jenkinsci/plugins/workflow/steps/durable_task/DurableTaskStep.java#L99

You do not need to look at `DurableTaskStep` implementation at all.
Just implement `SimpleBuildWrapper`, which is quite straightforward.
Or if you need Pipeline specifics, a block-scoped `Step`; there are
plenty of examples, such as in the `ant` plugin (whose tests also
demonstrate usage).

You _can_ implement `SynchronousNonBlockingStepExecution` but then
your step will not survive a Jenkins restart, so this should not be
done if you expect it to take more than a fraction of a second, which
I understand is not at all the case for you.

[Devin Nusbaum]

Nikhil, maybe to make it clearer, what we mean is that instead of using your step like this:

```

runMATLABCommand "disp('************TIMEOUT STARTED********');disp(datetime('now'));pause(6*60);disp('************TIMEOUT ENDED********');disp(datetime('now'))"

```

You would implement a SimpleBuildWrapper or a block-scoped step so that users would instead do something like this (no idea how the Matlab executable works or how it accepts arguments, my escaping is probably wrong as well):

```
withMatlab(…) {
sh(‘matlab "disp('************TIMEOUT STARTED********');disp(datetime(\’now\'));pause(6*60);disp('************TIMEOUT ENDED********');disp(datetime(\‘now\’))"')
}
```

Maybe with a small helper step to convert a matlab command into a shell command if escaping for both Matlab and the shell is tedious, or if there are a bunch of arguments that the step needs to add automatically:

```
withMatlab(…) {
sh(createMatlabCommand("disp('************TIMEOUT STARTED********');disp(datetime('now'));pause(6*60);disp('************TIMEOUT ENDED********');disp(datetime('now'))")
}
```

With these kinds of approaches you don’t have to actually implement the complex pieces of the `sh` step yourself. I mentioned the `sh` step to show you how complicated it is to implement something similar, but I should have been clearer that you should really reconsider your step’s design; trying to reimplement the `sh` step does not make sense.

> --
> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr001XJJ4GS-MOQ2tEw8ef7wvtP2XMYh94gey_EhqK5kug%40mail.gmail.com.

[Nikhil Bhoski]

Thanks Alot, Jesse and Davin for all you time and helping me learn it better. 

I cannot use SimpleBuildWrapper as it it has the restriction of carrying Context env variable (JENKINS-29144). Which i had discussed with Jesse in below thread

https://groups.google.com/forum/?nomobile=true#!topic/jenkinsci-dev/yOOPaxN-1vk

I am looking for Pipeline specific solution for this and SynchronousNonBlockingStepExecution looks promising but yes my execution may vary in time definitely would take more than a fraction of second. I guess i am left with only choice of trying block-scoped `Step`

> To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.

[Nikhil Bhoski]

Hi Jesse,

SynchronousNonBlockingStepExecution   works perfectly fine. however i am interested in learning and implementing it using block-scoped step . as suggested i went through ant plugin code but i could not find the reference of getContext().newBodyInvoker(). I am still no clear that how i could invoke my external process using BodyInvoker(). if you could point out to me a few examples of plugins that would be really great.

Regards

Nikhil 

[Jesse Glick]

On Fri, Jul 24, 2020 at 9:14 AM Nikhil Bhoski <nikhil...@gmail.com> wrote:
> SynchronousNonBlockingStepExecution works perfectly fine

…until you restart Jenkins, at which point your build will fail. So do
not do this.

> i went through ant plugin code but i could not find the reference of getContext().newBodyInvoker()

Sorry, that was a `SimpleBuildWrapper` actually. Try:

https://github.com/jenkinsci/groovy-plugin/pull/22/files#diff-50f6bc8a021eb526aa29d3be83dc8e0bR148

[Nikhil Bhoski]

Thanks, Jesse

I will try and understand this. however first few attempts always showed below exception when i invoked my pipeline step as 

runMATLABCommand "Command" 

java.lang.IllegalStateException: There is no body to invoke

at org.jenkinsci.plugins.workflow.cps.CpsStepContext.newBodyInvoker(CpsStepContext.java:282)

Meanwhile, I found out this https://github.com/jenkinsci/workflow-durable-task-step-plugin/blob/master/src/main/java/org/jenkinsci/plugins/workflow/steps/durable_task/DurableTaskStep.java

This works perfectly fine with BourneShellScript  class invocation however  I am kind of stuck at figuring out the cleanup activities 

after execution is done (like deleteing all temp folders and files i created) Class DurableTask does not seem to have a way to override cleanup() method. any idea on how we could do the cleanup after public Controller launch(EnvVars env, FilePath workspace, Launcher launcher,

            TaskListener listener) throws IOException, InterruptedException{} method gets executed.

Thanks & Regards

Nikhil

[Jesse Glick]

On Wed, Jul 29, 2020 at 7:27 AM Nikhil Bhoski <nikhil...@gmail.com> wrote:
> first few attempts always showed below exception when i invoked my pipeline step as
> runMATLABCommand "Command"
> java.lang.IllegalStateException: There is no body to invoke
> at org.jenkinsci.plugins.workflow.cps.CpsStepContext.newBodyInvoker(CpsStepContext.java:282)

Did you forget to mark your `StepDescriptor` as block-scoped? Or
forget to actually pass a block?

> This works perfectly fine with BourneShellScript class invocation however I am kind of stuck at figuring out the cleanup activities
> after execution is done (like deleteing all temp folders and files i created) Class DurableTask does not seem to have a way to override cleanup() method.

Again, do not go there. You just need a wrapper step. You do not need
to touch or even look at `DurableTaskStep`.