[JIRA] (JENKINS-44924) pipeline groovy script - Sort a list with custom comparator or closure not sorting

[andrew.bayer@gmail.com (JIRA)]

Andrew Bayer assigned an issue to Andrew Bayer

Jenkins / JENKINS-44924 pipeline groovy script - Sort a list with custom comparator or closure not sorting Change By: Andrew Bayer Assignee: Andrew Bayer Add Comment

This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)

[andrew.bayer@gmail.com (JIRA)]

Andrew Bayer started work on JENKINS-44924

Change By: Andrew Bayer Status: Reopened In Progress

Add Comment

[jakub@pawlinski.pl (JIRA)]

Jakub Pawlinski commented on JENKINS-44924

Re: pipeline groovy script - Sort a list with custom comparator or closure not sorting Just lost two hours thinking that I'm doing something wrong. the org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: unclassified field java.lang.Integer Name is ** anything but helpful. tried all possible sorts with closure, with comparator, with OrderBy, finally falled back to Script Console to learn that each attempt I made works fine. Decided to submit a bug, but found its already here. Would love to have proper "@NonCPS" requirement messages/exceptions or even a list to refer to rather than integer error.

Add Comment

[scm_issue_link@java.net (JIRA)]

SCM/JIRA link daemon commented on JENKINS-44924

Re: pipeline groovy script - Sort a list with custom comparator or closure not sorting

Code changed in jenkins User: Andrew Bayer Path: src/main/resources/org/jenkinsci/plugins/scriptsecurity/sandbox/whitelists/generic-whitelist http://jenkins-ci.org/commit/script-security-plugin/15c8d24fc2e46b705d461b72f0d9908ed7530c68 Log: JENKINS-44924 Whitelist DefaultGroovyMethods for sort, unique, etc

Add Comment

[scm_issue_link@java.net (JIRA)]

SCM/JIRA link daemon commented on JENKINS-44924

Re: pipeline groovy script - Sort a list with custom comparator or closure not sorting

Code changed in jenkins User: Andrew Bayer Path: src/main/resources/org/jenkinsci/plugins/scriptsecurity/sandbox/whitelists/generic-whitelist

http://jenkins-ci.org/commit/script-security-plugin/ffbfc0adda6af7c787c0569bc2196d2343c3a259 Log: Merge pull request #191 from abayer/add-more-dgm-methods

JENKINS-44924 Whitelist DefaultGroovyMethods for sort, unique, etc

Compare: https://github.com/jenkinsci/script-security-plugin/compare/47d7d1a2df7a...ffbfc0adda6a

Add Comment

[scof@rambler.ru (JIRA)]

Roman Sinyakov commented on JENKINS-44924

Re: pipeline groovy script - Sort a list with custom comparator or closure not sorting

Andrew Bayer IS there any plans to close the issue. And wonder why is it considered a "Minor"?? The trivial correct groovy code just doesn't (silently!) work and that's a "small problem"?? Amazing...

Add Comment

[scof@rambler.ru (JIRA)]

Roman Sinyakov edited a comment on JENKINS-44924

Re: pipeline groovy script - Sort a list with custom comparator or closure not sorting

[~abayer] IS Is there any plans to close the issue . ? And wonder why is it considered a "Minor"?? The trivial correct groovy code just doesn't (silently!) work and that's a "small problem"?? Amazing...

Add Comment

[vivek.pandey@gmail.com (JIRA)]

Vivek Pandey updated an issue

Jenkins / JENKINS-44924

pipeline groovy script - Sort a list with custom comparator or closure not sorting

Change By: Vivek Pandey Labels: groovy pipeline triaged-2018-11

Add Comment

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

[issac1993@gmail.com (JIRA)]

Daniel Issac commented on JENKINS-44924

Re: pipeline groovy script - Sort a list with custom comparator or closure not sorting Any updates on this.?I agree with Roman Sinyakov.This seems to be a serious issue which needs to be fixed.

Add Comment

[joerg.baeuerle@berthold.com (JIRA)]

Joerg Baeuerle commented on JENKINS-44924

Re: pipeline groovy script - Sort a list with custom comparator or closure not sorting

Yes, I also agree. This has to be fixed. I've just spent two days of my life trying to figure out what was wrong with my pipeline script. My Groovy code worked perfectly in Jenkin's Script Console, but once put in a Jenkinsfile strange things happened. In my case it was not sort() but toSorted() which has the same problem. The following code snipplet List<String> foo = ["hello","hi","hey"].toSorted { a, b -> a.length() <=> b.length() } println foo.toString() taken straight from the toSorted() specification in the Groovy Docs. In Jenkin's Script Console the output was [hi, hey, hello] which is correct. When run from a Jenkinsfile the build's output, however, was -1  This is not acceptable and certainly not a "Minor" issue, because it costs a lot of time and money to hunt down these kinds of bugs!

Add Comment

[joerg.baeuerle@berthold.com (JIRA)]

Joerg Baeuerle edited a comment on JENKINS-44924

Re: pipeline groovy script - Sort a list with custom comparator or closure not sorting

Yes, I also agree. This has to be fixed. I've just spent two days of my life trying to figure out what was wrong with my pipeline script. My Groovy code worked perfectly in Jenkin's Script Console, but once put in a Jenkinsfile strange things happened. In my case it was not {{sort()}} but {{toSorted()}} which has the same problem. The following code snipplet snippet {code :groovy }

List<String> foo = ["hello","hi","hey"].toSorted { a, b -> a.length() <=> b.length() } println foo.toString()

{code} taken straight from the toSorted() specification in the [Groovy Docs|http://docs.groovy-lang.org/latest/html/api/org/codehaus/groovy/runtime/DefaultGroovyMethods.html#toSorted(java.lang.Iterable,groovy.lang.Closure)].

In Jenkin's Script Console the output was

{code :java } [hi, hey, hello] {code}

which is correct. When run from a Jenkinsfile the build's output, however, was

{code :java } -1 {code}

This is not acceptable and certainly not a "Minor" issue, because it costs a lot of time and money to hunt down these kinds of bugs!

Add Comment

[joerg.baeuerle@berthold.com (JIRA)]

Joerg Baeuerle edited a comment on JENKINS-44924

Re: pipeline groovy script - Sort a list with custom comparator or closure not sorting

Yes, I also agree. This has to be fixed. I've just spent two days of my life trying to figure out what was wrong with my pipeline script. My Groovy code worked perfectly in Jenkin's Script Console, but once put in a Jenkinsfile strange things happened. In my case it was not {{sort()}} but {{toSorted()}} which has the same problem. The following code snippet {code :java } List<String> def foo = ["hello","hi","hey"].toSorted { a, b -> a.length() <=> b.length() }

println foo.toString() {code} taken straight from the toSorted() specification in the [Groovy Docs|http://docs.groovy-lang.org/latest/html/api/org/codehaus/groovy/runtime/DefaultGroovyMethods.html#toSorted(java.lang.Iterable,groovy.lang.Closure)]. In Jenkin's Script Console the output was {code:java} [hi, hey, hello] {code} which is correct. When run from a Jenkinsfile the build's output, however, was {code:java} -1 {code}  This is not acceptable and certainly not a "Minor" issue, because it costs a lot of time and money to hunt down these kinds of bugs!

Add Comment

[joerg.baeuerle@berthold.com (JIRA)]

Joerg Baeuerle edited a comment on JENKINS-44924

Re: pipeline groovy script - Sort a list with custom comparator or closure not sorting

Yes, I also agree. This has to be fixed. I've just spent two days of my life trying to figure out what was wrong with my pipeline script. My Groovy code worked perfectly in Jenkin's Script Console, but once put in a Jenkinsfile strange things happened. In my case it was not {{sort()}} but {{toSorted()}} which has the same problem , e . The g. with the following code snippet {code:java}

def foo = ["hello","hi","hey"].toSorted { a, b -> a.length() <=> b.length() } println foo.toString() {code} taken straight from the toSorted() specification in the [Groovy Docs|http://docs.groovy-lang.org/latest/html/api/org/codehaus/groovy/runtime/DefaultGroovyMethods.html#toSorted(java.lang.Iterable,groovy.lang.Closure)]. In Jenkin's Script Console the output was {code:java} [hi, hey, hello] {code} which is correct. When run from a Jenkinsfile the build's output, however, was {code:java} -1 {code}  This is not acceptable and certainly not a "Minor" issue, because it costs a lot of time and money to hunt down these kinds of bugs!

Add Comment

[patrick@intrepid.io (JIRA)]

Pat Butkiewicz commented on JENKINS-44924

Re: pipeline groovy script - Sort a list with custom comparator or closure not sorting

Just want to make sure I'm understanding this correctly. Jenkins has broken custom sorting in the Groovy language, expects people to write their Jenkinsfiles and Pipeline libraries in Groovy, and then marks this as a minor issue? Every on this thread agrees that's unacceptable, yes?

Add Comment

[patrick@intrepid.io (JIRA)]

Pat Butkiewicz edited a comment on JENKINS-44924

Re: pipeline groovy script - Sort a list with custom comparator or closure not sorting

Just want to make sure I'm understanding this correctly. Jenkins has broken custom sorting in the Groovy language, expects people to write their Jenkinsfiles and Pipeline libraries in Groovy, and then marks this as a minor issue? Every Everyone on this thread agrees that's unacceptable, yes?

Add Comment

[andrew.bayer@gmail.com (JIRA)]

Andrew Bayer commented on JENKINS-44924

Re: pipeline groovy script - Sort a list with custom comparator or closure not sorting

If I could have viably fixed this, I would have already - the transformation done to Groovy code to be able to be durable makes a lot of things more or less impossible to get working correctly. We've dealt with as many of them as we could without having to basically write our own implementation of all of Groovy, but there are still edge cases (sorry, but this is a comparative edge case) where we just can't fix them. We're sorry, but this is what it is.

Add Comment

[konstanty.karagiorgis@gmail.com (JIRA)]

Konstanty Karagiorgis commented on JENKINS-44924

Re: pipeline groovy script - Sort a list with custom comparator or closure not sorting

Could it at least be somehow reflected in https://github.com/jenkinsci/JenkinsPipelineUnit so I could unit test all the hacks I need to do to make my code running on jenkins? What's even the purpose of writing unit tests if I can't guarantee, that the code will be running the same on target environment? It gets even worse, it breaks silently, which wastes people's time and money.

Add Comment

[konstanty.karagiorgis@gmail.com (JIRA)]

Konstanty Karagiorgis updated an issue

Jenkins / JENKINS-44924

pipeline groovy script - Sort a list with custom comparator or closure not sorting

Change By: Konstanty Karagiorgis Comment:

Could it at least be somehow reflected in [https://github.com/jenkinsci/JenkinsPipelineUnit] so I could unit test all the hacks I need to do to make my code running on jenkins? What's even the purpose of writing unit tests if I can't guarantee, that the code will be running the same on target environment? It gets even worse, it breaks silently, which wastes people's time and money.

Add Comment

[neal.macdonald@prismtech.com (JIRA)]

Neal Macdonald commented on JENKINS-44924

Re: pipeline groovy script - Sort a list with custom comparator or closure not sorting I couldn't get a pipeline script to work with map sort() at all until I came across [this| https://stackoverflow.com/questions/52946282/jenkins-sort-method-behaviour-not-as-expected ]. This might be a workaround for the issue above. Calling sort from a function with @NonCPS annotation eg: @NonCPS def sortExample(items) { def itemsSorted = items.sort{ it['val'] } println(itemsSorted) } node('targetnode') { def m = [ [name: 'abc', val: '123'], [name: 'zwe', val: '934'], [name: 'wxc', val: '789']] println m sortExample(m) } resulted in a full map sort dependent on the value of a specific key: [Pipeline] echo [{name=abc, val=123}, {name=zwe, val=934}, {name=wxc, val=789}] [Pipeline] echo [{name=abc, val=123}, {name=wxc, val=789}, {name=zwe, val=934}]

Add Comment

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)

[neal.macdonald@prismtech.com (JIRA)]

Neal Macdonald edited a comment on JENKINS-44924

Re: pipeline groovy script - Sort a list with custom comparator or closure not sorting

I couldn't get a pipeline script to work with map sort() at all until I came across [this| [https://stackoverflow.com/questions/52946282/jenkins-sort-method-behaviour-not-as-expected]

]. This might be a workaround for the issue above. Calling sort from a function with @NonCPS annotation eg:

{noformat}

@NonCPS def sortExample(items) {     def itemsSorted = items.sort{ it['val'] }     println(itemsSorted) } node('targetnode') {     def m = [ [name: 'abc', val: '123'], [name: 'zwe', val: '934'], [name: 'wxc', val: '789']]     println m     sortExample(m) }

{noformat}

resulted in a full map sort dependent on the value of a specific key:

{noformat}

[Pipeline] echo [{name=abc, val=123}, {name=zwe, val=934}, {name=wxc, val=789}] [Pipeline] echo [{name=abc, val=123}, {name=wxc, val=789}, {name=zwe, val=934}]

{noformat}

Add Comment

[neal.macdonald@prismtech.com (JIRA)]

[lucas.cimon@gmail.com (JIRA)]

Lucas Cimon commented on JENKINS-44924

Re: pipeline groovy script - Sort a list with custom comparator or closure not sorting

I found bit of helpful context about this issue : https://wiki.jenkins.io/display/JENKINS/Pipeline+CPS+method+mismatches#PipelineCPSmethodmismatches-Callingnon-CPS-transformedmethodswithCPS-transformedarguments

Add Comment

[lucas.cimon@gmail.com (JIRA)]

Lucas Cimon edited a comment on JENKINS-44924

Re: pipeline groovy script - Sort a list with custom comparator or closure not sorting

I found bit of helpful context about this issue : [ https://wiki.jenkins.io/display/JENKINS/Pipeline+CPS+method+mismatches#PipelineCPSmethodmismatches-Callingnon-CPS-transformedmethodswithCPS-transformedarguments ]   Edit: captain obvious... I did not notice the link to this exact issue on the wiki :)

Add Comment

[lucas.cimon@gmail.com (JIRA)]

Lucas Cimon edited a comment on JENKINS-44924

Re: pipeline groovy script - Sort a list with custom comparator or closure not sorting

I found bit of helpful context about this issue : [https://wiki.jenkins.io/display/JENKINS/Pipeline+CPS+method+mismatches#PipelineCPSmethodmismatches-Callingnon-CPS-transformedmethodswithCPS-transformedarguments]   Edit: captain obvious... I did not notice the link to this exact issue on the wiki :)

Here is the workaround I used to sort a list of maps base on a String field: {code:java} pipeline {     stages {         stage("Test") {             steps {                 script {                     def propList = [[name: 'B'], [name: 'A']]                     sortByField(propList, 'name')                     propList.eachWithIndex { entry, i ->                      println "${i}: ${entry}"                     }                 }             }         }     } } @NonCPS static sortByField(list, fieldName) {     list.sort{ it[fieldName] } } {code}

Add Comment

[lucas.cimon@gmail.com (JIRA)]

Lucas Cimon edited a comment on JENKINS-44924

Re: pipeline groovy script - Sort a list with custom comparator or closure not sorting

I found bit of helpful context about this issue : [https://wiki.jenkins.io/display/JENKINS/Pipeline+CPS+method+mismatches#PipelineCPSmethodmismatches-Callingnon-CPS-transformedmethodswithCPS-transformedarguments] Edit: captain obvious... I did not notice the link to this exact issue on the wiki :)

Here is the workaround I used to sort a list of maps base Map objects, based on a String field:

{code:java} pipeline {     stages {         stage("Test") {             steps {                 script {                     def propList = [[name: 'B'], [name: 'A']]                     sortByField(propList, 'name')                     propList.eachWithIndex { entry, i ->                      println "${i}: ${entry}"                     }                 }             }         }     } } @NonCPS static sortByField(list, fieldName) {     list.sort{ it[fieldName] } } {code}

Add Comment

[ianfixes@gmail.com (JIRA)]

Ian Katz commented on JENKINS-44924

Re: pipeline groovy script - Sort a list with custom comparator or closure not sorting

If I could have viably fixed this, I would have already - the transformation done to Groovy code to be able to be durable makes a lot of things more or less impossible to get working correctly. We've dealt with as many of them as we could without having to basically write our own implementation of all of Groovy, but there are still edge cases (sorry, but this is a comparative edge case) where we just can't fix them. If the sort function can't work under any circumstances, shouldn't the correct behavior be to have it throw some exception rather than just pretend to work?  (Emphasis for meaning, not for tone.) I lost about a day's work here trying permutations of the following documented descriptions of array sort: http://docs.groovy-lang.org/next/html/documentation/working-with-collections.html#_sorting https://mrhaki.blogspot.com/2015/03/groovy-goodness-new-methods-to-sort-and.html https://mrhaki.blogspot.com/2011/09/groovy-goodness-sort-or-remove.html https://stackoverflow.com/a/20386474/2063546 I would have greatly preferred an exception that said "We're sorry, but this is what it is."  Instead I assumed that I wasn't supplying the correct arguments, mutating instead of returning a value, etc.  That was a very unpleasant rabbit hole. Can you talk more about what makes function impossible to support?  For example, is it related to a threaded library or some other implementation detail that prevents the built-in function from every being translated to CPS?  Is it a lack of support for the "spaceship operator"?  Or is it something inherent to the sort algorithm that would make it fail even if I tried to implement my own array sort function by hand?

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-44924

Re: pipeline groovy script - Sort a list with custom comparator or closure not sorting

If you are using a reasonably recent version of workflow-cps it should have printed a warning with a link to a wiki page explaining this class of issue and workarounds.

Add Comment

[ianfixes@gmail.com (JIRA)]

Ian Katz commented on JENKINS-44924

Re: pipeline groovy script - Sort a list with custom comparator or closure not sorting

I'm on 2.190.2 of Jenkins, with BlueOcean 1.21.0. Not sure how to find workflow-cps version. I didn't see a warning, but my point is that it should produce an error, not a warning. What is the justification for allowing me to call a function that you know in advance will not do its job?

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-44924

Re: pipeline groovy script - Sort a list with custom comparator or closure not sorting

/pluginManager/installed; text console log in classic view, not Blue Ocean; and because the detector has false positives (it is very complicated).

Add Comment