Help with NotSerializableException in pipeline
Slava Dubrovskiy
Here is my test pipeline:
#!groovy
def behatList =['AutoSuiteSet_0', 'AutoSuiteSet_1', 'AutoSuiteSet_2', 'AutoSuiteSet_3', 'AutoSuiteSet_4', 'AutoSuiteSet_5']
def suitesStat=[AutoSuiteSet_0:0, AutoSuiteSet_1:1, AutoSuiteSet_2:2, AutoSuiteSet_3:3, AutoSuiteSet_4:4, AutoSuiteSet_5:5]
stage("test") {
node('master') {
behatList2=sortSuites(behatList, suitesStat)
echo "SUITES2=${behatList2}"
}
}
@NonCPS
def sortSuites(suites, suites_time){
timeLimit = suites_time.values().max()
def suitesMap= [:]
for(s in suites){
x=suites_time.find{ artifact -> artifact.key == s}
if(x){
suitesMap.put(x.key, x.value)
}else{
suitesMap.put(s, timeLimit)
}
}
tasks = [suitesMap]
timeLimit = suitesMap.values().max()
while(canSplit()) {
tasks = tasks.collect { t ->
if(checkLimit(t)){
t = splitTo2(t)
}
t
}.flatten()
}
tasks.sort { a, b -> b.values().sum() <=> a.values().sum() }
tasks = tasks.collect { t -> t.keySet()}
return tasks
}
@NonCPS
def checkLimit(t) {
if(t.values().sum()>timeLimit && t.size()>1){
return true
}else{
return false
}
}
@NonCPS
def canSplit() {
for(t in tasks) {
if(checkLimit(t)){
return true
}
}
return false
}
@NonCPS
def splitTo2(int_map) {
A=[:]
B=[:]
for(n in int_map.sort{it.value}) {
if (A.size() < B.size()) {
A.put(n.key, n.value)
}else{
B.put(n.key, n.value)
}
}
return [A, B]
}
an exception which occurred:
in field delegate
in field closures
in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@7fd2cde1
Caused: java.io.NotSerializableException: java.util.LinkedHashMap$Entry
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:860)
at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOu
...
Michael Pailloncy
--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/e7a79c56-512f-4706-ab65-9a347966abb3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
jer...@bodycad.com
Viacheslav Dubrovskyi
Hi
> Also is the suitesStat properly defined when declared?
Maybe not relevent, but I would declare the function above/before the main stage. Maybe this won't change anything. Also is the suitesStat properly defined when declared? [AutoSuiteSet_0:0 ...]
--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/196f9dc8-3445-4fca-8030-475caaec4191%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
-- WBD, Viacheslav Dubrovskyi
Viacheslav Dubrovskyi
Hi All,
Thank you for answers. Finally I found correct way. Only map.sort
was necessary move under @NonCPS function.
My work pipeline looks:
def suitesMap = [:]
if (suites_time.size()>0){
timeLimit = suites_time.values().max()
for(s in suites){
x = suites_time.find{ artifact -> artifact.key == s}
if(x){
suitesMap.put(x.key, x.value)
}else{
suitesMap.put(s, timeLimit)
}
}
suitesList = suites.collect { t -> [t] }
return suitesList
}
tasks = divideList(suitesMap)
tasks = sortListInMap(tasks)
tasksList = tasks.collect { t -> t.keySet()}
return tasksList
}
def divideList(inputMap){
inputMap=sortMap(inputMap)
maxValue = inputMap.values().max()
def outputSet = []
def outputMap = [:]
for (inputElement in inputMap) {
possibleMap = outputMap + [inputElement]
possibleMapSum = possibleMap.values().sum()
if (possibleMapSum < maxValue) {
outputMap = possibleMap
}else if(possibleMapSum == maxValue) {
outputSet.add(possibleMap)
outputMap = [:]
}else if(possibleMapSum > maxValue) {
outputSet.add(outputMap)
outputMap = [:]
outputMap.put(inputElement.key, inputElement.value)
}
}
if (outputMap) {
outputSet.add(outputMap)
}
return outputSet
}
@NonCPS
def sortMap(map) {
map.sort{a, b -> b.value <=> a.value }
}
@NonCPS
def sortListInMap(map) {
map.sort { a, b -> b.values().sum() <=> a.values().sum() }
}
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAPO77c0_uu7GwQGL3GNGoyh-D878mrbiMj-1fUZefgN0bRPmsA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
-- WBD, Viacheslav Dubrovskyi
Viacheslav Dubrovskyi
Hi Michael,
We have list of tests which I would like run in parallel. The
Parallel Test Executor Plugin doesn't work in our case, because we
don't use junit and the list of tests (behatList) is
result of script work. I can't run all tests on one node because
it will take about 16 hours.
So I save results of previous builds to groovy file as map and
load it before (suitesStat) . Then I group and sort tests
(sortSuites(suites, suites_time) in previous post). And
then I use this list for create parallel run.
def enviroments_b = [failFast: failFirstError]
for (int j = 0; j < behatList.size() ; j++) {
int index_b=j
enviroments_b["TestEnv behat=${behatList[index_b]}"] = {
node('behat') {
for (int j2 = 0; j2 < behatList[index_b].size() ; j2++) {
int index_b2=j2
sh "./run_test ${behatList[index_b][index_b2]}"
}
}
}
}
parallel enviroments_b
}
...
However, what do you want to achieve exactly ?
-- WBD, Viacheslav Dubrovskyi
Michael Pailloncy
--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/bad445be-212a-6f0b-7001-0c9212761be1%40gmail.com.
Viacheslav Dubrovskyi
Hi, thanks for the full explanation of your use case :-)How do you keep the Groovy file containing your previous builds result ? inside an agent's workspace ?
def stat_file = "suite_statics.groovy"
for (t2 in behatList) {
suitesStat.put(behatList[t][t2], duration_s)
}
writeFile file: "stats/${stat_file}", text: "suitesStat=${suitesStat}\nreturn this;"
and then save stats/${stat_file} as artifact
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAPO77c3TeEWZEsxNHXb-y0nRg%3DZ5%2BVnveqVcZuiRBj7GkttjBQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
-- WBD, Viacheslav Dubrovskyi
Viacheslav Dubrovskyi
Hi Michael,
I had work pipeline job. But after add more environments (envList) job broke with log:
[Pipeline] End of Pipeline
an exception which occurred:
in field delegate
in field closures
in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@1845f26
Caused: java.io.NotSerializableException:
java.util.LinkedHashMap$Entry
at
org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:860)
at
org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
at
org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
at
org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)
at
org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)
at
java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
at
java.util.LinkedHashMap.internalWriteEntries(LinkedHashMap.java:333)
at java.util.HashMap.writeObject(HashMap.java:1362)
at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown
Source)
I use NonCPS function sortSuites() and it always return list. I don't have more ideas how to fix. Can you help?
Thank you!
My test pipeline:
#!groovy
def envList = []
envList.add('NODE=behat3 ORO_PHP=7.1
ORO_APP=application/commerce-crm-ee ORO_TEST_SUITE=behat
ORO_DB=pgsql BEHAT_MODE=standalone')
envList.add('NODE=docker-doc ORO_TEST_SUITE=documentation
ORO_APP=documentation/crm ORO_DOC=OroCRM_Documentation')
envList.add('NODE=docker-doc ORO_TEST_SUITE=documentation
ORO_APP=documentation/commerce ORO_DOC=OroCommerce_Documentation')
envList.add('NODE=func_2cpu ORO_PHP=7.1
ORO_APP=application/crm-enterprise ORO_TEST_SUITE=functional
ORO_DB=PG ORO_INSTALLED=crm-enterprise_1.12 ORO_SE=ES
ES_VER=2.4.3 UPGRADE=true
TEST_RUNNER_OPTIONS=--group=search,dist')
envList.add('NODE=func_2cpu ORO_PHP=7.1
ORO_APP=application/crm-enterprise ORO_TEST_SUITE=functional
ORO_DB=PG ORO_INSTALLED=crm-enterprise_2.0.0 ORO_SE=ES
ES_VER=2.4.3')
envList.add('NODE=func_2cpu ORO_PHP=7.0
ORO_APP=application/crm-enterprise ORO_TEST_SUITE=functional
ORO_DB=PG ORO_INSTALLED=crm-enterprise_1.12 UPGRADE=true')
envList.add('NODE=func ORO_PHP=7.1
ORO_APP=application/commerce-enterprise ORO_TEST_SUITE=functional
ORO_DB=PG ORO_INSTALLED=commerce_1.3.0 ORO_SE=ES
ES_VER=2.3.5')
envList.add('NODE=func ORO_PHP=7.0
ORO_APP=application/commerce-enterprise ORO_TEST_SUITE=functional
ORO_DB=MYSQL ORO_INSTALLED=commerce_1.0.0')
envList.add('NODE=func ORO_PHP=7.1
ORO_APP=application/commerce ORO_TEST_SUITE=functional
ORO_DB=PG ORO_INSTALLED=commerce_1.0.0')
envList.add('NODE=func ORO_PHP=7.1
ORO_APP=application/commerce-crm ORO_TEST_SUITE=functional
ORO_DB=PG ORO_ENABLE_PRICE_SHARDING=true')
envList.add('NODE=func ORO_PHP=7.1
ORO_APP=application/commerce-crm-ee ORO_TEST_SUITE=functional
ORO_DB=PG ORO_INSTALLED=commerce_1.3.0')
envList.add('NODE=func ORO_PHP=7.1
ORO_APP=application/commerce-crm-ee ORO_TEST_SUITE=functional
ORO_DB=MYSQL ORO_INSTALLED=crm-enterprise_1.12 UPGRADE=true')
envList.add('NODE=func_2cpu ORO_PHP=7.1
ORO_APP=application/platform ORO_TEST_SUITE=functional
ORO_DB=PG')
envList.add('NODE=func_2cpu ORO_PHP=7.1
ORO_APP=application/crm ORO_TEST_SUITE=functional
ORO_DB=MYSQL')
envList.add('NODE=func_2cpu ORO_PHP=7.1
ORO_TEST_SUITE=behat_wiring')
envList.add('NODE=func ORO_PHP=7.1 ORO_TEST_SUITE=javascript
JS=YES CS=YES')
envList.add('NODE=func_2cpu ORO_PHP=7.0
ORO_APP=application/commerce-crm-ee ORO_TEST_SUITE=unit')
envList.add('NODE=func_2cpu ORO_PHP=7.1
ORO_APP=application/commerce-crm-ee ORO_TEST_SUITE=unit')
def behatList = []
def enviroments = [:]
for (int i = 0; i < envList.size() ; i++) {
int index=i, e = i+1
enviroments["TestEnv_${e}"] = {
stage ("TestEnv_${e}"){
withEnv(["NETWORK=${index}"] + ["TESTENV=TestEnv_${e}"] +
envList[index].tokenize()) {
if (env.ORO_TEST_SUITE == 'behat') {
node() {
ws("${HOME}/workspace/dev_${EXECUTOR_NUMBER}") {
sh """
echo "Install"
"""
behatStr = sh (
script: 'echo AutoSuiteSet_1 AutoSuiteSet_2
AutoSuiteSet_3',
returnStdout: true
).trim()
}
}
behatList=behatStr.tokenize()
echo "SUITES=${behatList}"
behatList=sortSuites(behatList)
echo "SUITES SORTED=${behatList}"
echo "behatList TYPE=${behatList.getClass()}"
def enviroments_b = [:]
for (int j = 0; j < behatList.size() ; j++) {
int index_b=j
enviroments_b["TestEnv_${e}
behat=${behatList[index_b]}"] = {
withEnv(["BEHAT_SUIT=${behatList[index_b]}"])
{
node() {
ws("${HOME}/workspace/dev_${EXECUTOR_NUMBER}") {
sh """
echo "Run
BEHAT_SUIT=$BEHAT_SUIT"
"""
} //workspace
} //node
} //withEnv
} //enviroments_b
} //for
try {
parallel enviroments_b
} catch (error) {
throw (error)
}
} else if (env.ORO_TEST_SUITE == 'functional' ||
env.ORO_TEST_SUITE == 'unit' || env.ORO_TEST_SUITE ==
'behat_wiring' || env.ORO_TEST_SUITE == 'javascript'){
node() {
ws("${HOME}/workspace/dev_${EXECUTOR_NUMBER}") {
sh """
echo "Run $ORO_TEST_SUITE"
"""
}
}
} else {
node() {
ws("${HOME}/workspace/dev_${EXECUTOR_NUMBER}") {
sh """
echo "Run $ORO_TEST_SUITE"
"""
}
}
}
}
}
}
}
try {
parallel enviroments
} catch (error) {
throw (error)
}
@NonCPS
def sortSuites(suites){
def suitesStat=[AutoSuiteSet_1:435, AutoSuiteSet_2:323,
AutoSuiteSet_3:363]
def suites_time = suitesStat
// def suites_time = [:]
def suitesMap = [:]
if (suites_time.size()>0){
def timeLimit = suites_time.values().max() //find max
value
for(s in suites){
x = suites_time.find{ artifact -> artifact.key == s}
if(x){
suitesMap.put(x.key, x.value)
}else{
suitesMap.put(s, timeLimit)
}
}
}else{
def suitesList = suites.collect { t -> [t] }
return suitesList
}
tasks = divideList(suitesMap)
tasks = sortListInMap(tasks)
tasksList = tasks.collect { t -> t.keySet()}
return tasksList
}
@NonCPS
def divideList(inputMap){
// Sort an incoming array in descending order
inputMap=sortMap(inputMap)
maxValue = inputMap.values().max()
// Resulting set of maps
def outputSet = []
// Variable for storing an individual map
def outputMap = [:]
for (inputElement in inputMap) {
// We add to the temporary map and consider the resulting sum of the maps items
possibleMap = outputMap + [inputElement]
possibleMapSum = possibleMap.values().sum()
// If the amount in the temporary map is less than maxValue, go to the next iteration, we will try to supplement the map with one more element
if (possibleMapSum < maxValue) {
outputMap = possibleMap
// If the amount in the temporary map is maxValue, add a temporary map to the result set and zero the temporary map
}else if(possibleMapSum == maxValue) {
outputSet.add(possibleMap)
outputMap = [:]
// If the amount in the temporary map exceeded maxValue, add the previous version of the temporary map to the result set (without the current element), and create a new temporary map with the current element
}else if(possibleMapSum > maxValue) {
outputSet.add(outputMap)
outputMap = [:]
outputMap.put(inputElement.key, inputElement.value)
}
}
if (outputMap) {
outputSet.add(outputMap)
}
return outputSet
}
@NonCPS
def sortMap(map) {
map.sort{a, b -> b.value <=> a.value }
}
@NonCPS
def sortListInMap(map) {
map.sort { a, b -> b.values().sum() <=>
a.values().sum() }
}
It seems like there are some variables used out of their scope and some misused Groovy methods.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAPO77c0_uu7GwQGL3GNGoyh-D878mrbiMj-1fUZefgN0bRPmsA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
-- WBD, Viacheslav Dubrovskyi
Reinhold Fuereder
Hi
I think it is the usage of the (default?) map in pipeline code, e.g. "def enviroments = [:]" which according to the StackTrace results in a LinkedHashMap?
According to https://jenkins.io/blog/2017/02/01/pipeline-scalability-best-practice/#appendix-serializable-vs.-non-serializable-types "Maps: normal Groovy Map, HashMap, TreeMap" are said to be okay though -- not sure if this is still up-to-date…
But maybe you wanted to use a List, or re-structure your pipeline code completely? Or moved some of the logic into a shared pipeline library instead?
Cu Reinhold
Viacheslav Dubrovskyi
Hi Reinhold,
Thank you for answer.
>I think it is the usage of the (default?) map in pipeline code, e.g. "def enviroments = [:]" which according to the StackTrace results in a LinkedHashMap?
Hm. If I use empty map suites_time = [:] in function sortSuites(), everything working good.
But all maps used only in @NonCPS functions. It's why I'm confused.
Will try to move some logic into pipeline library.
--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/C3D41C0B1D5991438F1A43F586F4B47F7F8731D5%40TIGER2010.xortex.local.
For more options, visit https://groups.google.com/d/optout.
-- WBD, Viacheslav Dubrovskyi