[JIRA] (JENKINS-57173) Issue with Darcs SCM plugin saving build records

11 views
Skip to first unread message

msollanych@dwavesys.com (JIRA)

unread,
Apr 24, 2019, 2:55:02 PM4/24/19
to jenkinsc...@googlegroups.com
Mike Sollanych created an issue
 
Jenkins / Bug JENKINS-57173
Issue with Darcs SCM plugin saving build records
Issue Type: Bug Bug
Assignee: Sven Strittmatter
Components: darcs-plugin
Created: 2019-04-24 18:54
Environment: Jenkins 2.164.2 with darcs plugin 0.3.11
(Behaviour has been the same for the past few Jenkins releases)
Labels: darcs
Priority: Minor Minor
Reporter: Mike Sollanych

When building a job backed on the Darcs SCM plugin, we get this nasty exception. The result is that the build.xml file isn't always saved properly, and we get disappearing builds in the history.

 

Apr 24, 2019 11:43:41 AM org.jenkinsci.plugins.darcs.DarcsChangeLogParser parse
INFO: Parsing changelog file /var/lib/jenkins/jobs/Processor Development (PD)/jobs/Typhon/jobs/Tests/jobs/Load Tests/builds/50/changelog.xml...
Apr 24, 2019 11:43:41 AM org.jenkinsci.plugins.darcs.DarcsSaxHandler endElement
INFO: Ignored tag <CHANGELOG>!
Apr 24, 2019 11:43:41 AM jenkins.plugins.slack.logging.SlackNotificationsLogger info
INFO: Processor Development (PD) » Typhon » Tests » Load Tests #50 found #49 as previous completed, non-aborted build
Apr 24, 2019 11:43:41 AM hudson.model.Run execute
SEVERE: Failed to save build record
java.io.IOException: java.lang.RuntimeException: Failed to serialize hudson.model.AbstractBuild#scm for class hudson.model.FreeStyleBuild
at hudson.XmlFile.write(XmlFile.java:200)
at hudson.model.Run.save(Run.java:2012)
at hudson.model.Run.execute(Run.java:1873)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: java.lang.RuntimeException: Failed to serialize hudson.model.AbstractBuild#scm for class hudson.model.FreeStyleBuild
at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)
at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)
at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)
at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
at hudson.XmlFile.write(XmlFile.java:193)
... 5 more
Caused by: java.lang.RuntimeException: Failed to serialize org.jenkinsci.plugins.darcs.DarcsChangeLogParser#sanitizer for class org.jenkinsci.plugins.darcs.DarcsChangeLogParser
at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
... 18 more
Caused by: java.lang.RuntimeException: Failed to serialize org.jenkinsci.plugins.darcs.DarcsXmlSanitizer#decoders for class org.jenkinsci.plugins.darcs.DarcsXmlSanitizer
at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
... 27 more
Caused by: java.lang.UnsupportedOperationException: Refusing to marshal sun.nio.cs.UTF_8$Decoder for security reasons; see https://jenkins.io/redirect/class-filter/
at hudson.util.XStream2$BlacklistedTypesConverter.marshal(XStream2.java:546)
at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88)
at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64)
at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:74)
at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
... 36 more

Apr 24, 2019 11:44:09 AM org.jenkinsci.plugins.darcs.DarcsSaxHandler fatalError
WARNING: org.xml.sax.SAXParseException; lineNumber: 167902; columnNumber: 1; An invalid XML character (Unicode: 0x3) was found in the element content of the document.
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

msollanych@dwavesys.com (JIRA)

unread,
Apr 29, 2019, 12:51:04 PM4/29/19
to jenkinsc...@googlegroups.com
Mike Sollanych commented on Bug JENKINS-57173
 
Re: Issue with Darcs SCM plugin saving build records

Just to follow up, I implemented the workaround for serializing classes via these JVM options:

 

-Dhudson.remoting.ClassFilter=org.jenkinsci.plugins.darcs.DarcsXmlSanitizer,org.jenkinsci.plugins.darcs.DarcsChangeLogParser,sun.nio.cs.UTF_8

 

and while it cleared up the logs, the build history still selectively disappears - particularly for jobs that succeed, it seems, while failed jobs are sticking around.

ich@weltraumschaf.de (JIRA)

unread,
Apr 29, 2019, 2:19:02 PM4/29/19
to jenkinsc...@googlegroups.com
Sven Strittmatter updated an issue
 
Change By: Sven Strittmatter
When building a job backed on the Darcs SCM plugin, we get this nasty exception. The result is that the build.xml file isn't always saved properly, and we get disappearing builds in the history.

  {code}

{code}

ich@weltraumschaf.de (JIRA)

unread,
Apr 29, 2019, 2:22:02 PM4/29/19
to jenkinsc...@googlegroups.com
Sven Strittmatter commented on Bug JENKINS-57173
 
Re: Issue with Darcs SCM plugin saving build records

First I have to admit that I'm really impressed that some still uses this plugin. I'm quite a little bit out of this plugin. I'm also impressed that the plugin still works current Jnkins versions. Anyway...

I'll take a look at it in the next days.

msollanych@dwavesys.com (JIRA)

unread,
Apr 29, 2019, 2:25:02 PM4/29/19
to jenkinsc...@googlegroups.com

Thanks, Sven. I will admit that I had never worked with Darcs before my current employer, and was amazed there even was a Jenkins plugin. I'll do my best to prioritize testing anything you suggest. If you need logs or anything, let me know.

ich@weltraumschaf.de (JIRA)

unread,
May 5, 2019, 3:36:02 PM5/5/19
to jenkinsc...@googlegroups.com

I digged into it: 0x3 seems to be a control character (ETX) which is not allowed in XML. There is one in the parsed log output. Could you please look into the Darcs XML log output at given line/column mentioned at the bottom of the exception?

Maybe darcs is screwing up the log output?

I'm not sure if I can handle this malformed XML input in an useful way.

msollanych@dwavesys.com (JIRA)

unread,
May 6, 2019, 11:52:02 AM5/6/19
to jenkinsc...@googlegroups.com

Thanks for looking into this. I'm a bit new to Darcs myself, so I'm not sure how to determine what commands the Darcs SCM plugin is running to get the xml output. I have used simple xml output commands from Darcs for other purposes unrelated to this issue and have parsed the output with no issue, so there must be some particular command that would reveal where this weird byte is coming from, so I can track its origin down.

 

We do have a variety of bits of older non-Jenkins build automation around that might be automatically adding comments or similar that may be causing an issue with this, so I'll definitely be following up internally with those who know more, but if you have any pointers let me know.

ich@weltraumschaf.de (JIRA)

unread,
May 7, 2019, 3:01:03 AM5/7/19
to jenkinsc...@googlegroups.com
Sven Strittmatter started work on Bug JENKINS-57173
 
Change By: Sven Strittmatter
Status: Open In Progress

ich@weltraumschaf.de (JIRA)

unread,
May 7, 2019, 3:08:03 AM5/7/19
to jenkinsc...@googlegroups.com

The used Darcs command is darcs changes --xml-output --summary.

ich@weltraumschaf.de (JIRA)

unread,
May 7, 2019, 9:15:03 AM5/7/19
to jenkinsc...@googlegroups.com

Did this issue start occuring after upgrading your Jenkins?

msollanych@dwavesys.com (JIRA)

unread,
May 7, 2019, 2:17:01 PM5/7/19
to jenkinsc...@googlegroups.com

Hah. We have traced this to a nasty Unicode character in a commit from 2009 - for whatever reason the SCM plugin pulls the entire history of the repo, which actually takes quite a while to run. I think Darcs has some functions for limiting this in the time domain, so if you are working on the plugin that'd be something to consider adding in.

 

We're going to see if we can find a way to remove the byte. Darcs itself should be escaping it, e.g. as an ` ` or similar, but I suppose it's too edgy of a case for that.

 

We upgrade Jenkins regularly and this issue reared its head as a part of the recent trust / serializer changes, but I think may have been an issue even beforehand.

msollanych@dwavesys.com (JIRA)

unread,
May 7, 2019, 6:26:02 PM5/7/19
to jenkinsc...@googlegroups.com

But maybe all of the failures to serialize come back to that bad commit, as opposed to whitelisting?

msollanych@dwavesys.com (JIRA)

unread,
May 7, 2019, 6:26:03 PM5/7/19
to jenkinsc...@googlegroups.com

The Broken Build plugin also generates these typical JEP-200 errors in conjunction with the Darcs plugin, so we're seeing aspects of that issue since this plugin was seemingly never update for the marshalling whitelisting change.

 

In particular, my Java command line option of:

 

-Dhudson.remoting.ClassFilter=org.jenkinsci.plugins.darcs.DarcsXmlSanitizer,org.jenkinsci.plugins.darcs.DarcsChangeLogParser,sun.nio.cs.UTF_8,hudson.model.AbstractBuild,hudson.model.FreeStyleBuild,hudson.util.RobustReflectionConverter

is not enough to not still get hit with

May 07, 2019 3:15:45 PM hudson.model.Run execute


SEVERE: Failed to save build record
java.io.IOException: java.lang.RuntimeException: Failed to serialize hudson.model.AbstractBuild#scm for class hudson.model.FreeStyleBuild

ich@weltraumschaf.de (JIRA)

unread,
May 8, 2019, 3:28:02 AM5/8/19
to jenkinsc...@googlegroups.com

I think we have two issues here:

  1. A bad character in the log output.
  2. A regression due to JEP-200.

Why I slurp the whole log I can't tell. I guess I had a reason at the time I developed the plugin.

I try to figure out how to fix the JEP-200 issue. The problem that I face is that it is long time ago I did the plugin and worked with Darcs. I do not use Darcs or the plugin since years. So it takes some time for me to understand what I did and why I did it.

msollanych@dwavesys.com (JIRA)

unread,
May 8, 2019, 12:31:02 PM5/8/19
to jenkinsc...@googlegroups.com

OK, sounds good. I really appreciate your work on this. If there's some way I can buy you a coffee, let me know

 

I'm working on putting in a quick filter - just a wrapper for Darcs that sed's out the bad character in question. If I can get that to work, we'll see if the serializer is happy and in turn stops deleting build history on me. If so, it's possible that pushing upstream on the Darcs maintainers could get us a newer version where the character is properly escaped before hitting XML, as it's actually invalid XML to emit like that instead of as an entity.... but there's no entity for it.

 

From one of my colleagues here:

"It looks like darcs handling of char 03 is a feature, not a bug. Darcs does not restrict what characters can be used in comments, but can't generate valid XML output if a comment uses an invalid XML character. Apparently, char 03 has no XML representation. is not allowed, because character references must represent valid characters."

 

Not sure I'd call it a feature, but hey. I'll let you know if the wrapper-filter works.

msollanych@dwavesys.com (JIRA)

unread,
May 15, 2019, 12:26:02 PM5/15/19
to jenkinsc...@googlegroups.com

My colleague has produced a fix for the plugin that seems to work. I'll encourage him to create a PR for you, or do it myself. Is there some process I should be aware of, or just fork and submit one?

iperminov@dwavesys.com (JIRA)

unread,
May 15, 2019, 2:03:02 PM5/15/19
to jenkinsc...@googlegroups.com
Ilya Perminov updated an issue
 
Change By: Ilya Perminov
Attachment: invalid-xml-char.patch
Attachment: modernization.patch
Attachment: serialization.patch

iperminov@dwavesys.com (JIRA)

unread,
May 15, 2019, 2:10:01 PM5/15/19
to jenkinsc...@googlegroups.com
Ilya Perminov commented on Bug JENKINS-57173
 
Re: Issue with Darcs SCM plugin saving build records

I work for the same company as Mike. I fixed the issues locally. The fixes are trivial, but making the plugin compile with the modern versions of  Jenkins and some libraries took some time. My fixes seem to work fine. I attached 3 patches.

 

ich@weltraumschaf.de (JIRA)

unread,
May 16, 2019, 3:57:02 AM5/16/19
to jenkinsc...@googlegroups.com

I'm not shure how the process is nowadays. As mentioned: It's really long time ago that I made this plugin

Patch files are a god way: I'll look inside them at the weekend. And then I figure out how to release and deploy a fixed version. Also I tried to make the plugin compatibility test work, but it looks like that this is a major issue. This darcs plugin is really outdated.

ich@weltraumschaf.de (JIRA)

unread,
May 22, 2019, 3:42:02 AM5/22/19
to jenkinsc...@googlegroups.com

In principle the patches looks good. Is it by accident that there are no commit messages?

What I understand: invalid-xml-char.patch and serialization.patch are the required solutions to solve your issues. modernization.patch is not that essential.

So plan is: Branch 3.3.x version and incorporate invalid-xml-char.patch and serialization.patch into it and try to release a 3.3.12 of the plugin.

The modernization is a not appropriate for a patchlevel release. AFAICS it is at least a minor maybe a mojor. There is also the chance of breaking changes. Also there is an old branch 4.0.0 in which I started to clean up things, add more documentation and write more tests. So my plan is to resurect the 4.0.0 branch and incorporate the modernization patch there and got to 1.0.0. Because I do not plan to add new features. But I won't to fullfil the plugin compatibility test, so that the few users of this plugin can use it safe in the future.

But I beg for patience! I have the find out all the rhings I did years ago and why in this plugin and it is not that clean coded as I would expect it nowadays from me

iperminov@dwavesys.com (JIRA)

unread,
May 22, 2019, 2:28:03 PM5/22/19
to jenkinsc...@googlegroups.com

The "modernization" patch is actually a set of changes I had to make to be able to compile the plugin using Java 8 and the current versions of Jenknis and Maven. I do not have any experience building Jenkins plugins, so it is very likely I setup my environment incorrectly. If you do not need these changes to successfully build the plugin, the patch is not needed.

The serialization patch is the main one as it makes the plugin compatible with JEP-200 Jenkins versions.

 

ich@weltraumschaf.de (JIRA)

unread,
May 30, 2019, 2:33:02 AM5/30/19
to jenkinsc...@googlegroups.com

I incorporated your fixes into the master https://github.com/Weltraumschaf/darcs-plugin. You can try to build the plugin from my sources and test it. I try to figure out how to release and deploy it. It's so long ago, I have no clue how to do it

For the Maven build to work you need a Mirror in your settings.xml:

<mirror>
  <id>repo.jenkins-ci.org</id>
  <mirrorOf>m.g.o-public</mirrorOf>
  <url>http://repo.jenkins-ci.org/public/</url>
</mirror>

iperminov@dwavesys.com (JIRA)

unread,
May 30, 2019, 6:27:01 PM5/30/19
to jenkinsc...@googlegroups.com

I compiled your new version without any issues and tested it with the latest Jenkins LTS. It works correctly. Thank you very much!
Reply all
Reply to author
Forward
0 new messages