Reading a file from the workspace

[Misty Stanley-Jones]

I set up a CSV file with my job parameters, and all the jobs are built from there. I originally had this weird system that turned the CSV into XML and used the XML slurper to read the values in. That worked OK for some reason, but reading the CSV directly is not. Here is some more info:

Both the DSL (build_all_jobs.groovy) and the CSV (doc_builds.csv) are in the jenkins_templates/ directory in the workspace (from Git).

In the Jenkins job, I configured it to "look on the filesystem" for the DSL, and pointed it to jenkins_templates/build_all_jobs.groovy.

I'm having trouble getting the DSL to "see" the CSV file. I load it like this (I also tried "doc_builds.csv" and "jenkins_templates/doc_builds.csv" in the File() invocation):

new File( "${WORKSPACE}/jenkins_templates/doc_builds.csv" ).withReader { r ->
    parseCsv( r ).each { ...

In the Jenkins job, I added a step to run some commands before the DSL, so I could debug what's visible to the Jenkins job itself:

Current working directory is /var/lib/jenkins/workspace/generate_doc_jobs
Contents of the jenkins_templates directory are
total 132
-rw-rw-r-- 1 jenkins jenkins 27109 Apr 20 21:06 build_all_jobs.groovy
-rw-rw-r-- 1 jenkins jenkins 10373 Apr 20 20:40 doc_builds.csv
-rw-rw-r-- 1 jenkins jenkins 13443 Apr 20 20:40 doc_builds.json
-rw-rw-r-- 1 jenkins jenkins 45056 Apr 20 20:40 doc_builds.xls
-rw-rw-r-- 1 jenkins jenkins  2535 Apr 20 20:40 GroovyExcelParser.groovy
-rw-rw-r-- 1 jenkins jenkins  5794 Apr 20 20:40 html_build.dsl
-rw-rw-r-- 1 jenkins jenkins  5569 Apr 20 20:40 pdf_build.dsl
-rw-rw-r-- 1 jenkins jenkins  6052 Apr 20 20:40 qa_build.dsl
-rw-rw-r-- 1 jenkins jenkins   474 Apr 20 20:40 README.adoc


Then it sets up Groovy and launches the DSL, which prints out the contents of user.dir and user.home to start with, then tries to load the CSV and bombs out with a FileNotFoundException:


Setting GROOVY_1_8_7_HOME=/var/lib/jenkins/tools/hudson.plugins.groovy.GroovyInstallation/groovy_1.8.7
Working directory is /
Home directory is /var/lib/jenkins
FATAL: Unable to run script
java.io.IOException: Unable to run script
	at javaposse.jobdsl.dsl.DslScriptLoader.runDslEngineForParent(DslScriptLoader.java:74)
	at javaposse.jobdsl.dsl.DslScriptLoader.runDslEngine(DslScriptLoader.java:93)
	at javaposse.jobdsl.plugin.ExecuteDslScripts.perform(ExecuteDslScripts.java:150)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:785)
	at hudson.model.Build$BuildExecution.build(Build.java:199)
	at hudson.model.Build$BuildExecution.doRun(Build.java:160)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:566)
	at hudson.model.Run.execute(Run.java:1665)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:246)
Caused by: java.io.FileNotFoundException: /var/lib/jenkins/workspace/generate_doc_jobs/jenkins_templates/doc_builds.csv (No such file or directory)
	at java.io.FileInputStream.open(Native Method)
	at java.io.FileInputStream.<init>(FileInputStream.java:146)
	at groovy.util.CharsetToolkit.<init>(CharsetToolkit.java:69)
	at org.codehaus.groovy.runtime.DefaultGroovyMethods.newReader(DefaultGroovyMethods.java:16042)
	at org.codehaus.groovy.runtime.DefaultGroovyMethods.withReader(DefaultGroovyMethods.java:16099)
	at org.codehaus.groovy.runtime.dgm$914.invoke(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271)
	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
	at build_all_jobs.run(build_all_jobs.groovy:8)
	at javaposse.jobdsl.dsl.DslScriptLoader.runDslEngineForParent(DslScriptLoader.java:69)
	... 11 more

Any ideas what dumb thing I am doing wrong this time?

[Misty Stanley-Jones]

I found a workaround, reading the CSV file as a URL straight from Github:

new URL( "http://my_repo/raw/master/jenkins_templates/doc_builds.csv" ).withReader { r ->
    parseCsv( r ).each {

It works but there is an obvious potential for a race condition if someone pushes an update to that file after the job is triggered by Github push and before it has read the file. Is there a better workaround I could use for this? If this is the only problem, I think I can live with it. I already have it managing 118 jobs for me!!! I'm actually using this to manage documentation building jobs using the DITA Open Toolkit, and I'd like to do a write-up about it once things are a bit more established.

Thanks,
Misty

[Marshall Pierce]

Workarounds are good, but solving the initial problem is better… :) 

Can you verify that you can read *other* files in the same directory? If so, if you touch a new file in that directory, can you read it too? I’m wondering if despite your logging you’re actually reading from somewhere unexpected.

-- 
You received this message because you are subscribed to the Google Groups "job-dsl-plugin" group.
To unsubscribe from this group and stop receiving emails from it, send an email to job-dsl-plugi...@googlegroups.com.
To post to this group, send email to job-dsl...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/job-dsl-plugin/a52900a2-96ff-4285-a531-9e775b927988%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Misty Stanley-Jones]

The Bash script sure can read other files. I'm so new to Groovy that I'm not 100% sure how I would go about reading a file, besides the way I tried to do it above. FWIW, that method works in a small testing script I wrote. I can't run the DSL script outside of Jenkins, I think. So the type of testing I can do is limited. I'm open to suggestions.

[Marshall Pierce]

You can run the DSL outside of Jenkins, actually. At some point I’ll write up what we do here with pre-testing our DSL before it gets deployed, but (1) make sure you’re using a pretty recent version of the DSL and (2) try doing something like these snippets:

----------

JobManagement jm = new MemoryJobManagement();

URL urlRoot = yourJobRepoRoot();
Path jobFilePath = pathToDsl.groovy
ScriptRequest scriptRequest =
        new ScriptRequest(null, new String(readAllBytes(jobFilePath), UTF_8), urlRoot);

DslScriptLoader.runDslEngine(scriptRequest, jm);

----------

The way you’re reading files with groovy is perfectly reasonable. If you can paste more of your bash and groovy it might help.

To view this discussion on the web visit https://groups.google.com/d/msgid/job-dsl-plugin/6dec18dc-9da9-43fb-89df-a11c95cc45b6%40googlegroups.com.

[Misty]

We are definitely not running the newest version of the DSL plugin. We are running 1.20, and I think that our admin may have made some local changes to it as well. I've asked for it to be upgraded (not least of which because I would like some of the Github convenience APIs), but it has not been done yet.

I tried running the script locally like you said, but I'm not sure how to get the MemoryJobManagement package imported. I also don't know what the yourJobRepoRoot() should be, just a quoted string? i get:

/var/lib/jenkins/workspace/generate_doc_jobs/test3.groovy: 1: unable to resolve class MemoryJobManagement
 @ line 1, column 20.

   JobManagement jm = new MemoryJobManagement();

I'll be unavailable most of tomorrow but I'll try to look at it in the next few days. In the meantime, the URL seems good enough. My next hurdle is how to handle the one field in the CSV that needs to be a space-separated list (within the one field). That's very inelegant and is going to cause headaches down the lane. It's always something. ;)

Thanks,

Misty

[Misty Stanley-Jones]

Solved! With careful reading of the docs, I came up with this:

streamFileFromWorkspace('jenkins_templates/doc_builds.csv').withReader { r ->

Posted for any other poor newbies who can't figure it out.

On Tuesday, April 21, 2015 at 2:26:11 PM UTC+10, Misty Stanley-Jones wrote: