[JIRA] [workflow-plugin] (JENKINS-27413) Handle file parameters

[yann@pleiades.fr.eu.org (JIRA)]

Yann Rouillard commented on JENKINS-27413

Re: Handle file parameters I am not sure I understand the issue's description but we do encounter a problem with workflow and file parameters. We observe that the file parameter is not at all taken into account in the workflow parameter: there is no environment variable defined at all concerning the parameter, the file uploaded is not copied into the workflow workspace. Which basically makes the file parameter completely useless for a job. The same setup works for a standard job. Are we are doing something wrong or is it a bug in the workflow plugin as this current issue seems to indicate ? Add Comment

This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-27413

Re: Handle file parameters Also need to check how we could send file parameters from a Workflow build, using the build step. Presumably you would want the actual file to come from the contextual workspace.

Add Comment

[johan.warlander@kambi.com (JIRA)]

Johan Wärlander commented on JENKINS-27413

Re: Handle file parameters Is there any way to work around this issue while keeping everything in a pipeline job? I have a process where we run a job to pre-process some data, then save the result as an artifact; we then use the Promoted Builds plugin to indicate that the result was OK (after a manual review), and to trigger a second job to load that data using an ETL tool. The second job, then, has to use the Copy Artifact plugin. While this works, it feels less direct than I'd like it to be; also, the user has to go to the second job to see the actual result of the ETL load.

Add Comment

[j.r.hodgson@gmail.com (JIRA)]

Jon Hodgson commented on JENKINS-27413

Re: Handle file parameters I've just run into this issue, in fact I've run into it at an earlier level that the file just doesn't seem to be uploaded at all (you can't view it in the "parameters" page of the build, the file name and link are there, but clicking it gives an error rather than the file contents). This is a major issue, because I need to set up try-before-commit functionality.

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick updated an issue

Jenkins / JENKINS-27413 Handle file parameters Change By: Jesse Glick Component/s: pipeline-build-step-plugin Component/s: workflow-cps-plugin Component/s: pipeline

Add Comment

This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)

[oren@chapo.co.il (JIRA)]

Oren Chapo commented on JENKINS-27413

Re: Handle file parameters I'm in the same boat with Jon Hodgson: trying to set up try-before-commit. All my jobs are pipelines, most of them are pretty complex so I can't go back to freestyle jobs... I'm stuck with missing file upload functionality. For the sake of documentation, if someone knows of a workaround please post it here.

Add Comment

[j.r.hodgson@gmail.com (JIRA)]

Jon Hodgson commented on JENKINS-27413

Re: Handle file parameters I bent over backwards trying to get it to work with string parameters, but between python (which I use for my launch script), Groovy and HTTP trying to be helpful something always got auto-converted (early tests would seem great, then I would try a real-world diff and there would be something in there that broke, such as accented characters in files that had different formats) In the end I went with a smple FTP serverr on the same machine as Jenkins, and uploading a file with a name given by a job I pass as a string parameter. It's not the most neat and tidy solution, but it works at least... well so far.

Add Comment

[johan.warlander@kambi.com (JIRA)]

Johan Wärlander commented on JENKINS-27413

Re: Handle file parameters FYI, if you only need to deal with manual input.. file parameter can be caught in an 'input' step, and then passed on: stage('file input') { node { // Get file using input step, will put it in build directory def inputFile = input message: 'Upload file', parameters: [file(name: 'data.txt')] // Read contents and write to workspace writeFile(file: 'data.txt', text: inputFile.readToString()) // Stash it for use in a different part of the pipeline stash name: 'data', includes: 'data.txt' } } stage('do something with data') { node { // Unstash the file into an 'input' directory in the workspace dir('input') { unstash 'data' } // do something useful sh "ls -lR input/" } } Normally you would have the 'input' step outside of a node, but since we want to put our file in a workspace and stash (and / or archive) it, we'll need the node here.

Add Comment

[geoffrey.arthaud@developpement-durable.gouv.fr (JIRA)]

Geoffrey ARTHAUD commented on JENKINS-27413

Re: Handle file parameters Thank you Johan Wärlander for this solution which works well with text files. But I didn't find any simple solution for binary files, encoding parameter of writeFile is ignored (https://issues.jenkins-ci.org/browse/JENKINS-27094). Here is one solution I've found, by using FilePath API directly : Unable to find source-code formatter for language: groovy. Available languages are: actionscript, html, java, javascript, none, sql, xhtml, xml stage('file input') { node('master') { def inputFile = input message: 'Upload file', parameters: [file(name: 'data.zip')] unzip dir: '', glob: '', zipFile: inputFile.remote } } But two drawbacks : This works only on the master node (I didn't manage to access to the FilePath channel of the workspace from Pipeline plugin) getRemote() method from FilePath is not whitelisted Any suggestion ?

Add Comment

[geoffrey.arthaud@developpement-durable.gouv.fr (JIRA)]

Geoffrey ARTHAUD edited a comment on JENKINS-27413

Re: Handle file parameters

Thank you Johan Wärlander for this solution which works well with text files. But I didn't find any simple solution for binary files, encoding parameter of writeFile is ignored (https://issues.jenkins-ci.org/browse/JENKINS-27094). Here is one solution I've found, by using FilePath API directly :

{code: groovy java }

stage('file input') {   node('master') {     def inputFile = input message: 'Upload file', parameters: [file(name: 'data.zip')]     unzip dir: '', glob: '', zipFile: inputFile.remote   } }

{code} But two drawbacks : * This works only on the master node (I didn't manage to access to the FilePath channel of the workspace from Pipeline plugin) * getRemote() method from FilePath is not whitelisted Any suggestion ?

Add Comment

[geoffrey.arthaud@developpement-durable.gouv.fr (JIRA)]

Geoffrey ARTHAUD edited a comment on JENKINS-27413

Re: Handle file parameters

Thank you Johan Wärlander for this solution which works well with text files. But I didn't find any simple solution for binary files, encoding parameter of writeFile is ignored (https://issues.jenkins-ci.org/browse/JENKINS-27094). Here is one solution I've found, by using FilePath API directly :

{code:java}

stage('file input') {   node('master') {     def inputFile = input message: 'Upload file', parameters: [file(name: 'data.zip')]     unzip dir: '', glob: '', zipFile: inputFile.remote   } } {code} But two drawbacks : * This works only on the master node (I didn't manage to access to the FilePath channel of the workspace from Pipeline plugin) * getRemote() method from FilePath is not whitelisted

Using FilePath.copyTo() or FilePath.unzip() instead of unzip has similar drawbacks. Any suggestion ?

Add Comment

[johan.warlander@kambi.com (JIRA)]

Johan Wärlander commented on JENKINS-27413

Re: Handle file parameters Geoffrey Arthaud, does Base64 encoding change anything in that scenario? Eg. something like: writeFile(file: 'data.zip.b64', text: inputFile.read().getBytes().encodeBase64().toString()) ..and then stash it, and do the decoding / unzipping it in a separate node section? I don't know if it makes a difference w/ master vs slave node execution, as we only have a master node so far in our setup.

Add Comment

[j.r.hodgson@gmail.com (JIRA)]

Jon Hodgson commented on JENKINS-27413

Re: Handle file parameters I personally gave up trying to pass the file as a parameter, as a text file it always got messed up somewhere along the line (it's a diff, so although it's text the encoding can vary depending on the source files). So I siwtched to creating a zip with a uri as the filename, uploading it with ftp, and passing the uri as a parameter. It's perhaps not as elegant as doing it all in jenkins, but it works.

Add Comment

[geoffrey.arthaud@developpement-durable.gouv.fr (JIRA)]

Geoffrey Arthaud commented on JENKINS-27413

Re: Handle file parameters Base64 encoding/decoding should do the trick on any master or slave node. Thank you !

Add Comment

[burtsevyg@mail.ru (JIRA)]

Yuriy Burtsev commented on JENKINS-27413

Re: Handle file parameters "file parameter" and "stash" enough for load file to job running on different nodes.

Add Comment

This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)

[myxal.mxl@gmail.com (JIRA)]

Michal Zatloukal commented on JENKINS-27413

Re: Handle file parameters Any news on this? For the record, I noticed that the file provided through parameter is only missing when using Pipeline from SCM - when using Pipeline script, the file is present and this trivial pipeline completes successfully: parameters { file description: 'Blah', name: 'custom.xml' } pipeline { agent any stages { stage ("cat-file") { steps { powershell 'Get-Content .\\custom.xml' } } } }

Add Comment

[csnyder@us.ibm.com (JIRA)]

Chris Snyder commented on JENKINS-27413

Re: Handle file parameters Took a while to find out that this was an issue. Plus one for getting this fixed in an upcoming release.  UnstashParam worked for me. Thanks.

Add Comment

This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)

[eygohlolz@gmail.com (JIRA)]

Emmanuel Goh commented on JENKINS-27413

Re: Handle file parameters Can confirm that this is still broken - my minimal Jenkinsfile below yields "echo null":   pipeline {   parameters {     file(name: 'ZIP') {{  }}}   }}{{stages {     stage('Test file') { {{      steps { sh "echo ${params.ZIP}" }}} {{    }}} {{  }}} }

Add Comment

[eygohlolz@gmail.com (JIRA)]

Emmanuel Goh edited a comment on JENKINS-27413

Re: Handle file parameters

Can confirm that this is still broken - my minimal Jenkinsfile below yields "echo null":

{ { code:java} pipeline { }}   agent any {{   parameters { }}

{{     file(name: 'ZIP') }} {{   } }} {{   }}{{ stages { }} {{     stage('Test file') { }} {{      steps \ { sh "echo ${params.ZIP}" } } } {{    } }} {{  } }}

{ { code } }}

Add Comment

[baurmatic@gmail.com (JIRA)]

Bauyrzhan Makhambetov commented on JENKINS-27413

Re: Handle file parameters  works in freesyle project but in pipeline is not working.   tried with `ls -l .` with no success, as if there is no file at all.

Add Comment

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-27413

Re: Handle file parameters File parameters are not currently supported. I am not particularly inclined to fix this per se since support for them is quite complicated architecturally—the implementation for freestyle projects does not generalize naturally and relies on a special file upload web request handler. Simpler, safer, and more flexible would be to introduce a plugin defining a Base64-encoded string parameter type, with a form submission GUI and CLI that make it convenient to obtain the value from a local file, as well as a SimpleBuildWrapper (usable as a Pipeline block-scoped step) that decodes the value to a local temporary file within its body.

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick assigned an issue to Unassigned

Jenkins / JENKINS-27413 Handle file parameters

Change By: Jesse Glick Assignee: Jesse Glick

Add Comment

[baurmatic@gmail.com (JIRA)]

Bauyrzhan Makhambetov commented on JENKINS-27413

Re: Handle file parameters Thank you for pointing that out.

Add Comment

[kuisathaverat@gmail.com (JIRA)]

Ivan Fernandez Calvo edited a comment on JENKINS-27413

Re: Handle file parameters my 5 cents, this code shows how to load a properties file into the environment, in the example the properties come from a String but also you can load a file from disk with the readProperties pipeline utility step ``` {code} node {    def configPrperties = """    VAR01 = value1    VAR02 = value2    """    def config = readProperties(text: configPrperties)    config.each { k, v ->        env."${k}" = v    }    sh 'export' } ``` {code}

Add Comment

[kuisathaverat@gmail.com (JIRA)]

Ivan Fernandez Calvo commented on JENKINS-27413

Re: Handle file parameters

my 5 cents, this code shows how to load a properties file into the environment, in the example the properties come from a String but also you can load a file from disk with the readProperties pipeline utility step

```

node { def configPrperties = """ VAR01 = value1 VAR02 = value2 """ def config = readProperties(text: configPrperties) config.each { k, v -> env."${k}" = v } sh 'export' } ```

Add Comment

[alexhaynes93@gmail.com (JIRA)]

Alex Haynes commented on JENKINS-27413

Re: Handle file parameters If there is no plan to support the file parameter option, it would be nice to remove it from the documentation here: https://jenkins.io/doc/book/pipeline/syntax/#parameters

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-27413

Re: Handle file parameters Alex Haynes good point! You can help.

Add Comment

[james.hogarth@gmail.com (JIRA)]

James Hogarth commented on JENKINS-27413

Re: Handle file parameters Avoid actively breaking it though please as as we do actually use this functionality with the help of our global library.

Add Comment

[james.hogarth@gmail.com (JIRA)]

James Hogarth edited a comment on JENKINS-27413

Re: Handle file parameters

Avoid actively breaking it though please as as we do actually use this functionality with the help of our global library.

{code:java} def inputGetFile(String savedfile = null) { def filedata = null def filename = null

// Get file using input step, will put it in build directory

// the filename will not be included in the upload data, so optionally allow it to be specified if (savedfile == null) { def inputFile = input message: 'Upload file', parameters: [file(name: 'library_data_upload'), string(name: 'filename', defaultValue: 'uploaded-file-data')] filedata = inputFile['library_data_upload'] filename = inputFile['filename'] } else { def inputFile = input message: 'Upload file', parameters: [file(name: 'library_data_upload')] filedata = inputFile filename = savedfile

} // Read contents and write to workspace

writeFile(file: filename, encoding: 'Base64', text: filedata.read().getBytes().encodeBase64().toString()) // Remove the file from the master to avoid stuff like secret leakage filedata.delete() return filename }{code} And this library code is used with this jenkinsfile snippet: {code:java} stage('request a file with chosen filename') { node { deleteDir() uploaded_file = library.inputGetFile('my-file-here') sh "file '${uploaded_file}'" sh "ls -la ." } }{code}

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-27413

Re: Handle file parameters Another plugin idea, useful for uploads too large to be reasonably handled as Base64 and environment variables: a parameter type which lets you upload a file to an S3 (or MinIO) bucket. Could either upload from the master upon form/CLI submission; or, more efficiently, provide a special UI & API endpoint to allocate a blob and then hand you a presigned URL good for upload only, valid for an hour. The parameter value would be a presigned URL good for download only, again valid only for a day or so. Similarly for Azure etc. (No need to delete the blob in a RunListener after the build completes. That would break Replay / Rebuild and, as with artifact-manager-s3, would force the master to have a more dangerous IAM permission. Anyway S3 can be configured to automatically expunge or archive old blobs.)

Add Comment

[cfrolik@gmail.com (JIRA)]

Chris Frolik commented on JENKINS-27413

Re: Handle file parameters "I am not particularly inclined to fix this per se" That is very disappointing, and a stumbling block for those trying to convert their freestyle jobs to declarative pipeline jobs. I really hope you change your mind on this. At the very least, the documentation should mention that it isn't supported.

Add Comment

[cfrolik@gmail.com (JIRA)]

Chris Frolik edited a comment on JENKINS-27413

Re: Handle file parameters {quote} "I am not particularly inclined to fix this per se" {quote}

That is very disappointing, and a stumbling block for those trying to convert their freestyle jobs to declarative pipeline jobs. I really hope you change your mind on this. At the very least, the documentation should mention that it isn't supported.

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite commented on JENKINS-27413

Re: Handle file parameters Removed from documentation on jenkins.io July 25, 2019 by PR-2388

Add Comment

[jan.vrany@fit.cvut.cz (JIRA)]

Jan Vrany commented on JENKINS-27413

Re: Handle file parameters Just a notice: I have moved my workaround library with `unstashParam` step to GitHub, new locations is https://github.com/janvrany/jenkinsci-unstashParam-library I also renamed it as this issue seems to be unlikely "fixed".

Add Comment

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)

[reach.vinay.av01@gmail.com (JIRA)]

Vin Win commented on JENKINS-27413

Re: Handle file parameters Hi James Hogarth - Does the solution works for binary/zip file. Tried using groovy script & respective one on Jenkins pipleline, but it doesn't seem to work for zip (Can not open file 'fileName.zip' as archive') or binary file uploaded. Would like to know if this is been considered in future releases ?

Add Comment

[paul.sharpe@zf.com (JIRA)]

Paul Sharpe commented on JENKINS-27413

Re: Handle file parameters Vin Win: The unstashParam workaround works great for us, including zip files. We are using a windows instance of Jenkins (in case that is significant).

Add Comment

[andreas.schmid@bd.so.ch (JIRA)]

Andreas Schmid commented on JENKINS-27413

Re: Handle file parameters I don't quite understand if this is just not implemented yet, or if it's impossible to implement at all. If it is possible, we might be able to fund the development of this feature, as we need it too. As I am still rather new to the Jenkins community: Can anyone recommend me any companies that could do that?

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-27413

Re: Handle file parameters Andreas Schmid as to technical design, see my comments of 2018-10-24 and 2019-06-21. I am interpreting the requirement more broadly than the original statement: a user or script should be able to trigger a Pipeline job with a build parameter that includes the contents of a (possibly large, possibly binary, but not secret) file, using any common mechanism (GUI, HTTP POST, build-job CLI command, build step), in such a way that some convenient (Scripted and/or Declarative) syntax may be used to retrieve that file in a designated workspace location at a designated time, including in future rebuilds.

Add Comment