[JIRA] [workflow-plugin] (JENKINS-34564) Give the ability to choose how the multibranch subprojects will be named.

[grandemange.kevin@gmail.com (JIRA)]

Grandemange Kevin created an issue

Jenkins / JENKINS-34564 Give the ability to choose how the multibranch subprojects will be named. Issue Type: Improvement Assignee: Jesse Glick Components: workflow-plugin Created: 2016/May/03 12:12 PM Priority: Blocker Reporter: Grandemange Kevin The %2F encoding of / breaks many software (msbuild for example) and can't be changed. Being able to provide a rule to build the subproject names from the branch name instead of just encoding things would permit to give the user the power to workaround their problems with this kind of software. This would an advanced option in which one can write something like a sed regex. Add Comment

This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)

[grandemange.kevin@gmail.com (JIRA)]

Grandemange Kevin updated an issue

Jenkins / JENKINS-34564 Give the ability to choose how the multibranch subprojects will be named.

Change By: Grandemange Kevin

The %2F encoding of / breaks many software (msbuild for example) and can't be changed.

Being able to provide a rule to build the subproject names from the branch name instead of just encoding things would permit to give the user the power to  workaround  work around  their problems with this kind of software. This would  be  an advanced option in which one can write something like a sed regex.

Add Comment

[mneale@cloudbees.com (JIRA)]

Michael Neale commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named. This also breaks tools like "npm run" as well (even on filesystems where it is ok). "%" is not recommended as a path name on any platform, even if it is tolerated in some cases.

Add Comment

[mneale@cloudbees.com (JIRA)]

Michael Neale edited a comment on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

This also breaks tools like "npm run" as well (even on filesystems where it is ok).  "%" is not recommended as a path name on any platform, even if it is tolerated in some cases.

I think the %2F causes more problems that people realise, so picking something else would make more sense vs making it an "advanced" option.

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

Not easily changed. There is a bidirectional mapping between branch name and job name which must be recoverable. Also existing branch projects must be preserved, and these are already using URL encoding. So I do not plan to change the job names. The workspace name is another matter. Possibly Jenkins core could just use a different algorithm for Jenkins/Slave.getWorkspaceFor which uses blander characters.

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick updated an issue

Jenkins / JENKINS-34564

Give the ability to choose how the multibranch subprojects will be named.

Change By: Jesse Glick Labels: multibranch

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick updated an issue

Jenkins / JENKINS-34564 Give the ability to choose how the multibranch subprojects will be named.

Not a blocker since you can trivially customize the workspace you use from within Jenkinsfile itself, using the ws step. Change By: Jesse Glick Priority: Blocker Major

Add Comment

[mneale@cloudbees.com (JIRA)]

Michael Neale commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named. yes I think the only real problem is % in the path. This is the best I can find (it isn't much) for what to not use in a path: https://en.wikipedia.org/wiki/Filename#Reserved_characters_and_words I would class this as an "ongoing irritation" to keep an eye on.

Add Comment

[thxmasj@gmail.com (JIRA)]

Thomas Johansen commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

This issue hits me when when I run the test phase, as I have plenty of code using getClass().getResource("someResourceName").getFile(), which results in wrong file names. I am aware of the badness of such code, but I want to avoid changes in the code base at the time. Jesse Glick: How do I customize the workspace in the Jenkinsfile with a ws step? Can you provide an example, or even point me to documentation?

Add Comment

[mneale@cloudbees.com (JIRA)]

Michael Neale commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

Thomas Johansen ws('custom path goes here') { // some block } Is that what you mean? The "snippet builder" has some docs for it (depending on what version of Jenkins you are running depends how obvious the location of that snippet thing is).

Add Comment

[thxmasj@gmail.com (JIRA)]

Thomas Johansen commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

Michael Neale: Yes. Would there be any reason to have the workspace name unique for the branch, or maybe derived from the branch name (let's say branchName.replaceAll("/", "_")). Is that possible?

Add Comment

[mneale@cloudbees.com (JIRA)]

Michael Neale commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

Thomas Johansen It can be an absolute path - and will be locked. If it isn't unique it appends @N to the end. So there will be no branch names in the workspace (as far as I understand it) only what you specify, so you can be creative. From the snippet docs: "You can instead specify a path here and that workspace will be locked instead. (The path may be relative to the slave root, or absolute.) If concurrent builds ask for the same workspace, a directory with a suffix such as @2 may be locked instead. Currently there is no option to wait to lock the exact directory requested; if you need to enforce that behavior, you can either fail (error) when pwd indicates that you got a different directory, or you may enforce serial execution of this part of the build by some other means such as stage name: '…', concurrency: 1."

Add Comment

[mneale@cloudbees.com (JIRA)]

Michael Neale edited a comment on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

[~thxmasj]  It can be an absolute path - and will be locked. If it  isn't unique  already locked when  it  is needed it  appends @N to the end. So there will be no branch names in the workspace (as far as I understand it) only what you specify, so you can be creative.

From the snippet docs:  "You can instead specify a path here and that workspace will be locked instead. (The path may be relative to the slave root, or absolute.) If concurrent builds ask for the same workspace, a directory with a suffix such as @2 may be locked instead. Currently there is no option to wait to lock the exact directory requested; if you need to enforce that behavior, you can either fail (error) when pwd indicates that you got a different directory, or you may enforce serial execution of this part of the build by some other means such as stage name: '…', concurrency: 1."

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

My current proposal is to fix this in the core code for default workspace selection, since I have heard no reports of the project name per se being a problem—only buggy external tools which cannot cope with % in workspace pathnames.

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick updated an issue

Jenkins / JENKINS-34564

Give the ability to choose how the multibranch subprojects will be named.

Change By: Jesse Glick Component/s: core Component/s: workflow-plugin Labels: multibranch  workflow

Add Comment

[mneale@cloudbees.com (JIRA)]

Michael Neale commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named. Jesse Glick yes I have not heard any complaints about the naming, only when things break on the path.

Add Comment

[emanuelez@java.net (JIRA)]

emanuelez commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

One notable failure comes from ld: /workspace/realm/realm-java/ez%2Fstandard-jenkinsfile/realm/realm-jni/build/standalone-toolchains/arm/bin/../lib/gcc/arm-linux-androideabi/4.9/../../../../arm-linux-androideabi/bin/ld: error: /workspace/realm/realm-java/ez: could not load plugin library: /workspace/realm/realm-java/ez: cannot open shared object file: No such file or directory

Add Comment

This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)

[emanuelez@java.net (JIRA)]

emanuelez edited a comment on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

One notable failure comes from ld in a github org folder where ws{} doesn't really work :

/workspace/realm/realm-java/ez%2Fstandard-jenkinsfile/realm/realm-jni/build/standalone-toolchains/arm/bin/../lib/gcc/arm-linux-androideabi/4.9/../../../../arm-linux-androideabi/bin/ld: error: /workspace/realm/realm-java/ez: could not load plugin library: /workspace/realm/realm-java/ez: cannot open shared object file: No such file or directory

Add Comment

[emanuelez@java.net (JIRA)]

emanuelez edited a comment on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

One notable failure comes from ld in a github org folder where ws{} doesn't really work: /workspace/realm/realm-java/ez%2Fstandard-jenkinsfile/realm/realm-jni/build/standalone-toolchains/arm/bin/../lib/gcc/arm-linux-androideabi/4.9/../../../../arm-linux-androideabi/bin/ld: error: /workspace/realm/realm-java/ez: could not load plugin library: /workspace/realm/realm-java/ez: cannot open shared object file: No such file or directory

So from this point of view it actually is a blocking issue.

Add Comment

[alastair@d-silva.org (JIRA)]

Alastair D'Silva commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

the % encoding also breaks some makefiles, where the path ends up in the (generated) makefile - '%' is a special character and is misinterpreted.

Add Comment

[emanuelez@java.net (JIRA)]

emanuelez commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

As a workaround I dockerized the build, which men allows to add a custom mount point in the container using the `ws` step

Add Comment

[alastair@d-silva.org (JIRA)]

Alastair D'Silva commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

Proposed fix here: https://github.com/jenkinsci/branch-api-plugin/pull/46 It adds a couple of different user-selectable encoding options in addition to the the default one.

Add Comment

[alastair@d-silva.org (JIRA)]

Alastair D'Silva commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

Note that the Multibranch project plugin is also affected by this (and I can't see any way to customise the workspace path). I do think that Jesse's idea of fixing it in the core is worthwhile though, as there are other avenues that troublesome characters can get into the paths (eg. Matrix axes).

Add Comment

[alastair@d-silva.org (JIRA)]

Alastair D'Silva edited a comment on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

Note that the [Multibranch project plugin|https://wiki.jenkins-ci.org/display/JENKINS/Multi-Branch+Project+Plugin] is also affected by this (and I can't see any way to customise the workspace path). I do think that Jesse's idea of fixing it in the core is worthwhile though the best way forward , as there are other avenues that troublesome characters can get into the paths (eg. Matrix axes).

Add Comment

[jdumay@cloudbees.com (JIRA)]

James Dumay commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

Jesse Glick I don't think making this configurable is going to help the situation - users will run into it then have to find this config option to make it behave well. The best option is no configuration: lets just substitute / with something path safe.

Add Comment

[alastair@d-silva.org (JIRA)]

Alastair D'Silva commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

To be clear, It's not just /, but many special characters that may cause problems, eg. for makefiles, % and $ will cause problems if the paths make it into the makefile (eg. through generation). I expect there will be many others that need to be handled for a variety of languages. James Dumay, I'm not suggesting that this is a final solution, but it's better than what is currently available, at least until someone is willing to put in the time to provide a migration path for the dependent plugins. Matt points out on Github that this is non-trivial.

Add Comment

[alastair@d-silva.org (JIRA)]

Alastair D'Silva edited a comment on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

To be clear, It's not just /, but many special characters that may cause problems, eg. for makefiles, % and $ will cause problems if the paths make it into the makefile (eg. through generation). I expect there will be many others that need to be handled for a variety of languages.

[~jamesdumay], I'm not suggesting that this is a final solution, but it's better than what is currently available, at least until someone is willing to put in the time to provide a migration path for the dependent plugins. Matt points out on Github that this is non-trivial. (later) Those complexities may not be relevant in an upgrade situation, where no jobs can be running.

Add Comment

[janarend@gmail.com (JIRA)]

Jan Arend Jansen commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

Why not use a SHA hash on the name?

Add Comment

[alastair@d-silva.org (JIRA)]

Alastair D'Silva commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

Jan Arend Jansen SHA1s are not ideal as the obfuscate the paths in artifact names, which makes is a lot harder to figure out what is going on when copying artifacts from other jobs. One could argue, it's worse than the Base64 encoding suggested by Matthew DeTullio as it obfuscates, and introduces a (low) probability of a collision. The best I could come up with that avoids collisions & still maintains some semblance of readability is the 'Strip' method implemented in the PR above.

Add Comment

[alastair@d-silva.org (JIRA)]

Alastair D'Silva edited a comment on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

[~jajansen] SHA1s are not ideal as the they obfuscate the paths in artifact names, which makes is a lot harder to figure out what is going on when copying artifacts from other jobs. One could argue, it's worse than the Base64 encoding suggested by [~mjdetullio] as it obfuscates, and introduces a (low) probability of a collision.

The best I could come up with that avoids collisions & still maintains some semblance of readability is the 'Strip' method implemented in the PR above.

Add Comment

[mneale@cloudbees.com (JIRA)]

Michael Neale commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

I am not sure if "$" could reasonably be discounted as a path name. That is quite an annoying bug if tools don't cope with that as many other tools to use it in path names. There is no real canonical list of things to not use as path names, but https://en.wikipedia.org/wiki/Filename#Reserved_characters_and_words lists a few things to avoid. Unfortunately URL encoding stomps all over this and isn't an ideal choice. Once this is solved, it will have to involve migration IMO, as the number of people being bit by this is growing with the usage of multibranch and git flow.

Add Comment

[jdumay@cloudbees.com (JIRA)]

James Dumay commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

I think we should separate the artifact passing between jobs as a separate concern - there may be a nicer way we can provide that to you that doesn't rule out using a automatically generated directory name for these jobs (for example, Pipeline already has `stash` and `unstash` within pipelines).

Add Comment

[jdumay@cloudbees.com (JIRA)]

James Dumay edited a comment on JENKINS-34564

[jdumay@cloudbees.com (JIRA)]

James Dumay edited a comment on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

I think we should separate the artifact passing between jobs as a separate concern - there may be a nicer way we can provide that to you that doesn't rule out using a automatically generated directory name for these jobs (for . For example, Pipeline already has `stash` and `unstash` within pipelines ) - theres no reason why we couldn't extend this to `unstash $jobsName/$artifactName` . Referring to known artifacts on disk can be a bit horrifying mess in practice. What if your job gets moved so it runs on a different machine? Is it guaranteed to be there? Unlikely. Are you introducing races into your build process? You probably want the one produced by the immediate upstream job, not some random binary on disk that could have been produced by a failing run.

Add Comment

[jdumay@cloudbees.com (JIRA)]

James Dumay edited a comment on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

I think we should separate the artifact passing between jobs as a separate concern - there may be a nicer way we can provide that to you that doesn't rule out using a automatically generated directory name for these jobs. For example, Pipeline already has `stash` and `unstash` within pipelines - theres no reason why we couldn't extend this to `unstash $jobsName/$artifactName`. That way you could always rely on the artifact being created by a job that is upstream of the current job or fall back to the most recently successful run of the named job.

Referring to known artifacts on disk can be a bit horrifying mess in practice. What if your job gets moved so it runs on a different machine? Is it guaranteed to be there? Unlikely. Are you introducing races into your build process? You probably want the one produced by the immediate upstream job, not some random binary on disk that could have been produced by a failing run.

Add Comment

[alastair@d-silva.org (JIRA)]

Alastair D'Silva commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

I think you've misunderstood our use case. We are using Multibranch Project (as we do not have control over a number of the projects that we build, so we can't drop in Jenkinsfiles, which rules out pipelines), and rely on the "Copy artifacts from another project" build step, eg. Project name: gcc_binutils_toolchain/$ {GCC_BRANCH} /HOST_OS=$ {HOST_OS} ,HOST_PLATFORM=$ {HOST_PLATFORM},TARGET_PLATFORM=${HOST_PLATFORM} (Where GCC_BRANCH is an axis on the current project). As you can see, the branch name is (and should be, for readability) part of the path by which the artifacts are referenced.

Add Comment

[alastair@d-silva.org (JIRA)]

Alastair D'Silva edited a comment on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

I think you've misunderstood our use case. We are using Multibranch Project (as we do not have control over a number of the projects that we build, so we can't drop in Jenkinsfiles, which rules out pipelines), and rely on the "Copy artifacts from another project" build step, eg.

Project name: {noformat} gcc_binutils_toolchain/${GCC_BRANCH}/HOST_OS=${HOST_OS},HOST_PLATFORM=${HOST_PLATFORM},TARGET_PLATFORM=${HOST_PLATFORM} {noformat}

(Where GCC_BRANCH is an axis on the current project). As you can see, the branch name is (and should be, for readability) part of the path by which the artifacts are referenced.

Add Comment

[mneale@cloudbees.com (JIRA)]

Michael Neale commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

Alastair D'Silva oh I think that the name of the encoding on disk (or in URI) for that matter should not impact how you interact with a named job ideally anyway (that is its own serious concern).

Add Comment

[alastair@d-silva.org (JIRA)]

Alastair D'Silva commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

Michael Neale As currently implemented, the same encoded name is used by the Multibranch Project for all of the directories (top level job, slave workspaces) and the branch name (eg. GCC_BRANCH above).

Add Comment

[alkhimey@yahoo.com (JIRA)]

alkhimey alkhimey commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

Replacing "/" with "" instead of "%2" is a good and quick solution but it might be still problematic for some users using "" and "/" in their branching schemes. An alternative way might be to replace "/" with hierarchy. For example if the branch name is "feature/xyz", the workspace will be located in "C:\Jenkins\workspace\feature\xyz". I also agree with @Michael that %2 behavior should not be the default as it causes errors that are hard to figure out for a user that is unaware that % in the path might be problematic for its tools.

Add Comment

[mneale@cloudbees.com (JIRA)]

Michael Neale commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

I think there are 2 unsolved problems still: 1) what is the exact scheme to use that works best encodes most of the branch name in a non opaque way doesn't use path name elements which are known problematic 2) How to migrate existing jobs once #1 is solved. I think it may be reasonable if #1 can be solved for new jobs, but it is not clear to me how automatic to make migration. This is paths on disk, so migration means moving things around. To me it would be acceptable if both old and new worked but new branches defaulted to the "corrected" encoding, but I imagine this would be messy and also problematic.

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

To Michael Neale:

encodes most of the branch name in a non opaque way

The choice of encoding is a bit subtle, since you want to ensure that if two jobs have distinct fullName, they must also use distinct workspaces on a given node. This rules out simply replacing / with _, for example; you would have to more cautiously replace _ with __, etc. There have also been reports of Windows users running up against the infamous 260-character path limit, and even on Linux users of the sshagent step in conjunction with docker.Image.inside will hit a 108-character limit (UNIX_PATH_MAX), so it may be necessary to use a hybrid scheme: an SHA-256 hash of the Item.fullName in Base64, plus up to a couple dozen informational scrubbed characters from the tail of the fullName to make it easier for humans to recognize the directory. For example, the following algorithm produces unique, safe, pretty recognizable workspace names no longer than 80 characters: static String ws(String name) { return name.replaceAll("(%[0-9A-F]{2}|[^a-zA-Z0-9-_.])+", "_").replaceFirst(".*?(.{0,36}$)", "$1") + "-" + Base64.encode(Hashing.sha256().hashString(name).asBytes()).replace('/', '_').replace('+', '.').replaceFirst("=+$", ""); }

it is not clear to me how automatic to make migration

Migration is a nonissue since this is only about workspace names, and workspaces are dispensable. (Yes WorkspaceCleanupThread will lose track of the old directory, but this is already true for all sorts of more common cases such as job deletion so there is little sense worrying about it here; as a last resort we would need to just blow away the whole workspace root directory once a year or so.) To James Dumay:

I don't think making this configurable is going to help the situation

I never suggested making it configurable, except perhaps via system property as an escape hatch. Perhaps the original reporter of this issue assumed that would be the solution. To myself: you can trivially customize the workspace you use from within Jenkinsfile itself, using the ws step Some people have since pointed out that there are instances of this problem affecting the initial checkout, due to buggy SCMs (or SCM plugins). Implementations of JENKINS-33273 would help in that regard, by avoiding the need for the @script workspace to begin with, but in the meantime these use cases are truly blocked. My current proposal is to fix this in the core code for default workspace selection It might be better to just implement a WorkspaceLocator in branch-api active on branch projects, as this could work even on current cores, and leave matrix-project to its own devices. For a Slave, it could refer to getWorkspaceRoot(). For Jenkins, I would not bother supporting the archaic customizable Jenkins.workspaceDir on master (there is no equivalent for agents and anyway you should be doing builds on agents); should suffice to hardcode $JENKINS_HOME/workspace. (See JENKINS-21942 for the sordid story of why this is not necessarily even the default location.)

Add Comment

[mneale@cloudbees.com (JIRA)]

Michael Neale commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

Thanks Jesse Glick, good point in migration. I like your suggestion for a hybrid approach. I would have thought the first part of the name but you are right, the last part is more likely to be human recognisable/identifiable.

Add Comment

[jdumay@cloudbees.com (JIRA)]

James Dumay commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

Jesse Glick ahh sorry for misattributing that to you!

Add Comment

[grandemange.kevin@gmail.com (JIRA)]

Grandemange Kevin commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

??To James Dumay:

I don't think making this configurable is going to help the situation

I never suggested making it configurable, except perhaps via system property as an escape hatch. Perhaps the original reporter of this issue assumed that would be the solution.?? Yes, I was the one that gave this possible workaround. This didn't solve the real underlying problem, but as a user my problem was that the path was not good for some uses I thought that if there were others with the same problem, the problem would be treated in depth. And it happened very quickly ! Thanks a lot !

Add Comment

[grandemange.kevin@gmail.com (JIRA)]

Grandemange Kevin edited a comment on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

?? To James Dumay:

I don't think making this configurable is going to help the situation I never suggested making it configurable, except perhaps via system property as an escape hatch. Perhaps the original reporter of this issue assumed that would be the solution.

??

Yes, I was the one that gave this possible workaround. This didn't solve the real underlying problem, but as a user my problem was that the path was not good for some uses I thought that if there were others with the same problem, the problem would be treated in depth. And it happened very quickly ! Thanks a lot !

Add Comment

[grandemange.kevin@gmail.com (JIRA)]

Grandemange Kevin edited a comment on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

{quote}

To James Dumay:         I don't think making this configurable is going to help the situation         I never suggested making it configurable, except perhaps via system property  as an escape hatch. Perhaps the original reporter of this issue assumed that would be the solution.

{quote}

Yes, I was the one that gave this possible workaround. This didn't solve the real underlying problem, but as a user my problem was that the path was not good for some uses I thought that if there were others with the same problem, the problem would be treated in depth. And it happened very quickly ! Thanks a lot !

Add Comment

[yury.zaytsev@traveltainment.de (JIRA)]

Yury Zaytsev commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

Well, I appreciate the desire to treat the problem in depth, but given the nature of the issue, I'd think it warrants a quick workaround first, such that a better solution can be developed without so much pressure. I think the obvious workaround is to use "" instead of "%" in escapes and screen "" with another "_"; this is as close to the current scheme as possible and will keep the bi-directionality of the mapping. (Actually, I personally would have used a modified PunyCode scheme in the first place, but ...). Most build systems don't have problems with underscores, the changes required are minimal and one can then take time to think of and properly implement a better scheme. Currently, multibranch subprojects do not work with the following build systems: Make (it interprets % in rule targets as a wildcard, and no escaping that I've tried seems to work) Boost Jam (fails with obscure error messages, have not identified the underlying problem yet) CMake (as reported in related bugs) MSBuild (as reported in related bugs) Apparently, a number of other build systems / application containers, etc. Large number of plugins (cppcheck, cifs, ...) Of course, one could rightfully argue that all this software is broken, but I hope that some practical solution can be found quickly. This issue is absolutely critical to the usability of the multibranch stuff.

Add Comment

[yury.zaytsev@traveltainment.de (JIRA)]

Yury Zaytsev commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

For the benefit of other affected users, I'm posting a workaround suggested by @rrodriguez__ on #jenkins IRC based on the comment in a linked ticket ( https://issues.jenkins-ci.org/browse/JENKINS-30744?focusedCommentId=247893#comment-247893 ): node(agent) { def workspace_orig = pwd() def workspace_sane = workspace_orig.replaceAll("%", "_") ws(workspace_sane) { // ... } } I've also filed a bug report against Boost: https://svn.boost.org/trac/boost/ticket/12448 , but I'm not sure if one should really expect them to fix this anytime soon.

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick updated an issue

Jenkins / JENKINS-34564

Give the ability to choose how the multibranch subprojects will be named.

Change By: Jesse Glick Component/s: branch-api-plugin Component/s: core

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

I'd think it warrants a quick workaround first, such that a better solution can be developed without so much pressure.

Well I suspect the solution I proposed would be an hour’s work for me to write, I just have not prioritized this yet, but it seems it has a lot of votes.

Add Comment

[mneale@cloudbees.com (JIRA)]

Michael Neale commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

Jesse Glick yes it seems a lot of tools don't cope with funky names, so yeah, lots of votes - cest la vie.

Add Comment

[cobexer@gmail.com (JIRA)]

Christoph Obexer commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

Matrix projects have such an option called "Use custom child workspace". If multibranch projects had such an option it could be used to solve this issue and the feature branch disk space problem at the same time. I hope "My current proposal is to fix this in the core code for default workspace selection, since I have heard no reports of the project name per se being a problem" means that this will then simply be the default? aka all branches building in the same folder called after the project?

Add Comment

[hessm@us.ibm.com (JIRA)]

Matt Hess commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

Please fix.. this defect renders Jenkins at my current location as "not the way forward".

Add Comment

[hessm@us.ibm.com (JIRA)]

Matt Hess commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

I just tried the work around above by Yury Zaytsev ( explained more in JENKINS-30744)... this works... Thank you!!! However, it would still be nice to have an actual fix and get this out of the jenkinFile.

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick updated JENKINS-34564

Jenkins / JENKINS-34564

Give the ability to choose how the multibranch subprojects will be named.

Change By: Jesse Glick Status: In Progress Review

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick started work on JENKINS-34564

Change By: Jesse Glick Status: Open In Progress

Add Comment

[scm_issue_link@java.net (JIRA)]

SCM/JIRA link daemon commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named. Code changed in jenkins User: Jesse Glick Path: src/main/java/jenkins/branch/WorkspaceLocatorImpl.java src/test/java/jenkins/branch/WorkspaceLocatorImplTest.java src/test/java/jenkins/branch/harness/BranchProjectFactoryImpl.java http://jenkins-ci.org/commit/branch-api-plugin/040578616a1a7f66605203ee15ed2e9445f171ba Log: [FIXED JENKINS-34564] For branch projects, pick workspace names which are bounded in length and avoid unusual characters.

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick updated JENKINS-34564

Jenkins / JENKINS-34564

Give the ability to choose how the multibranch subprojects will be named.

Change By: Jesse Glick Status: In Review Resolved Resolution: Fixed

Add Comment

[yury.zaytsev@traveltainment.de (JIRA)]

Yury Zaytsev commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named. Awesome, thank you very much! Could you please tell in which plugin / release it will become available? Many thanks!

Add Comment

[scm_issue_link@java.net (JIRA)]

SCM/JIRA link daemon commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

Code changed in jenkins User: Jesse Glick Path: pom.xml src/test/java/org/jenkinsci/plugins/workflow/multibranch/WorkflowBranchProjectFactoryTest.java http://jenkins-ci.org/commit/workflow-multibranch-plugin/7c908dc9409eda22ead2885af3616f435b13ec27 Log: JENKINS-34564 Adjusted integration test to match modified behavior.

Add Comment

[scm_issue_link@java.net (JIRA)]

SCM/JIRA link daemon commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

Code changed in jenkins User: Jesse Glick Path: pom.xml src/test/java/org/jenkinsci/plugins/workflow/multibranch/WorkflowBranchProjectFactoryTest.java

http://jenkins-ci.org/commit/workflow-multibranch-plugin/dba7f3b9732ecf01b8a6e99f6ac10826394edd55 Log: Merge pull request #33 from jglick/encoded-workspace-names-JENKINS-34564

JENKINS-34564 Adjusted integration test to match modified behavior

Compare: https://github.com/jenkinsci/workflow-multibranch-plugin/compare/34c7574b0cdc...dba7f3b9732e

Add Comment

[espen@albrektsen.net (JIRA)]

espenalb commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

The current fix (Just updated to Pipeline+Multibranch+Plugin 2.9) is still causing problems on Windows. It is nice that the %2F is gone - but there is now another problem - i am hitting the 260-character path length limit. I have a fairly typical .Net application - on my slave the workspace becomes {:\Jenkins\workspace\AS_hotfix_unit-test-failure-DZA4W4JGIWQ64M7NMWU3XPVUCYQQSZIABUVZWCGG373RLGNCFODQ } which is ok - but very long. But when running tests, I hit the error: c:\Jenkins\workspace\AS_hotfix_unit-test-failure-DZA4W4JGIWQ64M7NMWU3XPVUCYQQSZIABUVZWCGG373RLGNCFODQ\MYPROJ.Net.Tests\\bin\Debug\\MYPROJ.Net.Tests.dll System.IO.FileLoadException: The path is too long after being fully qualified. Make sure the full path is less than 260 characters and the directory name is less than 248 characters. File name: 'nunit.framework, Version=3.2.1.0, Culture=neutral, PublicKeyToken=2638cd05610744eb' ---> System.IO.FileLoadException: The path is too long after being fully qualified. Make sure the full path is less than 260 characters and the directory name is less than 248 characters. File name: 'nunit.framework' A little strange - the path is well within the 248char limit, but my build fails anyway. I solve it with this workaround: def ensureSafeWorkspace(Closure block) { def maxWorkspacePathLen = 60; def isUnsafe = pwd().contains("%") || (pwd().length()>maxWorkspacePathLen); if (isUnsafe) { // Then we will request a new workspace... ws(safePath(env.JOB_NAME).take(maxWorkspacePathLen)) { block(); } } else { // Just call the closure in the current workspace (Avoid unnecessary master@2 workspace for example) block(); } } Could we not just get rid of the excessively long string? Jenkins already seems to have features for preventing workspace sharing...

Add Comment

[espen@albrektsen.net (JIRA)]

espenalb edited a comment on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

The current fix (Just updated to Pipeline+Multibranch+Plugin 2.9) is still causing problems on Windows. It is nice that the {{%2F}} is gone - but there is now another problem - i am hitting the 260-character path length limit.

I have  a fairly typical .Net application - on my slave the workspace becomes { {c :\Jenkins\workspace\AS_hotfix_unit-test-failure-DZA4W4JGIWQ64M7NMWU3XPVUCYQQSZIABUVZWCGG373RLGNCFODQ } } which is ok - but very long.

But when running tests, I hit the error:

{code}

c:\Jenkins\workspace\AS_hotfix_unit-test-failure-DZA4W4JGIWQ64M7NMWU3XPVUCYQQSZIABUVZWCGG373RLGNCFODQ\MYPROJ.Net.Tests\\bin\Debug\\MYPROJ.Net.Tests.dll System.IO.FileLoadException: The path is too long after being fully qualified.  Make sure the full path is less than 260 characters and the directory name is less than 248 characters. File name: 'nunit.framework, Version=3.2.1.0, Culture=neutral, PublicKeyToken=2638cd05610744eb' ---> System.IO.FileLoadException: The path is too long after being fully qualified.  Make sure the full path is less than 260 characters and the directory name is less than 248 characters. File name: 'nunit.framework'

{code}

A little strange - the path is well within the 248char limit, but my build fails anyway. I solve it with this workaround:

{code}

def ensureSafeWorkspace(Closure block) { def maxWorkspacePathLen = 60; def isUnsafe = pwd().contains("%") || (pwd().length()>maxWorkspacePathLen); if (isUnsafe) { // Then we will request a new workspace...   ws(safePath(env.JOB_NAME).take(maxWorkspacePathLen)) {    block();   }   } else { // Just call the closure in the current workspace (Avoid unnecessary master@2 workspace for example)   block(); } }

{code}

Could we not just get rid of the excessively long string? Jenkins already seems to have features for preventing workspace sharing...

Add Comment

[ben.herfurth@develop-group.de (JIRA)]

Ben Herfurth commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

As espenalb has already commented: on Windows, my build breaks, because the path is too long, because the workspace now contains this ridiculously long strings. Can we somehow get rid of them?

Add Comment

[yury.zaytsev@traveltainment.de (JIRA)]

Yury Zaytsev commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

It seems that fixing this issue can't be too easy apparently the default 80-char limit was way too generous for Windows slaves. But does it really warrant decreasing it? Or rather advice users to not to bury workspace too deep in the hierarchy? Ben Herfurth, in light of all problems with Windows path length limitations, why wouldn't you just give Jenkins agent a reasonable workspace path, e.g. C:\Jenkins\workspace <--- this should work.

Add Comment

[ben.herfurth@develop-group.de (JIRA)]

Ben Herfurth commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

my jenkins workspace is located under: d:\install\jenkins\workspace does not work ... since the directory structure is quite deep. (java, javascript, npm ... etc)

Add Comment

[bradley.cooley@viasat.com (JIRA)]

Bradley Cooley commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

Maybe we could make this a configurable option?

Add Comment

[ilya.lukyanov@gmail.com (JIRA)]

Ilya Lukyanov commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

I sincerely respect the work done, but I think it can't be left in the current state. Before the change we were able to give projects and branches meaningful names. Only things to avoid were non-alphanumeric characters. It was an annoying issue that we couldn't use some special characters, I agree. However, in my opinion after the change we have a bigger one - in order to fit into the path length limit the only things we can do are: Place Jenkins home dir into C:\J or something like that. Weird, but not a big deal. I would also place workspace directory into one with a short name, BUT because of [this line](https://github.com/jenkinsci/branch-api-plugin/pull/50/files#diff-dc7cf3673f26f05af47a328e34d6df20R72) the setting is ignored so it's not possible. It's a separate issues, but should be fixed I believe. Give projects short/one-letter names. Give branches short/one-letter names. Obviously, the last two are much worse limitations than just avoiding special chars. Of course, we could have some workarounds inside Jenkins file, but I think it's ideologically wrong to have workarounds for CI server or its plugins in source code or pipeline definition. I'm not a java developer, but am happy to contribute once we all agree on the best approach. I'll have a play with the code and will submit a PR if I get any ideas.

Add Comment

[ilya.lukyanov@gmail.com (JIRA)]

Ilya Lukyanov edited a comment on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

I sincerely respect the work done, but I think it can't be left in the current state. Before the change we were able to give projects and branches meaningful names. Only things to avoid were non-alphanumeric characters. It was an annoying issue that we couldn't use some special characters, I agree. However, in my opinion after the change we have a bigger one - in order to fit into the path length limit the only things we can do are:

* Place Jenkins home dir into *_C:\J_* or something like that. Weird, but not a big deal. * I would also place workspace directory into one with a short name, *BUT* because of [this line ]( | https://github.com/jenkinsci/branch-api-plugin/pull/50/files#diff-dc7cf3673f26f05af47a328e34d6df20R72 ) ] the setting is ignored so it's not possible. It's a separate issues, but should be fixed I believe. * Give projects short/one-letter names. * Give branches short/one-letter names.

Obviously, the last two are much worse limitations than just avoiding special chars. Of course, we could have some workarounds inside Jenkins file, but I think it's ideologically wrong to have workarounds for CI server or its plugins in source code or pipeline definition. I'm not a java developer, but am happy to contribute once we all agree on the best approach. I'll have a play with the code and will submit a PR if I get any ideas.

Add Comment

[mneale@cloudbees.com (JIRA)]

Michael Neale commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

Does it matter what the name is on disk? Should the JENKINS_HOME be as transparent as this? I am curious as to what people are doing with it (this change only impacts how it appears on disk if I understand it correctly)

Add Comment

[mneale@cloudbees.com (JIRA)]

Michael Neale commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

espenalb Ben Herfurth perhaps open a new ticket for the path limit on windows? as it may be able to be reduced. I note that windows 10 claims to have removed that limit: https://mspoweruser.com/ntfs-260-character-windows-10/

Add Comment

[hans.schulz@sap.com (JIRA)]

Hans Schulz commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

The new solution has multiple issues which makes it unusable for us: You have no simple way to get workspaces from org/project/branch/build which you can use in e.g. scripts. The workspaces are neither reused nor cleaned up, causing a "filesystem leak". No simple way to write a cleanup script because of #1 You cannot use them on the console, not even with shell expansion, due to some starting with a "-" (ex. rm -r * in JENKINS_HOME/workspace fails) They do not reside in the job's directory anymore, so deleting a job does not delete the workspaces (and again, no simple mapping possible) The filename is verry long. While not being an issue for us, many users have reported hitting MAXPATH with their projects on Windows. Why was just changing the escaping strategy to more sane characters not an option?

Add Comment

[ict@iwolszewski.pl (JIRA)]

Iwo Olszewski commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

The new workspace naming schema can be disabled by setting the system property jenkins.branch.WorkspaceLocatorImpl.PATH_MAX to 0. Other values should control max length of generated directory name in new schema (but I didn't verified it). To be effective, this system property need to be set during Jenkins startup (e.g. by adding -Djenkins.branch.WorkspaceLocatorImpl.PATH_MAX=0 as a param).

Add Comment

[hans.schulz@sap.com (JIRA)]

Hans Schulz edited a comment on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

The new solution has multiple issues which makes it unusable for us:

# You have no simple way to get workspaces from org/project/branch/build which you can use in e.g. scripts. # The workspaces are neither reused nor cleaned up, causing a "filesystem leak". No simple way to write a cleanup script because of #1 # You cannot use them on the console, not even with shell expansion, due to some starting with a "-" (ex. rm -r * in JENKINS_HOME/workspace fails) # They do not reside in the job's directory anymore, so deleting a job does not delete the workspaces (and again, no simple mapping possible) # The filename is verry very long. While not being an issue for us, many users have reported hitting MAXPATH with their projects on Windows.

Why was just changing the escaping strategy to more sane characters not an option?

Add Comment

[hans.schulz@sap.com (JIRA)]

Hans Schulz edited a comment on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

# The new solution has multiple issues which makes it unusable for us:

# You have no simple way to get workspaces from org/project/branch/build which you can use in e.g. scripts. # The workspaces are neither reused nor cleaned up, causing a "filesystem leak". No simple way to write a cleanup script because of #1 # You cannot use them on the console, not even with shell expansion, due to some starting with a "-" (ex. rm -r * in JENKINS_HOME/workspace fails) # They do not reside in the job's directory anymore, so deleting a job does not delete the workspaces (and again, no simple mapping possible)

# The filename is verry long. While not being an issue for us, many users have reported hitting MAXPATH with their projects on Windows.

Why was just changing the escaping strategy to more sane characters not an option?

Add Comment

[yury.zaytsev@traveltainment.de (JIRA)]

Yury Zaytsev reopened an issue

So, in addition to causing massive problems on Windows, this also causes trouble on Unix, if the workspace contains shell scripts which use binaries residing in the workspace as interpreters. That is, something like what follows: $ cat foo #!/home/jenkins/agent/workspace/xxx-O7T6IFSVMNBO5RYGRVNFUGEOPAZAIQ4GQRK47UUDIOZMEN3K2OAA/source/lib/python-san/install/bin/python-dbg One might think this ought to happen very rarely, but that's not the case! Basically, when you install any Python modules in a local prefix tree, it hardcodes the absolute path to the interpreter into the scripts and are you are screwed See here for the details on `#!` maximum length limits (127 bytes on Linux): http://www.in-ulm.de/~mascheck/various/shebang/

Jenkins / JENKINS-34564

Give the ability to choose how the multibranch subprojects will be named.

Change By: Yury Zaytsev Resolution: Fixed Status: Resolved Reopened

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named. (although in the case of Windows there are workarounds IIRC involving a magic path prefix that switches paths to UNC syntax internally)

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick resolved as Fixed

bq, You have no simple way to get workspaces from org/project/branch/build which you can use in e.g. scripts. You should not be doing so.

The workspaces are neither reused

Yes they are. nor cleaned up If the job is deleted while the agent is online, the workspace is removed. Jenkins has more general issues here.

You cannot use them on the console, not even with shell expansion, due to some starting with a "-" (ex. rm -r * in JENKINS_HOME/workspace fails)

In general you should use --. Patches to ensure that the trimmed name does not start with - will be entertained of course (follow-up issues). The filename is very long. While not being an issue for us, many users have reported hitting MAXPATH with their projects on Windows. Yes; in general there is no foolproof solution for operating systems which impose path length limitations.

Jenkins / JENKINS-34564

Give the ability to choose how the multibranch subprojects will be named.

Change By: Jesse Glick Status: Reopened Resolved Resolution: Fixed

Add Comment

[sorin.sbarnea@gmail.com (JIRA)]

Sorin Sbarnea reopened an issue

I am reopening this issue because clearly the new current behaviour is even worse than it was when the bug was opened in the first place. If we generate directory names that are ~80 characters long we can expect trouble. Not to mention that putting all of them in a single folder could generate some other scalability issues. Until we have a proper change of the default behaviour (what we really should have!) people encountering this can do add this to /etc/default/jenkins: -Djenkins.branch.WorkspaceLocatorImpl.PATH_MAX=40 – probably it would be safe to put even a shorter value, like 20. The default number seems to be ridiculous high, somewhere between 80-90. This feature did broke out Jenkins which was already tuned to workaround the shebang PATH limitations by moving the home directory of Jenkins user to /j folder, as opposed to /var/lib/jenkins. This was enough for us but now is not anymore.

Jenkins / JENKINS-34564 Give the ability to choose how the multibranch subprojects will be named.

Change By: Sorin Sbarnea

Resolution: Fixed Status: Resolved Reopened

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

If we generate directory names that are ~80 characters long we can expect trouble.

Depends entirely on what you are doing inside those directories. 80 characters is not really that long, and prior to the change directories prefixes could be much longer—unbounded, in fact (as noted in some complaints prior to the fix). (Note that you can customize the prefix length via system property.) PATH_MAX=40 No, anything less than 54 will not work at all, I just neglected to have that issue a warning.

already tuned to workaround the shebang PATH limitations

This really sounds like a separate issue, possibly in durable-task plugin, which should be filed separately with steps to reproduce from scratch—unless it is a limitation in the tools you are using: all I have seen so far is

when you install any Python modules in a local prefix tree, it hardcodes the absolute path to the interpreter into the scripts

which sounds like a bug in Python: it should use /usr/bin/env.

Add Comment

[yury.zaytsev@traveltainment.de (JIRA)]

Yury Zaytsev commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

> which sounds like a bug in Python: it should use /usr/bin/env. Just for the record: this isn't a bug by any means. They do it on purpose, because installed modules should be bound to a specific installing interpreter for many reasons. The question is, can they apply some hacks on their side to work around limited shebang path length issue or not, but it's a tangential one, and, in any case, the latest status was that if one could think of something, they'd welcome patches.

Add Comment

[lebreton.gabriel@gmail.com (JIRA)]

Gabriel Le Breton commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

I updated my plugins recently and my builds stopped working. I tried Jesse Glick's suggestion and it worked. Branches names are now stripped Instead. Before plugin update (was working): /var/lib/jenkins/workspace/project/project/branch_name@script After update (failing): /var/lib/jenkins/workspace/project_project_master-0GR3K62UHM3ITQD3DYGVNX7HHRXYWN3NP946WC282P28PH4B061E Adding the following line to /etc/default/jenkins: JAVA_ARGS="-Djenkins.branch.WorkspaceLocatorImpl.PATH_MAX=30" Results in this: r-MQDDZ79AS4LC4V4O49CTXGW52GDS8XDTMLMAIS05Q0ZHJ2NKK19N@script Only one letter is kept from my branch but I don't really care as long as it works. Thanks for this. At this point, I would only keep generated hash. In case it doesn't solve for other people, Using a shorter path for workspace sounds a good idea too:

/j folder, as opposed to /var/lib/jenkins.

But not quite sure what's the best way to achieve this. Related SO Questions: http://stackoverflow.com/questions/17310959/is-there-any-way-to-change-the-working-directory-of-a-jenkins-maven-build http://stackoverflow.com/questions/21839538/change-jenkins-home-on-red-hat-linux

Add Comment

[lebreton.gabriel@gmail.com (JIRA)]

Gabriel Le Breton edited a comment on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

I updated my plugins recently and my builds stopped working. I tried [Jesse Glick's suggestion|https://issues.jenkins-ci.org/browse/JENKINS-34564?focusedCommentId=272709&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-272709] and it worked. Branches names are now stripped

Instead. Before plugin update (was working):

{code} /var/lib/jenkins/workspace/project/project/branch_name@script {code} After update (failing): {code} /var/lib/jenkins/workspace/project_project_master-0GR3K62UHM3ITQD3DYGVNX7HHRXYWN3NP946WC282P28PH4B061E {code}

Adding the following line to {{/etc/default/jenkins}}:

{code} JAVA_ARGS="-Djenkins.branch.WorkspaceLocatorImpl.PATH_MAX=30" {code} Results in this: {code} /var/lib/jenkins/workspace/ r-MQDDZ79AS4LC4V4O49CTXGW52GDS8XDTMLMAIS05Q0ZHJ2NKK19N@script {code}

Only one letter is kept from my branch but I don't really care as long as it works. Thanks for this. At this point, I would only keep generated hash. In case it doesn't solve for other people, Using a shorter path for workspace sounds a good idea too:

{quote} {{/j}} folder, as opposed to {{/var/lib/jenkins}}. {quote}

But not quite sure what's the best way to achieve this. Related SO Questions: http://stackoverflow.com/questions/17310959/is-there-any-way-to-change-the-working-directory-of-a-jenkins-maven-build http://stackoverflow.com/questions/21839538/change-jenkins-home-on-red-hat-linux

Add Comment

[lebreton.gabriel@gmail.com (JIRA)]

Gabriel Le Breton edited a comment on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

I updated my plugins recently and my builds stopped working. I tried [Jesse Glick's suggestion|https://issues.jenkins-ci.org/browse/JENKINS-34564?focusedCommentId=272709&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-272709] and it worked. Branches names are now stripped Instead. Before plugin update (was working): {code} /var/lib/jenkins/workspace/project/project/branch_name@script {code} After update (failing): {code}

/var/lib/jenkins/workspace/project_project_master-0GR3K62UHM3ITQD3DYGVNX7HHRXYWN3NP946WC282P28PH4B061E @script

{code} Adding the following line to {{/etc/default/jenkins}}: {code} JAVA_ARGS="-Djenkins.branch.WorkspaceLocatorImpl.PATH_MAX=30" {code} Results in this: {code} /var/lib/jenkins/workspace/r-MQDDZ79AS4LC4V4O49CTXGW52GDS8XDTMLMAIS05Q0ZHJ2NKK19N@script {code} Only one letter is kept from my branch but I don't really care as long as it works. Thanks for this. At this point, I would only keep generated hash. In case it doesn't solve for other people, Using a shorter path for workspace sounds a good idea too: {quote} {{/j}} folder, as opposed to {{/var/lib/jenkins}}. {quote} But not quite sure what's the best way to achieve this. Related SO Questions: http://stackoverflow.com/questions/17310959/is-there-any-way-to-change-the-working-directory-of-a-jenkins-maven-build http://stackoverflow.com/questions/21839538/change-jenkins-home-on-red-hat-linux

Add Comment

[lebreton.gabriel@gmail.com (JIRA)]

Gabriel Le Breton edited a comment on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

I updated my plugins recently and my builds stopped working. I tried [Jesse Glick's suggestion|https://issues.jenkins-ci.org/browse/JENKINS-34564?focusedCommentId=272709&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-272709] and it worked. Branches names are now stripped Instead.

Before plugin update {color:green} (was working , 56 characters ) {color} : {code} /var/lib/jenkins/workspace/project/project/ branch_name master @script {code} After update {color:red} (failing , 109 characters ) {color} : {code} /var/lib/jenkins/workspace/project_project_master-0GR3K62UHM3ITQD3DYGVNX7HHRXYWN3NP946WC282P28PH4B061E@script

{code} Adding the following line to {{/etc/default/jenkins}}: {code} JAVA_ARGS="-Djenkins.branch.WorkspaceLocatorImpl.PATH_MAX=30" {code}

Results in this {color : green}(working, 88 characters){color}:

{code} /var/lib/jenkins/workspace/r-MQDDZ79AS4LC4V4O49CTXGW52GDS8XDTMLMAIS05Q0ZHJ2NKK19N@script {code} Only one letter is kept from my branch but I don't really care as long as it works. Thanks for this. At this point, I would only keep generated hash. In case it doesn't solve for other people, Using a shorter path for workspace sounds a good idea too: {quote} {{/j}} folder, as opposed to {{/var/lib/jenkins}}. {quote} But not quite sure what's the best way to achieve this. Related SO Questions: http://stackoverflow.com/questions/17310959/is-there-any-way-to-change-the-working-directory-of-a-jenkins-maven-build http://stackoverflow.com/questions/21839538/change-jenkins-home-on-red-hat-linux

Add Comment

[lebreton.gabriel@gmail.com (JIRA)]

Gabriel Le Breton edited a comment on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

I updated my plugins recently and my builds stopped working. I tried [Jesse Glick's suggestion|https://issues.jenkins-ci.org/browse/JENKINS-34564?focusedCommentId=272709&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-272709] and it worked . Branches , but branches names are now stripped Instead (I'm ok with that) .

Before plugin update {color:green}(was working, 56 characters){color}: {code}

/var/lib/jenkins/workspace/project/project/master@script {code} After update {color:red}(failing, 109 characters){color}:

{code} /var/lib/jenkins/workspace/project_project_master-0GR3K62UHM3ITQD3DYGVNX7HHRXYWN3NP946WC282P28PH4B061E@script {code} Adding the following line to {{/etc/default/jenkins}}: {code} JAVA_ARGS="-Djenkins.branch.WorkspaceLocatorImpl.PATH_MAX=30" {code} Results in this {color:green}(working, 88 characters){color}: {code} /var/lib/jenkins/workspace/r-MQDDZ79AS4LC4V4O49CTXGW52GDS8XDTMLMAIS05Q0ZHJ2NKK19N@script {code} Only one letter is kept from my branch but I don't really care as long as it works. Thanks for this. At this point, I would only keep generated hash. In case it doesn't solve for other people, Using a shorter path for workspace sounds a good idea too: {quote} {{/j}} folder, as opposed to {{/var/lib/jenkins}}. {quote} But not quite sure what's the best way to achieve this. Related SO Questions: http://stackoverflow.com/questions/17310959/is-there-any-way-to-change-the-working-directory-of-a-jenkins-maven-build http://stackoverflow.com/questions/21839538/change-jenkins-home-on-red-hat-linux

Add Comment

[lebreton.gabriel@gmail.com (JIRA)]

Gabriel Le Breton edited a comment on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

I updated my plugins recently and my builds stopped working. I tried [Jesse Glick's suggestion|https://issues.jenkins-ci.org/browse/JENKINS-34564?focusedCommentId=272709&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-272709] and it worked, but branches names are now stripped (I'm ok with that). For the record, in my case, it was failing to create a python virtualenv inside working directory (path too long), during {{pip install -r requirements.txt}}: {code} bad interpreter: Permission denied {code}

[lebreton.gabriel@gmail.com (JIRA)]

Gabriel Le Breton edited a comment on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

I updated my plugins recently and my builds stopped working. I tried [Jesse Glick's suggestion|https://issues.jenkins-ci.org/browse/JENKINS-34564?focusedCommentId=272709&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-272709] and it worked, but branches names are now stripped (I'm ok with that).

For the record, in my case, it was failing to create install packages in a python * virtualenv inside working directory * (path too long), during {{pip install -r requirements.txt}}:

[dbeck@cloudbees.com (JIRA)]

Daniel Beck commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

Jesse Glick

80 characters is not really that long

It's a third of the total possible path length on Windows. Some "special" tools on Linux are also shitty about this. When I was running on Windows, I set the agent home to D:\J, and the hudson.model.Slave.workspaceRoot to ws, to waste as few chars as possible. This change would have wrecked my builds because Java devs routinely use up ~200 chars in deeply nested, verbosely named folder hierarchies.

Add Comment

[mneale@cloudbees.com (JIRA)]

Michael Neale commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

perhaps its time to have a shorter hash-only thing to not eat the path length... (given many people are setting it to shorter and don't miss the job name)

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

IIRC there is some trick to make Windows behave properly here: prepending \\?\ or the like to the agent FS root. In general we cannot protect you from the consequences of having deep workspace path hierarchies. The former behavior (using raw Item.fullName) was shorter in some cases, but also unbounded in length and so caused problems for people with, for example, long branch names; the current behavior at least guarantees a bound on the part Jenkins controls.

Add Comment

[mneale@cloudbees.com (JIRA)]

Michael Neale commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

Yes as per: http://stackoverflow.com/questions/1880321/why-does-the-260-character-path-length-limit-exist-in-windows \\?\ is prefix to use. This seems to imply that on windows 10 you can just change the registry and you don't have to do anything else: http://betanews.com/2016/05/29/long-paths-windows-10/

Add Comment

[ben.herfurth@develop-group.de (JIRA)]

Ben Herfurth commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

Why cant the hash just be shortened? It worked for all of us before the change. Now it does not work anymore ==> change it back please.

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

Why cant the hash just be shortened?

It is already as short as it can be while guaranteeing that there is no collision between jobs.

Add Comment

[lebreton.gabriel@gmail.com (JIRA)]

Gabriel Le Breton commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

Is the lenght limit documented somewhere? There's usually no collision with git sha hash and it's 41 characters long. Jenkins seems to use a 53 character long.

Add Comment

[lebreton.gabriel@gmail.com (JIRA)]

Gabriel Le Breton edited a comment on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

Is the lenght length limit documented somewhere? There's usually no collision with git sha hash and it's 41 characters long. Jenkins seems to use a 53 character characters long hash .

Add Comment

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

This is using SHA-256 encoded in Base32, it is different.

Add Comment

[robertdw@twasink.net (JIRA)]

Robert Watkins commented on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

This is using SHA-256 encoded in Base32

So use SHA-256 encoded in RFC-4648-flavoured Base64 - it'll be half the length.

Add Comment

[robertdw@twasink.net (JIRA)]

Robert Watkins edited a comment on JENKINS-34564

Re: Give the ability to choose how the multibranch subprojects will be named.

bq. This is using SHA-256 encoded in Base32 So use SHA-256 encoded in [ RFC-4648 |https://en.wikipedia.org/wiki/Base64#RFC_4648] -flavoured Base64 - it'll be half the length.

Add Comment