[JIRA] (JENKINS-39668) Attempts to copy files across jobs.

1 view
Skip to first unread message

andwho@gmail.com (JIRA)

unread,
Nov 11, 2016, 6:35:02 AM11/11/16
to jenkinsc...@googlegroups.com
Anders Hoglund created an issue
 
Jenkins / Bug JENKINS-39668
Attempts to copy files across jobs.
Issue Type: Bug Bug
Assignee: Ulli Hafner
Components: warnings-plugin
Created: 2016/Nov/11 11:34 AM
Environment: Ubuntu 16.04LTS
Priority: Major Major
Reporter: Anders Hoglund

The warnings-plugin seems to access files and tries to make copies accross job boundries. Noticed this when new files pulled in one job are not yet available in the other, then I get a lot of false warnings added to the statistics. The current job name is "BorisB_BetaFlight" and from there the warnings-plugin tries to access files in a job named "AndWho_Betaflight_buildsystem"

See it all here: http://andwho.sytes.net:8080/job/BorisB_BetaFlight/697/warnings27Result/source.-8247652556134220142/#16

 01 Copying the source file '/var/lib/jenkins/jobs/AndWho_Betaflight_buildsystem/workspace/src/test/./src/main/target/KAKUTEF4-V1/target.c' from the workspace to the build folder 'f16d8a0.tmp' on the Jenkins master failed.
02 Is the file '/var/lib/jenkins/jobs/AndWho_Betaflight_buildsystem/workspace/src/test/./src/main/target/KAKUTEF4-V1/target.c' a valid filename?
03 If you are building on a slave: please check if the file is accessible under '$JENKINS_HOME/[job-name]//var/lib/jenkins/jobs/AndWho_Betaflight_buildsystem/workspace/src/test/./src/main/target/KAKUTEF4-V1/target.c'
04 If you are building on the master: please check if the file is accessible under '$JENKINS_HOME/[job-name]/workspace//var/lib/jenkins/jobs/AndWho_Betaflight_buildsystem/workspace/src/test/./src/main/target/KAKUTEF4-V1/target.c'
05 java.io.IOException: Failed to copy /var/lib/jenkins/jobs/AndWho_Betaflight_buildsystem/workspace/src/test/src/main/target/KAKUTEF4-V1/target.c to /var/lib/jenkins/jobs/BorisB_BetaFlight/builds/697/workspace-files/f16d8a0.tmp
06   at hudson.FilePath.copyTo(FilePath.java:2018)
07   at hudson.plugins.analysis.util.Files.copyFilesWithAnnotationsToBuildFolder(Files.java:80)
08   at hudson.plugins.analysis.core.HealthAwareRecorder.copyFilesWithAnnotationsToBuildFolder(HealthAwareRecorder.java:333)
09   at hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:89)
10   at hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:280)
11   at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:78)
12   at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
13   at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
14   at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:720)
15   at hudson.model.Build$BuildExecution.post2(Build.java:185)
16   at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:665)
17   at hudson.model.Run.execute(Run.java:1745)
18   at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
19   at hudson.model.ResourceController.execute(ResourceController.java:98)
20   at hudson.model.Executor.run(Executor.java:404)
21 Caused by: java.io.FileNotFoundException: /var/lib/jenkins/jobs/AndWho_Betaflight_buildsystem/workspace/src/test/src/main/target/KAKUTEF4-V1/target.c (No such file or directory)
22   at java.io.FileInputStream.open0(Native Method)
23   at java.io.FileInputStream.open(FileInputStream.java:195)
24   at java.io.FileInputStream.<init>(FileInputStream.java:138)
25   at hudson.FilePath$41.invoke(FilePath.java:2044)
26   at hudson.FilePath$41.invoke(FilePath.java:2039)
27   at hudson.FilePath.act(FilePath.java:1018)
28   at hudson.FilePath.act(FilePath.java:996)
29   at hudson.FilePath.copyTo(FilePath.java:2039)
30   at hudson.FilePath.copyTo(FilePath.java:2013)
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

andwho@gmail.com (JIRA)

unread,
Nov 11, 2016, 6:54:01 AM11/11/16
to jenkinsc...@googlegroups.com
Anders Hoglund commented on Bug JENKINS-39668
 
Re: Attempts to copy files across jobs.

Problem seems to have been introduced Nov 9 in a plugin update. Job 971 fails, prev job 970 is OK, only job relative oaths here:
http://andwho.sytes.net:8080/job/BorisB_BetaFlight/670/warnings27Result/

andwho@gmail.com (JIRA)

unread,
Nov 11, 2016, 6:55:01 AM11/11/16
to jenkinsc...@googlegroups.com
Anders Hoglund edited a comment on Bug JENKINS-39668
Problem seems to have been introduced Nov 9 in a plugin update. Job 971 fails, prev job 970 is OK, only job relative oaths paths here:
http://andwho.sytes.net:8080/job/BorisB_BetaFlight/670/warnings27Result/

andwho@gmail.com (JIRA)

unread,
Nov 11, 2016, 9:27:01 AM11/11/16
to jenkinsc...@googlegroups.com
Anders Hoglund edited a comment on Bug JENKINS-39668
Problem seems to have been introduced Nov 9 in a plugin update. Job 971 671 fails, prev job 970 670 is OK, only job relative paths here:
http://andwho.sytes.net:8080/job/BorisB_BetaFlight/670/warnings27Result/

ullrich.hafner@gmail.com (JIRA)

unread,
Nov 13, 2016, 5:46:01 PM11/13/16
to jenkinsc...@googlegroups.com

There was no update at this time. Is this due to a core change? Or did you change the folder structure?

In your cases the relative paths are expanded to absolute paths (since gcc did not provide absolute ones), seems that this computation now guesses the wrong path.

andwho@gmail.com (JIRA)

unread,
Nov 14, 2016, 11:12:02 AM11/14/16
to jenkinsc...@googlegroups.com

I have no record of what was updated in Jenkins at that time. Usually only plugin updates. There are other jobs/projects with same kind of interference. For axample this one: http://andwho.sytes.net:8080/job/AndWho_Betaflight_buildsystem/375/warnings27Result/. Next build was OK.
I have also found really old builds with this problem, so my Nov9 assumption seems false. Look here, Sept29 : http://andwho.sytes.net:8080/job/rs2k_raceflight/2/warnings27Result/ Project "rs2k_raceflight" accessing files in neighbour project "AndWho_Betaflight_BUZZERM6"
As if there is a clash on the WORKSPACE env. variable.

ullrich.hafner@gmail.com (JIRA)

unread,
Nov 27, 2016, 8:12:03 AM11/27/16
to jenkinsc...@googlegroups.com

I see. Relative paths arevsomewhat difficult to handle, normally make should print the path before gcc is running. See https://github.com/jenkinsci/warnings-plugin/blob/master/src/test/resources/hudson/plugins/warnings/parser/gnuMakeGcc.txt for an example.

Did you use any special make options?

ullrich.hafner@gmail.com (JIRA)

unread,
Nov 27, 2016, 8:13:01 AM11/27/16
to jenkinsc...@googlegroups.com
Ulli Hafner edited a comment on Bug JENKINS-39668
I see. Relative paths arevsomewhat are somewhat difficult to handle, normally make should print the path before gcc is running. See https://github.com/jenkinsci/warnings-plugin/blob/master/src/test/resources/hudson/plugins/warnings/parser/gnuMakeGcc.txt for an example.


Did you use any special make options?

andwho@gmail.com (JIRA)

unread,
Nov 27, 2016, 12:39:01 PM11/27/16
to jenkinsc...@googlegroups.com

Well, make is run recursively. Typical top level is just with goal/target arg "make all" or "make REVO" etc. Second level uses some extra args. And the the -j flag for full parallellism in the bottom layer.
Typically it looks like this:

$ make REVO | grep make
   make binary hex TARGET=REVO && \
make[1]: Entering directory '/home/andho/projects/cleanflight/andwho/betaflight'
make -j ./obj/betaflight_3.1.0_REVO.bin
make[2]: Entering directory '/home/andho/projects/cleanflight/andwho/betaflight'
make[2]: Leaving directory '/home/andho/projects/cleanflight/andwho/betaflight'
make -j ./obj/betaflight_3.1.0_REVO.hex
make[2]: Entering directory '/home/andho/projects/cleanflight/andwho/betaflight'
make[2]: Leaving directory '/home/andho/projects/cleanflight/andwho/betaflight'
make[1]: Leaving directory '/home/andho/projects/cleanflight/andwho/betaflight'

The compile command line looks typically like this:

ccache ./tools/gcc-arm-none-eabi-5_4-2016q3/bin/arm-none-eabi-gcc -c -o obj/main/REVO/telemetry/esc_telemetry.o -mthumb -mcpu=cortex-m4 -march=armv7e-m -mfloat-abi=hard -mfpu=fpv4-sp-d16 -fsingle-precision-const
ant -Wdouble-promotion -flto -fuse-linker-plugin -Os  -I./src/main -I./src/main/target -I./lib/main/STM32F4xx_StdPeriph_Driver/inc -I./lib/main/STM32_USB_OTG_Driver/inc -I./lib/main/STM32_USB_Device_Library/Core
/inc -I./lib/main/STM32_USB_Device_Library/Class/cdc/inc -I./lib/main/STM32_USB-FS-Device_Driver/inc -I./lib/main/CMSIS/CM4/CoreSupport -I./lib/main/CMSIS/CM4/DeviceSupport/ST/STM32F4xx -I./src/main/vcpf4 -I./li
b/main/MAVLink -I./src/main/target/REVO -ggdb3 -DDEBUG -std=gnu99 -Wall -Wextra -Wunsafe-loop-optimizations -Wdouble-promotion -ffunction-sections -fdata-sections -pedantic -DSTM32F40_41xxx -DHSE_VALUE=8000000 -
DFLASH_SIZE=1024 -DHSE_VALUE=8000000 -DUSE_STDPERIPH_DRIVER -DREVO  -D'__FORKNAME__="betaflight"' -D'__TARGET__="REVO"' -D'__REVISION__="04ea404"' -save-temps=obj -MMD -MP ./src/main/telemetry/esc_telemetry.c

/A

ullrich.hafner@gmail.com (JIRA)

unread,
Nov 28, 2016, 6:58:01 AM11/28/16
to jenkinsc...@googlegroups.com

Hmm, there are several 'Entering directory' messages in the log. Seems that the GCC+Make parser does not work correctly here...

andwho@gmail.com (JIRA)

unread,
Nov 28, 2016, 9:17:01 AM11/28/16
to jenkinsc...@googlegroups.com

Yes, those are absolute full paths. But still only for the current project directory. No reference into any neighbouring project.
Also, I have not seen this problem for some time now. Could it be showing up only in certain situations, like two projects being built/checked at the same time? I have two executors in Jenkins.

ullrich.hafner@gmail.com (JIRA)

unread,
Nov 28, 2016, 6:05:03 PM11/28/16
to jenkinsc...@googlegroups.com

What I wanted to say: if the parser correctly reads the part

make[1]: Entering directory '/home/andho/projects/cleanflight/andwho/betaflight'

then the absolute path to the source ./src/main/telemetry/esc_telemetry.c is obtained by a simple concatenation. However, sometimes in your case that part does not work and only the relative path ./src/main/telemetry/esc_telemetry.c is used. In this case the workspace is scanned for matching files.

I don#t find any 'Entering directory' messages in your reported build: http://andwho.sytes.net:8080/job/BorisB_BetaFlight/697/console
Is this build different?

ullrich.hafner@gmail.com (JIRA)

unread,
Nov 28, 2016, 6:06:03 PM11/28/16
to jenkinsc...@googlegroups.com
Ulli Hafner edited a comment on Bug JENKINS-39668
What I wanted to say: if the parser correctly reads the part
{noformat}

make[1]: Entering directory '/home/andho/projects/cleanflight/andwho/betaflight'
{noformat}
then the absolute path to the source {{./src/main/telemetry/esc_telemetry.c}} is obtained by a simple concatenation. However, sometimes in your case that part does not work sometimes and only then the relative path {{./src/main/telemetry/esc_telemetry.c}} is used. In this case the workspace is scanned for matching files.

I don#t find any 'Entering directory' messages in your reported build: http://andwho.sytes.net:8080/job/BorisB_BetaFlight/697/console
Is this build different?

ullrich.hafner@gmail.com (JIRA)

unread,
Nov 28, 2016, 6:07:01 PM11/28/16
to jenkinsc...@googlegroups.com
Ulli Hafner edited a comment on Bug JENKINS-39668
What I wanted to say: if the parser correctly reads the part
{noformat}
make[1]: Entering directory '/home/andho/projects/cleanflight/andwho/betaflight'
{noformat}
then the absolute path to the source {{./src/main/telemetry/esc_telemetry.c}} is obtained by a simple concatenation. However, in your case that part does not work sometimes and then the relative path {{./src/main/telemetry/esc_telemetry.c}} is used. In this case the workspace is scanned for matching files.
(Which also obtained a wrong result but This is an other problem: normally the absolute path should be automatically computed).

I don#t find any 'Entering directory' messages in your reported build: http://andwho.sytes.net:8080/job/BorisB_BetaFlight/697/console
Is this build different?

ullrich.hafner@gmail.com (JIRA)

unread,
Nov 28, 2016, 6:07:02 PM11/28/16
to jenkinsc...@googlegroups.com
Ulli Hafner edited a comment on Bug JENKINS-39668
What I wanted to say: if the parser correctly reads the part
{noformat}
make[1]: Entering directory '/home/andho/projects/cleanflight/andwho/betaflight'
{noformat}
then the absolute path to the source {{./src/main/telemetry/esc_telemetry.c}} is obtained by a simple concatenation. However, in your case that part does not work sometimes and then the relative path {{./src/main/telemetry/esc_telemetry.c}} is used. In this case the workspace is scanned for matching files. (Which also obtained a wrong result but This this is an other problem: normally the absolute path should be automatically computed).


I don#t find any 'Entering directory' messages in your reported build: http://andwho.sytes.net:8080/job/BorisB_BetaFlight/697/console
Is this build different?

ullrich.hafner@gmail.com (JIRA)

unread,
Nov 28, 2016, 6:08:02 PM11/28/16
to jenkinsc...@googlegroups.com
Ulli Hafner edited a comment on Bug JENKINS-39668
What I wanted to say: if the parser correctly reads the part
{noformat}
make[1]: Entering directory '/home/andho/projects/cleanflight/andwho/betaflight'
{noformat}
then the absolute path to the source {{./src/main/telemetry/esc_telemetry.c}} is obtained by a simple concatenation. However, in your case that part does not work sometimes and then the relative path {{./src/main/telemetry/esc_telemetry.c}} is used. In this case the workspace is scanned for matching files. (Which also obtained a wrong result but this is an other problem: normally the absolute path should be automatically computed).

I don # ' t find any 'Entering directory' messages in your reported build: http://andwho.sytes.net:8080/job/BorisB_BetaFlight/697/console
Is this build different?

andwho@gmail.com (JIRA)

unread,
Nov 28, 2016, 7:45:01 PM11/28/16
to jenkinsc...@googlegroups.com

Oh, sorry. Should have mentioned that this printout was not done by Jenkins, it was a manual build in my home-dir with full verbosity just to show the flags used and the recursion of make. Jenkins builds are done with default verbosity, no such "Entering Dir..." messages logged. Later builds have an "make V=0 ...." option for even less verbosity. As you noticed.
If the warning plugin depends on this printout being logged, then you might just have found something here. But I do not quite understand why it is needed. There must be better ways of finding out the project root dir, if needed at all. But OK, it may be cases where you need to follow when make recurses into dirs below the root.
Please advice if I should update the makefiles to show this kind of messages even at low verbosity settings. Or even turn on full verbosity, but that will cause a large console log....

ullrich.hafner@gmail.com (JIRA)

unread,
Nov 30, 2016, 8:41:01 AM11/30/16
to jenkinsc...@googlegroups.com

Actually the make/gcc parser is written by a volunteer, since I don't use these tools on my own. I think it would improve the stability of the results (and the speed!) if you would start make with an option so that the current directory is shown.

andwho@gmail.com (JIRA)

unread,
Nov 30, 2016, 2:14:01 PM11/30/16
to jenkinsc...@googlegroups.com

OK, but $WORKSPACE env. var. would be much better for the root dir. Everything is relative to that. Unless make recurses down lower to dir levels.
I think there is still something wrong, one project knowing about the $WORKSPACE of another project. The root dir should/could always be set correctly.

ullrich.hafner@gmail.com (JIRA)

unread,
Nov 30, 2016, 4:18:02 PM11/30/16
to jenkinsc...@googlegroups.com

Yes, I think there are two different problems here.

kierzo@kierzo.com (JIRA)

unread,
Jul 27, 2018, 11:35:03 AM7/27/18
to jenkinsc...@googlegroups.com

I have a similar issue compiling using msbuild, the parser is including part of the output line as part of the file name e.g. including "134>" because the msbuild compiler sometimes does not put a space between the process number and filename  :

e.g.

134>F:\Jenkins\workspace\project\code\colr.for(8): remark #7712: This variable has not been used. [COLSET]

01 Copying the source file '134>F:\Jenkins\workspace\project\code\colr.for' from the workspace to the build folder '1f8ace0e.tmp' on the Jenkins master failed.
02 Is the file '134>F:\Jenkins\workspace\project\code\colr.for' a valid filename?

 
This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)

ullrich.hafner@gmail.com (JIRA)

unread,
Nov 7, 2018, 8:51:02 AM11/7/18
to jenkinsc...@googlegroups.com
Ulli Hafner updated an issue
 
Change By: Ulli Hafner
Component/s: warnings-ng-plugin
Component/s: warnings-plugin
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

ullrich.hafner@gmail.com (JIRA)

unread,
Jan 15, 2019, 6:35:02 AM1/15/19
to jenkinsc...@googlegroups.com
Ulli Hafner resolved as Fixed
 

I think that should work now in the warnings-ng plugin.
Change By: Ulli Hafner
Status: Open Resolved
Resolution: Fixed
Reply all
Reply to author
Forward
0 new messages