How to avoid parallel execution of certain commands
Slawomir Czarko
Mike Shal
On Linux & OSX tup will re-direct all temporary & output files to a
private namespace in .tup/tmp so that parallel jobs don't conflict.
Eg: you could have two tests in parallel that write to "foo.txt" in
the same directory and they won't stomp on each other (assuming the
temporary files are removed before the process exits, otherwise tup
will complain that you're writing to outputs you didn't specify). Is
there some other way that you find they are conflicting? If so I'd
rather try to address that in tup itself so that all jobs can always
run in parallel if the dependencies allow it. Otherwise developers
will be randomly surprised when the conflicting jobs happen to get
scheduled at the same time, which is a design failure (of tup) in my
book.
If you're on Windows, I've been attempting to fix that on a branch,
but I haven't got all the details sorted out yet. It will be supported
eventually though.
-Mike
Slawomir Czarko
On Thursday, April 12, 2012 5:22:02 PM UTC+2, mar...@gmail.com wrote:
On Thu, Apr 12, 2012 at 4:30 AM, Slawomir Czarko
Mike Shal
It will if tup is running the sub-processes in a chroot environment.
By default it doesn't do that since it requires privileges to do so,
but then you could do:
Tupfile:
: |> ^c^ sh ok.sh 1 |>
: |> ^c^ sh ok.sh 2 |>
ok.sh:
pid=$$
echo "$1 My pid is $pid" > /tmp/output.txt
sleep 1
echo "Me is: $1 $pid"
cat /tmp/output.txt
if ! grep "$1 My pid is $pid" /tmp/output.txt > /dev/null; then
echo "Error: Expected $pid but got:" 1>&2
cat /tmp/output.txt 1>&2
exit 1
fi
rm /tmp/output.txt
With the ^c^ flag the /tmp/output.txt files get mapped, and the two
scripts can run simultaneously without issue. Without the ^c^ flag,
the absolute path doesn't go through fuse so tup doesn't see it.
I would prefer if tup could always run sub-processes in a chroot, but
I don't want to limit it to people who have root access to enable
those privileges, so unfortunately it is a manual thing that must be
done at the moment :(
-Mike
Slawomir Czarko
Slawomir Czarko
Mike Shal
<slawomi...@gmail.com> wrote:
> On Thursday, April 12, 2012 8:10:45 PM UTC+2, Slawomir Czarko wrote:
>>
>> When I use ^c^ (after chmod u+s /usr/bin/tup) I get:
>>
>> fusermount: option allow_other only allowed if 'user_allow_other' is set
>> in /etc/fuse.conf
>> fuse_mount: Operation not permitted
>> tup error: Unable to mount FUSE on .tup/mnt
>>
>> -Slawomir
>
>
> I changed /etc/fuse.conf and now I get this:
>
> 1) [0.125s] build/configurations/autotest_unit_test/c_ipc:
> ../run_one_test.sh unit_test_runner c_ipc c_ipc/_test_ok
>
> ../run_one_test.sh: line 35: /dev/fd/62: Permission denied
Assuming your system is like mine and /dev/fd is a symlink to
/proc/self/fd, I think I just pushed a fix for this. The problem is
that when the sub-process would try to read /proc/self, tup's fuse
process is the one actually doing the readlink() call, and so the
kernel would assign tup's pid for 'self' rather than the sub-processes
pid. The fuse server now just makes an explicit check for this and
writes in the correct pid. Let me know if there is still an issue
here.
> *** tup errors ***
> tup warning: Writing to hidden file
> '/sandbox/slawomir/deve/repos/cyclades-trunk/development/.log/host_unit_test_stdout.log'
> tup warning: Writing to hidden file
> '/sandbox/slawomir/deve/repos/cyclades-trunk/development/.log/host_unit_test_unwanted_stdout.log'
> [ ] 100%
> tup warning: Update resulted in 2 warnings
>
> line 35 is where I actually run a test:
>
> run_unit_test $PKG $UNIT_TEST_RUNNER_PROG > >(tee --append ${STDOUT_LOG}) 2>
>>(tee --append ${STDERR_LOG} >&2)
>
> If I remove redirections I get to see many tests failing due to changed
> behaviour in chroot. For example calls to mkfifo fail.
I also pushed a fix for creating temporary fifos. It still won't let
you create a fifo as an output file (I could be convinced to change
this if someone has a compelling use-case), but a sub-process can
create a fifo and use it internally, then delete it when it is
finished and it should work.
Thanks for the reports - let me know what else you run into.
-Mike
Slawomir Czarko
Assuming your system is like mine and /dev/fd is a symlink to
/proc/self/fd, I think I just pushed a fix for this. The problem is
that when the sub-process would try to read /proc/self, tup's fuse
process is the one actually doing the readlink() call, and so the
kernel would assign tup's pid for 'self' rather than the sub-processes
pid. The fuse server now just makes an explicit check for this and
writes in the correct pid. Let me know if there is still an issue
here.
> *** tup errors ***
> tup warning: Writing to hidden file
> '/sandbox/slawomir/deve/repos/cyclades-trunk/development/.log/host_unit_test_stdout.log'
> tup warning: Writing to hidden file
> '/sandbox/slawomir/deve/repos/cyclades-trunk/development/.log/host_unit_test_unwanted_stdout.log'
> [ ] 100%
> tup warning: Update resulted in 2 warnings
I also pushed a fix for creating temporary fifos. It still won't let
you create a fifo as an output file (I could be convinced to change
this if someone has a compelling use-case), but a sub-process can
create a fifo and use it internally, then delete it when it is
finished and it should work.
Mike Shal
<slawomi...@gmail.com> wrote:
> On Saturday, April 14, 2012 3:28:39 PM UTC+2, mar...@gmail.com wrote:
>>
>> Assuming your system is like mine and /dev/fd is a symlink to
>> /proc/self/fd, I think I just pushed a fix for this. The problem is
>> that when the sub-process would try to read /proc/self, tup's fuse
>> process is the one actually doing the readlink() call, and so the
>> kernel would assign tup's pid for 'self' rather than the sub-processes
>> pid. The fuse server now just makes an explicit check for this and
>> writes in the correct pid. Let me know if there is still an issue
>> here.
>
> With the latest version I'm getting this:
>
> * 1) build/configurations/autotest_unit_test/t_sdd: ../run_one_test.sh
> unit_test_runner t_sdd t_sdd/_test_ok
>
> *** tup errors ***
> tup error: File '/proc/19560/fd/pipe:[11179468]' was written to, but is not
> in .tup/db. You probably should specify it as an output
Do you know what library calls it is using to get this behavior? Or
what it's trying to do? I'm having trouble trying to reproduce this.
>
>
>
>>
>> > *** tup errors ***
>> > tup warning: Writing to hidden file
>> >
>> > '/sandbox/slawomir/deve/repos/cyclades-trunk/development/.log/host_unit_test_stdout.log'
>> > tup warning: Writing to hidden file
>> >
>> > '/sandbox/slawomir/deve/repos/cyclades-trunk/development/.log/host_unit_test_unwanted_stdout.log'
>> > [ ] 100%
>> > tup warning: Update resulted in 2 warnings
>
> Can I get rid of these warnings?
I just pushed a new option to master - now you can set
updater.warnings=0 in your tup options.
-Mike
Slawomir Czarko
On Tue, Apr 17, 2012 at 5:05 AM, Slawomir Czarko
wrote:
> On Saturday, April 14, 2012 3:28:39 PM UTC+2, mar...@gmail.com wrote:
>>
>> Assuming your system is like mine and /dev/fd is a symlink to
>> /proc/self/fd, I think I just pushed a fix for this. The problem is
>> that when the sub-process would try to read /proc/self, tup's fuse
>> process is the one actually doing the readlink() call, and so the
>> kernel would assign tup's pid for 'self' rather than the sub-processes
>> pid. The fuse server now just makes an explicit check for this and
>> writes in the correct pid. Let me know if there is still an issue
>> here.
>
> With the latest version I'm getting this:
>
> * 1) build/configurations/autotest_unit_test/t_sdd: ../run_one_test.sh
> unit_test_runner t_sdd t_sdd/_test_ok
>
> *** tup errors ***
> tup error: File '/proc/19560/fd/pipe:[11179468]' was written to, but is not
> in .tup/db. You probably should specify it as an outputDo you know what library calls it is using to get this behavior? Or
what it's trying to do? I'm having trouble trying to reproduce this.
Mike Shal
Ahh, I can reproduce it with that. But it seems this doesn't work with
a chroot in the fuse example file-system either, so I'm not sure how
tup could support it. I can't figure out why exactly it's failing
though - if I modify the fuse example to handle /proc/self correctly,
and run everything as root, it still isn't able to create the
pipe:[12345] file.
Are you just trying to log a specific command's output here? Or do you
do this for every command? Since tup already captures the output to a
file, maybe it would make sense to put logging there. I think it would
essentially just involve a rename() instead of unlink() on the output
file when the command finishes. Though tup puts both stdout and stderr
to the same file to keep them chronological, so it would be different
from what you're trying to do.
Also I feel like we just went down this path because of commands
trying to use the same filenames in /tmp. How much effort would be
involved in having those commands use a proper unique filename?
-Mike
Slawomir Czarko
On Tuesday, April 24, 2012 5:17:29 AM UTC+2, mar...@gmail.com wrote:
On Fri, Apr 20, 2012 at 7:06 AM, Slawomir Czarko
Also I feel like we just went down this path because of commands
trying to use the same filenames in /tmp. How much effort would be
involved in having those commands use a proper unique filename?
Mike Shal
work if I bind-mount /proc (currently I just bind-mount /dev into the
chroot). However, it seems tup is having trouble waiting for the full
process to complete. What does this style of redirection actually do?
Any idea why I can't waitpid() on the initial process and have it wait
until both the main command and the tee process complete? If we can
get that piece, then I think we can get this working...
-Mike
Slawomir Czarko
On Thursday, April 26, 2012 3:28:39 AM UTC+2, mar...@gmail.com wrote:
I took another stab at this, and it seems I can get the redirection to
work if I bind-mount /proc (currently I just bind-mount /dev into the
chroot). However, it seems tup is having trouble waiting for the full
process to complete. What does this style of redirection actually do?
Any idea why I can't waitpid() on the initial process and have it wait
until both the main command and the tee process complete? If we can
get that piece, then I think we can get this working...
Mike Shal
http://lists.gnu.org/archive/html/bug-bash/2011-11/msg00137.html
The problem I am running into is that the tee process is run in a
second shell that is "disowned" from the parent shell. This means when
tup does a waitpid() on the process it created, it may return before
the tee command ever even runs. You can see this in a shell just by
doing:
true> >(sleep 1; echo foo)
The command prompt will return before the echo runs. The problem for
tup is that it thinks the sub-process has already finished, then tee
may try to create the file afterward.
It seems this method of redirection causes problems for others as well:
http://stackoverflow.com/questions/4489139/bash-process-substitution-and-syncing
That thread has a few recommendations to try. I couldn't get the
sync.pipe one to work, but does this way work for you?
{ { command | tee -- "$stdoutF" 1>&3; } 2>&1 \
| tee -- "$stderrF" 1>&2; } 3>&1
(Note you need a space after the {'s otherwise it may fail in
non-sensical ways). And if you use the PIPESTATUS variable I think you
need to do a 'set -o pipefail' in your bash script.
Or if you can find a way to properly synchronize the tee processes
with the >( redirection so that waitpid() won't return until they are
finished, then I can apply the patch to tup for bind-mounting /proc
and then it should work...
Let me know what you come up with, and if I should apply that patch
(maybe I'll just do that anyway...)
-Mike
Slawomir Czarko
Mike Shal
probably why it's complaining - if they aren't specified as outputs
and a command tries to overwrite files, tup won't allow it.
Maybe it would help to create a log file for each unit test, and then
group them together with a different command? Something like:
: foreach *.exe |> unit test | tee %o |> %B-output.log
: *-output.log |> cat %f > %o |> summaried-output.log
If this gets too tricky, it would probably make sense to look into
adding log capturing as a feature of tup.
-Mike