parallel vala builds: dependencies

[Simon Werbeck]

Hi, I managed to setup parallel builds for vala and tup. But for now I
need to rebuild the whole project whenever a file changes.
Basically, in one command I generate the interface (`fast' vapi) files
from the .vala source files. After that I run a script to generate a
rule for each source file, specifying all vapis files except it's own
as order-only inputs.
The problem is that the vala compiler needs to read from every vapi in
order to find the required symbols. That means tup will execute the
command if any vapi file changes.
That said I think the problem could be solved by introducing a new
kind of `weak' order-only input similar to makes order-only-
prerequisites http://www.gnu.org/software/make/manual/make.html#Prerequisite-Types
which would not cause the command to be reexecuted but allows it to
reading from the `weak' input.

Not shure I thought this all through but that's all I can think of for
now.
Thanks in advance!

[Mike Shal]

Hi Simon,

On Mon, May 28, 2012 at 9:16 AM, Simon Werbeck
<simon....@googlemail.com> wrote:
> Hi, I managed to setup parallel builds for vala and tup. But for now I
> need to rebuild the whole project whenever a file changes.
> Basically, in one command I generate the interface (`fast' vapi) files
> from the .vala source files. After that I run a script to generate a
> rule for each source file, specifying all vapis files except it's own
> as order-only inputs.

If I understand you correctly, you have a single command like this:

: *.vala |> cmd... |> 1.vapi 2.vapi ...

This would necessarily re-create every .vapi file everytime any .vala
file changed. Have you tried using a foreach instead? You can see an
example of this that Emmanuel Oga created:
https://github.com/EmmanuelOga/valatup

I don't know how well that scales up to a real project but maybe it would help.

> The problem is that the vala compiler needs to read from every vapi in
> order to find the required symbols. That means tup will execute the
> command if any vapi file changes.
> That said I think the problem could be solved by introducing a new
> kind of `weak' order-only input similar to makes order-only-
> prerequisites http://www.gnu.org/software/make/manual/make.html#Prerequisite-Types
> which would not cause the command to be reexecuted but allows it to
> reading from the `weak' input.

If the vala compiler is reading from every vapi file, presumably you
*want* to re-run the vala compiler every time any vapi file changes?
If not, why is the compiler reading from it if it has no effect on the
output?

-Mike

[Simon Werbeck]

Am Freitag, den 01.06.2012, 13:39 -0400 schrieb Mike Shal:
> Hi Simon,
>
> On Mon, May 28, 2012 at 9:16 AM, Simon Werbeck
> <simon....@googlemail.com> wrote:
> > Hi, I managed to setup parallel builds for vala and tup. But for now I
> > need to rebuild the whole project whenever a file changes.
> > Basically, in one command I generate the interface (`fast' vapi) files
> > from the .vala source files. After that I run a script to generate a
> > rule for each source file, specifying all vapis files except it's own
> > as order-only inputs.
>
> If I understand you correctly, you have a single command like this:
>
> : *.vala |> cmd... |> 1.vapi 2.vapi ...

No, I'm already using foreach, sorry I didn't write that clearly enough.
OK, let's say I have the command:

!fast-vapi = |> valac --fast-vapi=%B.vapi %f |> %B.vapi

and I run that on each .vala file using foreach:
:foreach *.vala |> !fast-vapi |>

After that, to compile each vala file separately I use:

hello.vala | square.vapi |> valac --use-fast-vapi=square.vapi -c %f |>
%b.o
square.vala |> valac -c %f |> %b.o

Note that I have to explicitly specify the vapi file as order-only
dependency, as it was auto-generated.

but what happens if I have another file circle.vala and I modify
hello.vala to depend on that file? Then I have to edit the Tupfile to
look like this:

hello.vala | square.vapi circle.vapi |> valac
--use-fast-vapi=square.vapi --use-fast-vapi=circle.vapi -c %f |> %b.o

Writing that by hand will become tedious once your project grows. So of
course I tried writing a script to automate that. The script would grab
all fast-vapi files in the directory except the own (hello.vala doesn't
need hello.vapi) and use them as order-only inputs and write the
--use-fast-vapi flags. But then, what if I have another file `cube.vala'
that is not used by hello.vala? My script would still list cube.vapi
file as order-only input and recompile hello.vala on every change of
cube.vapi. Clearly, I needed dependency information.

> If the vala compiler is reading from every vapi file, presumably you
> *want* to re-run the vala compiler every time any vapi file changes?
> If not, why is the compiler reading from it if it has no effect on the
> output?

The purpose of fast-vapi files from what I can tell is, that they're
`faster' parsable than .vala files, as they only contain interface
definitions `local' to that single file. But since they contain no
#include directives you have to pass every single one of them to every
compilation command that might need it and the compiler then picks out
the definitions it needs. That is also the recommended way according to
this post (in combination with make dependenies, see below)
http://web.archiveorange.com/archive/v/vLjdSKbOYpjdV2gTRldx

valac has a --deps flag to output Makefile dependency information. I
tried parsing them to generate the build rules. But they are only
generated after compilation i.e. you can't generate them in a seperate
step. As a result I ran into a cyclic dependency problem, so this is no
option as well.

Wheew, I hope some of that makes sense. In conclusion I think the
options I have are:
1. For each vala file, maintain a handwritten deps file, parse it to
generate build rules (kind of like #include rules in .c files)
2. Use valac's --deps flag, run a seperate script to generate build
rules from that, modify by hand on changes in the source file
3. The afore mentioned `weak' order-only dependencies. They determine
only order of execution but don't force a rebuild.
4. Recompile everything everytime (right now the easiest way)

I think none of the options is optimal, so I'm thankful for any
suggestion.

Thanks,
Simon

[Mike Shal]

Hi Simon,

On Sat, Jun 2, 2012 at 9:00 AM, Simon Werbeck

Listing an unused file as an input (or order-only input) is fine - the
only downside to over-specifying inputs is that it could limit
parallelism. The inputs you specify in the Tupfile aren't used for
determining when to rebuild, they are only used for ordering. So if I
understand your script correctly, all it would mean is that all .vapi
files are created before valac runs, even if there is technically a
valac that could run simultaneously with an unreleated vapi
generation.

The actual dependencies that determine when to rebuild are determined
by the file accesses of the process. The reason you see everything
rebuilding when an "unused" vapi file is regenerated is because valac
is actually reading from every vapi file. I suspect this is because
your script is specifying every .vapi file in the --use-fast-vapi
flag.

>
>> If the vala compiler is reading from every vapi file, presumably you
>> *want* to re-run the vala compiler every time any vapi file changes?
>> If not, why is the compiler reading from it if it has no effect on the
>> output?
> The purpose of fast-vapi files from what I can tell is, that they're
> `faster' parsable than .vala files, as they only contain interface
> definitions `local' to that single file. But since they contain no
> #include directives you have to pass every single one of them to every
> compilation command that might need it and the compiler then picks out
> the definitions it needs. That is also the recommended way according to
> this post (in combination with make dependenies, see below)
> http://web.archiveorange.com/archive/v/vLjdSKbOYpjdV2gTRldx

Hmm, it seems tup is a little at odds with vala here, since it isn't
going to use their generated deps file. Tup makes the determination of
when to rebuild solely based on the files accessed by valac, which you
list explicitly with the --use-fast-vapi flag.

>
> valac has a --deps flag to output Makefile dependency information. I
> tried parsing them to generate the build rules. But they are only
> generated after compilation i.e. you can't generate them in a seperate
> step. As a result I ran into a cyclic dependency problem, so this is no
> option as well.
>
> Wheew, I hope some of that makes sense. In conclusion I think the
> options I have are:
> 1. For each vala file, maintain a handwritten deps file, parse it to
> generate build rules (kind of like #include rules in .c files)
> 2. Use valac's --deps flag, run a seperate script to generate build
> rules from that, modify by hand on changes in the source file
> 3. The afore mentioned `weak' order-only dependencies. They determine
> only order of execution but don't force a rebuild.
> 4. Recompile everything everytime (right now the easiest way)
>
> I think none of the options is optimal, so I'm thankful for any
> suggestion.

If I understand what you mean by "weak" dependencies, *every* input
link is "weak" in tup (internally, these are called sticky links). Tup
will upgrade a sticky link to a normal link if the sub-process
actually reads from the file. Note that a regular input vs. order-only
input doesn't matter here - the only difference between those is that
regular inputs show up in %-flags (like %f), but order-only inputs do
not.

So, I think you could leave the script to list every vapi file as an
input without much concern (maybe it would get slow if you have lots
and lots of vala files in a single directory). The focus should
probably be on finding a way to only list the --use-fast-vapi flags
that are actually necessary, so that valac will only read from those
files. This might need to be something like option 1) - which as you
said would be similar to writing #includes. Though instead of going in
the vala file, maybe they would just go in the Tupfile:

# includes for foo.vala
VALAFLAGS_foo.vala += --use-fast-vapi=bar.vapi

# includes for bar.vala
VALAFLAGS_bar.vala += --use-fast-vapi=baz.vapi
VALAFLAGS_bar.vala += --use-fast-vapi=baz2.vapi

: foreach *.vala | *.vapi |> valac %f $(VALAFLAGS_%f) ... |> ...

You could put those VALAFLAGS definitions in their own files (like
foo.inc, bar.inc, or whatever you want to call it) if you don't want
them all in a single file. I don't really know how this will play out
in practice. Please post back if you get a nice working set of
rules/scripts that work with vala (I'm sure this would help others),
or if you get stuck again maybe there are other things we can try.

Thanks,
-Mike

[encodr]

Hi there

I've pretty much moved adopted tup as my preferred way to build code, on Linux and OSX.
For various reasons I now need to look at FreeBSD, and I'm confused regarding the possibility of running tup on it.
I see contradictory messages about the ability of FreeBSD to run fuse at all (some reports say it supports fuse for years, yet there is a google SOC project from last year apparently to make it happen).

Can anyone tell me
* can I run fuse on FreeBSD?
* if yes: can I run tup on the fuse that runs on FreeBSD?
* if no: can I use an older tup of ldpreload vintage on FreeBSD?
or more simply

- can I run tup on FreeBSD?

thanks

e

[Sean]

> I've pretty much moved adopted tup as my preferred way to build code, on Linux and OSX.

It's pretty nice, isn't it? The two platforms I spend time on are OS-X
(dev) and FreeBSD (production).

> For various reasons I now need to look at FreeBSD, and I'm confused regarding the possibility of running tup on it.
> I see contradictory messages about the ability of FreeBSD to run fuse at all (some reports say it supports fuse for years, yet there is a google SOC project from last year apparently to make it happen).
>
> Can anyone tell me
> * can I run fuse on FreeBSD?

You can, or at least I know that there's a FUSE port for FreeBSD. I
can't use FUSE in production, however.

> * if yes: can I run tup on the fuse that runs on FreeBSD?

Possibly.

> * if no: can I use an older tup of ldpreload vintage on FreeBSD?
> or more simply

This would be nice.

> - can I run tup on FreeBSD?

A more accurate question is, "Can I run tup on a system without
FUSE?" At present, the answer is no, however Michael and I have had a
few off-list discussions about this so I don't imagine this will be
the case forever, but at present, FUSE is required.

-sc

[Mike Shal]

On Sun, Jun 17, 2012 at 6:54 PM, Sean <se...@chittenden.org> wrote:
>> I've pretty much moved adopted tup as my preferred way to build code, on Linux and OSX.
>
> It's pretty nice, isn't it? The two platforms I spend time on are OS-X
> (dev) and FreeBSD (production).
>
>> For various reasons I now need to look at FreeBSD, and I'm confused regarding the possibility of running tup on it.
>> I see contradictory messages about the ability of FreeBSD to run fuse at all (some reports say it supports fuse for years, yet there is a google SOC project from last year apparently to make it happen).
>>
>> Can anyone tell me
>> * can I run fuse on FreeBSD?
>
> You can, or at least I know that there's a FUSE port for FreeBSD. I
> can't use FUSE in production, however.

IIRC in our offline discussions we talked about using a one-shot build
that would run tup as a single-use shell script rather than doing any
dependency checking. Would this be sufficient for the production
installs? Eg you could do:

tar -xvf mypkg-1.2.3.tar.gz
cd mypkg-1.2.3
tup one-shot-build (or whatever)

This would mean incremental builds don't work at all, so if you need
to rebuild it would involve wiping out the 'mypkg-1.2.3' directory and
unzipping/re-execute one-shot-build. In other words, you couldn't
really use this to do development, but you could use tup as the
baseline installer to build things on the production server without
involving fuse.

>
>> * if yes: can I run tup on the fuse that runs on FreeBSD?
>
> Possibly.

I would like to get it working with fuse if possible for those who
have non-production FreeBSD machines to use it. I am open to
alternatives if fuse can't be made to work, but I think that since
fuse already runs on Linux & OSX it would probably be the path of
least resistance.

>
>> * if no: can I use an older tup of ldpreload vintage on FreeBSD?
>> or more simply
>
> This would be nice.

A few years ago I tried to run tup on FreeBSD (or maybe it was
OpenBSD?) and the ldpreload method worked, at least minimally.
Unfortunately all of the binaries that I would be interested in
running (eg: gcc) were statically linked, which doesn't work with the
ldpreload method. Is it still the case that these tools are statically
linked? If so then this avenue is probably not worth pursuing.

>
>> - can I run tup on FreeBSD?
>
> A more accurate question is, "Can I run tup on a system without
> FUSE?"  At present, the answer is no, however Michael and I have had a
> few off-list discussions about this so I don't imagine this will be
> the case forever, but at present, FUSE is required.

Correct. I don't think it would be too much of a stretch given the
similarities between FreeBSD and Linux/OSX, but neither I nor Sean
have had much time to delve into the details at present :(

-Mike

[encodr]

Thanks for the replies (but I dont understand what is meant by "one-shot builds")

I've made two attempts on FreeBSD.
1 - fuse
Difficulties in the use of mount with MS_BIND, which doesn't exist in FreeBSD. At the shell there is a command mount_nullfs, which might provide the same effect using different technology. Possibly. I can't see how to invoke it from C. Maybe I'll try via exec.
I've got so far as building tup, but it doesn't run properly. After fuse_mount I see that /dev/fuse0 is mounted at .tup/mnt, but that directory is not visible to the program. An attempt to chdir or otherwise refer to the ".tup/mnt" yields an invalid fd or inappropriate action or something like that - I dont have the error message to hand. Also unmount of ".tup/mnt" fails in tup, but works at the command line. Interestingly(?), in tup I *can* unmount if I provide the *absolute* path to ".tup/mnt".
I suspect that unmount now requires the directory name rather than the device name.
I suspect my mount without bind isn't having the desired effect.

I began to feel the hole was getting deeper, so I stopped digging and turned to ldpreload.
I understand entirely why you prefer to pursue fuse, but I hoped there might be fewer mysteries in the older approach.

2 - ldpreload
I started from the last commit of tup before the merge of the fuse branch.
tup can be persuaded to build relatively easily.
e.g. -ldl is not needed, compat files are not needed
Running it is another matter. It took me quite a bit of head scratching and extra debug before I realised that the reason tup never saw a file being created by the compiler was ... gcc is statically linked. Doh!
I fixed that by CC = "g++ -x c" and LD = "g++".
Operations performed by the compiler *are* communicated to tup via tup-ldpreload.so.
But now I have another problem.

The very first compilation of tup by tup fails on the first line of the source file.
i.e. tup/src/access_event/send_event.c which starts with
#include "tup/access_event.h"
The compiler complains that it cant find tup/access_event.h
The compiler has been provided with -I../../../src
Conversely, if I copy the given command line and execute it from shell in the tup/access_event/ directory, then the compilation succeeds.
Strange?
It seems as though the compiler, as invoked by tup, finds the source file effectively at "./send_event.c" but then cant navigate to the related directory containing headers because the "current directory" is somewhere else. As though it is not ".". or is unset. Or something. What could be going on?

I'm still waiting for inspiration about that.

If anyone can cast any light ....

e

> --
> tup-users mailing list
> email: tup-...@googlegroups.com
> unsubscribe: tup-users+...@googlegroups.com
> options: http://groups.google.com/group/tup-users?hl=en

[Sean Chittenden]

>> You can, or at least I know that there's a FUSE port for FreeBSD. I
>> can't use FUSE in production, however.
>
> IIRC in our offline discussions we talked about using a one-shot build
> that would run tup as a single-use shell script rather than doing any
> dependency checking. Would this be sufficient for the production
> installs? Eg you could do:
>
> tar -xvf mypkg-1.2.3.tar.gz
> cd mypkg-1.2.3
> tup one-shot-build (or whatever)

Close. The pipe-dream would be that tup generates a shell script that has a similar interface to autoconf's ./configure script so that there was zero external dependencies when building and deploying.

> This would mean incremental builds don't work at all, so if you need
> to rebuild it would involve wiping out the 'mypkg-1.2.3' directory and
> unzipping/re-execute one-shot-build. In other words, you couldn't
> really use this to do development, but you could use tup as the
> baseline installer to build things on the production server without
> involving fuse.

Right, my gripe is that FUSE is a requirement. I split development between OS-X and FreeBSD. Dtrace support on FreeBSD, along with the other goodies that come with FreeBSD that aren't available on Linux, it's a PITA to use Linux for development, nevermind production.

>>> * if yes: can I run tup on the fuse that runs on FreeBSD?
>>
>> Possibly.
>
> I would like to get it working with fuse if possible for those who
> have non-production FreeBSD machines to use it. I am open to
> alternatives if fuse can't be made to work, but I think that since
> fuse already runs on Linux & OSX it would probably be the path of
> least resistance.

Feh. FUSE being present should enable functionality, but I don't see why it needs to be required. If I omit something in my tupfile, that's my issue for being lazy and relying on Tup's ability to implicitly stitch together the dependencies. FUSE acts as a bandaid for poorly written or incomplete Tupfiles, or as you said, Tupfiles act as a hint.

I could FUSE being required for automatically generating Tupfiles, but if I have an explicitly written/generated Tupfile, I don't want to rely on loading a kernel module just to invoke clang.

>>> * if no: can I use an older tup of ldpreload vintage on FreeBSD?
>>> or more simply
>>
>> This would be nice.
>
> A few years ago I tried to run tup on FreeBSD (or maybe it was
> OpenBSD?) and the ldpreload method worked, at least minimally.
> Unfortunately all of the binaries that I would be interested in
> running (eg: gcc) were statically linked, which doesn't work with the
> ldpreload method. Is it still the case that these tools are statically
> linked? If so then this avenue is probably not worth pursuing.

That's correct and done to ensure consistent ABIs are used when compiling programs.

-sc

--
Sean Chittenden
se...@chittenden.org

[Mike Shal]

On Fri, Jun 22, 2012 at 2:31 PM, encodr <enc...@googlemail.com> wrote:
> Thanks for the replies (but I dont understand what is meant by "one-shot builds")
>
> I've made two attempts on FreeBSD.
> 1 - fuse
> Difficulties in the use of mount with MS_BIND, which doesn't exist in FreeBSD. At the shell there is a command mount_nullfs, which might provide the same effect using different technology. Possibly. I can't see how to invoke it from C. Maybe I'll try via exec.

Are you building tup as root or with suid set? The MS_BIND parameter
is only used for mounting the /dev file-system within the fuse fs to
run processes in a chroot. Even as root or with suid, the command
still needs to explicitly request running in a chroot (with the ^c^
flag), or with the full_deps option enabled. Otherwise it shouldn't be
going through this code path. It might be easier to try to get it
working without this feature first.

> I've got so far as building tup, but it doesn't run properly. After fuse_mount I see that /dev/fuse0 is mounted at .tup/mnt, but that directory is not visible to the program. An attempt to chdir or otherwise refer to the ".tup/mnt" yields an invalid fd or inappropriate action or something like that - I dont have the error message to hand. Also unmount of ".tup/mnt" fails in tup, but works at the command line. Interestingly(?), in tup I *can* unmount if I provide the *absolute* path to ".tup/mnt".
> I suspect that unmount now requires the directory name rather than the device name.
> I suspect my mount without bind isn't having the desired effect.

By "not visible to the program" do you mean the sub-process that tup
runs? (ie: is gcc or whatever failing to access the directory?) By
design, tup blocks out other processes from accessing the fuse
filesystem (such as your user shell, so you can't manually chdir into
it) so that the dependency monitoring isn't affected.

What unmount code did you add to tup to get the fuse_server.c to
build? Right now there is only code for linux (fusermount -u -z
.tup/mnt) and OSX (unmount(TUP_MNT, MNT_FORCE). Presumably one of
these, or something similar, would need to be coded in an #ifdef for
FreeBSD.

>
> I began to feel the hole was getting deeper, so I stopped digging and turned to ldpreload.
> I understand entirely why you prefer to pursue fuse, but I hoped there might be fewer mysteries in the older approach.
>
> 2 - ldpreload
> I started from the last commit of tup before the merge of the fuse branch.
> tup can be persuaded to build relatively easily.
> e.g. -ldl is not needed, compat files are not needed
> Running it is another matter. It took me quite a bit of head scratching and extra debug before I realised that the reason tup never saw a file being created by the compiler was ... gcc is statically linked. Doh!
> I fixed that by CC =  "g++ -x c" and LD = "g++".
> Operations performed by the compiler *are* communicated to tup via tup-ldpreload.so.
> But now I have another problem.

Yes, that was the problem I ran into as well :(. I didn't think to try
g++, though.

>
> The very first compilation of tup by tup fails on the first line of the source file.
> i.e. tup/src/access_event/send_event.c which starts with
> #include "tup/access_event.h"
> The compiler complains that it cant find tup/access_event.h
> The compiler has been provided with -I../../../src
> Conversely, if I copy the given command line and execute it from shell in the tup/access_event/ directory, then the compilation succeeds.
> Strange?
> It seems as though the compiler, as invoked by tup, finds the source file effectively at "./send_event.c" but then cant navigate to the related directory containing headers because the "current directory" is somewhere else. As though it is not ".".  or is unset. Or something. What could be going on?
>
> I'm still waiting for inspiration about that.
>
> If anyone can cast any light ....

Hmm, that's confusing. As I recall, tup hardly did any interfering
with the sub-process when using ldpreload - pretty much everything
passed straight through to the real libc calls. I don't know why that
would be causing the command to fail as you describe.

-Mike

[encodr]

Thanks Mike. Some progress

On 22 Jun 2012, at 20:40, Mike Shal wrote:

> On Fri, Jun 22, 2012 at 2:31 PM, encodr <enc...@googlemail.com> wrote:
>> Thanks for the replies (but I dont understand what is meant by "one-shot builds")
>>
>> I've made two attempts on FreeBSD.
>> 1 - fuse
>> Difficulties in the use of mount with MS_BIND, which doesn't exist in FreeBSD. At the shell there is a command mount_nullfs, which might provide the same effect using different technology. Possibly. I can't see how to invoke it from C. Maybe I'll try via exec.
>
> Are you building tup as root or with suid set? The MS_BIND parameter
> is only used for mounting the /dev file-system within the fuse fs to
> run processes in a chroot. Even as root or with suid, the command
> still needs to explicitly request running in a chroot (with the ^c^
> flag), or with the full_deps option enabled. Otherwise it shouldn't be
> going through this code path. It might be easier to try to get it
> working without this feature first.

I'm not building as root. I'm not aware of having suid set.

>
>> I've got so far as building tup, but it doesn't run properly. After fuse_mount I see that /dev/fuse0 is mounted at .tup/mnt, but that directory is not visible to the program. An attempt to chdir or otherwise refer to the ".tup/mnt" yields an invalid fd or inappropriate action or something like that - I dont have the error message to hand. Also unmount of ".tup/mnt" fails in tup, but works at the command line. Interestingly(?), in tup I *can* unmount if I provide the *absolute* path to ".tup/mnt".
>> I suspect that unmount now requires the directory name rather than the device name.
>> I suspect my mount without bind isn't having the desired effect.
>
> By "not visible to the program" do you mean the sub-process that tup
> runs? (ie: is gcc or whatever failing to access the directory?) By
> design, tup blocks out other processes from accessing the fuse
> filesystem (such as your user shell, so you can't manually chdir into
> it) so that the dependency monitoring isn't affected.

No, I mean that tup itself cant access .tup/mnt

>
> What unmount code did you add to tup to get the fuse_server.c to
> build? Right now there is only code for linux (fusermount -u -z
> .tup/mnt) and OSX (unmount(TUP_MNT, MNT_FORCE). Presumably one of
> these, or something similar, would need to be coded in an #ifdef for
> FreeBSD.

My reading suggested that simply calling unmount() would work. And it does, providing I provide an absolute path to the dir..

Ahem. Mea culpa. Don't call in Mulder and Scully yet.
When I modified Tuprules.tup to add g++ etc, I moved the order of CFLAGS in the !cc macro. Unfortunately I moved it into the ^ ^ section at the beginning, i.e. out of the actual command. Sigh.

Latest result is that the compile works, tup sees many types of file access occurring, but is not seeing the creation of the .o.
Feels like I'm getting close ...

e

[Mike Shal]

On Fri, Jun 22, 2012 at 2:58 PM, Sean Chittenden <se...@chittenden.org> wrote:
>>> You can, or at least I know that there's a FUSE port for FreeBSD. I
>>> can't use FUSE in production, however.
>>
>> IIRC in our offline discussions we talked about using a one-shot build
>> that would run tup as a single-use shell script rather than doing any
>> dependency checking. Would this be sufficient for the production
>> installs? Eg you could do:
>>
>> tar -xvf mypkg-1.2.3.tar.gz
>> cd mypkg-1.2.3
>> tup one-shot-build (or whatever)
>
> Close. The pipe-dream would be that tup generates a shell script that has a similar interface to autoconf's ./configure script so that there was zero external dependencies when building and deploying.

Ahh, ok. I think this would still have to be a separate command (ie:
not generated as part of 'tup upd'), since I think it would
necessitate loading all of the commands from the db. Perhaps 'tup
generate-script foo.sh' could be added as part of whatever process you
have to create a tarball, assuming that's the method of deployment.

>> I would like to get it working with fuse if possible for those who
>> have non-production FreeBSD machines to use it. I am open to
>> alternatives if fuse can't be made to work, but I think that since
>> fuse already runs on Linux & OSX it would probably be the path of
>> least resistance.
>
> Feh. FUSE being present should enable functionality, but I don't see why it needs to be required. If I omit something in my tupfile, that's my issue for being lazy and relying on Tup's ability to implicitly stitch together the dependencies. FUSE acts as a bandaid for poorly written or incomplete Tupfiles, or as you said, Tupfiles act as a hint.

I really don't view it this way. I think automated file-level
dependency detection is a baseline requirement for build systems. Even
if it is possible for a human to correctly list the dependencies for a
single build, it really isn't possible to maintain as files are
added/deleted/moved around. You will make a mistake, and when you do
the build system should fail loudly and tell you of the error.

Again I don't really care if it's fuse or not, but dependency
detection needs to be there in some form. This is one of my primary
gripes with make, and I don't want to take tup in a direction that
relies solely on the developer for this information.

>
> I could FUSE being required for automatically generating Tupfiles, but if I have an explicitly written/generated Tupfile, I don't want to rely on loading a kernel module just to invoke clang.

That is a good point. In a private email you had mentioned kqueue as a
possible alternative - does that also involve a kernel module? Is it
ok to use in a production environment or is it also too troublesome?

>
>>>> * if no: can I use an older tup of ldpreload vintage on FreeBSD?
>>>> or more simply
>>>
>>> This would be nice.
>>
>> A few years ago I tried to run tup on FreeBSD (or maybe it was
>> OpenBSD?) and the ldpreload method worked, at least minimally.
>> Unfortunately all of the binaries that I would be interested in
>> running (eg: gcc) were statically linked, which doesn't work with the
>> ldpreload method. Is it still the case that these tools are statically
>> linked? If so then this avenue is probably not worth pursuing.
>
> That's correct and done to ensure consistent ABIs are used when compiling programs.

Ahh, ok. I thought it might be for recovery so if libc.so got hosed
you could still run gcc to build a new one.

Thanks,
-Mike

[Sean Chittenden]

>> Close. The pipe-dream would be that tup generates a shell script that has a similar interface to autoconf's ./configure script so that there was zero external dependencies when building and deploying.
>
> Ahh, ok. I think this would still have to be a separate command (ie:
> not generated as part of 'tup upd'), since I think it would
> necessitate loading all of the commands from the db. Perhaps 'tup
> generate-script foo.sh' could be added as part of whatever process you
> have to create a tarball, assuming that's the method of deployment.

Or export the contents of the db to something that can be sourced via sh(1). This would be a killer feature.
\

>> Feh. FUSE being present should enable functionality, but I don't see why it needs to be required. If I omit something in my tupfile, that's my issue for being lazy and relying on Tup's ability to implicitly stitch together the dependencies. FUSE acts as a bandaid for poorly written or incomplete Tupfiles, or as you said, Tupfiles act as a hint.
>
> I really don't view it this way. I think automated file-level
> dependency detection is a baseline requirement for build systems. Even
> if it is possible for a human to correctly list the dependencies for a
> single build, it really isn't possible to maintain as files are
> added/deleted/moved around. You will make a mistake, and when you do
> the build system should fail loudly and tell you of the error.

'ya know, UNIX solved the detection of this stuff a *LONG* time ago... it's called atime, and it exists for a reason. :~] I see FUSE as a very heavyweight, micro performance optimization for all scenarios but the largest of large projects (e.g. an entire kernel build). Even building boost of FreeBSD w/ >50K files, it takes 1-2min. Not seconds, granted, but portability trumps performance.

And to be clear, I like tup and the way that it builds files... just not the way that it detects stale dependencies because of the weight of the mechanism that detects this. I'd be *VERY* happy with a series of recursive stat(2) calls when tup is in "development mode."

> Again I don't really care if it's fuse or not, but dependency
> detection needs to be there in some form. This is one of my primary
> gripes with make, and I don't want to take tup in a direction that
> relies solely on the developer for this information.

When doing development, yeah. cmake does this differently in that it parses the source file, but if you touch a header file, it'll pick up the change and recompile all of the dependent .o files. Searching for #include files isn't that hard when parsing something slurped in by cpp(1).

>> I could FUSE being required for automatically generating Tupfiles, but if I have an explicitly written/generated Tupfile, I don't want to rely on loading a kernel module just to invoke clang.
>
> That is a good point. In a private email you had mentioned kqueue as a
> possible alternative - does that also involve a kernel module? Is it
> ok to use in a production environment or is it also too troublesome?

kqueue(2) is a system call, just like epoll(2). Nothing special needs to be done, it just works. Mac has the same API.

>>>>> * if no: can I use an older tup of ldpreload vintage on FreeBSD?
>>>>> or more simply
>>>>
>>>> This would be nice.
>>>
>>> A few years ago I tried to run tup on FreeBSD (or maybe it was
>>> OpenBSD?) and the ldpreload method worked, at least minimally.
>>> Unfortunately all of the binaries that I would be interested in
>>> running (eg: gcc) were statically linked, which doesn't work with the
>>> ldpreload method. Is it still the case that these tools are statically
>>> linked? If so then this avenue is probably not worth pursuing.
>>
>> That's correct and done to ensure consistent ABIs are used when compiling programs.
>
> Ahh, ok. I thought it might be for recovery so if libc.so got hosed
> you could still run gcc to build a new one.

Naw, it's because FreeBSD treats ABI as an immutable property of major release versions. For example, the base compiler doesn't get updated between minor releases, only major releases. It's one of the *very* nice things about FreeBSD.

[Mike Shal]

On Fri, Jun 22, 2012 at 3:55 PM, encodr <enc...@googlemail.com> wrote:
> Thanks Mike. Some progress
>
> On 22 Jun 2012, at 20:40, Mike Shal wrote:
>
>> On Fri, Jun 22, 2012 at 2:31 PM, encodr <enc...@googlemail.com> wrote:
>>> Thanks for the replies (but I dont understand what is meant by "one-shot builds")
>>>
>>> I've made two attempts on FreeBSD.
>>> 1 - fuse
>>> Difficulties in the use of mount with MS_BIND, which doesn't exist in FreeBSD. At the shell there is a command mount_nullfs, which might provide the same effect using different technology. Possibly. I can't see how to invoke it from C. Maybe I'll try via exec.
>>
>> Are you building tup as root or with suid set? The MS_BIND parameter
>> is only used for mounting the /dev file-system within the fuse fs to
>> run processes in a chroot. Even as root or with suid, the command
>> still needs to explicitly request running in a chroot (with the ^c^
>> flag), or with the full_deps option enabled. Otherwise it shouldn't be
>> going through this code path. It might be easier to try to get it
>> working without this feature first.
>
> I'm not building as root. I'm not aware of having suid set.

Is the MS_BIND thing a build issue or run-time error? If it's a build
issue you can just temporarily comment it out for now.

>
>>
>>> I've got so far as building tup, but it doesn't run properly. After fuse_mount I see that /dev/fuse0 is mounted at .tup/mnt, but that directory is not visible to the program. An attempt to chdir or otherwise refer to the ".tup/mnt" yields an invalid fd or inappropriate action or something like that - I dont have the error message to hand. Also unmount of ".tup/mnt" fails in tup, but works at the command line. Interestingly(?), in tup I *can* unmount if I provide the *absolute* path to ".tup/mnt".
>>> I suspect that unmount now requires the directory name rather than the device name.
>>> I suspect my mount without bind isn't having the desired effect.
>>
>> By "not visible to the program" do you mean the sub-process that tup
>> runs? (ie: is gcc or whatever failing to access the directory?) By
>> design, tup blocks out other processes from accessing the fuse
>> filesystem (such as your user shell, so you can't manually chdir into
>> it) so that the dependency monitoring isn't affected.
>
> No, I mean that tup itself cant access .tup/mnt

Hmm, strange. Any idea what version of libfuse this is? I wonder if
tup is doing something that requires a newer version...

>> Hmm, that's confusing. As I recall, tup hardly did any interfering
>> with the sub-process when using ldpreload - pretty much everything
>> passed straight through to the real libc calls. I don't know why that
>> would be causing the command to fail as you describe.
>
> Ahem.  Mea culpa. Don't call in Mulder and Scully yet.
> When I modified Tuprules.tup to add g++ etc, I moved the order of CFLAGS in the !cc macro. Unfortunately I moved it into the ^  ^ section at the beginning, i.e. out of the actual command. Sigh.
>
> Latest result is that the compile works, tup sees many types of file access occurring, but is not seeing the creation of the .o.
> Feels like I'm getting close ...

When trying to debug issues like this with ldpreload, I would find it
helpful by running 'nm' on the binary that is supposed to write the
file (either g++, or maybe 'as' if it's the assembler actually writing
the .o) and look for symbols with the string "open" in them. When the
sub-process calls "open" or "fopen", it may actually map to the symbol
"__open" or "open64", or something else. That's why there's a bunch of
different symbol names in ldpreload.c that all do similar things.
Perhaps FreeBSD just has a slightly different symbol for open that g++
is using and currently isn't wrapped.

-Mike

[encodr]

On 22 Jun 2012, at 21:10, Mike Shal wrote:

> On Fri, Jun 22, 2012 at 3:55 PM, encodr <enc...@googlemail.com> wrote:
>> Thanks Mike. Some progress
>>
>> On 22 Jun 2012, at 20:40, Mike Shal wrote:
>>
>>> On Fri, Jun 22, 2012 at 2:31 PM, encodr <enc...@googlemail.com> wrote:
>>>> Thanks for the replies (but I dont understand what is meant by "one-shot builds")
>>>>
>>>> I've made two attempts on FreeBSD.
>>>> 1 - fuse
>>>> Difficulties in the use of mount with MS_BIND, which doesn't exist in FreeBSD. At the shell there is a command mount_nullfs, which might provide the same effect using different technology. Possibly. I can't see how to invoke it from C. Maybe I'll try via exec.
>>>
>>> Are you building tup as root or with suid set? The MS_BIND parameter
>>> is only used for mounting the /dev file-system within the fuse fs to
>>> run processes in a chroot. Even as root or with suid, the command
>>> still needs to explicitly request running in a chroot (with the ^c^
>>> flag), or with the full_deps option enabled. Otherwise it shouldn't be
>>> going through this code path. It might be easier to try to get it
>>> working without this feature first.
>>
>> I'm not building as root. I'm not aware of having suid set.
>
> Is the MS_BIND thing a build issue or run-time error? If it's a build
> issue you can just temporarily comment it out for now.

Thanks, I'll try that later

>
>>
>>>
>>>> I've got so far as building tup, but it doesn't run properly. After fuse_mount I see that /dev/fuse0 is mounted at .tup/mnt, but that directory is not visible to the program. An attempt to chdir or otherwise refer to the ".tup/mnt" yields an invalid fd or inappropriate action or something like that - I dont have the error message to hand. Also unmount of ".tup/mnt" fails in tup, but works at the command line. Interestingly(?), in tup I *can* unmount if I provide the *absolute* path to ".tup/mnt".
>>>> I suspect that unmount now requires the directory name rather than the device name.
>>>> I suspect my mount without bind isn't having the desired effect.
>>>
>>> By "not visible to the program" do you mean the sub-process that tup
>>> runs? (ie: is gcc or whatever failing to access the directory?) By
>>> design, tup blocks out other processes from accessing the fuse
>>> filesystem (such as your user shell, so you can't manually chdir into
>>> it) so that the dependency monitoring isn't affected.
>>
>> No, I mean that tup itself cant access .tup/mnt
>
> Hmm, strange. Any idea what version of libfuse this is? I wonder if
> tup is doing something that requires a newer version...

fusefs-libs 2.7.4

>
>>> Hmm, that's confusing. As I recall, tup hardly did any interfering
>>> with the sub-process when using ldpreload - pretty much everything
>>> passed straight through to the real libc calls. I don't know why that
>>> would be causing the command to fail as you describe.
>>
>> Ahem. Mea culpa. Don't call in Mulder and Scully yet.
>> When I modified Tuprules.tup to add g++ etc, I moved the order of CFLAGS in the !cc macro. Unfortunately I moved it into the ^ ^ section at the beginning, i.e. out of the actual command. Sigh.
>>
>> Latest result is that the compile works, tup sees many types of file access occurring, but is not seeing the creation of the .o.
>> Feels like I'm getting close ...
>
> When trying to debug issues like this with ldpreload, I would find it
> helpful by running 'nm' on the binary that is supposed to write the
> file (either g++, or maybe 'as' if it's the assembler actually writing
> the .o) and look for symbols with the string "open" in them. When the
> sub-process calls "open" or "fopen", it may actually map to the symbol
> "__open" or "open64", or something else. That's why there's a bunch of
> different symbol names in ldpreload.c that all do similar things.
> Perhaps FreeBSD just has a slightly different symbol for open that g++
> is using and currently isn't wrapped.
>
> -Mike

If only, Unfortunately, the inability to detect open on *.o is because g++ uses 'as' and that is - again - statically linked.

Then I had a thought - what about clang?
Is it installed - yes.
Is it dynamically linked - yes
does it work -- YES

... until it tries to link something. Then it wants to use /usr/bin/ld - sigh - statically linked again.

With clang tup "works" in the sense of detecting which files need to get created. It creates all the .o.
Change a .c, the correct .o gets created.
But it can't build a program or a library.

So, Tup *can* work on FreeBSD 9.0 with ldpreload (I'm using GhostBSD)
BUT if I want to get any further I suspect I'll have to rebuild the toolchain, or at least 'ld', in dynamic form.
Not a "production" approach, I suspect.

I might pursue this, since I'm not at the moment concerned about "production" on BSD.
Or - maybe I'll have another look at fuse ...

e

[Mike Shal]

On Fri, Jun 22, 2012 at 4:09 PM, Sean Chittenden <se...@chittenden.org> wrote:
>>> Close. The pipe-dream would be that tup generates a shell script that has a similar interface to autoconf's ./configure script so that there was zero external dependencies when building and deploying.
>>
>> Ahh, ok. I think this would still have to be a separate command (ie:
>> not generated as part of 'tup upd'), since I think it would
>> necessitate loading all of the commands from the db. Perhaps 'tup
>> generate-script foo.sh' could be added as part of whatever process you
>> have to create a tarball, assuming that's the method of deployment.
>
> Or export the contents of the db to something that can be sourced via sh(1). This would be a killer feature.
> \

Isn't this what "foo.sh" would presumably be in this case? I was
thinking if you had a Tupfile like this:

: foreach *.c |> gcc -c %f -o %o |> %B.o
: *.o |> gcc %f -o %o |> program

Then 'tup generate-script foo.sh' would create:

foo.sh:
#! /bin/sh
gcc -c foo.c -o foo.o
gcc -c bar.c -o bar.o
gcc foo.o bar.o -o program

Is that what you're thinking or am I misunderstanding?

>
>>> Feh. FUSE being present should enable functionality, but I don't see why it needs to be required. If I omit something in my tupfile, that's my issue for being lazy and relying on Tup's ability to implicitly stitch together the dependencies. FUSE acts as a bandaid for poorly written or incomplete Tupfiles, or as you said, Tupfiles act as a hint.
>>
>> I really don't view it this way. I think automated file-level
>> dependency detection is a baseline requirement for build systems. Even
>> if it is possible for a human to correctly list the dependencies for a
>> single build, it really isn't possible to maintain as files are
>> added/deleted/moved around. You will make a mistake, and when you do
>> the build system should fail loudly and tell you of the error.
>
> 'ya know, UNIX solved the detection of this stuff a *LONG* time ago... it's called atime, and it exists for a reason. :~] I see FUSE as a very heavyweight, micro performance optimization for all scenarios but the largest of large projects (e.g. an entire kernel build). Even building boost of FreeBSD w/ >50K files, it takes 1-2min. Not seconds, granted, but portability trumps performance.
>
> And to be clear, I like tup and the way that it builds files... just not the way that it detects stale dependencies because of the weight of the mechanism that detects this. I'd be *VERY* happy with a series of recursive stat(2) calls when tup is in "development mode."

I don't see how atime would really solve this -- it seems more geared
towards knowing if a specific file has been read or not. Tup is
interested in knowing what files a particular process has accessed,
not that a particular file has been accessed. To use it in tup it
would have to scan the whole tree after each job, and everything would
have to run serially. You'd also have to avoid opening any files while
the build was going, otherwise it would start adding spurious
dependencies.

>
>
>> Again I don't really care if it's fuse or not, but dependency
>> detection needs to be there in some form. This is one of my primary
>> gripes with make, and I don't want to take tup in a direction that
>> relies solely on the developer for this information.
>
> When doing development, yeah. cmake does this differently in that it parses the source file, but if you touch a header file, it'll pick up the change and recompile all of the dependent .o files. Searching for #include files isn't that hard when parsing something slurped in by cpp(1).

Yeah, but that is limited to C/C++ files. I think it is better to do
this generically, so you don't have to implement a dependency parser
for every application in the build system.

>
>
>
>>> I could FUSE being required for automatically generating Tupfiles, but if I have an explicitly written/generated Tupfile, I don't want to rely on loading a kernel module just to invoke clang.
>>
>> That is a good point. In a private email you had mentioned kqueue as a
>> possible alternative - does that also involve a kernel module? Is it
>> ok to use in a production environment or is it also too troublesome?
>
> kqueue(2) is a system call, just like epoll(2). Nothing special needs to be done, it just works. Mac has the same API.

Oops, just responded to this part in the private email chain. In
summary; I tried it out on a Mac, but haven't figured out how to get
it to work for tup yet :(

-Mike

[Mike Shal]

Doh :). I am also starting to setup a FreeBSD vm so hopefully I can
help out with trying to get fuse working there soon (maybe later this
week?). If you are able to give it a try before then and make any
progress please let me know!

-Mike

[Sean Chittenden]

>> Or export the contents of the db to something that can be sourced via sh(1). This would be a killer feature.
>
> Isn't this what "foo.sh" would presumably be in this case? I was
> thinking if you had a Tupfile like this:
>
> : foreach *.c |> gcc -c %f -o %o |> %B.o
> : *.o |> gcc %f -o %o |> program
>
> Then 'tup generate-script foo.sh' would create:
>
> foo.sh:
> #! /bin/sh
> gcc -c foo.c -o foo.o
> gcc -c bar.c -o bar.o
> gcc foo.o bar.o -o program
>
> Is that what you're thinking or am I misunderstanding?

That'd work. A slightly more flushed out example:

# Generate a script that can detect platform specific items
`tup generate-configure configure`

configure does the "does this platform have kqueue or epoll" type of work and, ideally, would produce another shell script that could be sourced as config.sh.

# Generates the build sequence
`tup generate-build build.sh`

Then the "all" target for Makefile would be:

all:
sh build.sh

And build.sh would source config.sh for things such as $CC and whether or not individual files should be built. This would be much nicer than make. build.sh:

#!/bin/sh

. config.sh

$CC -c -o foo.o foo.c
RET=$?
if [ $RET != 0 ]; then
echo "Building foo.c failed"
exit 1
else
OBJS="$OBJS foo.o"
fi

$CC -c -o bar.o foo.c
RET=$?
if [ $RET != 0 ]; then
echo "Building bar.c failed"
exit 1
else
OBJS="$OBJS bar.o"
fi

if [ x"$USE_KQUEUE" = x1 ]; then
$CC -c -o kqueue.o kqueue.c
if [ $RET != 0 ]; then
echo "Building kqueue.c failed"
exit 1
fi
OBJS="$OBJS kqueue.o"
fi

$LD $OBJS -o program${EXE}
RET=$?
...


Something like that. Sane and hackable with all of the dependencies very easily traceable.

>> And to be clear, I like tup and the way that it builds files... just not the way that it detects stale dependencies because of the weight of the mechanism that detects this. I'd be *VERY* happy with a series of recursive stat(2) calls when tup is in "development mode."
>
> I don't see how atime would really solve this -- it seems more geared
> towards knowing if a specific file has been read or not. Tup is
> interested in knowing what files a particular process has accessed,
> not that a particular file has been accessed. To use it in tup it
> would have to scan the whole tree after each job, and everything would
> have to run serially. You'd also have to avoid opening any files while
> the build was going, otherwise it would start adding spurious
> dependencies.

For doing normal builds, not a big deal. But if you need to "rescan to regenerate build dependencies" then you would have to do things serially. Most recompiles don't require that.

>> When doing development, yeah. cmake does this differently in that it parses the source file, but if you touch a header file, it'll pick up the change and recompile all of the dependent .o files. Searching for #include files isn't that hard when parsing something slurped in by cpp(1).
>
> Yeah, but that is limited to C/C++ files. I think it is better to do
> this generically, so you don't have to implement a dependency parser
> for every application in the build system.

*nods* I dabble with D, so I understand this.

>>>> I could FUSE being required for automatically generating Tupfiles, but if I have an explicitly written/generated Tupfile, I don't want to rely on loading a kernel module just to invoke clang.
>>>
>>> That is a good point. In a private email you had mentioned kqueue as a
>>> possible alternative - does that also involve a kernel module? Is it
>>> ok to use in a production environment or is it also too troublesome?
>>
>> kqueue(2) is a system call, just like epoll(2). Nothing special needs to be done, it just works. Mac has the same API.
>
> Oops, just responded to this part in the private email chain. In
> summary; I tried it out on a Mac, but haven't figured out how to get
> it to work for tup yet :(

*nods* -sc



--
Sean Chittenden
se...@chittenden.org

[encodr]

On 26 Jun 2012, at 03:09, Mike Shal wrote:

>> <snip>

>> So, Tup *can* work on FreeBSD 9.0 with ldpreload (I'm using GhostBSD)
>> BUT if I want to get any further I suspect I'll have to rebuild the toolchain, or at least 'ld', in dynamic form.
>> Not a "production" approach, I suspect.
>>
>> I might pursue this, since I'm not at the moment concerned about "production" on BSD.
>> Or - maybe I'll have another look at fuse ...
>
> Doh :). I am also starting to setup a FreeBSD vm so hopefully I can
> help out with trying to get fuse working there soon (maybe later this

> week?). If you are able to give it a try before then and make any(@

> progress please let me know!

This bsd comes with gcc 4.2.1, statically linked.
I installed binutils and gcc from ports. This delivers gcc 4.6.3 dynamically linked.
With small config mods Tup builds itself with the dynamically linked toolchain.
(without monitor - that requires inotify, not present in BSD).

Returning to fuse:
fuse_mount(TUP_MNT, &args) reports success, and
if I exit the program after this, then I see that /dev/fuse0 is mounted on /some/where/.tup/mnt

However, when I let the program continue normally, then
virt_tup_open()
fails to
openat(top_top_fd(), TUP_MNT, O_RDONLY),
reporting
./tup/mnt: Invalid argument

my config in build.sh looks like
FreeBSD)
plat_files="$plat_files ../src/compat/dir_mutex.c "
plat_files="$plat_files ../src/compat/clearenv.c "
plat_files="$plat_files ../src/compat/utimensat.c"
plat_cflags="$plat_cflags -include ../src/compat/freebsd.h"
plat_cflags="$plat_cflags -DS_IFDIR=0040000"
plat_cflags="$plat_cflags -DAT_SYMLINK_NOFOLLOW=0x100"
plat_cflags="$plat_cflags -DAT_REMOVEDIR=0x200"
plat_cflags="$plat_cflags -Dd_ino=d_fileno"
which, of course, might be nonsense.

e

[Mike Shal]

This was failing because FreeBSD doesn't accept AT_SYMLINK_NOFOLLOW in
faccessat(), which is what was ultimately returning the "Invalid
argument".

>
> my config in build.sh looks like
>      FreeBSD)
>        plat_files="$plat_files ../src/compat/dir_mutex.c "
>        plat_files="$plat_files ../src/compat/clearenv.c "
>        plat_files="$plat_files ../src/compat/utimensat.c"
>        plat_cflags="$plat_cflags -include ../src/compat/freebsd.h"
>        plat_cflags="$plat_cflags -DS_IFDIR=0040000"
>        plat_cflags="$plat_cflags -DAT_SYMLINK_NOFOLLOW=0x100"
>        plat_cflags="$plat_cflags -DAT_REMOVEDIR=0x200"
>        plat_cflags="$plat_cflags -Dd_ino=d_fileno"
> which, of course, might be nonsense.

I used a slightly different version here - I used the linux wrapper
for utimensat since FreeBSD has futimesat, so we don't need the
dir_mutex stuff. Also the S_IFDIR and such flags were missing because
I was defining POSIX_SOURCE for some reason, but that no longer is
necessary.

I pushed a few commits to master - the bootstrapped tup builds and can
pass some test cases, but it can't yet build tup itself. Here's what's
left as far as I can tell:

1) ar prints "ar: warning: can't mmap file: bin.o: Operation not
supported by device" and no objects are in the archive, though the
command returns successfully for some reason. Not sure if this is a
tup issue or a fuse issue
2) linking tup fails with "fatal: Unable to read current working
directory: No such file or directory". Removing the 'git describe'
command (used to get the version info) from the linker line makes this
go away, and then the linker command fails with a bunch of unresolved
symbols because libtup.a is empty (due to the above issue)
3) there are innocuous "umount: /dev/fuse0: unknown file system"
errors after tup quits (the file-system is successfully unmounted,
though)
4) tests randomly fail with ".tup/mnt: Operation not supported by
device". This happens quite often in some tests - eg: t2066 worked
3/10 tries when I tried to run it a bunch in a row. Seems like a race
condition, but I dunno where that would be yet.

Anyway, it's getting a bit closer. If you have any ideas for these
issues, let me know. I'll give it another try tomorrow.

-Mike

[Mike Shal]

On Mon, Jun 25, 2012 at 10:30 PM, Sean Chittenden <se...@chittenden.org> wrote:
>>> Or export the contents of the db to something that can be sourced via sh(1). This would be a killer feature.
>>
>> Isn't this what "foo.sh" would presumably be in this case? I was
>> thinking if you had a Tupfile like this:
>>
>> : foreach *.c |> gcc -c %f -o %o |> %B.o
>> : *.o |> gcc %f -o %o |> program
>>
>> Then 'tup generate-script foo.sh' would create:
>>
>> foo.sh:
>> #! /bin/sh
>> gcc -c foo.c -o foo.o
>> gcc -c bar.c -o bar.o
>> gcc foo.o bar.o -o program
>>
>> Is that what you're thinking or am I misunderstanding?
>
> That'd work. A slightly more flushed out example:
>
> # Generate a script that can detect platform specific items
> `tup generate-configure configure`
>
> configure does the "does this platform have kqueue or epoll" type of work and, ideally, would produce another shell script that could be sourced as config.sh.

I think you'd still want to use autoconf (or whatever) to do this kind
of introspective configuration. I don't plan to integrate that sort of
thing with tup.

Hmm this is a bit more complicated than I was initially thinking. How
would tup know to generate the configurable bits like checking
USE_KQUEUE? After parsing a Tupfile, tup just stores the fully
expanded command string in the database, and adds links to the inputs
& outputs. The if-statements and $-variables in the Tupfile (which
aren't the same as the environment variables in a shell script) are
only stored temporarily in memory as the Tupfile is read in. If you

had a Tupfile like this:

ifeq ($(USE_KQUEUE),1)
: kqueue.c |> gcc ... |> kqueue.o
endif

and USE_KQUEUE was 0, then after the Tupfile is parsed there is no
knowledge that this rule exists at all in tup, so it would be quite
difficult to generate that statement in the shell script.

It might be easier to still run tup on the production system, but have
it load fuse when it is used in a development capacity at runtime (eg:
'tup upd' will dlopen libfuse.so). Another mode could be introduced
that does a single build by parsing all the Tupfiles and executing the
shell commands, rather than generating a separate shell script.
Essentially, this would just be parsing the Tupfiles as if they were a
less feature-ful version of sh. This would mean you'd still have to
install tup on the production machine to build the package, but
without fuse it would be a single-shot build. Does that sound
reasonable at all?

-Mike

[Mike Shal]

On Thu, Jun 28, 2012 at 10:14 PM, Mike Shal <mar...@gmail.com> wrote:
> I pushed a few commits to master - the bootstrapped tup builds and can
> pass some test cases, but it can't yet build tup itself. Here's what's
> left as far as I can tell:
>
> 1) ar prints "ar: warning: can't mmap file: bin.o: Operation not
> supported by device" and no objects are in the archive, though the
> command returns successfully for some reason. Not sure if this is a
> tup issue or a fuse issue
> 2) linking tup fails with "fatal: Unable to read current working
> directory: No such file or directory". Removing the 'git describe'
> command (used to get the version info) from the linker line makes this
> go away, and then the linker command fails with a bunch of unresolved
> symbols because libtup.a is empty (due to the above issue)
> 3) there are innocuous "umount: /dev/fuse0: unknown file system"
> errors after tup quits (the file-system is successfully unmounted,
> though)
> 4) tests randomly fail with ".tup/mnt: Operation not supported by
> device". This happens quite often in some tests - eg: t2066 worked
> 3/10 tries when I tried to run it a bunch in a row. Seems like a race
> condition, but I dunno where that would be yet.

Well I was able to fix part 4) - it was a race condition between
starting the fuse thread and trying to open the first file. It seems
after the first failure, the fs runs correctly. For some reason the fs
still isn't ready even after the init() callback, so in FreeBSD I just
have it ignore the first ENODEV error. I think number 1) is the only
major issue at the moment, the others are just kinda annoying...

-Mike

[encodr]

I replaced 'ar' with /usr/local/bin/ar, referencing the more recent command I acquired by installing binutils from ports.
It does not suffer from the mmap issues (or does not complain)

Now link fails due to some missing entries - I think I haven't got the compat files correct when built by tup.
Hopefully easy to fix (but that's all for today)

e

> 2) linking tup fails with "fatal: Unable to read current working
> directory: No such file or directory". Removing the 'git describe'
> command (used to get the version info) from the linker line makes this
> go away, and then the linker command fails with a bunch of unresolved
> symbols because libtup.a is empty (due to the above issue)
> 3) there are innocuous "umount: /dev/fuse0: unknown file system"
> errors after tup quits (the file-system is successfully unmounted,
> though)
> 4) tests randomly fail with ".tup/mnt: Operation not supported by
> device". This happens quite often in some tests - eg: t2066 worked
> 3/10 tries when I tried to run it a bunch in a row. Seems like a race
> condition, but I dunno where that would be yet.
>
> Anyway, it's getting a bit closer. If you have any ideas for these
> issues, let me know. I'll give it another try tomorrow.
>
> -Mike
>

[encodr]

An update of sorts:

I started again from your commit.
Added a new freebsd section to src/compat/Tupfile.
Commented out the git version command.
Set my PATH to /usr/local/bin so that my gcc46 etc get used.

Tup builds itself ok, but fails on some tests.
I haven't run them all yet: some cause the system to hang, so that kinda slows things down.
Failed:
2099, 2101, 2102, 2105, 2106, 2116, 2117, 2119, 2123, 3041, 4024

System Hang:
4020, 4021, 4026, and one of those marked Fail above - I forget which . :-(

for tests that pass, the only complaint is at the end:

umount: /dev/fuse0: unknown file system

but everything looks ok. No /dev/fuse is left mounted.

e

[Mike Shal]

On Sat, Jun 30, 2012 at 1:34 PM, encodr <enc...@googlemail.com> wrote:
> I started again from your commit.
> Added a new freebsd section to src/compat/Tupfile.

Yeah I realized this was missing after I pushed out the last round of
commits. I'll have it updated once I get power back at my house...

> Commented out the git version command.
> Set my PATH to /usr/local/bin so that my gcc46 etc get used.
>
> Tup builds itself ok, but fails on some tests.
> I haven't run them all yet: some cause the system to hang, so that kinda slows things down.
> Failed:
> 2099, 2101, 2102, 2105, 2106, 2116, 2117, 2119, 2123, 3041, 4024

I think some of these (at least the 2* tests) were failing because
something in FreeBSD is doing extra readdirs() that aren't present on
other systems, and that throws off the run-scripts. I haven't had time
yet to narrow it down to figure out what exactly is doing the readdir,
and whether it can be avoided.

>
> System Hang:
> 4020, 4021, 4026, and one of those marked Fail above - I forget which . :-(

Hmm, this is strange. The only one I found that was hanging was t4027,
which I have already disabled in FreeBSD (it is also disabled on OSX).
I don't recall if these other ones you list failed for me, but they
weren't hanging. I don't think there is anything particularly crazy in
these that other tests don't also do. Do they always hang or is it
random? Maybe a previous hang was locking something (ie: was /dev/fuse
still mounted in another test case? Perhaps that would be an issue).

>
> for tests that pass, the only complaint is at the end:
> umount: /dev/fuse0: unknown file system
> but everything looks ok. No /dev/fuse is left mounted.

I'm not sure what that's about yet but I see it too. I figure it isn't
causing a problem so it can be dismissed as an innocuous warning for
the moment until the rest of it works :)

Thanks for trying it out,
-Mike

[encodr]

?-{ :-( ;-/

On 1 Jul 2012, at 01:10, Mike Shal wrote:

> snip

>>
>> System Hang:
>> 4020, 4021, 4026, and one of those marked Fail above - I forget which . :-(
>
> Hmm, this is strange. The only one I found that was hanging was t4027,
> which I have already disabled in FreeBSD (it is also disabled on OSX).
> I don't recall if these other ones you list failed for me, but they
> weren't hanging. I don't think there is anything particularly crazy in
> these that other tests don't also do. Do they always hang or is it
> random? Maybe a previous hang was locking something (ie: was /dev/fuse
> still mounted in another test case? Perhaps that would be an issue).

I modified the t4020 test script so it creates the test files but does not invoke tup.
Then run it, to create the tuptesttmp directory.
Then installed fusefs-fusexmp_fh from ports. This provides an example fuse server that mounts / on a nominated directory and then reports access to it.
Fire that up and cd via the fuse-mediated directory to the waiting tuptesttmp-t4020-open-rdwr.sh directory.
In this instance, tup is not involved:

$ ./prog.exe
system hangs
(this is in a vm: from the host point of view, the vm is maxing out 1 core)

Worse, on reboot there is a significant amount of filesystem damage, and not just "local". e.g. all the icons in the gnome menus have gone.

( I had previously noticed, after other tests had hung, the reboot sometimes had to repair more than a few blocks.)

I suppose this might be an issue with freebsd on virtualbox, but I'm inclined to suspect bugs in the freebsd fuse.

Next time I'll make a snapshot before running fuse. Now I have to reinstall the system, I think. :-(

e

[Mike Shal]

I am also running FreeBSD in a VirtualBox VM (on an OSX host, which I
highly disrecommend unless you like kernel panics). My FreeBSD version
is 9.0-RELEASE, and I have the same fusefs-libs-2.7.4 that you have.
My fusefs-kmod shows:

DISTVERSION= 0.3.9-pre1.20080208
PORTREVISION= 8

(From the Makefile - is that the appropriate way to get the version?).
Is that the same as yours? Or, maybe I am just lucky? Or something
else is the culprit?

I was able to build a newer version of the fuse libs (2.8.6) without
any modification, though I didn't notice any difference in tup. Maybe
that is worth trying out.

>
> Next time I'll make a snapshot before running fuse. Now I have to reinstall the system, I think. :-(

Sorry :(. Perhaps I should point out the "NO WARRANTY" clause in the
license and run away...

-Mike

[encodr]

;-)

I have been using a recent GhostBSD (FreeBSD 9) in VirtualBox.
I re-installed it on two other types of VM - in one the networking was unreliable, and in the other the panic was precisely the same as in VirtualBox. So I guess that rules out the VM as the problem.
I theorise: that leaves fuse or the kernel.

I installed PCBSD 9 in VirtualBox - which is a whole other sort of grief, damn thing (I think, the KDE desktop) keeps pausing when it becomes "idle" regardless of power management settings - BUT the kernel+fuse implementation may be better, because the "hanging" tests no longer hang.

Perhaps the guys from GhostBSD have built their own kernel and made some small mistake.

Now: the following fail for one reason or another:

t2099-run9.sh
t2102-include-rules-different-depth.sh
t2105-node-var-runscr.sh
t2106-node-var-runscr2.sh
t2116-node-var-overwrite-with-normal-var.sh
t2117-node-var-overwrite-append-with-normal-var.sh
t2119-normal-var-overwrite-with-node-var.sh
t2123-node-var-generated-file-from-include.sh
t4024-program-chdir2.sh
t4040-statfs.sh
t5023-ghost-lib.sh
t5059-ar.sh
t5069-transitive-dep.sh
t5073-double-failure.sh
t6023-ldpreload-fopen.sh
t6052-broken-update31.sh
t8005-variant2.sh
t8029-variant-generated-overwrite.sh
t8030-variant-generated-overwrite2.sh
t8031-variant-generated-overwrite3.sh
t8067-node-var-non-generated-file.sh
t8068-node-var-generated-file-from-include.sh

I don't know how many of these are "significant" one is due to attempt to get version from git (as you had to remove from the attempt to build tup itself) ; several others complain of the mount directory name being too long, some are failure of mmap.
I can let you have the error output and/or the contents of the tuptesttmp* directories if you like, perhaps by direct mail instead of cluttering up the mail group.
Or do you already have exactly this list?

Reminder - this is all using gcc46 and recent binutils, dynamically linked, rather than the default gcc,ar, etc in /usr/bin

On the bright side - rather a lot if it is working! :-)

e

[Mike Shal]

On Fri, Jul 6, 2012 at 5:14 PM, encodr <enc...@googlemail.com> wrote:
> I have been using a recent GhostBSD (FreeBSD 9) in VirtualBox.
> I re-installed it on two other types of VM - in one the networking was unreliable, and in the other the panic was precisely the same as in VirtualBox. So I guess that rules out the VM as the problem.
> I theorise: that leaves fuse or the kernel.
>
> I installed PCBSD 9 in VirtualBox - which is a whole other sort of grief, damn thing (I think, the KDE desktop) keeps pausing when it becomes "idle" regardless of power management settings - BUT the kernel+fuse implementation may be better, because the "hanging" tests no longer hang.
>
> Perhaps the guys from GhostBSD have built their own kernel and made some small mistake.

For the kind of issues you described I would suspect a problem in the
fuse kernel part, rather than the user space library. Is there a
difference in the fusefs-kmod between GhostBSD and PCBSD?

Also I'm not really familiar with all the FreeBSD variants - the one I
am using is from here:
ftp://ftp.freebsd.org/pub/FreeBSD/releases/i386/i386/ISO-IMAGES/9.0/
(the dvd iso). Is that PCBSD, or something else?

>
> Now: the following fail for one reason or another:
>
> t2099-run9.sh
> t2102-include-rules-different-depth.sh
> t2105-node-var-runscr.sh
> t2106-node-var-runscr2.sh
> t2116-node-var-overwrite-with-normal-var.sh
> t2117-node-var-overwrite-append-with-normal-var.sh
> t2119-normal-var-overwrite-with-node-var.sh
> t2123-node-var-generated-file-from-include.sh
> t4024-program-chdir2.sh
> t4040-statfs.sh
> t5023-ghost-lib.sh
> t5059-ar.sh
> t5069-transitive-dep.sh
> t5073-double-failure.sh
> t6023-ldpreload-fopen.sh
> t6052-broken-update31.sh
> t8005-variant2.sh
> t8029-variant-generated-overwrite.sh
> t8030-variant-generated-overwrite2.sh
> t8031-variant-generated-overwrite3.sh
> t8067-node-var-non-generated-file.sh
> t8068-node-var-generated-file-from-include.sh

I have a pretty similar list:
t2099-run9
t2101-run10
t2105-node-var-runscr
t2106-node-var-runscr2
t2117-node-var-overwrite-append-with-normal-var
t2123-node-var-generated-file-from-include
t4015-link-order
t4024-program-chdir2
t4031-chown
t4040-statfs
t5023-ghost-lib
t5059-ar
t5069-transitive-dep
t5073-double-failure
t6023-ldpreload-fopen
t6052-broken-update31
t8005-variant2
t8068-node-var-generated-file-from-include
t9000-client-var
t9001-client-var-change
t9004-client-var-change-subdir
t9005-variant-client

I don't know yet why we would have some slight differences, though :).
I think around 4-5 were failing because of the ar mmap thing, and I
need to dig into the rest.

>
> I don't know how many of these are "significant" one is due to attempt to get version from git (as you had to remove from the attempt to build tup itself) ; several others complain of the mount directory name being too long, some are failure of mmap.
> I can let you have the error output and/or the contents of the tuptesttmp* directories if you like, perhaps by direct mail instead of cluttering up the mail group.
> Or do you already have exactly this list?

I will try to make some progress on my list since I can (obviously)
reproduce them. Then we can see if you still have others that are
failing which work for me.

>
> Reminder - this is all using gcc46 and recent binutils, dynamically linked, rather than the default gcc,ar, etc in /usr/bin
>
> On the bright side - rather a lot if it is working! :-)

Okie - thanks for the help!

-Mike