Support for Go (golang)

[Bryan Matsuo]

I'm new to fabricate (enjoying it so far). But I am having trouble integrating it into a project I have that uses Go. I was wondering if there was something I am missing.

The problem I'm facing is that Go projects are typically compiled by running  the command "go build" which performs its own dependency analysis to speed up compilation. The effect of this is that fabricate.run() cannot properly detect Go program dependencies and it can prevent the Go compiler from rebuilding binaries in some cases when it is required.

The only built-in workaround I've seen for this is to invoke "go build" using the AlwaysRunner class or by calling frabicate.shell(). But both methods seems to have the disadvantage that fabricate.autoclean() will not remove the compiled outputs.

Have others had experience using fabricate with Go or a similar tool which has undetectable dependencies? Is there a workaround that I'm missing?

It seems like what I want is a kind of "force run" keyword argument to fabricate.run() that will run the command even if its dependencies have not changed.

[Simon Alford]

What platform are you using fabricate on?

If you are using Linux and I were to guess why fabricate is not working correctly for "go build" it would be because "go build" is using some system calls that fabricate does not yet recognise as file operations. Therefore it cannot workout what the dependencies are. If you can get an strace output of "go build" it may give us a clue as to what is wrong. Have a look in the source of farbicate.py if you want to know the options given to strace.

Simon.

--
You received this message because you are subscribed to the Google Groups "fabricate users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fabricate-use...@googlegroups.com.
To post to this group, send email to fabrica...@googlegroups.com.
Visit this group at http://groups.google.com/group/fabricate-users.
For more options, visit https://groups.google.com/d/optout.

[Lex Trotman]

If `go build` checks dependencies and doesn't re-build them each time
its run then it may not be actually accessing the dependency source
every time the `go build` command is run, so fabricate won't see the
source as a dependency or the product of that source as an output. So
the next time it won't notice that source has changed and so your
command won't get run.

This is supported by your note that the clean does not know to remove
products of the dependencies, if they were not touched last time the
`go build` ran fabricate doesn't know about them any more.

Not sure how to cure that one.

Cheers
Lex

[Simon Alford]

HI again,

 

I believe what Lex is saying is true for the ATimesRunner, as the files may not be accesses by the 'go build' process if they have not changed. The StraceRunner tracks the 'stat' system call, this means any file that even just has its modification time checked becomes a dependency.

ATimesRunner is normally used on Windows (if atimes are enabled), where as StraceRunner is normally used on Linux (if strace is installed). 

Let us know your platform, and we may be able to help further.

Simon.

[Bryan Matsuo]

Thanks for the thoughtful responses. It actually seems like the problem was in my understanding of fabricate. And that Go was just a red herring. Indeed, Go should (at a minimum) stat the source files every time it compiles. Sorry to misdirect you.

From inspecting the .deps file it appears that fabricate only tracks system calls on files located in the working directory or below. And in the case I was dealing with the Go program had a dependency in the same repository but not under the directory that contained the build script. Adding the flag "-d/path/to/repository" to the build script invocation fixed the problem and run('go', 'build') worked as expected.

[Lex Trotman]

Simon,

Following a similar issue in another context it made me think some
more, and I think there is still a potential problem. Yes, if go stats
the inputs they will be recorded, but if it does not touch the output
it will not be recorded for the command, or if go stats the output
then it will be recorded as an input.

If the output is not correctly recorded then clean won't work.

Cheers
Lex

[Simon Alford]

I think you are correct. The second run of the command would would likely change most the outputs to inputs in the .deps file. Assuming only a few files are rebuilt.

To fix that it may require some analysis of the old dependencies when a command is re-run. Original outputs should remain outputs even if they are just listed as inputs on the re-run.

Thinking about it there may be other cases where this is a problem. E.g. Re-run of command due to deps change produces an output file with a different name. The new dependencies will not include the original output file (which is correct) but the original output will not get cleaned. That may need a new type in the deps file 'old-output', which will get cleaned but not cause a re-run if it is changed or removed.

Simon.

[Bryan Matsuo]

Thinking about it there may be other cases where this is a problem. E.g. Re-run of command due to deps change produces an output file with a different name

The issue you mention here happens to be affecting a different aspect of our current build setup. It hasn't been unbearable. But the potential 'old-output' field you mention does sound like it would improve the experience.