"Function not implemented" build error
Anatol Pomozov
I am trying to build tup tool using ./bootstrap.sh and it fails. My
system is ubuntu 10.04 TLS/gcc 4.4.3
anatol:tup $ ./bootstrap.sh
mkdir build
cd build
bootstrap CC (unoptimized) ../src/linux/rbtree.c
bootstrap CC (unoptimized) ../src/tup/bin.c
bootstrap CC (unoptimized) ../src/tup/config.c
bootstrap CC (unoptimized) ../src/tup/create_name_file.c
bootstrap CC (unoptimized) ../src/tup/db.c
bootstrap CC (unoptimized) ../src/tup/db_util.c
bootstrap CC (unoptimized) ../src/tup/debug.c
bootstrap CC (unoptimized) ../src/tup/delete_name_file.c
bootstrap CC (unoptimized) ../src/tup/dircache.c
bootstrap CC (unoptimized) ../src/tup/entry.c
bootstrap CC (unoptimized) ../src/tup/file.c
bootstrap CC (unoptimized) ../src/tup/fslurp.c
bootstrap CC (unoptimized) ../src/tup/getexecwd.c
bootstrap CC (unoptimized) ../src/tup/graph.c
bootstrap CC (unoptimized) ../src/tup/init.c
bootstrap CC (unoptimized) ../src/tup/lock.c
bootstrap CC (unoptimized) ../src/tup/parser.c
bootstrap CC (unoptimized) ../src/tup/path.c
bootstrap CC (unoptimized) ../src/tup/platform.c
bootstrap CC (unoptimized) ../src/tup/server.c
bootstrap CC (unoptimized) ../src/tup/string_tree.c
bootstrap CC (unoptimized) ../src/tup/tupid_tree.c
bootstrap CC (unoptimized) ../src/tup/updater.c
bootstrap CC (unoptimized) ../src/tup/vardb.c
bootstrap CC (unoptimized) ../src/tup/tup/main.c
bootstrap CC (unoptimized) ../src/tup/monitor/null.c
bootstrap CC (unoptimized) ../src/tup/colors/colors.c
bootstrap CC (unoptimized) ../src/sqlite3/sqlite3.c
bootstrap CC (unoptimized) ../src/ldpreload/ldpreload.c
bootstrap LD.so tup-ldpreload.so
bootstrap LD tup
Error: tup database already exists in directory: /usr/local/google/
home/anatol/sources/opensource/build_tools/tup
[ tup ] Scanning filesystem...0.012s
[ tup ] No tup.config changes.
[ tup ] No Tupfiles to parse.
[ tup ] No files to delete.
[ tup ] Executing Commands...
[ 0/1 ] LINK tup
tup error: chdir() is not supported.
fatal: Could not switch to '.': Function not implemented
Error: Expected to write to file 'tup' from cmd 707 but didn't
Error: Expected to write to file 'tup-version.o' from cmd 707 but
didn't
*** Command 707 failed with return value 128: version=`git describe`;
echo "const char *tup_version(void) {return \"$version\";}" | gcc -x c
-c - -o tup-version.o; gcc src/tup/tup/main.o libtup.a tup-version.o -
o tup -lpthread
Mike Shal
<anatol....@gmail.com> wrote:
> Hi,
>
> I am trying to build tup tool using ./bootstrap.sh and it fails. My
> system is ubuntu 10.04 TLS/gcc 4.4.3
>
> anatol:tup $ ./bootstrap.sh
> [ tup ] Executing Commands...
> [ 0/1 ] LINK tup
> tup error: chdir() is not supported.
> fatal: Could not switch to '.': Function not implemented
> Error: Expected to write to file 'tup' from cmd 707 but didn't
> Error: Expected to write to file 'tup-version.o' from cmd 707 but
> didn't
> *** Command 707 failed with return value 128: version=`git describe`;
> echo "const char *tup_version(void) {return \"$version\";}" | gcc -x c
> -c - -o tup-version.o; gcc src/tup/tup/main.o libtup.a tup-version.o -
> o tup -lpthread
Hi Anatol,
I just upgraded my ubuntu vm to 10.04.1, and was able to bootstrap tup
without this error. Can you check your git and gcc executables to see
if they are similar to mine? I'm wondering if one of them has some
wrapper script around it that ends up doing a 'chdir' for some reason.
I have this:
$ which git
/usr/bin/git
$ file `which git`
/usr/bin/git: ELF 32-bit LSB executable, Intel 80386, version 1
(SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.15,
stripped
$ which gcc
/usr/bin/gcc
$ file `which gcc`
/usr/bin/gcc: symbolic link to `gcc-4.4'
$ file `which gcc-4.4`
/usr/bin/gcc-4.4: ELF 32-bit LSB executable, Intel 80386, version 1
(SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.15,
stripped
A simple work-around might just be to have tup ignore chdir's to ".",
but I'd like to know why it's happening first, since I'm not able to
reproduce it.
Thanks,
-Mike
Anatol Pomozov
Thanks for the quick response.
/usr/bin/git: ELF 64-bit LSB executable, x86-64, version 1 (SYSV),
dynamically linked (uses shared libs), for GNU/Linux 2.6.15, not
stripped
I built git from sources, that is why it is a little bit different.
$ git --version
git version 1.7.4.rc1.7.g2cf08
> $ which gcc
> /usr/bin/gcc
/usr/bin/gcc
> $ file `which gcc`
> /usr/bin/gcc: symbolic link to `gcc-4.4'
/usr/bin/gcc: symbolic link to `gcc-4.4'
> $ file `which gcc-4.4`
> /usr/bin/gcc-4.4: ELF 32-bit LSB executable, Intel 80386, version 1
> (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.15,
> stripped
/usr/bin/gcc-4.4: ELF 64-bit LSB executable, x86-64, version 1 (SYSV),
dynamically linked (uses shared libs), for GNU/Linux 2.6.15, stripped
> A simple work-around might just be to have tup ignore chdir's to ".",
> but I'd like to know why it's happening first, since I'm not able to
> reproduce it.
If I run
$ version=`git describe`; echo "const char *tup_version(void) {return
\"$version\";}" | gcc -x c -c - -o tup-version.o; gcc
src/tup/tup/main.o libtup.a tup-version.o -o tup -lpthread
It works fine and I see 'tup' binary in the root folder.
$ ./tup --version
tup v0.3-152-g094d952
Later this evening I'll try building tup on my home computer (Ubuntu 10.10 x64).
Ricardo M. Correia
> tup error: chdir() is not supported.
> fatal: Could not switch to '.': Function not implemented
> Error: Expected to write to file 'tup' from cmd 707 but didn't
> Error: Expected to write to file 'tup-version.o' from cmd 707 but
> didn't
> *** Command 707 failed with return value 128: version=`git describe`;
> echo "const char *tup_version(void) {return \"$version\";}" | gcc -x c
> -c - -o tup-version.o; gcc src/tup/tup/main.o libtup.a tup-version.o -
> o tup -lpthread
If I remember correctly, this happens when you try to bootstrap tup
outside of a git repository.
One thing you can do is to replace "`git describe`" with some hard-coded
version string, e.g. "v0.3-wip"
Ricardo M. Correia
> Not sure that i understand it. I just cloned tup repository and trying
> to build the project. I have .git directory in the root.
Ah, then perhaps you have some extension in your ~/.gitconfig which
causes git to chdir() out of the git repo?
Or maybe the git implementation that you have has some new behavior (you
compiled from source, right?).
> > One thing you can do is to replace "`git describe`" with some hard-coded
> > version string, e.g. "v0.3-wip"
>
> Thanks, it allowed to fix the problem.
No problem :-) Obviously it's not really a fix, but more of a
workaround.
Cheers,
Ricardo
Anatol Pomozov
> On Wed, 2011-01-12 at 15:47 -0800, Anatol Pomozov wrote:
>> Not sure that i understand it. I just cloned tup repository and trying
>> to build the project. I have .git directory in the root.
>
> Ah, then perhaps you have some extension in your ~/.gitconfig which
> causes git to chdir() out of the git repo?
No, I don't see anything suspicious in my ~/.gitconfig
I run git with strace and what I have locally
$ strace -f -e trace=chdir git describe
chdir(".") = 0
chdir("/home/anatol/sources/opensource/build_tools/tup") = 0
v0.3-152-g094d952
And this is what I have on a computer with git from *.deb package
$ strace -f -e trace=chdir git describe
v2.6.31-8676-g7ca263cd
It means clean 'git describe' does not call chdir().
> Or maybe the git implementation that you have has some new behavior (you
> compiled from source, right?).
Maybe it has been changed. Is there an easy way to find who and where
exactly calls chdir() method?
Anatol Pomozov
Here is my gdb session. It seems that chdir comes from git internals
(abspath.c, function make_absolute_path). So your build probably will
be broken in the next git release.
anatol:tup $ gdb git
GNU gdb (GDB) 7.2-gg6
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
Type "show copying" and "show warranty" for licensing/warranty details.
This GDB was configured as "x86_64-linux".
<http://wiki/Main/GnuDebugger FAQ: http://go/gdb Email: gdb-team>
Hey, I'm GDB 7.x. Check me out! http://wiki/Main/Gdb7x
Reading symbols from /usr/bin/git...done.
(gdb) set args describe
(gdb) break chdir
Breakpoint 1 at 0x7ffff736ee60: file
../sysdeps/unix/syscall-template.S, line 82.
(gdb) run
Starting program: /usr/bin/git describe
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Breakpoint 1, chdir () at ../sysdeps/unix/syscall-template.S:82
82 ../sysdeps/unix/syscall-template.S: No such file or directory.
in ../sysdeps/unix/syscall-template.S
(gdb) where
#0 chdir () at ../sysdeps/unix/syscall-template.S:82
#1 0x0000000000465e31 in make_absolute_path (path=<value optimized
out>) at abspath.c:46
#2 0x0000000000489207 in set_git_work_tree (new_work_tree=0x74d180
".") at environment.c:151
#3 0x00000000004ba4eb in setup_discovered_git_dir (nongit_ok=0x0) at
setup.c:423
#4 setup_git_directory_gently_1 (nongit_ok=0x0) at setup.c:543
#5 setup_git_directory_gently (nongit_ok=0x0) at setup.c:582
#6 0x0000000000404d44 in run_builtin (argc=1, argv=0x7fffffffe240) at git.c:268
#7 handle_internal_command (argc=1, argv=0x7fffffffe240) at git.c:448
#8 0x0000000000404e98 in run_argv (argc=1, argv=0x7fffffffe240) at git.c:492
#9 main (argc=1, argv=0x7fffffffe240) at git.c:565
Abdulla Kamar
/usr/local/google/home/anatol/sources/opensource/build_tools/tup
--
tup-users mailing list
email: tup-...@googlegroups.com
unsubscribe: tup-users+...@googlegroups.com
options: http://groups.google.com/group/tup-users?hl=en
--
Abdulla
Mike Shal
<anatol....@gmail.com> wrote:
> Ok,
>
> Here is my gdb session. It seems that chdir comes from git internals
> (abspath.c, function make_absolute_path). So your build probably will
> be broken in the next git release.
>
Ahh, thanks for digging into this! I can reproduce the behavior with
the latest git as well. I will try to get in a fix soon hopefully...
-Mike
Anatol Pomozov
On Wed, Jan 12, 2011 at 6:58 PM, Abdulla Kamar <abdull...@gmail.com> wrote:
>> /usr/local/google/home/anatol/sources/opensource/build_tools/tup
Oops, forgot to remove it.
> Off topic, but out of curiosity, are you allowed to mention what Google uses
> tup for?
Well, Google has huge internal codebase and having a tool that works
fast on multi-gigabyte source-tree is highly important. Internally
Google used its own closed build tool. The tool emphasizes on
distributed build process i.e. breaking build into many pieces and
running it on dedicated build machines in parallel. Something like
distcc but in lager scale and for many compilers/scripts.
Also it is uses distributed cache for the build system based on the
BigTable. It is basically huge map of
HASH(sources+dependencies)->HASH(output) and if the build tool finds
that somebody already built the same sources it reuses cached objects
instead of building it.
Yet another interesting idea is trying to make the build hermetic i.e.
put as much dependencies into repository as possible -
gcc/clang/javac/go compiler/all libraries even zip tool, everything
that affects output. So everyone has the same sources/tools and does
not depend on version installed locally via apt-get.
All said above the tool is for internal sourcetree. Developers are
quite happy with it and I don't think that it will be changed any time
soon.
Different story is for opensource projects
(Android/Chrome/ChromeOS/...). It cannot use the internal tool and it
needs something else. Android used Make, Chrome uses GYP + Make
(http://code.google.com/p/gyp/w/list). Even taking into account that
Makefiles for Android very well written it still far from ideal. This
is because of Android has huge source tree http://source.android.com/
On my 2cpu x 8 core machine it take 40 minutes to build android from
scratch, incremental build takes ~10-15 minutes. And even if I did not
change anything gmake still takes a couple of minutes to run (mostly
parsing Makefiles, doing a lot of stats() etc...).
Everything said above was just a preamble. The answer to your question
is AFAIK nobody at Google uses it. But opensource projects are looking
for something better than gmake. There are many build tools around
-Autotools/CMake/waf/scons/premake/... These tools work fine for small
projects but when it comes to 100K+ files they become really slow and
difficult to maintain.
I am investigating Tup at my personal time. I am trying to convert
parts of Android build to Tup and see if the tool can handle it. I
don't want to tell anything about future of Tup at Google, but
personally I like this tool.
Abdulla Kamar
--
tup-users mailing list
email: tup-...@googlegroups.com
unsubscribe: tup-users+...@googlegroups.com
options: http://groups.google.com/group/tup-users?hl=en
--
Abdulla
Mike Shal
> I also admire tup's design, though I'm not enamoured with it's build
> scripts. From my experience, that's one of the areas where the internal
> build tool excels. (Personal opinion, YMMV.)
The build scripts in tup started as a natural way to supply info to
the DAG based on it's model of a build. Ie: everything in tup is
handled as "inputs --> command --> outputs", which led me to the
syntax:
: inputs |> command |> outputs
The only real limitation here that tup adds is that outputs must be in
the same directory as the Tupfile (otherwise figuring out when to
re-parse gets tricky). I don't see any way of changing this particular
part of the build script without radically changing tup's design (you
can think of it as a string-ification of a build object, or
something).
The rest of the stuff in the Tupfiles (ie: %-flags, $-variables, and
things like Tuprules.tup) are just there to remove some of the
redundancy in writing :-rules. This part you could argue about all day
long about expressiveness and what-not. A while back, Ryan Rusaw put
together a patch to add in-line lua support. I also tried to add a
generic 'run' command that would execute an external script (like
foo.py, or something). That script would write out :-rules one line at
a time to stdout to get the info in the tup database (internally, it
could be reading things from a command config file and such). It
didn't seem like there was much interest in either approach, though.
You can check out Ryan's patch in the 'Files' section on the tup-users
group, and the 'run' command exists in the 'runscr' branch of tup.
-Mike
Anatol Pomozov
On Fri, Jan 14, 2011 at 8:44 PM, Abdulla Kamar <abdull...@gmail.com> wrote:
> Oh that's great to hear. I was
> watching: http://www.infoq.com/presentations/Development-at-Google, and I
> got the impression that the internal build tool, or some variant of it,
> would be open-sourced. Did I misjudge Ashish's intentions?
The tool is a clean reimplementation. The issue is that it is not
released yet and there is no clear information when it will happen.
While tup is here and can be used right now.
> I also admire tup's design, though I'm not enamoured with it's build
> scripts. From my experience, that's one of the areas where the internal
> build tool excels. (Personal opinion, YMMV.)
I wrote ~30 Tupfiles last couple of days and I wouldn't say that the
syntax is bad. It is clean and very usable.
From other side I can see that people will say "I do not like its
syntax I want to write build rules in my favorite language -
Python/OCaml/Lua/...". This is the reason why all these
Waf/SCons/CMake/Premake/Rake/.. exist.
But the main difference of tup is not even syntax. The main difference
is its backend:
- it does not keep graph of dependencies in RAM, instead it stores it
on disk (SQLite)
- it does not parse buildfiles every time you build. Instead it
parses files only when they are changed and then it updates the graph
- it has monitor tool (inotify daemon)
All this stuff makes incremental build very fast even for huge projects.
Regarding its syntax - if tup would have a huge userbase and there
will be enough energy I imagine that it would be possible to implement
pluggable Tupfiles parsers. This parser reads files in your favorite
language, converts it to a set of (input-> command -> output) and then
updates SQLite db with this data. Profit!
Abdulla Kamar
But the main difference of tup is not even syntax. The main difference
is its backend:
- it does not keep graph of dependencies in RAM, instead it stores it
on disk (SQLite)
- it does not parse buildfiles every time you build. Instead it
parses files only when they are changed and then it updates the graph
- it has monitor tool (inotify daemon)
All this stuff makes incremental build very fast even for huge projects.
Regarding its syntax - if tup would have a huge userbase and there
will be enough energy I imagine that it would be possible to implement
pluggable Tupfiles parsers. This parser reads files in your favorite
language, converts it to a set of (input-> command -> output) and then
updates SQLite db with this data. Profit!
--
Abdulla
Mike Shal
<anatol....@gmail.com> wrote:
> Regarding its syntax - if tup would have a huge userbase and there
> will be enough energy I imagine that it would be possible to implement
> pluggable Tupfiles parsers. This parser reads files in your favorite
> language, converts it to a set of (input-> command -> output) and then
> updates SQLite db with this data. Profit!
This is what the 'runscr' branch that I mentioned is supposed to
allow. You would still have to stick a Tupfile in each directory you
use that does an 'include_rules', and a Tuprules.tup at the top-level
of your project could do 'run ./my-build-script.sh' or 'run
./my-build-script.py' or whatever. Those scripts would have the actual
build logic, and just print a bunch of :-rules to stdout so tup can
stick them in the database. I've never actually tried to use it for a
project though, so I don't know how it would behave in practice.
-Mike
Anatol Pomozov
More information: I did git bisect and found that this issue is caused
by following change
http://odin3.kernel.org/git-lewiemann/?p=git/git.git;a=commit;h=e6aea2dba27798f5d1eca32435e407541caca400
If understand the commit message correctly this is behavior is
temporary and it will gone once refactoring finished.
Adding the author of the commit to the loop.
Nguyễn, could you please confirm that chdir() syscalls will gone in
the near future? It breaks Tup tool. See the whole thread here
https://groups.google.com/forum/?fromgroups#!topic/tup-users/E4xSE5v4JoI
Mike Shal
> 2011/1/20 Anatol Pomozov <anatol....@gmail.com>:
>> Nguyễn, could you please confirm that chdir() syscalls will gone in
>> the near future? It breaks Tup tool. See the whole thread here
>> https://groups.google.com/forum/?fromgroups#!topic/tup-users/E4xSE5v4JoI
>
>
> Git does a lot of chdir()ing at startup, especially when GIT_DIR is
> not set and .git is not at current directory. Even if chdir() were
> removed from make_absolute_path() you would encounter it soon
> somewhere (28 occurrences according to git-grep).
Tup watches all spawned programs for file accesses in order to make
sure dependencies are accurate. Most tools used during the build
process (gcc, ar, etc) don't chdir, so it didn't need to be supported.
I use 'git describe' in the build for tup itself in order to generate
the version string, which worked until the latest git. I don't think
it will be a huge deal to support chdir in tup, now that there's a
reason for it. Just need to set aside some time to do it :)
Thanks for the info,
-Mike
Mike Shal
I just pushed some test cases for chdir(). Only chdir() (not fchdir())
is supported, and there are still some corner cases that will not work
(for example, if you chdir outside of the tup hierarchy and then try
to write files back in tup, it won't work correctly). Please let me
know if anyone comes across a build tool that actually does something
like this.
I tested the tup build itself with the latest git and it seems to have
addressed the issue.
Thanks Anatol for catching this before the new git version is released!
-Mike