redo partially broken on OS X Mavericks

[gepa...@gmail.com]

I discovered a fairly serious problem with redo on Mavericks. It seems to have something to do with nested redo calls and file descriptor operations with fcntl in jwack.py. I managed an extremely rough workaround by commenting out lines 104-106 (from https://github.com/apenwarr/redo), which prevents MAKEFLAGS from being set with jobserver file descriptors — as far as I can tell, my workaround prevents multiple parallel jobs (the -j flag) from working properly.

The failure looks like this:

redo  all

Traceback (most recent call last):

  File "/opt/brew/bin/redo-ifchange", line 5, in <module>

    import redo_ifchange

  File "/opt/brew/bin/../lib/redo/redo_ifchange.py", line 35, in <module>

    rv = builder.main(targets, should_build)

  File "/opt/brew/bin/../lib/redo/builder.py", line 319, in main

    jwack.get_token(t)

  File "/opt/brew/bin/../lib/redo/jwack.py", line 145, in get_token

    setup(1)

  File "/opt/brew/bin/../lib/redo/jwack.py", line 95, in setup

    raise ValueError('broken --jobserver-fds from make; prefix your Makefile rule with a "+"')

ValueError: broken --jobserver-fds from make; prefix your Makefile rule with a "+"

default.do:202:in `block in <main>': failed to rebuild augen1 (RuntimeError)

Trying "redo -j20 test" causes a different set of failures:

redo          Documentation/redo-ood.md.tmp

Traceback (most recent call last):

  File "/opt/brew/bin/redo-ifchange", line 5, in <module>

    import redo_ifchange

  File "/opt/brew/bin/../lib/redo/redo_ifchange.py", line 35, in <module>

    rv = builder.main(targets, should_build)

  File "/opt/brew/bin/../lib/redo/builder.py", line 337, in main

    BuildJob(t, f, lock, shouldbuildfunc, done).start()

  File "/opt/brew/bin/../lib/redo/builder.py", line 97, in start

    self._start_do()

  File "/opt/brew/bin/../lib/redo/builder.py", line 173, in _start_do

    state.commit()

  File "/opt/brew/bin/../lib/redo/state.py", line 107, in commit

    db().commit()

sqlite3.OperationalError: disk I/O error

Both of these errors go away with the MAKEFLAGS code commented out.

At this point, I'm not sure if it's a Mavericks (10.9.0) bug, or if something else has gone wrong. Either way, suggestions for a better solution than removing parallel job support are welcome.

Many thanks,

Constantine

[William Casarin]

On 2013-10-25, at 2:02 AM, gepa...@gmail.com wrote:

sqlite3.OperationalError: disk I/O error

My coworker was getting this error when running parallel builds, but I haven't seen the first one yet, and we have a pretty nested set of do files. I'm hesitant to update to Mavericks just yet because of it. We use redo extensively throughout our codebase and this bug is pretty crippling to our build times.

[Robert Bocchino]

Have you tried using a different version of python?  The python version bundled with Mavericks is 2.7.5.  For me, using python 2.7.8 (installed via MacPorts) made the disk I/O error go away for redo -j2 on my 2-core MacBook pro. I have not seen the first error.

Rob