Calling redo from xonsh

33 views
Skip to first unread message

Norman Kabir

unread,
Jan 14, 2019, 2:48:10 PM1/14/19
to redo
I am attempting to execute redo-ifchange from within a demo xonsh script:

demo.do

#!/usr/bin/env xonsh

redo-ifchange demo.c

echo "demo" 1>&2


Running "redo demo" throws an error

$ redo demo
redo  demo
Traceback (most recent call last):
  File "/home/nkabir/.local/bin/redo-ifchange", line 11, in <module>
    sys.exit(main())
  File "/home/nkabir/.local/lib/python2.7/site-packages/redo/cmd_ifchange.py", line 41, in main
    jobserver.setup(0)
  File "/home/nkabir/.local/lib/python2.7/site-packages/redo/jobserver.py", line 240, in setup
    'prefix your Makefile rule with a "+"')
ValueError: broken --jobserver-auth from make; prefix your Makefile rule with a "+"
demo


I am confused by this error because I am not using a Makefile (yet). Any ideas on how to resolve this? Otherwise, xonsh pairs very nicely with redo.

Avery Pennarun

unread,
Jan 14, 2019, 8:24:08 PM1/14/19
to Norman Kabir, redo
On Mon, Jan 14, 2019 at 9:48 AM Norman Kabir <nka...@rocketknowledge.com> wrote:
I am attempting to execute redo-ifchange from within a demo xonsh script:
[...]
$ redo demo
redo  demo
Traceback (most recent call last):
  File "/home/nkabir/.local/bin/redo-ifchange", line 11, in <module>
    sys.exit(main())
  File "/home/nkabir/.local/lib/python2.7/site-packages/redo/cmd_ifchange.py", line 41, in main
    jobserver.setup(0)
  File "/home/nkabir/.local/lib/python2.7/site-packages/redo/jobserver.py", line 240, in setup
    'prefix your Makefile rule with a "+"')
ValueError: broken --jobserver-auth from make; prefix your Makefile rule with a "+"
demo


I am confused by this error because I am not using a Makefile (yet). Any ideas on how to resolve this? Otherwise, xonsh pairs very nicely with redo.

It looks like xonsh is closing all "extra" file descriptors when it runs a subprocess, such as redo-ifchange. This is extremely bad behaviour in a program that is supposed to replace a shell.

(I just tried out xonsh on my machine, and it does indeed close extra file descriptors.  It's probably using the default, and very evil, close_fds=True option in python3's subprocess.Popen.  This used to default to False back in python2, as the various deities intended.)

redo (and make) use file descriptor inheritance for their parallelism.  They pass the numbers of those file descriptors in an environment variable named MAKEFLAGS.  redo also has a second set of file descriptors in REDO_CHEATFDS, which must also be present when that environment variable is set.

You can temporarily work around the problem by doing this at the top of all your xonsh .do files:

del $MAKEFLAGS
del $REDO_CHEATFDS

...but that's ugly and it will still prevent redo from running more than one job at a time (ie. everything will be serialized).

Fundamentally this is a very serious bug in xonsh, and you should complain to them about it :)  

(While we're here, their fd redirection syntax is extremely sloppy; 1>&2 works, but 1>&5 writes to a file named &5, even if there is an fd#5 open.  I don't think this is going to win any awards for a more predictable syntax than sh.)

Meanwhile, I'll clarify the error message in redo to give a better explanation when it happens.

Have fun,

Avery

Norman Kabir

unread,
Jan 15, 2019, 8:27:33 AM1/15/19
to Avery Pennarun, redo
Thank you for your lucid response! I've opened an issue here:


The workaround is much appreciated. It will do for now.

The redo repository on Github doesn't seem to have Issues enabled:


Is this intentional? I wanted to add a reference to the issue above to track progress.

Cheers,
--Norm


--
You received this message because you are subscribed to the Google Groups "redo" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redo-list+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Avery Pennarun

unread,
Jan 15, 2019, 12:55:14 PM1/15/19
to Norman Kabir, redo
On Tue, Jan 15, 2019 at 3:27 AM Norman Kabir <nka...@rocketknowledge.com> wrote:
The redo repository on Github doesn't seem to have Issues enabled:


Is this intentional? I wanted to add a reference to the issue above to track progress.

At one point I read a Linus Torvalds rant about how bug tracking systems don't actually solve problems in open source, unless people are paid to monitor and sort them.  I think it might have been this one: https://yarchive.net/comp/linux/bug_tracking.html

There's room to criticize his tone, but his overall thesis that open source bug tracking systems are where bug reports go to die seems to be correct; you can see it in the ever-growing bug database for any popular open source project.  Fundamentally, open source developers fix whatever they want to fix.  Ironically, they are actually *more* likely to fix important problems if they are periodically reminded about them on a mailing list because there was no tracking system.

So I disabled the bug trackers on all my github projects and overall it has worked pretty well.

(When it comes to corporate-sponsored projects, I feel exactly the opposite.  Bug trackers are really useful for prioritization and for predicting schedules, both of which are valuable for businesses.  And you can pay people to sort through them.  I wrote about corporate bug tracking processes here: https://apenwarr.ca/log/20171213)

Have fun,

Avery

Norman Kabir

unread,
Jan 15, 2019, 8:07:55 PM1/15/19
to Avery Pennarun, redo
Thank you. I am happy to follow any process you choose. I just wanted to confirm that I hadn't missed an issue tracker.-)

Cheers,
--Norm

Avery Pennarun

unread,
Jan 17, 2019, 7:14:20 PM1/17/19
to Norman Kabir, redo
On Tue, Jan 15, 2019 at 3:27 AM Norman Kabir <nka...@rocketknowledge.com> wrote:
Thank you for your lucid response! I've opened an issue here:


The workaround is much appreciated. It will do for now.

I've updated redo in current master so that you don't have to unset the undocumented REDO_CHEATFDS, and to clarify the error message when MAKEFLAGS file descriptors have a problem.  The new error message links here for more details: https://redo.readthedocs.io/en/latest/FAQParallel/#MAKEFLAGS

Have fun,

Avery

Norman Kabir

unread,
Feb 4, 2019, 7:24:24 PM2/4/19
to redo
Hi Avery.

I've been working through the issue with the Anthony Scopatz of xonsh. I now find myself caught in the middle of very useful tools whose internals I do not know very well. I created a minimal Github project to reproduce the behavior:


I'm including a response from Anthony in the event you might be able to shed some light on the underlying issue:

...

---------- Forwarded message ---------
From: Anthony Scopatz <notifi...@github.com>
Date: Mon, Feb 4, 2019 at 5:08 PM
Subject: Re: [xonsh/xonsh] xonsh appears to be closing file descriptors in subprocesses (#2984)
To: xonsh/xonsh <xo...@noreply.github.com>


Oh interesting, so redo is actually launching xonsh, potentially via make. That is very interesting. I am still not sure why you / Avery think that this is a file descriptor issue. Can you rope Avery into this conversation to explain? Or is there a minimal Makefile that I can use to test this out?

I am not sure how many folks have tried to use xonsh inside of make or gdb or other tools with assume that $SHELL is a posix-compatible shell. We, of course, want xonsh to work with these tools.

xonsh has been a pleasure to use!

Thanks again!


...


Cheers,

--Norm

Avery Pennarun

unread,
Feb 4, 2019, 11:20:02 PM2/4/19
to Norman Kabir, redo
I posted a reply to the github thread. Looks like someone else beat
me to it with an even simpler example.

https://github.com/xonsh/xonsh/issues/2984#issuecomment-460506863
Reply all
Reply to author
Forward
0 new messages