Redo implementation in Bourne Shell

[Nils Dagsson Moskopp]

Hello redo list,


I have implemented DJB redo in Bourne Shell. It depends on GNU coreutils
or busybox. For dependency checking, it checks a dependencies' ctime
against the stored ctime and if the ctime differs, it checks the
dependencies' md5sum against the stored md5sum. Discuss!

<http://news.dieweltistgarnichtso.net/bin/redo.html>
<http://news.dieweltistgarnichtso.net/bin/redo-always.html>
<http://news.dieweltistgarnichtso.net/bin/redo-ifchange.html>
<http://news.dieweltistgarnichtso.net/bin/redo-ifcreate.html>
<http://news.dieweltistgarnichtso.net/bin/redo-sources.html>
<http://news.dieweltistgarnichtso.net/bin/redo-stamp.html>
<http://news.dieweltistgarnichtso.net/bin/redo-targets.html>


Greetings,
--
Nils Dagsson Moskopp // erlehmann
<http://dieweltistgarnichtso.net>

[Ryan Eckbo]

I'm having trouble downloading it - can you put it on github?

[Nils Dagsson Moskopp]

Ryan Eckbo <ryan....@gmail.com> writes:

> I'm having trouble downloading it - can you put it on github?

I can, but I will most certainly not, since GitHub has already suspended
my profile once while I was working on a project for a client. But since
you seem to have git, you can just clone the repository from my server!

On the page <http://news.dieweltistgarnichtso.net/bin/> there is written:
git clone http://news.dieweltistgarnichtso.net/bin.git

Alternatively, just use wget:
wget http://news.dieweltistgarnichtso.net/bin/redo.html
wget http://news.dieweltistgarnichtso.net/bin/redo-always
wget http://news.dieweltistgarnichtso.net/bin/redo-ifchange
wget http://news.dieweltistgarnichtso.net/bin/redo-ifcreate
wget http://news.dieweltistgarnichtso.net/bin/redo-sources
wget http://news.dieweltistgarnichtso.net/bin/redo-stamp
wget http://news.dieweltistgarnichtso.net/bin/redo-targets

> On Monday, September 15, 2014 6:49:47 PM UTC+10, lol internet wrote:

Funny that your email client thinks my name is “lol internet” :---DDDDD

>>
>> Hello redo list,
>>
>>
>> I have implemented DJB redo in Bourne Shell. It depends on GNU coreutils
>> or busybox. For dependency checking, it checks a dependencies' ctime
>> against the stored ctime and if the ctime differs, it checks the
>> dependencies' md5sum against the stored md5sum. Discuss!
>>
>> <http://news.dieweltistgarnichtso.net/bin/redo.html>
>> <http://news.dieweltistgarnichtso.net/bin/redo-always.html>
>> <http://news.dieweltistgarnichtso.net/bin/redo-ifchange.html>
>> <http://news.dieweltistgarnichtso.net/bin/redo-ifcreate.html>
>> <http://news.dieweltistgarnichtso.net/bin/redo-sources.html>
>> <http://news.dieweltistgarnichtso.net/bin/redo-stamp.html>
>> <http://news.dieweltistgarnichtso.net/bin/redo-targets.html>
>>
>>
>> Greetings,
>> --
>> Nils Dagsson Moskopp // erlehmann
>> <http://dieweltistgarnichtso.net>
>>
>

> --
> You received this message because you are subscribed to the Google Groups "redo" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to redo-list+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Nils Dagsson Moskopp]

Nils Dagsson Moskopp <ni...@dieweltistgarnichtso.net> writes:

> I have implemented DJB redo in Bourne Shell.

Performance surprise: My bourne shell redo implementation takes around
twice the time of apenwarr's Python implementation for returning from a
dependency check for up to date builds while using far fewer resources.

I therefore do recommend to use this version instead of apenwarr's do.

For testing, I repeatedly built a roguelike a friend of mine wrote in C
with both implementations: <https://github.com/plomlompom/plomrogue>

Data:
• redo-targets reports 33 targets
• redo-sources reports 127 sources
• a target has, on average, 37 dependencies
• build check time with apenwarr redo: 0.2s on Thinkpad T60
• build check time with my redo implementation: 0.44s on Thinkpad T60

For comparison, the full build takes over 10.5 seconds on Thinkpad T60.

I stopped optimizing my implementation when I got to execution times of
under half a second for the plomrogue test case. The bottleneck of my
redo implementation seems to be interprocess messaging. According to
strace, it spends 99,99% of time in wait4(2) and 0,01% in read(2).

[Nils Dagsson Moskopp]

I have created a new page for my DJB implementation in Bourne Shell;
please provide criticism if anything about it is not totally clear:
<http://news.dieweltistgarnichtso.net/bin/redo-sh.html>

Nils Dagsson Moskopp <ni...@dieweltistgarnichtso.net> writes:

[necron...@gmail.com]

Hello creator of BASH inplementation (lol internet),

A quick question: is your implementation compatible with the original implementation by python?

I have an example of very basic .do files that work under one version but does not work under another. See blow:

Files: ./input/1.dat, ./input/2.dat ./default.proc.do

1.dat and 2.dat can be anything (contents not used here)

default.proc.do contents:
    redo-ifchange $2.dat
    echo A > $3

redo ./input/1.proc works under python version but doesn't work under bash version.

Here is the error I get:
redo 1.proc
redo: Users/karolis/temp/redo/second/1.dat: no .do file
redo: input/1.proc: got exit code 1.

Any insights into why this is? Or are the versions not supposed to be compatible?

[necron...@gmail.com]

I think I broke the scripts..

I changed the default.proc.do to:
    redo-ifchange input/$2.dat
    echo A > $3

and then redo ./input/1.proc started working.

But then I tried redo-ifchange ./input/1.proc and it have this error:
line 32: /.dependencies.tmp: Permission denied

As well as created several files under ./input:
1.dat.ctime, 1.dat.md5sum, 1.png.dependencies, etc.

Probably should not have happened. Did I do something terribly wrong?

[Nils Dagsson Moskopp]

necron...@gmail.com writes:

> Hello creator of BASH inplementation (lol internet),

It is not bash, but bourne shell.

> A quick question: is your implementation compatible with the original
> implementation by python?

With the exception of parallel processing, it should do. Be aware,
however, that it handles up-to-date checks differently. The Python
version is IMHO wrong in assuming that a changed ctime means the
dependency was changed, as lots of things can change the ctime.

I have written a test suite for redo implementations you can get with:
git clone http://daten.dieweltistgarnichtso.net/src/redo-testcases.git

> I have an example of very basic .do files that work under one version but
> does not work under another. See blow:
>
> Files: ./input/1.dat, ./input/2.dat ./default.proc.do
>
> 1.dat and 2.dat can be anything (contents not used here)
>
> default.proc.do contents:
> redo-ifchange $2.dat
> echo A > $3
>
> redo ./input/1.proc works under python version but doesn't work under bash
> version.
>
> Here is the error I get:
> redo 1.proc
> redo: Users/karolis/temp/redo/second/1.dat: no .do file
> redo: input/1.proc: got exit code 1.
>
> Any insights into why this is? Or are the versions not supposed to be
> compatible?

I will test my implementation with the information given by you.

> On Tuesday, July 7, 2015 at 4:39:06 AM UTC+3, lol internet wrote:
>>
>> I have created a new page for my DJB implementation in Bourne Shell;
>> please provide criticism if anything about it is not totally clear:
>> <http://news.dieweltistgarnichtso.net/bin/redo-sh.html>
>>

>> Nils Dagsson Moskopp <ni...@dieweltistgarnichtso.net <javascript:>>

>> writes:
>>
>> > Hello redo list,
>> >
>> >
>> > I have implemented DJB redo in Bourne Shell. It depends on GNU coreutils
>> > or busybox. For dependency checking, it checks a dependencies' ctime
>> > against the stored ctime and if the ctime differs, it checks the
>> > dependencies' md5sum against the stored md5sum. Discuss!
>> >
>> > <http://news.dieweltistgarnichtso.net/bin/redo.html>
>> > <http://news.dieweltistgarnichtso.net/bin/redo-always.html>
>> > <http://news.dieweltistgarnichtso.net/bin/redo-ifchange.html>
>> > <http://news.dieweltistgarnichtso.net/bin/redo-ifcreate.html>
>> > <http://news.dieweltistgarnichtso.net/bin/redo-sources.html>
>> > <http://news.dieweltistgarnichtso.net/bin/redo-stamp.html>
>> > <http://news.dieweltistgarnichtso.net/bin/redo-targets.html>
>> >
>> >
>> > Greetings,
>> > --
>> > Nils Dagsson Moskopp // erlehmann
>> > <http://dieweltistgarnichtso.net>
>>
>> --
>> Nils Dagsson Moskopp // erlehmann
>> <http://dieweltistgarnichtso.net>
>>
>

> --
> You received this message because you are subscribed to the Google Groups "redo" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to redo-list+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Nils Dagsson Moskopp]

necron...@gmail.com writes:

> I think I broke the scripts..
>
> I changed the default.proc.do to:
> redo-ifchange input/$2.dat
> echo A > $3
>
> and then redo ./input/1.proc started working.
>
> But then I tried redo-ifchange ./input/1.proc and it have this error:
> line 32: /.dependencies.tmp: Permission denied

You should call redo-ifchange only from a redo skript. Did you do that?

> As well as created several files under ./input:
> 1.dat.ctime, 1.dat.md5sum, 1.png.dependencies, etc.
>
> Probably should not have happened. Did I do something terribly wrong?

Please send me a tar with the scripts you are using for your build. I am
going to take a look at them and try to figure out if you were wrong or
my implementation has a bug.

> --
> You received this message because you are subscribed to the Google Groups "redo" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to redo-list+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Nils Dagsson Moskopp]

I have no implemented a --parallel switch for my redo implementation. No
job count yet – patch welcome as long as it does not bloat dependencies.

[Nils Dagsson Moskopp]

The page for my redo implementation now has a FAQ section:
<http://news.dieweltistgarnichtso.net/bin/redo-sh.html#faq>

[Nils Dagsson Moskopp]

I have removed the --parallel switch for my redo implementation since I
consider it unmanageable bloat. Please use the shell builtins “&” and
“wait” or GNU parallel if you want to have async builds with redo.