Generate multiple file from a single file

[zesen...@gmail.com]

Hi List,

So I'm trying to use Shake in a project of mine. A step in the building process is to extract multiple files from a single file with a command (say ./extract.py whole.dat a.part b.part c.part).

What I want to achieve is, sometimes maybe only a.part is older than whole.dat, in which case I only want to extract a.part, to save some CPU cycle. The script extract.py is capable of partial extraction.

It worth mentioning that, the loading of whole.dat can take some time, so I don't want to have seperate rules for each part - that would invoke the loading seperately and a waste.

Thank you.

Best,
Zesen

[Neil Mitchell]

Hi Zesen,

Assuming that you extract the files at the same time if possible, how
would it be the case that b.part is valid but a.part has changed?

You might be looking for &%> or &?>, which are the multiple output
rules, see https://hackage.haskell.org/package/shake-0.16.4/docs/Development-Shake.html#v:-38--37--62-
and https://hackage.haskell.org/package/shake-0.16.4/docs/Development-Shake.html#v:-38--63--62-

In rarer cases you might want to look at the Batching features at
https://hackage.haskell.org/package/shake-0.16.4/docs/Development-Shake.html#g:18,
but I'd try and avoid that if possible - most things can be expressed
more easily using the other features.

Thanks, Neil

[zesen...@gmail.com]

Hi Neil,

Thanks for the reply.

On Tuesday, April 10, 2018 at 5:49:05 AM UTC-4, Neil Mitchell wrote:
> Hi Zesen,
>
> Assuming that you extract the files at the same time if possible, how
> would it be the case that b.part is valid but a.part has changed?

I feel this is possible - maybe I just manually delete a.part and b.part is still newer than whole.dat. I admit it's not a common scenario.

>
> You might be looking for &%> or &?>, which are the multiple output
> rules, see https://hackage.haskell.org/package/shake-0.16.4/docs/Development-Shake.html#v:-38--37--62-
> and https://hackage.haskell.org/package/shake-0.16.4/docs/Development-Shake.html#v:-38--63--62-

currently I'm using &%> to extract all files at once. It's working well.

> In rarer cases you might want to look at the Batching features at
> https://hackage.haskell.org/package/shake-0.16.4/docs/Development-Shake.html#g:18,
> but I'd try and avoid that if possible - most things can be expressed
> more easily using the other features.

Yeah I had difficulty understanding the semantics of batching. But anyway, by now &%> works well for me. I guess I'm trying too hard to optimize.

>
> Thanks, Neil
>
> On Mon, Apr 9, 2018 at 9:51 PM, <zesen...@gmail.com> wrote:
> > Hi List,
> >
> > So I'm trying to use Shake in a project of mine. A step in the building process is to extract multiple files from a single file with a command (say ./extract.py whole.dat a.part b.part c.part).
> >
> > What I want to achieve is, sometimes maybe only a.part is older than whole.dat, in which case I only want to extract a.part, to save some CPU cycle. The script extract.py is capable of partial extraction.
> >
> > It worth mentioning that, the loading of whole.dat can take some time, so I don't want to have seperate rules for each part - that would invoke the loading seperately and a waste.
> >
> > Thank you.
> >
> > Best,
> > Zesen

Best,
Zesen

[Neil Mitchell]

>> Assuming that you extract the files at the same time if possible, how
>> would it be the case that b.part is valid but a.part has changed?
>
> I feel this is possible - maybe I just manually delete a.part and b.part is still newer than whole.dat. I admit it's not a common scenario.

That seems unlikely - if it is happening with any regularity then
users are probably doing something untoward which needs to be captured
properly.

> Yeah I had difficulty understanding the semantics of batching. But anyway, by now &%> works well for me. I guess I'm trying too hard to optimize.

Batching isn't too hard - but the docs weren't great. I've pushed some
improvements to HEAD which hopefully will make it easier for those
that come after. However, in this case I don't think it's going to
help.