Jenga simple cp rule

[Ashish Agarwal]

I'm following the compile-and-link example, which doesn't work as expected. Here's a stripped down version, where I'm just set up a rule to copy a file a.ml to a.ml2. The copy happens the first time, but jenga doesn't that the copy should be done again when the source file changes. Why not? I specified `Dep.path src` as a dependency.

let cp (src:Path.t) (dst:Path.t) : Rule.t =

  let cmd =

    "cp" ^ " " ^

      (Path.to_string src) ^ " " ^

      (Path.to_string dst)

  in

  let action = bash ~dir:Path.the_root cmd in

  Rule.simple ~targets:[dst] ~deps:[Dep.path src] ~action

let scheme ~(dir:Path.t) : Scheme.t =

  let dir_str = Path.to_string dir in

  match Filename.parts dir_str with

  | "."::[] ->

    Scheme.rules [

      cp (Path.relative ~dir "a.ml") (Path.relative ~dir "a.ml2")

    ]

  | _ -> failwith ("unknown dir " ^ dir_str)

Now, when I run `jenga a.ml2`, the file a.ml2 gets created as a copy of a.ml. However, if I update a.ml, and run `jenga a.ml2` again, jenga does not update a.ml2.

[Valentin Gatien-Baron]

There is no obvious problem with your piece of code (except perhaps for the lack of escaping).

By "update a.ml" you mean change the contents right? If you just change timestamps or do a change and undo it every fast, jenga is not going to redo the copy.

If so, what if you delete a.ml2? Does jenga rebuild the file then?

--
You received this message because you are subscribed to the Google Groups "ocaml-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ocaml-core+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ashish Agarwal]

On Fri, Nov 14, 2014 at 12:18 PM, Valentin Gatien-Baron <vgatie...@janestreet.com> wrote:

 

There is no obvious problem with your piece of code (except perhaps for the lack of escaping).

What should I be escaping?

By "update a.ml" you mean change the contents right? If you just change timestamps or do a change and undo it every fast, jenga is not going to redo the copy.

I open an editor, add a line, and save the file. So the change is real and not too fast.

If so, what if you delete a.ml2? Does jenga rebuild the file then?

Yes, then it does get rebuilt.

[Ben Millwood]

Copying in Arseniy who's doing a bit of work on jenga.

[Valentin Gatien-Baron]

On Fri, Nov 14, 2014 at 12:26 PM, Ashish Agarwal <agarw...@gmail.com> wrote:

On Fri, Nov 14, 2014 at 12:18 PM, Valentin Gatien-Baron <vgatie...@janestreet.com> wrote:

 

There is no obvious problem with your piece of code (except perhaps for the lack of escaping).

What should I be escaping?

The pathnames. I am pretty sure our jengaroot gets it wrong because we don't really care, but if you have whitespace, backslashes, leading dashes and what not in your paths, you cp is not going to work right. Or call cp directly instead of using a shell (doesn't solve the leading dashes issue though).

 

By "update a.ml" you mean change the contents right? If you just change timestamps or do a change and undo it every fast, jenga is not going to redo the copy.

I open an editor, add a line, and save the file. So the change is real and not too fast.

Ok.

 

If so, what if you delete a.ml2? Does jenga rebuild the file then?

Yes, then it does get rebuilt.

And to be clear, a.ml is a source file, it is not generated by some rules that you haven't shown?

If you delete a.ml, what happens? Jenga should complain it doesn't know how to build it.

If it doesn't, there is an executable called jenga-offline which can be used to dump the persistent state of jenga. Can you check that in here, your copy rule appear and a.ml is listed as a dependency?

[Ashish Agarwal]

On Fri, Nov 14, 2014 at 12:41 PM, Valentin Gatien-Baron <vgatie...@janestreet.com> wrote:

What should I be escaping?

The pathnames.

Oh yes, I wasn't worried about this as I expect to use simple file names.

And to be clear, a.ml is a source file, it is not generated by some rules that you haven't shown?

Right. It is a source file.

If you delete a.ml, what happens? Jenga should complain it doesn't know how to build it.

Yes, I get

ERROR: a.ml: No rule or source for: a.ml

If it doesn't, there is an executable called jenga-offline which can be used to dump the persistent state of jenga. Can you check that in here, your copy rule appear and a.ml is listed as a dependency?

 $ jenga-offline cat-sexp

cat_sexp...

cat_sexp, loading...

cat_sexp, converting...

cat_sexp, output...

cat_sexp, waiting...

((fs_persist

  ((digests

    ((cache

      ((jengaroot.ml

        (((dev 16777220) (ino 122774164) (kind File) (size 4723)

          (mtime 1415979846))

         MD5))

       (a.ml

        (((dev 16777220) (ino 123389594) (kind File) (size 10)

          (mtime 1415988296))

         MD5))

       (a.ml2

        (((dev 16777220) (ino 123388093) (kind File) (size 11)

          (mtime 1415985923))

         MD5))))))

   (listings ((cache ())))))

 (build_persist

  ((generated ())

   (ruled

    ((a.ml2

      ((targets (((Path a.ml2) (Digest MD5))))

       (deps (((Path a.ml) (Digest MD5))))

       (action ((dir .) (prog bash) (args (-c "cp a.ml a.ml2"))))))))

   (actioned

    ((((dir .) (prog bash) (args (-c "rm -rf .jenga.debug")))

      ((deps ()) (stdout "")))

     (((dir .) (prog bash) (args (-c "rm -rf .jenga.db")))

      ((deps ()) (stdout "")))

     (((dir .) (prog bash) (args (-c "cp a.ml a.ml2")))

      ((deps (((Path a.ml) (Digest MD5)))) (stdout "")))

     (((dir .) (prog bash) (args (-c "rm -rf .jenga.plugin-cache")))

      ((deps ()) (stdout ""))))))))cat_sexp, done...

[Valentin Gatien-Baron]

Well I can't say I am very familiar with the information stored in there, but instead of MD5, you should have the actual md5 in there. That would certainly explain why changing the contents of files doesn't help.

Does the md5sum in the path just return MD5??

[Ashish Agarwal]

On Fri, Nov 14, 2014 at 1:11 PM, Valentin Gatien-Baron <vgatie...@janestreet.com> wrote:

Does the md5sum in the path just return MD5??

I'm on a Mac.

$ md5 a.ml
MD5 (a.ml) = 72becbd66aaafd35aa862711e483bd37

So the bug is here:

https://github.com/janestreet/jenga/blob/4bbf1bbdbcd85d57db1a6355edbe1fa33bf21bbb/lib/fs.ml#L287

It assumes the output of md5 on a Mac is formatted the same as that on Linux machines, which print:

$ md5sum a.ml
72becbd66aaafd35aa862711e483bd37  a.ml

Any reason not to always use OCaml's Digest.file function?

[Valentin Gatien-Baron]

On Fri, Nov 14, 2014 at 1:23 PM, Ashish Agarwal <agarw...@gmail.com> wrote:

On Fri, Nov 14, 2014 at 1:11 PM, Valentin Gatien-Baron <vgatie...@janestreet.com> wrote:

Does the md5sum in the path just return MD5??

I'm on a Mac.

$ md5 a.ml
MD5 (a.ml) = 72becbd66aaafd35aa862711e483bd37

So the bug is here:

https://github.com/janestreet/jenga/blob/4bbf1bbdbcd85d57db1a6355edbe1fa33bf21bbb/lib/fs.ml#L287

It assumes the output of md5 on a Mac is formatted the same as that on Linux machines, which print:

Ok. While Nick or Arseniy fix this, you can apparently say JENGA_SYSTEM=dumb to use the Digest module instead.

 

$ md5sum a.ml
72becbd66aaafd35aa862711e483bd37  a.ml

Any reason not to always use OCaml's Digest.file function?

That's what jenga used to do. Presumably it was made to spawn a separate process to not slow down the main process, or to be benefit from pallelelism, or both.

[Valentin Gatien-Baron]

On Fri, Nov 14, 2014 at 1:52 PM, Valentin Gatien-Baron <vgatie...@janestreet.com> wrote:

On Fri, Nov 14, 2014 at 1:23 PM, Ashish Agarwal <agarw...@gmail.com> wrote:

On Fri, Nov 14, 2014 at 1:11 PM, Valentin Gatien-Baron <vgatie...@janestreet.com> wrote:

Does the md5sum in the path just return MD5??

I'm on a Mac.

$ md5 a.ml
MD5 (a.ml) = 72becbd66aaafd35aa862711e483bd37

So the bug is here:

https://github.com/janestreet/jenga/blob/4bbf1bbdbcd85d57db1a6355edbe1fa33bf21bbb/lib/fs.ml#L287

It assumes the output of md5 on a Mac is formatted the same as that on Linux machines, which print:

Ok. While Nick or Arseniy fix this, you can apparently say JENGA_SYSTEM=dumb to use the Digest module instead.

Or more directly, JENGA_USE_OCAML_DIGEST set to anything should also work.

[Ashish Agarwal]

> JENGA_USE_OCAML_DIGEST set to anything should also work.

Great, thanks. That's an easy enough workaround.