Getting started: remote_file rule, extracting

48 views
Skip to first unread message

Collin Lefeber

unread,
Aug 3, 2019, 11:23:41 AM8/3/19
to please-build
One of the first dependencies in an existing Makefile is downloading a nodejs archive and extracting it for later use as a tool.

I started a BUILD file in a "buildtools/" directory/package with the following

remote_file(
    name = "node_archive",
    out = "node-v8.11.3-linux-x64",
    extract = True
)

Mostly to test out the "extract" option, but I think it's intended to extract just 1 file from the archive? Otherwise I wasn't sure what to use for "out = ". This was the error.

11:13:44.852    INFO: //buildtools:node_archive: Extracting...
11:13:45.324    INFO: //buildtools:node_archive: Collecting outputs...
11:13:45.358    INFO: Shutting down cache workers...
11:13:45.358    INFO: [7/7 tasks] //buildtools:node_archive: Build failed: Error moving outputs for target //buildtools:node_archive: Callback: lstat plz-out/tmp/buildtools/node_archive._build/lib: no such file or directory [0.5s]


So I was also trying the following

node_archive = remote_file(
    name = "node_archive",
    out = "node-v8.11.3-linux-x64.tar.gz",
)

genrule(
    name = "extract_node",
    srcs = [node_archive],
    deps = [":node_archive"],
    cmd = "tar -xvf $SRC",
)

genrule(
    name = "print_node",
    srcs = [node_archive],
    cmd = "echo $SRC",
)

Attempting to use the node_archive out as a src in the extraction rule. But, the tar extraction doesn't look to be running at all

11:18:23.207    INFO: //:pleasings: Building...
11:18:23.225    INFO: //:pleasings: Collecting outputs...
11:18:23.237   DEBUG: Outputs for //:pleasings have changed
11:18:23.248    INFO: //:pleasings: Storing...
11:18:23.248   DEBUG: Storing //:pleasings: pleasings in dir cache...
11:18:23.248    INFO: [5/8 tasks] //:pleasings: Built [0.0s]
11:18:23.401    INFO: //buildtools:node_archive: Collecting outputs...
11:18:23.401   DEBUG: Outputs for //buildtools:node_archive have changed
11:18:23.418    INFO: //buildtools:node_archive: Storing...
11:18:23.418   DEBUG: Need to rebuild //buildtools:extract_node, //buildtools:node_archive has changed
11:18:23.418   DEBUG: Storing //buildtools:node_archive: node-v8.11.3-linux-x64.tar.gz in dir cache...
11:18:23.418    INFO: [5/8 tasks] //buildtools:node_archive: Built [0.8s]
11:18:23.418   DEBUG: Need to rebuild //buildtools:print_node, //buildtools:node_archive has changed
11:18:23.419    INFO: //buildtools:extract_node: Preparing...
11:18:23.419    INFO: //buildtools:print_node: Preparing...
11:18:23.419    INFO: Shutting down cache workers...
11:18:23.419    INFO: [8/8 tasks] //buildtools:extract_node: Nothing to do [0.0s]
11:18:23.419   DEBUG: Shut down all cache workers
11:18:23.419    INFO: [8/8 tasks] //buildtools:print_node: Nothing to do [0.0s]
Build finished; total time 870ms, incrementality 0.0%. Outputs:
//:pleasings:
  plz-out/gen/pleasings
//buildtools:extract_node:
//buildtools:node_archive:
  plz-out/gen/buildtools/node-v8.11.3-linux-x64.tar.gz
//buildtools:print_node:

$ find plz-out/ | grep node
plz-out/tmp/buildtools/print_node._build
plz-out/tmp/buildtools/extract_node._build
plz-out/gen/buildtools/node-v8.11.3-linux-x64.tar.gz


I am definitely missing something but am a bit stuck at this point.

If I can work through this i'd like to contribute back a guide to using plz for front end focused, incremental builds.

Peter Ebden

unread,
Aug 5, 2019, 5:26:07 AM8/5/19
to Collin Lefeber, please-build
Hey, so you're missing outputs on your extract_node genrule; you probably want something like outs = ["node-v8.11.3-linux-x64.tar.gz"].

I'm not sure precisely what's wrong with the other option and why it's looking at a folder called lib, might need a little more investigation, but the above should get you going I think.

--
You received this message because you are subscribed to the Google Groups "please-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to please-build...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/please-build/4d0a1199-aaf8-4068-94d9-de07bbe2ba4c%40googlegroups.com.


--

Thought Machine is a limited company registered in England & Wales.
Registered number: 11114277
Registered Office: 5 New Street Square, London EC4A 3TW

Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Thought Machine. Employees of Thought Machine are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to Thought Machine policy and outside the scope of the employment of the individual concerned. Thought Machine will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or any other liability arising.



Thought Machine Group a limited company registered in England & Wales.
Registered number: 11114277. 

The content of this email is confidential and intended for the recipient specified in message only. It is strictly forbidden to share any part of this message with any third party, without a written consent of the sender. If you received this message by mistake, please reply to this message and follow with its deletion, so that we can ensure such a mistake does not occur in the future.

Collin Lefeber

unread,
Aug 6, 2019, 3:28:57 PM8/6/19
to Peter Ebden, please-build
Thans for the response Peter, i'll try that.

The extracted nodejs tar should produce 1 directory as output: "node-v8.11.3-linux-x64/" and the contents:

bin/
CHANGELOG.md
include/
LICENSE
README.md
lib/
share/

$ find . -type f | wc -l
4137

So would I need to enumerate every extracted file as an output?

Collin Lefeber

unread,
Aug 6, 2019, 9:57:46 PM8/6/19
to please-build
Was able to experiment tonight and adding outs worked!

genrule(
    name = "extract_node",
    srcs = [node_archive],
    deps = [":node_archive"],
    outs = ["node-v8.11.3-linux-x64/"],
    cmd = "tar -xvf $SRC",
)

$ plz clean -f; plz build
Build finished; total time 1.15s, incrementality 0.0%. Outputs:
//:pleasings:
  plz-out/gen/pleasings
//buildtools:extract_node:
  plz-out/gen/buildtools/node-v8.11.3-linux-x64
//buildtools:node_archive:
  plz-out/gen/buildtools/node-v8.11.3-linux-x64.tar.gz
//buildtools:print_node:
 
$ ls plz-out/gen/buildtools/node-v8.11.3-linux-x64
bin  CHANGELOG.md  include  lib  LICENSE  README.md  share

So that should hold me over for a bit!



To unsubscribe from this group and stop receiving emails from it, send an email to please-build+unsubscribe@googlegroups.com.

Peter Ebden

unread,
Aug 7, 2019, 5:51:58 AM8/7/19
to Collin Lefeber, please-build
Yep indeed, as you say just specifying the directory name as an output is fine. Glad it's working for you now!

To unsubscribe from this group and stop receiving emails from it, send an email to please-build...@googlegroups.com.


--

Thought Machine is a limited company registered in England & Wales.
Registered number: 11114277
Registered Office: 5 New Street Square, London EC4A 3TW

Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Thought Machine. Employees of Thought Machine are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to Thought Machine policy and outside the scope of the employment of the individual concerned. Thought Machine will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or any other liability arising.



Thought Machine Group a limited company registered in England & Wales.
Registered number: 11114277. 

The content of this email is confidential and intended for the recipient specified in message only. It is strictly forbidden to share any part of this message with any third party, without a written consent of the sender. If you received this message by mistake, please reply to this message and follow with its deletion, so that we can ensure such a mistake does not occur in the future.

--
You received this message because you are subscribed to the Google Groups "please-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to please-build...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/please-build/e61cdc19-df2a-489f-899b-b01cb977c953%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages