Generated file has read only access, how to overcome?

tsvi...@iota.org

unread,

Jul 27, 2018, 3:35:52 AM7/27/18

to bazel-discuss

running:

bazel test //common/storage/sql/sqlite3/...

on that test:

cc_test(
name = "test_sqlite3",
srcs = [
"test_sqlite3.c",
],
data = [":db_file"],
visibility = ["//visibility:public"],
deps = [
"//common/storage/sql/sqlite3:sqlite3_storage",
"//common/trinary:trit_ptrit",
"@unity",
],
)

genrule(
name = "db_file",
srcs = ["//common/storage/sql:schema"],
outs = ["ciri.db"],
cmd = "sqlite3 $@ < $<;",
local = 1,
output_to_bindir = 1,
)

Fails.

but when i exe `chmod 666 ciri.db` and then exe the test binary (test_sqlite3) manually, it passes.

also, when i change the change the `cmd` in genrule to:

cmd = "sqlite3 $(@D)/test.db < $<; cat $(@D)/test.db > $@; chmod 666 $(@D)/test.db",

and in my binary i use test.db instead of "ciri.db", it fails when i run "bazel test ..." but passes when i exe manually (without me having to chmod the db file)

I would be grateful for any help on that matter.
Thanks!

László Csomor

unread,

Jul 27, 2018, 4:20:45 AM7/27/18

to tsvi...@iota.org, bazel-discuss

Bazel makes output files readonly by design. I don't know all the reasons, but one is: this way only one action can write the file. If Bazel didn't make the file readonly and multiple actions could write the file, then there'd be a race condition in the build -- depending on which action ran first, the file's contents would be different, making the build non-deterministic.

Is the test trying to write to "ciri.db"?

FYI, dunno if you're aware of it, but to use data-dependencies (e.g. "ciri.db") in the test, you could use the C++ runfiles library: https://github.com/bazelbuild/bazel/blob/770b98717ddcff401eeb50ef7860cf6f1b852a24/tools/cpp/runfiles/runfiles.h#L15-L69 . I recommend using this library because it hides the complexities of finding the data-dependencies, and it works on all supported platforms.

Finally, if the genrule creates "test.db" too, then "test.db" should be in "outs". Otherwise Bazel is not aware that "test.db" is an output of the genrule and an input of the test rule. That's bad because in the next build, imagine Bazel rebuilds the genrule and "ciri.db" stays the same but "test.db" is different: for all Bazel knows, all the outputs of the genrule are the same, and all the inputs of the test rule are the same, so Bazel won't rerun the test.

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

--
*IOTΛ Foundation*
c/o Nextland
Strassburgerstraße 55
10405 Berlin · Germany

Board of Directors: Dominik Schiener, David Sønstebø, Ralf Rottmann

ID/Company No.: 3416/1234/2

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/deca8e81-33a1-4bdf-b6e4-332f02e949ee%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

László Csomor

unread,

Jul 27, 2018, 4:21:52 AM7/27/18

to tsvi...@iota.org, bazel-discuss

> for all Bazel knows, all the outputs of the genrule are the same, and all the inputs of the test rule are the same, so Bazel won't rerun the test.

I meant: all the outputs are the same *as in the last build*, so Bazel will think that the test is still up-to-date since the last build.

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

tsvi...@iota.org

unread,

Jul 27, 2018, 4:40:02 AM7/27/18

to bazel-discuss

Thanks a lot Laszlo!
test.db is just a dummy hack so i can "chmod", is there any C only solution?
i thought of maybe creating from within the test a copy of "ciri.db"
and use it since it's only for test purposes anyways and it shouldn't be persistent..

László Csomor

unread,

Jul 27, 2018, 5:19:56 AM7/27/18

to tsvi...@iota.org, bazel-discuss

> test.db is just a dummy hack so i can "chmod", is there any C only solution?

What do you mean by C only solution, and for what?

> i thought of maybe creating from within the test a copy of "ciri.db"

> and use it since it's only for test purposes anyways and it shouldn't be persistent..

Yes, that sounds good. Do not mutate input files and data dependencies -- in the lucky case the dependency is a generated file and attempting to write it fails (as you have seen), but in the unlucky case it is a file in your source tree...

Bazel always defines $TEST_TMPDIR for tests which is guaranteed to be writable. (Watch out where you write in it though: this directory may not be unique to the test, it may in fact be "/tmp".)

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/c8c8c9cf-85b7-4e74-8820-0a064ae8cd5a%40googlegroups.com.

Reply all

Reply to author

Forward