How to generate files in rule

[Andrew Katson]

Good afternoon, 

I am working on getting openssl working for bzlmod on Windows in https://github.com/raccoons-build/bazel-openssl-cc/pull/5. 

I wrote a custom rule that uses the perl scripts provided per architecture and os by openssl to generate assembly files needed by the :crypto target. Unfortunately, even though I use ctx.actions.declare_file and ctx.actions.run_shell and it all succeeds when I query for generated files from the rule I get empty output. 

Can someone look at my "perl_genrule" and see if there is anything obvious that I am doing wrong here? 

I read the rule tutorial docs and the rule succeeds so I am not quite sure what could be the issue. Shouldn't it fail if no files are output?

[Filip Filmar]

Did you print the contents of the variable `out_files` before the `return` in `_perl_genrule_impl` to verify that you actually have something in it?

You get an error if DefaultInfo.files is nonempty, and you don't have those files generated. But if out_files = [], it will all be silent.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/bazel-discuss/876c6090-79e4-4781-b0a5-85810a57bea1n%40googlegroups.com.

[Andrew Katson]

This is what is printed when I print the list of different info files

DEBUG: /home/runner/.cache/bazel/_bazel_runner/64369e5247be6db573f761c18506a749/external/openssl++_repo_rules+openssl-generated-overlay/perl_genrule.bzl:64:10: katsonandrew: [struct(data_runfiles = None, default_runfiles = None, files = depset([<generated file external/openssl+/crypto/aes/aes-x86_64.s>, <generated file external/openssl+/crypto/aes/aesni-mb-x86_64.s>, <generated file external/openssl+/crypto/aes/aesni-sha1-x86_64.s>, <generated file external/openssl+/crypto/aes/aesni-sha256-x86_64.s>, <generated file external/openssl+/crypto/aes/aesni-x86_64.s>, <generated file external/openssl+/crypto/aes/bsaes-x86_64.s>, <generated file external/openssl+/crypto/aes/vpaes-x86_64.s>, <generated file external/openssl+/crypto/bn/rsaz-2k-avx512.s>, <generated file external/openssl+/crypto/bn/rsaz-3k-avx512.s>, <generated file external/openssl+/crypto/bn/rsaz-4k-avx512.s>, <generated file external/openssl+/crypto/bn/rsaz-avx2.s>, <generated file external/openssl+/crypto/bn/rsaz-x86_64.s>, <generated file external/openssl+/crypto/bn/x86_64-gf2m.s>, <generated file external/openssl+/crypto/bn/x86_64-mont.s>, <generated file external/openssl+/crypto/bn/x86_64-mont5.s>, <generated file external/openssl+/crypto/camellia/cmll-x86_64.s>, <generated file external/openssl+/crypto/chacha/chacha-x86_64.s>, <generated file external/openssl+/crypto/ec/ecp_nistz256-x86_64.s>, <generated file external/openssl+/crypto/ec/x25519-x86_64.s>, <generated file external/openssl+/crypto/md5/md5-x86_64.s>, <generated file external/openssl+/crypto/modes/aes-gcm-avx512.s>, <generated file external/openssl+/crypto/modes/aesni-gcm-x86_64.s>, <generated file external/openssl+/crypto/modes/ghash-x86_64.s>, <generated file external/openssl+/crypto/poly1305/poly1305-x86_64.s>, <generated file external/openssl+/crypto/rc4/rc4-md5-x86_64.s>, <generated file external/openssl+/crypto/rc4/rc4-x86_64.s>, <generated file external/openssl+/crypto/sha/keccak1600-x86_64.s>, <generated file external/openssl+/crypto/sha/sha1-mb-x86_64.s>, <generated file external/openssl+/crypto/sha/sha1-x86_64.s>, <generated file external/openssl+/crypto/sha/sha256-mb-x86_64.s>, <generated file external/openssl+/crypto/sha/sha256-x86_64.s>, <generated file external/openssl+/crypto/whrlpool/wp-x86_64.s>, <generated file external/openssl+/crypto/x86_64cpuid.s>, <generated file external/openssl+/engines/e_padlock-x86_64.s>, <generated file external/openssl+/crypto/sha/sha512-x86_64.s>]), files_to_run = None), struct(compilation_context = <unknown object com.google.devtools.build.lib.rules.cpp.CcCompilationContext>, linking_context = <unknown object com.google.devtools.build.lib.rules.cpp.CcLinkingContext>)]

I ran this command from the openssl directory on my Windows machine

perl crypto/aes/asm/aes-x86_64.pl nasm crypto/aes/aes-x86_64.s

And it generated the file with no issues.

--

Andrew Katson

[Filip Filmar]

If DefaultInfo is not empty, I would expect that you either have these files generated, or if the rule somehow didn't generate them, you'd get an error message.

Now, I am not quite sure what is meant by "when I query for generated files from the rule I get empty output." So I'll give some general direction.

In this situation, I would usually use the flags `--subcommands` and `--sandbox_debug` when building, to see which files end up being put where.

F

[Andrew Katson]

What I meant is I run this query on the rule to see if it generates anything 

 bazel query --registry="file:///tmp/bazel-central-registry" 'kind("generated file", @openssl//:perlasm_genfiles)'

And it gives me empty results. 

Generation:
```
# Configuration: 0de042f7f216bd39770efda4fe9ac141f679167720307fb3637d83903e80fc71
# Execution platform: @@platforms//host:host
SUBCOMMAND: # @@openssl+//:perlasm_genfiles [action 'Generating file bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/sm4/sm4-armv8.S from script external/openssl+/crypto/aes/asm/vpaes-armv8.pl', configuration: 0de042f7f216bd39770efda4fe9ac141f679167720307fb36[37](https://github.com/raccoons-build/bazel-openssl-cc/actions/runs/13291627372/job/37113681694?pr=5#step:5:38)d83903e80fc71, execution platform: @@platforms//host:host, mnemonic: GenerateAssemblyFromPerlScripts]
(cd /private/var/tmp/_bazel_runner/c973e65f821f1a9c531cafa0f1d745d7/execroot/_main && \
  exec env - \
  /bin/bash -c 'perl external/openssl+/crypto/aes/asm/vpaes-armv8.pl ios64 bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/sm4/sm4-armv8.S')
```

Error:
```
ERROR: /private/var/tmp/_bazel_runner/c973e65f821f1a9c531cafa0f1d745d7/external/openssl+/BUILD.bazel:388:11: Compiling crypto/sm4/sm4-armv8.S failed: missing input file '@@openssl+//:crypto/sm4/sm4-armv8.S'
ERROR: /private/var/tmp/_bazel_runner/c973e65f821f1a9c531cafa0f1d745d7/external/openssl+/BUILD.bazel:388:11: Compiling crypto/sm4/sm4-armv8.S failed: 1 input file(s) do not exist
```

--

Andrew Katson

[Filip Filmar]

I am not an expert on the CcInfo provider, but I would check if the issue may be related to you putting the out_files (assembly files IIUC) in the `headers` attribute.

Headers are special because they do *not* appear in the compilation command line, but they are available in the build sandbox.  If indeed CcInfo is considering them to be headers, they won't take part in any command lines. This is why you want to establish the exact command lines used, the `--subcommands` flag is your friend.

F

On Wed, Feb 12, 2025 at 2:28 PM Andrew Katson <katsona...@gmail.com> wrote:

Oh but if i use cquery instead of query I do get files. So it is working! Just need to figure out why my cc_library cannot see it

INFO: Analyzed target @@openssl+//:perlasm_genfiles (7 packages loaded, 51 targets configured).
INFO: Found 1 target...
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/aes/aesv8-armx.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/aes/bsaes-armv8.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/aes/vpaes-armv8.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/arm64cpuid.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/bn/armv8-mont.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/chacha/chacha-armv8.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/ec/ecp_nistz256-armv8.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/ec/ecp_sm2p256-armv8.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/md5/md5-aarch64.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/modes/aes-gcm-armv8-unroll8_64.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/modes/aes-gcm-armv8_64.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/poly1305/poly1305-armv8.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/sha/keccak1600-armv8.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/sha/sha1-armv8.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/sha/sha256-armv8.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/sha/sha512-armv8.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/sm3/sm3-armv8.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/sm4/sm4-armv8.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/sm4/vpsm4-armv8.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/sm4/vpsm4_ex-armv8.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/modes/ghashv8-armx.S

--

Andrew Katson

[Filip Filmar]

I do not know anything about the specific genrules you are looking at.  I'm trying to help based on the general shape of what you attempted and the general shape of what comes out of it.

So far we established

• We resolved the issue of files not being generated. My understanding is that the files were, in fact, being generated, and that the method to look them up needed to be changed.
• The current issue is something about CcInfo.   I took a look at CcInfo, and it seems that I'm wrong in the sense that CcInfo seems to not be the place for sources at all. I suppose knowing what's in CcInfo is maybe table stakes for a bazel user, but I just don't compile that much C++ directly.
• What further issues do you observe? Can you share here what you tried and what you observed? "input files do not exist" can indicate that the files are not in the sandbox used for compilation, which is all too easy to happen. Building with the flag `--sandox_debug` will help you figure out what the compilation sandbox looks like.  I usually look up the top level dir for the sandbox and do `tree <that_top_level_dir>` to see the structure of the sandbox at a glance.

On Wed, Feb 12, 2025 at 3:07 PM Andrew Katson <katsona...@gmail.com> wrote:

Do you happen to know what the genrule does? Like the default one? I've looked at the source and it is fairly short so I couldn't glean much. But the mac and linux generation was handled by a genrule and that didn't work for Windows which is why I am trying this way. I will look at the cc info again and see if there is a more appropriate field! 

--

Andrew Katson

[Andrew Katson]

Do you happen to know what the genrule does? Like the default one? I've looked at the source and it is fairly short so I couldn't glean much. But the mac and linux generation was handled by a genrule and that didn't work for Windows which is why I am trying this way. I will look at the cc info again and see if there is a more appropriate field! 

--

Andrew Katson

[Andrew Katson]

Oh but if i use cquery instead of query I do get files. So it is working! Just need to figure out why my cc_library cannot see it

INFO: Analyzed target @@openssl+//:perlasm_genfiles (7 packages loaded, 51 targets configured).
INFO: Found 1 target...
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/aes/aesv8-armx.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/aes/bsaes-armv8.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/aes/vpaes-armv8.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/arm64cpuid.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/bn/armv8-mont.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/chacha/chacha-armv8.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/ec/ecp_nistz256-armv8.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/ec/ecp_sm2p256-armv8.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/md5/md5-aarch64.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/modes/aes-gcm-armv8-unroll8_64.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/modes/aes-gcm-armv8_64.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/poly1305/poly1305-armv8.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/sha/keccak1600-armv8.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/sha/sha1-armv8.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/sha/sha256-armv8.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/sha/sha512-armv8.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/sm3/sm3-armv8.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/sm4/sm4-armv8.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/sm4/vpsm4-armv8.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/sm4/vpsm4_ex-armv8.S
bazel-out/darwin_arm64-fastbuild/bin/external/openssl+/crypto/modes/ghashv8-armx.S

--

Andrew Katson

[Andrew Katson]

I actually have no idea what CcInfo is supposed to look like either. I've been trawling Github trying to figure out what ways people do it and so I added my assembly files to `direct_private_headers` so that they could be considered sources. I am actually running with that. This is the full command I run: Run mkdir workspace && cd workspace && echo 8.0.0 > .bazelversion && echo 'bazel_dep(name = "openssl", version = "3.3.1.bcr.nix.wip")' > MODULE.bazel && bazel test --verbose_failures --sandbox_debug -s --registry="file:///tmp/bazel-central-registry" -k @openssl//...

As far as tree goes here is the output: 

.
└── workspace
    ├── MODULE.bazel
    ├── MODULE.bazel.lock
    ├── bazel-bin -> /home/runner/.cache/bazel/_bazel_runner/64369e5247be6db573f761c18506a749/execroot/_main/bazel-out/k8-fastbuild/bin
    ├── bazel-out -> /home/runner/.cache/bazel/_bazel_runner/64369e5247be6db573f761c18506a749/execroot/_main/bazel-out
    ├── bazel-testlogs -> /home/runner/.cache/bazel/_bazel_runner/64369e5247be6db573f761c18506a749/execroot/_main/bazel-out/k8-fastbuild/testlogs
    └── bazel-workspace -> /home/runner/.cache/bazel/_bazel_runner/64369e5247be6db573f761c18506a749/execroot/_main

I ran it on the next step with continue-on-error so this should be everything. But oddly it stops at the symbolic links....

So next I will try tree /home/runner/.cache/ to see what is in the different bazel-* dirs. 

--

Andrew Katson

[Andrew Katson]

Printing out all the assembly files is here: https://github.com/raccoons-build/bazel-openssl-cc/actions/runs/13315068118/job/37187027044?pr=5

Here is a snippet. 

 ├── aesni-mb-x86_64.s -> /home/runner/.cache/bazel/_bazel_runner/64369e5247be6db573f761c18506a749/execroot/_main/bazel-out/k8-fastbuild/bin/external/openssl+/crypto/aes/aesni-mb-x86_64.s

--

Andrew Katson

[Filip Filmar]

This is going plenty in the weeds. Do you have a link to a hermetic repro?

[Andrew Katson]

I made one! https://github.com/andrewkatson/perl_genrule_problem.git

If you look at the github actions you will see the same error I get now.

--

Andrew Katson

[Filip Filmar]

Well, your assembly.pl is lying to you.

You tell it to put the output file to ./bazel-out/k8-fastbuild/bin/assembly.s, but it puts it at ./assembly.s.

I don't know if this is your actual issue, or a min repro case of the actual issue, but that is a problem with this build.

F

(exec env - \
    TMPDIR=/tmp \
  /home/fmil/.cache/bazel/_bazel_fmil/install/0b6397354bedad4359bdae4332ce8fc5/linux-sandbox -t 15 -w /dev/shm -w /home/fmil/.cache/bazel/_bazel_fmil/55ab4d922f0691b0d240e2cf00330667/sandbox/linux-sandbox/3/execroot/_main -w /tmp -M /home/fmil/.cache/bazel/_bazel_fmil/55ab4d922f0691b0d240e2cf00330667/sandbox/linux-sandbox/3/_hermetic_tmp -m /tmp -S /home/fmil/.cache/bazel/_bazel_fmil/55ab4d922f0691b0d240e2cf00330667/sandbox/linux-sandbox/3/stats.out -D /home/fmil/.cache/bazel/_bazel_fmil/55ab4d922f0691b0d240e2cf00330667/sandbox/linux-sandbox/3/debug.out -- /bin/sh -i)
ERROR: /home/fmil/tmp/perl_genrule_problem/BUILD:4:13: Generating file bazel-out/k8-fastbuild/bin/assembly.s from script assembly.pl failed: (Exit 1): linux-sandbox failed: error executing GenerateAssemblyFromPerlScripts command
  (cd /home/fmil/.cache/bazel/_bazel_fmil/55ab4d922f0691b0d240e2cf00330667/sandbox/linux-sandbox/3/execroot/_main && \
  exec env - \
    TMPDIR=/tmp \
  /home/fmil/.cache/bazel/_bazel_fmil/install/0b6397354bedad4359bdae4332ce8fc5/linux-sandbox -t 15 -w /dev/shm -w /home/fmil/.cache/bazel/_bazel_fmil/55ab4d922f0691b0d240e2cf00330667/sandbox/linux-sandbox/3/execroot/_main -w /tmp -M /home/fmil/.cache/bazel/_bazel_fmil/55ab4d922f0691b0d240e2cf00330667/sandbox/linux-sandbox/3/_hermetic_tmp -m /tmp -S /home/fmil/.cache/bazel/_bazel_fmil/55ab4d922f0691b0d240e2cf00330667/sandbox/linux-sandbox/3/stats.out -D /home/fmil/.cache/bazel/_bazel_fmil/55ab4d922f0691b0d240e2cf00330667/sandbox/linux-sandbox/3/debug.out -- ./perl_generate_file.sh perl assembly.pl bazel-out/k8-fastbuild/bin/assembly.s elf)
Assembly code has been written to assembly.s
bazel-out/k8-fastbuild/bin/assembly.s does not exist failing
Target //:perlasm_genfiles failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.105s, Critical Path: 0.02s
INFO: 2 processes: 2 internal.
ERROR: Build did NOT complete successfully
FAILED:
    Fetching repository @@rules_kotlin+; starting
    Fetching repository @@rules_java++toolchains+local_jdk; starting
┬─[fmil@fmil9:~/tmp/perl_genrule_problem]─[04:40:26 PM]
│ (g/b:main)
╰─>$ tree /home/fmil/.cache/bazel/_bazel_fmil/55ab4d922f0691b0d240e2cf00330667/sandbox/linux-sandbox/3/execroot/_main
/home/fmil/.cache/bazel/_bazel_fmil/55ab4d922f0691b0d240e2cf00330667/sandbox/linux-sandbox/3/execroot/_main
├── assembly.pl -> /home/fmil/.cache/bazel/_bazel_fmil/55ab4d922f0691b0d240e2cf00330667/execroot/_main/assembly.pl
├── assembly.s
├── bazel-out
│   └── k8-fastbuild
│       └── bin
└── perl_generate_file.sh -> /home/fmil/.cache/bazel/_bazel_fmil/55ab4d922f0691b0d240e2cf00330667/execroot/_main/perl_generate_file.sh

[Filip Filmar]

Another issue is, executable files should have the executable bit set.

[Filip Filmar]

On Thu, Feb 13, 2025 at 6:02 PM Andrew Katson <katsona...@gmail.com> wrote:

the others do not. So I think my genrule is just overwriting every time. Does that sound like something that can happen? 

Well, in run_generation, you have only one `out_as_file` for a loop of files.  If you didn't intend that, that would most likely result in things being overwritten.

[Filip Filmar]

Fixed at: https://github.com/andrewkatson/perl_genrule_problem/pull/1/files

[Andrew Katson]

I see.... I think it's just an error with the repro because when I use `tree` in the generation shell script it prints this out 

SUBCOMMAND: # @@openssl+//:perlasm_genfiles [action 'Generating file bazel-out/k8-fastbuild/bin/external/openssl+/crypto/bn/rsaz-avx2.s from script external/openssl+/crypto/bn/asm/rsaz-avx2.pl', configuration: 3723722276aac1314320ca9a9fb2a278d80e4eb3b422da8939e984231a537159, execution platform: @@platforms//host:host, mnemonic: GenerateAssemblyFromPerlScripts]
(cd /home/runner/.cache/bazel/_bazel_runner/64369e5247be6db573f761c18506a749/execroot/_main && \
  exec env - \
  external/openssl++_repo_rules+openssl-generated-overlay/perl_generate_file.sh perl external/openssl+/crypto/bn/asm/rsaz-avx2.pl bazel-out/k8-fastbuild/bin/external/openssl+/crypto/bn/rsaz-avx2.s elf)
# Configuration: 3723722276aac1314320ca9a9fb2a278d80e4eb3b422da8939e984231a537159

# Execution platform: @@platforms//host:host

INFO: From Generating file bazel-out/k8-fastbuild/bin/external/openssl+/crypto/bn/rsaz-4k-avx512.s from script external/openssl+/crypto/bn/asm/rsaz-4k-avx512.pl:
sh: 0: Illegal option --
sh: 0: Illegal option -
bazel-out/k8-fastbuild/bin/external/openssl+
`-- crypto
    `-- bn
        `-- rsaz-4k-avx512.s

HOWEVER, I noticed that every shell script sees a different "bazel-out/k8-fastbuild/bin/external/openssl+" somehow. Like each time it runs the currently generated file exists but the others do not. So I think my genrule is just overwriting every time. Does that sound like something that can happen? 

--

Andrew Katson

[Andrew Katson]

Wow!!! Thank you so much. I think this addresses my issue. 

--

Andrew Katson

[Filip Filmar]

Glad it worked. It would be interesting to read your summary of the issue here, when you find the time.

F

[Andrew Katson]

Ok! I am ready to do a write up now that the linux test shard is passing again. 

Here were the problems (thank you to Filip!)

1. Running over the srsc.files.to_list() let to redeclaration of the outfile so it was just causing unnecessary cycles

2. Running the generation script all in shell didn't work for Windows so a powershell script was needed

3. Running the generation script with the wrong assembly flavor led to files not generating

4. Running the generation script with the wrong output file was dumping them to the wrong place

5.  (Most importantly) The generation script was being run in a loop so each time it ran it would overwrite the directory it was putting the current assembly file into

Again, thank you to Filip for his help and patience!

--

Andrew Katson