patching sources of external repo

1,203 views
Skip to first unread message

Gregg Reynolds

unread,
May 13, 2018, 1:45:13 PM5/13/18
to bazel-discuss
Hi,

I have an external repo downloading and compiling just fine.  Problem is I need to apply a patch to it before compiling.  I'm not having any luck; the patch cmd runs but there are no input files.  When I examine the execroot using:

genrule(
    name = "gentest",
    srcs = glob(["include/**/*h"]),
    outs = ["gentest.log"],
    #cmd  = "pwd > $@"
    #cmd  = "ls -l external/mbedtls/include/mbedtls > $@"
    #cmd  = "ls -l bazel-out/darwin-fastbuild/genfiles/external/mbedtls > $@"
    cmd  = "find -L bazel-out -name *.h > $@"
)

I find no files from the external repo.

Is there a way to do this?

Thanks,

Gregg

My BUILD.mbedtls file looks like:

cc_library(
    name = "mbedtls-lib",
    copts = [
        # the library headers:
        "-Iexternal/mbedtls/include/mbedtls",
        "-Iexternal/mbedtls/include/",

        # custom config.h:
        "-Iconfig",
        "-Iconfig/mbedtls",
    ],
    srcs = glob(["library/*.c"]),
    hdrs = glob(["include/mbedtls/*.h"],
                exclude=["include/mbedtls/config.h"])
    + ["@//config/mbedtls:config.h",
    ],

    visibility = ["//visibility:public"]
)


genrule(
    name = "patch",
    srcs = [
        "include/mbedtls/certs.h",
        ...
         "@//config/mbedtls:ocf.patch"],
    outs = ["out/include/mbedtls/certs.h" ...],
    cmd = "patch -p1 -l -f < $(location @//config/mbedtls:ocf.patch"
)

Brian Silverman

unread,
May 13, 2018, 2:13:11 PM5/13/18
to Gregg Reynolds, bazel-discuss
On Sun, May 13, 2018 at 10:45 AM, Gregg Reynolds <d...@mobileink.com> wrote:
Hi,

I have an external repo downloading and compiling just fine.  Problem is I need to apply a patch to it before compiling.  I'm not having any luck; the patch cmd runs but there are no input files.  When I examine the execroot using:
I'm going to assume your external repo is called @mbedtls in my answers, although I'm not completely sure. 

genrule(
    name = "gentest",
    srcs = glob(["include/**/*h"]),
    outs = ["gentest.log"],
    #cmd  = "pwd > $@"
    #cmd  = "ls -l external/mbedtls/include/mbedtls > $@"
    #cmd  = "ls -l bazel-out/darwin-fastbuild/genfiles/external/mbedtls > $@"
    cmd  = "find -L bazel-out -name *.h > $@"
)

I find no files from the external repo.
What package is that genrule in? If it's in @mbedtls, then `find -L include -name *.h` should find the files. Otherwise, that glob is going to be empty (because glob doesn't match files in other packages) and it's not going to find anything.

More generally, after building, you can find the files for @mbedtls in bazel-out/../../../external/mbedtls (where bazel-out is the convenience symlink that ends up in your workspace).

Also, a tool I find helpful for debugging globs is `bazel query --output=build`, which will print your BUILD files with globs (and macros as a side effect) expanded. For example, try `bazel query --output=build @mbedtls//...` to see what the globs in your cc_library target are matching.

My BUILD.mbedtls file looks like:

cc_library(
    name = "mbedtls-lib",
    copts = [
        # the library headers:
        "-Iexternal/mbedtls/include/mbedtls",
        "-Iexternal/mbedtls/include/",
I think one or both of these need to be in includes instead. It seems like dependent rules are going to need to find these files too.

        # custom config.h:
        "-Iconfig",
        "-Iconfig/mbedtls",
I'm not sure these are going to find the folder[s] you want. Having -I paths in different packages is kind of unusual. Paths in the includes attr are definitely relative to the current package and can't include .., and I think these need to be in includes anyways. I would make a @//config/mbedtls:config_h cc_library which has `includes = ['.']` and put that in deps of this rule instead.
    ],
    srcs = glob(["library/*.c"]),
    hdrs = glob(["include/mbedtls/*.h"],
                exclude=["include/mbedtls/config.h"])
    + ["@//config/mbedtls:config.h",
    ],
I think you also need to exclude `include/mbedtls/certs.h` from your glob, and explicitly include `:patch` in hdrs.

    visibility = ["//visibility:public"]
)
`includes = ['out/include'] might also be necessary, depending on what path the other source files use to #include certs.h.


genrule(
    name = "patch",
    srcs = [
        "include/mbedtls/certs.h",
        ...
         "@//config/mbedtls:ocf.patch"],
    outs = ["out/include/mbedtls/certs.h" ...],
    cmd = "patch -p1 -l -f < $(location @//config/mbedtls:ocf.patch"
)
Have you tried `bazel build @mbedtls//:patch` and then looked at the output (path(s) will be printed as normal according to --show_result) to verify the patch command is actually working? It looks to me like it's trying to patch the input file in place, which isn't going to work. The way I've dealt with this before was copying the input to the output, and then running patch on that just-created output file.

I've done something similar before, except with .c files so they could just go in the same directory under different names. .h files get #included, so they need to be in a different directory under the same name, which makes it a bit tricker. I also kept the patches inline in the BUILD.mbedtls file to minimize the number of files to coordinate, which you might consider if your patches are short. This is what it looked like:
_xerbla_patch = '''
--- xerbla.c   2018-02-19 19:58:03.685420156 -0500
+++ xerbla.c   2018-02-19 19:59:02.993259128 -0500
@@ -55,7 +55,7 @@ static integer c__1 = 1;
 /*     .. */
 /*     .. Executable Statements .. */
 
-    printf("** On entry to %6s, parameter number %2i had an illegal value\\n",
+    printf("** On entry to %6s, parameter number %2li had an illegal value\\n",
\t\tsrname, *info);
 
 
'''
genrule(
  name = 'patch_xerbla',
  srcs = ['SRC/xerbla.c'],
  outs = ['patched_xerbla.c'],
  cmd = '\n'.join([
    'cp $< $@',
    'patch $@ - <<END',
    _xerbla_patch,
    'END',
  ]),
)

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAO40Mi%3D6JK%2B5P5UcdmNys0tGAnNQU8yuYotWk9JqS07%3Dgmn6MQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Jingwen Chen

unread,
May 13, 2018, 4:40:57 PM5/13/18
to Brian Silverman, Gregg Reynolds, bazel-discuss, aeh...@google.com
+aehlig

Klaus added the ability to apply patches to the Skylark http_archive implementation [1] a few months ago. Does this work for you?

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAP01z6KNwt20KjF61_B%2B9qmWntbS6ybtFzAwgB8WucvwKKJOBw%40mail.gmail.com.

Gregg Reynolds

unread,
May 13, 2018, 11:46:21 PM5/13/18
to Jingwen Chen, Brian Silverman, bazel-discuss, aeh...@google.com
On Sun, May 13, 2018 at 3:40 PM, Jingwen Chen <jin...@google.com> wrote:
+aehlig

Klaus added the ability to apply patches to the Skylark http_archive implementation [1] a few months ago. Does this work for you?

Nope, the patch seems to be broken.  WORKSPACE:

load(
    "@bazel_tools//tools/build_defs/repo:http.bzl",
    "http_archive",
    "http_file",
)

http_archive(
    name = "mbedtls",
    sha256 = "dacb9f5dd438c456b9ef6627637f46e16fd41e86d828217ec9f8547d3d22a338",
    strip_prefix = "mbedtls-mbedtls-2.4.2",
    patches = ["config/mbetdls/ocf.patch"],
    build_file = "config/mbedtls/BUILD",
)

config/mbedtls/BUILD:

genrule(
    name = "gentest",
    srcs = glob(["include/**/*h"]),
    outs = ["gentest.log"],
    cmd  = "find -L includes -name *.h > $@"
)


$ bazel build @mbedtls//:mbedtls-lib
ERROR: Skipping '@mbedtls//:mbedtls-lib': no such package '@mbedtls//': Traceback (most recent call last):
File "/private/var/tmp/_bazel_gar/a2778d8bc5379ccd6c684731e73b4da6/external/bazel_tools/tools/build_defs/repo/http.bzl", line 48
print(("ctx.attr.build_file %s, path %...))))
File "/private/var/tmp/_bazel_gar/a2778d8bc5379ccd6c684731e73b4da6/external/bazel_tools/tools/build_defs/repo/http.bzl", line 49, in print
ctx.path(ctx.attr.build_file)
Not a regular file: /Users/gar/iotivity/iotivity-fork/external/config/mbedtls.BUILD
WARNING: Target pattern parsing failed.


Gregg Reynolds

unread,
May 14, 2018, 12:32:41 AM5/14/18
to Brian Silverman, bazel-discuss
Thanks for the detailed reply.  Comments below.

BTW: bazel version 0.13.0, on OS X 10.12.6.

On Sun, May 13, 2018 at 1:12 PM, Brian Silverman <bsilve...@gmail.com> wrote:
On Sun, May 13, 2018 at 10:45 AM, Gregg Reynolds <d...@mobileink.com> wrote:
Hi,

I have an external repo downloading and compiling just fine.  Problem is I need to apply a patch to it before compiling.  I'm not having any luck; the patch cmd runs but there are no input files.  When I examine the execroot using:
I'm going to assume your external repo is called @mbedtls in my answers, although I'm not completely sure. 

genrule(
    name = "gentest",
    srcs = glob(["include/**/*h"]),
    outs = ["gentest.log"],
    #cmd  = "pwd > $@"
    #cmd  = "ls -l external/mbedtls/include/mbedtls > $@"
    #cmd  = "ls -l bazel-out/darwin-fastbuild/genfiles/external/mbedtls > $@"
    cmd  = "find -L bazel-out -name *.h > $@"
)

I find no files from the external repo.
What package is that genrule in? If it's in @mbedtls, then `find -L include -name *.h` should find the files. Otherwise, that glob is going to be empty (because glob doesn't match files in other packages) and it's not going to find anything.

WORKSPACE:
new_http_archive(
    name = "mbedtls",
    sha256 = "dacb9f5dd438c456b9ef6627637f46e16fd41e86d828217ec9f8547d3d22a338",
    strip_prefix = "mbedtls-mbedtls-2.4.2",
    build_file = "config/mbedtls/BUILD",
)

config/mbedtls/BUILD:

genrule(
    name = "gentest",
    srcs = glob(["include/**/*h"]),
    outs = ["gentest.log"],
    cmd  = "find -L includes -name *.h > $@"
)

Then $ bazel build @mbedtls//:gentest results in:

find: includes: No such file or directory
Target @mbedtls//:gentest failed to build

If I change the cmd to "find -L . -name *.h > $@" then I get lines like this in the output:

./external/mbedtls/include/mbedtls/certs.h
...

Also, a tool I find helpful for debugging globs is `bazel query --output=build`, which will print your BUILD files with globs (and macros as a side effect) expanded. For example, try `bazel query --output=build @mbedtls//...` to see what the globs in your cc_library target are matching.

Thanks, that helps a lot.

After some more poking around I finally got this to work:

genrule(
    name = "patch",
    srcs = glob(["**/*.c"])
         + glob(["**/*.h"])
         + glob(["**/*.data"])
         + glob(["**/*.function"])
         + glob(["**/*.sh"])
         + ["@//config/mbedtls:ocf.patch"],
    outs = ["patch.log"],
    cmd  = "patch -dexternal/mbedtls -p1 -l -f < $(location @//config/mbedtls:ocf.patch) > $@"
)

The srcs must list all the files listed in the patch file.  The patch file is config/mbedtls/ocf.patch

Thanks for your help!

Gregg

Gregg Reynolds

unread,
May 14, 2018, 12:29:04 PM5/14/18
to bazel-discuss
On Sun, May 13, 2018 at 12:45 PM, Gregg Reynolds <d...@mobileink.com> wrote:
Hi,

I have an external repo downloading and compiling just fine.  Problem is I need to apply a patch to it before compiling. 

Got it working, wrote a detailed blog about it:  Bazel: genrule patching an external repo

HTH,

Gregg

Klaus Aehlig

unread,
May 14, 2018, 12:40:32 PM5/14/18
to Gregg Reynolds, Jingwen Chen, Brian Silverman, bazel-discuss
> > Nope, the patch seems to be broken. WORKSPACE:
>
> [...]
>
> http_archive(
> name = "mbedtls",
> urls = ["https://github.com/ARMmbed/mbedtls/archive/mbedtls-2.4.2.zip"],
> sha256 =
> "dacb9f5dd438c456b9ef6627637f46e16fd41e86d828217ec9f8547d3d22a338",
> strip_prefix = "mbedtls-mbedtls-2.4.2",
> patches = ["config/mbetdls/ocf.patch"],
> build_file = "config/mbedtls/BUILD",

build_file has to be a fully qualified label here, starting with "@//" if it is
in the main repository.

> )

--
Klaus Aehlig
Google Germany GmbH, Erika-Mann-Str. 33, 80636 Muenchen
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschaeftsfuehrer: Paul Terence Manicle, Halimah DeLaine Prado
Reply all
Reply to author
Forward
0 new messages