implement rpm repository

[Jack47]

My C++ projects use libraries released on yum repository. rpm files are downloaded and uncompressed. Binary will use C++ header files and link .a or .so files from the uncompressed rpm package. I wonder how can I implement this process.
I can implement "new_http_rpm" just like the Bazel's "new_http_archive" rule and use BUILD file to specify which files we need. The another way is implement it as a skylark rule, input is rpm file name, output is downloaded and uncompressed files in rpm, and import to bazel using rule "new_local_repository". I believe implement "new_http_rpm" is better. Which way you prefer, or there is another better way to do this?

[Damien Martin-guillerez]

Are RPM zip with a standard layout? Can you automatically create the target corresponding to the rpm file depending on its content? If so then it might make sense to have a new_http_rpm but it won't be easy because we don't support creating remote repository rule using skylark so it has to be native java and I wonder if it's important enough to have a native support.

On Wed, May 6, 2015 at 1:55 AM Jack47 <csd...@126.com> wrote:

        My C++ projects use libraries released on yum repository. rpm files are  downloaded and uncompressed. Binary will use C++ header files and link .a or .so files from the uncompressed rpm package. I wonder how can I implement this process.
        I can implement "new_http_rpm" just like the Bazel's "new_http_archive" rule and use BUILD file to specify which files we need. The another way is implement it as a skylark rule, input is rpm file name, output is downloaded and uncompressed files in rpm, and import to bazel using rule "new_local_repository". I believe implement "new_http_rpm" is better. Which way you prefer, or there is another better way to do this?

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/53434416-D30B-4840-98FD-BDE1D838FDF7%40126.com.
For more options, visit https://groups.google.com/d/optout.

[Jack47]

Thanks for your reply, Damien.

I believe rpm is not a zip layout, the command to extract rpm package is :

rpm2cpio php-5.1.4-1.esp1.x86_64.rpm | cpio -idmv

I need to use a BUILD file to specify files in rpm which are needed in project. Now that skylark rule don't support creating remote repository, I will implement this feature in native java.  This feature is very important for me.

[Damien Martin-guillerez]

Sounds reasonable. You might want to discuss with Kristina on how to do it first though.

[Han-Wen Nienhuys]

Have you tried generating the .a files from the .rpm files using a
genrule? While that won't let you download the rpm files
automatically, it should provide you with most of the functionality
you need.

> https://groups.google.com/d/msgid/bazel-discuss/CAN3hOS-s_TSS111ydQPHHY6cd%3DWKz63K7YFnmFXPZ-9u0Fk-cA%40mail.gmail.com.

>
> For more options, visit https://groups.google.com/d/optout.

--
Han-Wen Nienhuys
Google Munich
han...@google.com

[Jack47]

Yes, use genrule to execute shell commands can extract .a files from .rpm files. But these .rpm files is stored on the yum repository, Bazel need to fetch it from yum repo. What's more, there are tens of rpm files and these rpm files is related, e.g, A.rpm need B.rpm, B.rpm needs C.rpm. Bazel should compute these dependencies, I need Bazel download these rpm files as concurrently as possible and cache the output result.

@ Kristina

I designed a new_http_rpm rule:
new_rpm_repository (
name = "search-utility-2.4.1", # name is the rpm file name, need to specify the exact rpm version
branch = "test", # test release or product release, default is product release
sha256 = "" # if not specified, will not compare sha256, so if search-utility-2.4.1_x86_64.rpm exists in the .external-repository, Bazel don't need to refetch the rpm from yum repository.
deps = "string-utility-1.0.0", # dependencies of this rule
build_file = "search-utility-2.4.1.BUILD" # A file to use as a BUILD file for this rpm repository.
)

To implement this feature, I need to implement these functions:
1. RpmDownloadFunction: find the rpm's download url according to the yum repository configs and then fetch this rpm to local directory.
2. implement rpm decompress class in DecompressFactory
3. import build file specified by build_file attribute.

If there is something wrong with my design, please correct me.

[Han-Wen Nienhuys]

How do you resolve dependencies? IIUC, you'll have to interpret the
format of both RPM (the dependencies field) and the format of the YUM
repository metadata. IIRC, the standard logic for this is written in
Python and/or C, so hooking this up with Bazel (Java) will be somewhat
challenging.

I'm a little concerned that on the long run, we will end up having to
implement support for every repository/packager under the sun (Suse,
debian apt-get, NPM docker, etc.), which is probably something Bazel
(a build system) should not attempt.

Kristina, is there a way to generalize remote repository support so we
don't have to implement the dependency/unpack logic inside the Bazel
Java code? Maybe we could have an interface to external programs or
skylark code that will implement the format specific logic?

[Lukács T. Berki]

It would be best if we could make this so that external repository formats can be implemented in Skylark. It's a bit hairy, though, because this by its nature requires talking to the external world, to which task Skylark in its current incarnation is not really suited for. The three options I see are:

1. Extend Skylark with network and file access functionality (available only to external repository implementations) and hope that we don't need Turing-completeness.
2. Make the interface to external repository implementations be a shell script or some other executable. This would make it hard to eventually port them to Windows, because bash is something alien to Windows machines. We'd need to package our own interpreter for hermeticity.
3. Make the interface be in Java, and let Bazel dynamically load .jar files or somesuch.

Out of these alternatives, I think (1) is the least bad.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAFQ2z_MM%3DCC0z2m9o5pEE99wSgSX%3DE0mb06LyxT7V_5jHZOvwg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Dienerstrasse 12 | 80331 München | Germany | Geschäftsführer: Graham Law, Christine Elizabeth Flores | Registergericht und -nummer: Hamburg, HRB 86891

Diese E-Mail ist vertraulich. Wenn Sie nicht der richtige Adressat sind, leiten Sie diese bitte nicht weiter, informieren Sie den Absender und löschen Sie die E-Mail und alle Anhänge. Vielen Dank.

       

This e-mail is confidential. If you are not the right addressee please do not forward it, please inform the sender, and please erase this e-mail including any attachments. Thanks.

[Jack47]

Yes, there should be a way to generalize remote repository support. RPM is used as a archive file, and rpm is just like the zip archive hosted on a remote server. Developer needs to inform Bazel the rpm's dependency by deps attribute. So we don't need Bazel to interpret the format of RPM's dependencies field.

[Han-Wen Nienhuys]

For the short-term, I think it would be best if we could just download
individual artifacts/source files as files. That doesn't offer
dependency management, but it will let you extract .a files from an
external .rpm in an automated fashion.

[Han-Wen Nienhuys]

On Thu, May 7, 2015 at 4:48 PM, Lukács T. Berki <lbe...@google.com> wrote:
> It would be best if we could make this so that external repository formats
> can be implemented in Skylark. It's a bit hairy, though, because this by its
> nature requires talking to the external world, to which task Skylark in its
> current incarnation is not really suited for. The three options I see are:
>
> Extend Skylark with network and file access functionality (available only to
> external repository implementations) and hope that we don't need
> Turing-completeness.

> Make the interface to external repository implementations be a shell script
> or some other executable. This would make it hard to eventually port them to
> Windows, because bash is something alien to Windows machines. We'd need to
> package our own interpreter for hermeticity.

> Make the interface be in Java, and let Bazel dynamically load .jar files or
> somesuch.
>
> Out of these alternatives, I think (1) is the least bad.

I don't see the problem with interfacing over the command line. Bazel
(which invokes compilers) is almost nothing but CLI interfaces. If
you're concerned about portability, a Python script would be
relatively easy to make portable.

[Lukács T. Berki]

Thinking again, we do have to make it possible to run non-hermetic binaries, since, well, that's how one discovers information about the host system. And given that we cannot prohibit people from invoking a binary that just writes the appropriate BUILD/WORKSPACE/whatever files into the places Bazel expects them to be written and that it's an obvious way understand archive formats (e.g. rpm) and network protocols (e.g. HTTP), I think the right thing to do is to just shell out to a binary, maybe with a light Skylark wrapper to e.g. decide which OS-specific binary to invoke and to tell Bazel about the new rules.

 

[Damien Martin-guillerez]

So, can't we just provide the possibility to create skylark extension for remote repository rules where the context given to those allows to call outside binary and do a lot of crazy stuff in order to generate a remote repository structure?

As you mentioned in the other thread, that would allow custom-made remote repository and would solve most of the issue for bazel init at the same time.

The only minor drawback I see is that we don't provide examples of tools directory anymore (it's probably fixable by having a generation flag if that's really a problem).

[Lukács T. Berki]

On Mon, May 11, 2015 at 11:42 AM, Damien Martin-guillerez <dmar...@google.com> wrote:

So, can't we just provide the possibility to create skylark extension for remote repository rules where the context given to those allows to call outside binary and do a lot of crazy stuff in order to generate a remote repository structure?

+1. I don't know how much work this is on the Skylark side, but Laurent certainly does.

As you mentioned in the other thread, that would allow custom-made remote repository and would solve most of the issue for bazel init at the same time.

The only minor drawback I see is that we don't provide examples of tools directory anymore (it's probably fixable by having a generation flag if that's really a problem).

Excuse me?

[Damien Martin-guillerez]

With the autoconf prototype we have, we generate a tools directory that could be used as starting point for people wanting to make a hermetic repo. It is not important and if it is becoming important then we can probably derive a flag / command to generate one based on a WORKSPACE file. Anyway that is not important.  

[Lukács T. Berki]

Oh, I see. I don't think this is a big deal, either.