How to use debian package as part of hermitic builds.

[Penugonda chenna reddy]

Dear Bazel Community,

 

 I'd like to incorporate a Debian package into my Bazel build tree without downloading it directly onto the system. If I use http_archive, how will the .deb package be extracted and utilized within the build? Any guidance on this would be appreciated.  

regards

PCReddy

[Filip Filmar]

On Tue, Nov 12, 2024 at 1:46 AM Penugonda chenna reddy <penug...@gmail.com> wrote:

I'd like to incorporate a Debian package into my Bazel build tree without downloading it directly onto the system. If I use http_archive, how will the .deb package be extracted and utilized within the build? Any guidance on this would be appreciated.  

Feel free to try it out.  The docs say that the archive will be downloaded and decompressed.  The spot where the contents of the package will be available is "%WORKSPACE%/bazel-<workspace_name>/external/<external_repo_name>".

It won't be used in any way in the build, unless you provide a BUILD file that makes its contents somehow addressable by the build, and then use those contents somewhere in your workspace. If you do, bazel will insert a WORKSPACE file as well as your provided BUILD file into the above mentioned dir.

Maybe back up a bit: what are you hoping to achieve? It is not immediately obvious to me what one could do with a .deb archive in a bazel build.

F

[Filip Filmar]

On Mon, Nov 18, 2024 at 9:37 PM Penugonda chenna reddy <penug...@gmail.com> wrote:

Let me clarify: Suppose my development environment requires certain apt packages, but I want to achieve hermetic builds without installing these packages at the system level. Is there a way to accomplish this in Bazel? 

This doesn't exist out of the box. Someone has to do the legwork to prepare the build environment this way.  In a professional setting, there's usually a toolchain team that makes this for you.

If you don't have a toolchain team, then you are your own toolchain team.

There are hermetic toolchains available for the frequent flyers such as GCC and clang.  If you want more, especially something that nobody has prepared yet, you're kind of on your own. 

But don't fret.

A while ago I showed how one can do this in a fairly universal way, relying on an ephemeral install of nix. 

The linked article explains how this approach differs from similar approaches that rely on a system-wide install of nix.

To illustrate the approach, here's a video that shows a one-liner bazel command, which downloads all the needed software, then builds, places, routes and programs a physical FPGA device.

At the beginning of the process, the only piece of software installed on the machine is bazel. At the end you can see the FPGA sending a greeting via the built-in UART module.

This generalizes well, at least for at-home workloads. Here's another video, showing a "zero to waveform viewer" workflow running a pre-synthesis simulation of a DDR3 memory controller, and the resulting signal waveform. Again, none of the tools used are installed on the system.

The setup is completely ephemeral and made automatically by bazel. This includes installing commercial software such as Vivado.

One `bazel clean --expunge` and everything is wiped away.  Next `bazel build` will revive all the needed pieces.

Caveat, at the latest bazelcon someone was not looking favorably at a mix of bazel and nix. I beg to differ, but my setup is not as demanding as theirs.

F

[Penugonda chenna reddy]

Thanks, Filip.

Let me clarify: Suppose my development environment requires certain apt packages, but I want to achieve hermetic builds without installing these packages at the system level. Is there a way to accomplish this in Bazel?  

regards

PCReddy

[Filip Filmar]

On Tue, Nov 19, 2024 at 11:01 PM Penugonda chenna reddy <penug...@gmail.com> wrote:

Have you explored using the Conan package manager as an alternative to Nix?

No. But apparently there is some interest in this elsewhere.

Additionally, if possible, could you provide insights on how to approach getting started with Bazel for C/C++ in a structured manner for a complex project? Specifically,

I'm not sure if I'd advise using bazel in a commercial setting without a resident bazel expert, and without having a very clear goal in mind.

Even then, it's tricky. Since you mention complex projects, some examples: Istio backpedaled from supporting bazel builds. Kubernetes did the same. Android too. They all had their reasons, and IIRC the reasons weren't the same but the outcome was.  Sometimes showstoppers can come from an unexpected direction. 

we need to handle cross-compilers and generated code, and our engineers currently have experience only with Makefiles. 

Mission-specific bazel help is usually the domain of paid consulting work.

F

[Penugonda chenna reddy]

Filip,

Thank you so much for your guidance. I’ll review the hermetic_cc toolchain and get back to you with my thoughts or any questions. This is an excellent starting point for me.

Have you explored using the Conan package manager as an alternative to Nix?

Additionally, if possible, could you provide insights on how to approach getting started with Bazel for C/C++ in a structured manner for a complex project? Specifically, we need to handle cross-compilers and generated code, and our engineers currently have experience only with Makefiles. 

I truly appreciate your support and expertise—it’s invaluable to learn from someone so experienced in the open-source community.

regards

[Adam Hamilton]

Regarding development environments, this is probably best achieved by using a Docker container and installing everything that is required inside that container, including Bazel. This way, you can control exactly which packages you need. You then expose your workspace folder as a bind mount from within the container I believe.

[Penugonda chenna reddy]

Filip,

I agree with your point that attempting a commercial project without an in-house Bazel expert might not be practical. However, given the potential benefits of Bazel, especially for distributed compilation, there’s a concern that not exploring it might leave us at a disadvantage. Commercial solutions for distributed compilation often don’t deliver sufficient value considering the licensing costs.

Would you be open to providing further guidance through paid consulting work? I deeply value your expertise and appreciate the time you’ve already dedicated to helping me.

Regards

PCReddy

[Penugonda chenna reddy]

  Thanks, Adam. We’ve been using Docker for toolchain isolation but are now exploring whether we can achieve the same isolation through hermetic builds. I just want to make sure we're considering all the options carefully, and any insights you can provide will be appreciated.

  

--
You received this message because you are subscribed to a topic in the Google Groups "bazel-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bazel-discuss/NS_N26tyXME/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bazel-discus...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/bazel-discuss/af4cadfc-b198-45fd-b2d8-9cd74a391789n%40googlegroups.com.

[Allan Clark]

On Wednesday, November 20, 2024 at 12:38:24 AM UTC-8 fil...@gmail.com wrote:

On Tue, Nov 19, 2024 at 11:01 PM Penugonda chenna reddy <penug...@gmail.com> wrote:

...we need to handle cross-compilers and generated code, and our engineers currently have experience only with Makefiles. 

Mission-specific bazel help is usually the domain of paid consulting work.

What build-environments and target environments are you looking at?

I've recently been working in cross-toolchains for some NAS devices (ARM variants), I might be able to help if you have a basic example.

(Sorry for jumping late, I only saw this on the nightly digest)

Currently unemployed, so I kinda have time :)

Allan

[Filip Filmar]

On Wed, Nov 20, 2024 at 12:45 PM Allan Clark <chicken...@gmail.com> wrote:

On Wednesday, November 20, 2024 at 12:38:24 AM UTC-8 fil...@gmail.com wrote:

On Tue, Nov 19, 2024 at 11:01 PM Penugonda chenna reddy <penug...@gmail.com> wrote:

...we need to handle cross-compilers and generated code, and our engineers currently have experience only with Makefiles. 

Mission-specific bazel help is usually the domain of paid consulting work.

What build-environments and target environments are you looking at? [...]  Currently unemployed, so I kinda have time :)

Well, there you go! Preparation just met opportunity.

F