Weekly TF.org native .rpm RHEL/Fedora {COPR services}

[Cristian Balint]

Dear All,

  Allow me to announce a small effort towards providing weekly native .rpm packages for RHEL (CentOS) & Fedora:


  (x) Covered packages from the TF.org ecosystem:

    - https://copr.fedorainfracloud.org/coprs/rezso/ML/package/tensorflow
    - https://copr.fedorainfracloud.org/coprs/rezso/ML/package/tensorboard
    - https://copr.fedorainfracloud.org/coprs/rezso/ML/package/bazel5


  (x) Components are from within larger ML effort, the repo with **instructions** is here:

    - https://copr.fedorainfracloud.org/coprs/rezso/ML

      The larger ML group holds mutual benefits as being shared to each other.

         Ex.: tensorflow & tflite is used by TVM efforts and other packages.

  (x) Outlines on tensorflow native packaging effort:

    * effort dates back to >3 years but since 6mo is public.
    * transparent and fully **automated** on free copr services.
    * build cycle is targeted as being **weekly**, that looks sufficient.
    * target systems: rhel8 <-> fedora-rawhide {x86_64, ppc64, aarch64}.
    * build relay on external system libraries (as possible) under LOCAL_LIBS.
    * tensorflow & tflites are generated and follow rhel specific packaging rules, i.e:

      https://copr-dist-git.fedorainfracloud.org/cgit/rezso/ML/tensorflow.git/tree/tensorflow.spec

To my knowledge there are no similar efforts up to this date.

It is possible to extend with other packages from the TF.org ecosystem, let me know of such.

For further interest: it is possible to set up dedicated CI builders and/or just feel free to reuse the binaries & specs & recipes available.


Thank you !
~cristian

[Mihai Maruseac]

Hi.

This sounds really great. A few questions:

- This seems to target a build every week. Do you just pick the latest commit when the build starts or are first looking to see if the commit is green in TF's CI (or another one)? We have the nightly branch that tracks the nightly releases (tf-nightly), which could be used to generate matching packages.

- Would it be better to target tensorflow releases (each one is in its own branch, e.g., TF 2.8 is from r2.8 branch and all patch releases to TF 2.8 will also be from that branch)

Thank you very much for this effort. If you can participate in the next SIG Build meeting to present this to the larger audience that would be great. Thank you very much for the update!

+Rostam Dinyari +Austin Anderson  FYI.

--

Tensors must flow securely

[Cristian Balint]

Hi,

Thank you Mihai for your feedback !

On Mon, Mar 28, 2022 at 11:11 PM Mihai Maruseac <mihaim...@google.com> wrote:

Hi.

This sounds really great. A few questions:

- This seems to target a build every week. Do you just pick the latest commit when the build starts or are first looking to see if the commit is green in TF's CI (or another one)? We have the nightly branch that tracks the nightly releases (tf-nightly), which could be used to generate matching packages.

 
    (1) Yes, it would be possible to *sync* up with mentioned tf-nightly (even if the cycle is weekly).

    Currently:

       * git main HEAD is picked up *as-is* at the time of build.
       * there are clear marks of git's hash & date on packages N-V-R itself.

 

- Would it be better to target tensorflow releases (each one is in its own branch, e.g., TF 2.8 is from r2.8 branch and all patch releases to TF 2.8 will also be from that branch)

   (2) Yes, as for STABLE & NIGHTLY releases would be possible in two ways:
            a) Specific repo way: emit only STABLE X.Y.Z tied to optional "modular streams" of **nighlies** (having higher N-V-R).
            b) Simply just have the "tensorflow-X.Y.Z" package (STABLE) and another  separate one called "tensorflow-nightly-gitXXX" (NIGHTLY).

   Currently:

       * only weekly packages are build out of git main head.

  (X)  One drawback of COPR services is that it keeps **only the latest** build of a package (or module), all rest are deleted (only the log entries are kept).

 

Thank you very much for this effort. If you can participate in the next SIG Build meeting to present this to the larger audience that would be great. Thank you very much for the update!

+Rostam Dinyari +Austin Anderson  FYI.

   If there is real interest in such .rpm native packages then it would be easy to implement #1 & #2{a||b} .

   To achieve #1 & #2{a||b} a separate/dedicated copr repo would be the best, where we may include many other tf.org related packages.

   As last mention, copr also offers builders targeting: opensuse, openmandriva and mageia {x86_64, ppc64le, aarch64} that can be enabled in the dedicated tf.org repository.

Thank you !

~cristian.

[Austin Anderson]

Cristian, this is indeed very cool! There's a list of projects and table of links on github.com/tensorflow/build that we use to showcase things like these -- are you interested in adding them there?

If you could come to the next SIG Build meeting (scheduled for April 5 at 2pm, see bit.ly/tf-sig-build-notes) it would be a good place to get more exposure for these.

Best,

Austin

[Cristian Balint]

Hi,

First, thank you Austin for the feedback !

On Wed, Mar 30, 2022 at 12:49 AM Austin Anderson <ange...@google.com> wrote:

Cristian, this is indeed very cool! There's a list of projects and table of links on github.com/tensorflow/build that we use to showcase things like these -- are you interested in adding them there?

* Will work to enable discussed proposals (by Mihai) here as: #1 & #2b.

* When complete a PR will follow on the pointed github TF-SIG page.

 

If you could come to the next SIG Build meeting (scheduled for April 5 at 2pm, see bit.ly/tf-sig-build-notes) it would be a good place to get more exposure for these.

* Will participate on Apr 5 (Tue) presenting 4 slides and discuss it in ~10 minutes:

      - fedora & rhel: advantage of native .rpm 

      - COPR current status caveats and challenges

      - future enhancement proposals for native package builds

      - possible coverage extensions beyond .rpm natives

Added myself to the sig-build-notes @Notes 2022-04-05 document.

Cheers,

~cristian.

--

Balint Cristian
e-mail: cristia...@gmail.com