Support for making s390x binaries available

[duanep...@gmail.com]

Hi,

We are planning to add s390x support to build binaries for Prometheus. We see multi-arch support for Prometheus building binaries for IBM ppcle64.
What process do you follow to build the same? Do you have a CI setup for it?

Regards,
Duane

[Martín Ferrari]

Hi Duane,

On 16/03/18 09:42, duanep...@gmail.com wrote:

> We are planning to add s390x support to build binaries for Prometheus. We see multi-arch support for Prometheus building binaries for IBM ppcle64.
> What process do you follow to build the same? Do you have a CI setup for it?

If it is of any use, Debian offers s390x binaries for prometheus and
related packages.

--
Martín Ferrari (Tincho)

[duanep...@gmail.com]

Hi,

We are looking to generate binaries for RHEL, SLES and UBUNTU. How are the binaries for Debian built for s390x, similarly we can do the same for the above distros.

Thanks & Regards,
Duane

[Martín Ferrari]

On 22/03/18 11:56, duanep...@gmail.com wrote:

> We are looking to generate binaries for RHEL, SLES and UBUNTU. How are the binaries for Debian built for s390x, similarly we can do the same for the above distros.

We have machines of that architecture running the debian build daemon
(sbuild), which uses the same scripts than for every other architecture.


--
Martín Ferrari (Tincho)

[duanep...@gmail.com]

Is there a chance that you would provide binaries for other distros like Ubuntu, RHEL and SLES for s390x too?

I do not see s390x listed here https://prometheus.io/download/ so a bit curious to know.

[Martín Ferrari]

On 22/03/18 12:33, duanep...@gmail.com wrote:

> Is there a chance that you would provide binaries for other distros like Ubuntu, RHEL and SLES for s390x too?
>
> I do not see s390x listed here https://prometheus.io/download/ so a bit curious to know.

I am only the Debian maintainer for prometheus, I don't even know how to
create an RPM :) But I can tell you that the binaries inside the debian
package should work out of the box in other distros, as they are
statically compiled:
https://packages.debian.org/sid/s390x/prometheus/download

Also, the Debian packages should be installable in Ubuntu without
issues. You should even have older versions of prometheus available from
the universe repositories: https://packages.ubuntu.com/bionic/prometheus

--
Martín Ferrari (Tincho)

[duanep...@gmail.com]

Thanks Martin. :)

[Matthias Rampke]

Hello,

the release binaries are built with the help of

https://github.com/prometheus/promu

which for the server is configured here

https://github.com/prometheus/prometheus/blob/master/.promu.yml

If Go supports the OS/arch combination you are after and it builds cleanly, it should be as simple as adding it there.

/MR

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/d171d6bf-f9a7-48f6-8a7b-64d1abdb9571%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Martin Schröder]

2018-03-22 12:56 GMT+01:00 <duanep...@gmail.com>:
> We are looking to generate binaries for RHEL, SLES and UBUNTU. How are the binaries for Debian built for s390x, similarly we can do the same for the above distros.

May I suggest using the Open Build Service: http://openbuildservice.org/

Although it's originally for SUSE, it can do everything you want
(including s390, I think).

Best
Martin

[duanep...@gmail.com]

Thanks for the info.

Could you elaborate a bit more?
After modifying the .promu.yml to include support for s390x (IBM Z), should we just be running a build using make?
Would that generate the prometheus binaries for s390x, or would we further need to make changes in the prometheus/promu and prometheus/golang-builder repositories adding support for s390x?

[Goutham Veeramachaneni]

Hi Duane,

So I am not too sure how the entire build process works here but I looked into all the relevant parts. So currently here is how our cross-compiling pipeline works:

1. promu looks at available architectures and the architectures listed.

2. Once there are matching archs, it downloads the relevant golang-builder docker images.

3. The compiles prometheus in that docker image.

Which essentially means you'll need to change things in 3 places:

1. promu: add s390x to the available archs. Make sure that there is a matching image for the it.

2. For there to be an image you would need to modify the golang-builder to add support: https://github.com/prometheus/golang-builder

3. Once both are done, just modify the prometheus .promu.yml for it to compile for s390x too.

Hopefully this is useful, and if anything is confusing, please feel free to reply here. I don't know the exact details of the process, but I'd love to find out to help you :)

Thanks,

Goutham.

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

[duanep...@gmail.com]

> > To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

>
> >
>
> > To post to this group, send email to prometheus...@googlegroups.com.
>
> >
>
> > To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/d171d6bf-f9a7-48f6-8a7b-64d1abdb9571%40googlegroups.com.
>
> >
>
> > For more options, visit https://groups.google.com/d/optout.
>
>
>
> Thanks for the info.
>
>
>
> Could you elaborate a bit more?
>
> After modifying the .promu.yml to include support for s390x (IBM Z), should we just be running a build using make?
>
> Would that generate the prometheus binaries for s390x, or would we further need to make changes in the prometheus/promu and prometheus/golang-builder repositories adding support for s390x?

Thanks Goutham, This is what i understood too, Any idea on how to go about testing the same?

[Goutham Veeramachaneni]

Hi Duane,

This would seem a bit of a hack, but I'd build the docker image from golang-builder, and push to a personal repo and then modify promu code to pick up from the personal repo. i.e, instead of using dockerBuilderImageName, I'd put "gouthamve/golang-builder" there. Once you confirm that promu is generating the right binaries and the whole process works, open PRs to the right repos. Feel free to tag me (@gouthamve on Github) on those PRs.

Thanks,

Goutham.

> > To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

[duanep...@gmail.com]

> > > To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

>
> >
>
> > >
>
> >
>
> > > To post to this group, send email to prometheus...@googlegroups.com.
>
> >
>
> > >
>
> >
>
> > > To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/d171d6bf-f9a7-48f6-8a7b-64d1abdb9571%40googlegroups.com.
>
> >
>
> > >
>
> >
>
> > > For more options, visit https://groups.google.com/d/optout.
>
> >
>
> >
>
> >
>
> > Thanks for the info.
>
> >
>
> >
>
> >
>
> > Could you elaborate a bit more?
>
> >
>
> > After modifying the .promu.yml to include support for s390x (IBM Z), should we just be running a build using make?
>
> >
>
> > Would that generate the prometheus binaries for s390x, or would we further need to make changes in the prometheus/promu and prometheus/golang-builder repositories adding support for s390x?
>
>
>
> Thanks Goutham, This is what i understood too, Any idea on how to go about testing the same?

Hi Goutham,

So I was able to build the golang-builder image locally, using the Debian:stretch (stable) docker image which does not have the crossbuild-essential-s390x package(https://packages.debian.org/search?suite=default&section=all&arch=any&searchon=names&keywords=crossbuild-essential. Although it is available in the buster (testing) and sid (unstable) images.


I have also built the promu binary, Ran the short tests which pass.
Although 'make test' fails due to -race only supported on linux/amd64, freebsd/amd64, darwin/amd64 and windows/amd64.

Any suggestions on proceeding further?

Regards,
Duane

[Simon Pasquier]

> > > To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

>
> >
>
> > >
>
> >
>
> > > To post to this group, send email to prometheus...@googlegroups.com.
>
> >
>
> > >
>
> >
>
> > > To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/d171d6bf-f9a7-48f6-8a7b-64d1abdb9571%40googlegroups.com.
>
> >
>
> > >
>
> >
>
> > > For more options, visit https://groups.google.com/d/optout.
>
> >
>
> >
>
> >
>
> > Thanks for the info.
>
> >
>
> >
>
> >
>
> > Could you elaborate a bit more?
>
> >
>
> > After modifying the .promu.yml to include support for s390x (IBM Z), should we just be running a build using make?
>
> >
>
> > Would that generate the prometheus binaries for s390x, or would we further need to make changes in the prometheus/promu and prometheus/golang-builder repositories adding support for s390x?
>
>
>
> Thanks Goutham, This is what i understood too, Any idea on how to go about testing the same?

Hi Goutham,

So I was able to build the golang-builder image locally, using the Debian:stretch (stable) docker image which does not have the crossbuild-essential-s390x package(https://packages.debian.org/search?suite=default&section=all&arch=any&searchon=names&keywords=crossbuild-essential. Although it is available in the buster (testing) and sid (unstable) images.


I have also built the promu binary, Ran the short tests which pass.
Although 'make test' fails due to -race only supported on linux/amd64, freebsd/amd64, darwin/amd64 and windows/amd64.

I'd say it is a bug in Makefile.common. It should only add the "-race" flag on the supported architectures.

Incorporating something along those lines would make sense IMO:

https://github.com/prometheus/node_exporter/blob/834e35112c4a7c7fbd3b6ff108d1042035297c55/Makefile#L34-L39

 

Any suggestions on proceeding further?

Regards,
Duane

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/81ae915b-6f91-4952-9b4b-75cb89555b24%40googlegroups.com.

[duanep...@gmail.com]

> > > > To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > To post to this group, send email to prometheus...@googlegroups.com.
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/d171d6bf-f9a7-48f6-8a7b-64d1abdb9571%40googlegroups.com.
>
> >
>
> > >
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > > > For more options, visit https://groups.google.com/d/optout.
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > Thanks for the info.
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > Could you elaborate a bit more?
>
> >
>
> > >
>
> >
>
> > > After modifying the .promu.yml to include support for s390x (IBM Z), should we just be running a build using make?
>
> >
>
> > >
>
> >
>
> > > Would that generate the prometheus binaries for s390x, or would we further need to make changes in the prometheus/promu and prometheus/golang-builder repositories adding support for s390x?
>
> >
>
> >
>
> >
>
> > Thanks Goutham, This is what i understood too, Any idea on how to go about testing the same?
>
>
>
> Hi Goutham,
>
>
>
> So I was able to build the golang-builder image locally, using the Debian:stretch (stable) docker image which does not have the crossbuild-essential-s390x package(https://packages.debian.org/search?suite=default&section=all&arch=any&searchon=names&keywords=crossbuild-essential. Although it is available in the buster (testing) and sid (unstable) images.
>
>
>
>
>
> I have also built the promu binary, Ran the short tests which pass.
>
> Although 'make test' fails due to -race only supported on linux/amd64, freebsd/amd64, darwin/amd64 and windows/amd64.
>
>
>
> I'd say it is a bug in Makefile.common. It should only add the "-race" flag on the supported architectures.
> Incorporating something along those lines would make sense IMO:
>
> https://github.com/prometheus/node_exporter/blob/834e35112c4a7c7fbd3b6ff108d1042035297c55/Makefile#L34-L39
>
>
>
>  
>
>
> Any suggestions on proceeding further?
>
>
>
> Regards,
>
> Duane
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

>
> To post to this group, send email to prometheus...@googlegroups.com.
>

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/81ae915b-6f91-4952-9b4b-75cb89555b24%40googlegroups.com.
>
>
>
> For more options, visit https://groups.google.com/d/optout.

Thanks Simon,

@Goutham any inputs regarding crossbuild?

[duanep...@gmail.com]

@Simon @Goutham I have a query related to the process used to build the golang-builder image and then building promu.

My understanding is as below:
1. Build the golang-builder-base image.
2. Using the golang-builder-base image we build powerpc, arm, mips etc.

In the base dockerfile(https://github.com/prometheus/golang-builder/blob/master/1.9/base/Dockerfile#L18) golang binaries used are amd64(https://golang.org/dl/go$GOLANG_VERSION.linux-amd64.tar.gz).

For s390x I have created a dockerfile sourced FROM debian:stable(s390x/debian:stable) and installed the respective go binary for s390x, therefore not building it from the golang-builder-base image.

It gets a bit tricky now when trying to make changes to promu, i.e. for crossbuilding (https://github.com/prometheus/promu/blob/master/cmd/crossbuild.go#L139).

Im a little confused with the fact that the golang-builder images for powerpc, arm, mips etc. are built from golang-builder-base and using this same amd64 go binary.

Does it not error out with an exec-format error when used on powerpc or other platforms?

[Goutham Veeramachaneni]

Hi Duane,

Go has really good cross compilation support built in :) See here: https://golang.org/doc/install/source#environment (GOOS, GOARCH). An outdated but still good article: https://dave.cheney.net/2015/08/22/cross-compilation-with-go-1-5

You'd be able to produce working binaries using GOOS=linux GOARCH=s390x

Thanks

Goutham.

[Matthias Rampke]

What is the final goal here? Are you trying to build on S390x yourself? In that case you are maybe better off using whatever your own normal build process is, most Prometheus projects are at their core very normal Go programs.

Or are you looking for release binaries for Linux/S390x? The release builds all run on Linux/amd64, so I don't know if you really need a special builder?

/MR

--

You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

To post to this group, send email to prometheus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/b6ab47a5-045c-483e-9e33-3ddd578b172e%40googlegroups.com.

[duanep...@gmail.com]

Hi,

@Goutham Thanks for the info.

I have created a folder for z and added/updated the required Dockerfile, Makefile and rootfs/builder.sh.

I updated the golang-builder/Makefile in order to verify build for Z/s390x as below:
-VARIANTS ?= base main arm powerpc mips
+VARIANTS ?= base z

On running 'make build', the base image is built with the base Dockerfile. It then tries to build quay.io/prometheus/golang-builder:1.9.7-z-master from the z Dockerfile and fails with error "E: Unable to locate package crossbuild-essential-s390x". This is because debian:stable for s390x does not include the package. Therefore using the debian:buster image which include crossbuild-essentials-s390x the build passes.

@Matthias, I am looking for releasing binaries for Linux/s390x.
Should I just make the changes related to Z and run the same on Linux/amd64?
How should we go about testing my changes to verify them before making PR's to the golang-builder, promu and prometheus repos?

Thanks,
Duane

[duanep...@gmail.com]

Hi,

We have successfully been able to build golang-builder-z docker image for z(s390x), by running the same on Linux/amd64.
The only change we had to make was use debian:buster over debian:stable in the golang-builder-base image Dockerfile to include the crossbuild-essentials-s390x package that is available in debian:buster.

Any comments/suggestions/issues regarding the use of debian:buster instead of debian:stable?

Regards,
Duane

[Goutham Veeramachaneni]

Hi Duane,

I've opened an issue and tagged the relevant maintainers for this: https://github.com/prometheus/golang-builder/issues/48 They'll reply with their thoughts soon.

Thanks,

Gouham.

[duanep...@gmail.com]

Hi Goutham,

Thanks. I had a look at the issue, went ahead and used dpkg-cross, g++-s390x-linux-gnu, and gcc-s390x-linux-gnu directly in the z Dockerfile.

Managed to build images for base arm powerpc mips z.
Ignored 'main' as it failed with the below error:

Step 6/8 : RUN mkdir -p /tmp/osxcross && cd /tmp/osxcross && curl -sSL "https://codeload.github.com/tpoechtrager/osxcross/tar.gz/${OSXCROSS_REV}" | tar -C /tmp/osxcross --strip=1 -xzf - && curl -sSLo tarballs/MacOSX${SDK_VERSION}.sdk.tar.xz ${OSXCROSS_SDK_URL} && UNATTENDED=yes ./build.sh >/dev/null && mv target "${OSXCROSS_PATH}" && rm -rf /tmp/osxcross "/usr/osxcross/SDK/MacOSX${SDK_VERSION}.sdk/usr/share/man"
---> Running in 766fe9ac6d51
curl: no URL specified!
curl: try 'curl --help' or 'curl --manual' for more information
The command '/bin/sh -c mkdir -p /tmp/osxcross && cd /tmp/osxcross && curl -sSL "https://codeload.github.com/tpoechtrager/osxcross/tar.gz/${OSXCROSS_REV}" | tar -C /tmp/osxcross --strip=1 -xzf - && curl -sSLo tarballs/MacOSX${SDK_VERSION}.sdk.tar.xz ${OSXCROSS_SDK_URL} && UNATTENDED=yes ./build.sh >/dev/null && mv target "${OSXCROSS_PATH}" && rm -rf /tmp/osxcross "/usr/osxcross/SDK/MacOSX${SDK_VERSION}.sdk/usr/share/man"' returned a non-zero code: 2

Thanks,
Duane

[Goutham Veeramachaneni]

Hi Duane,

I'm a little lost now as to the specific error. If you're now able to build the images for z, feel free to open PR's in the relevant repos. Maybe looking at the code and the error together would make it clear. Also, sorry it took so long for you to figure everything out.

Thanks,

Goutham.

--

You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

To post to this group, send email to prometheus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/4787b709-6f5e-45d5-8d47-605dcd0ed8d8%40googlegroups.com.

[Simon Pasquier]

Duane, this is because the build process needs access to credentials that are only known from the CI system.

For now, please send your PR to prometheus/golang-builder though you will most probably hit https://github.com/prometheus/golang-builder/issues/52

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

To post to this group, send email to prometheus-developers@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/4787b709-6f5e-45d5-8d47-605dcd0ed8d8%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/1536562936.local-82614ef5-a2f7-v1.4.2-f587b7b7%40getmailspring.com.

[duanep...@gmail.com]

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
> To post to this group, send email to prometheus...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/4787b709-6f5e-45d5-8d47-605dcd0ed8d8%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
>
> To post to this group, send email to prometheus...@googlegroups.com.

>
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/1536562936.local-82614ef5-a2f7-v1.4.2-f587b7b7%40getmailspring.com.
>
>
>
> For more options, visit https://groups.google.com/d/optout.

Hi Simon,Goutham,

Yes I will send a PR to prometheus/golang-builder as well as prometheus/promu.

I was successfully able to create the golang-builder docker image as well run the crossbuild for promu to generate binaries for linux-s390x.

Thanks,
Duane

[duanep...@gmail.com]

Hi guys,

I have created 3 pull requests for all 3 repositories.

prometheus/golang-builder#53(https://github.com/prometheus/golang-builder/pull/53)
prometheus/promu#107(https://github.com/prometheus/promu/pull/107)
prometheus/prometheus#4605(https://github.com/prometheus/prometheus/pull/4605)

Let me know if any changes need to be made.

Thanks,
Duane