[PATCH 1/3] scripts: new script to download and extract arbitrary Fedora rpm

[Waldemar Kozaczuk]

Signed-off-by: Waldemar Kozaczuk <jwkoz...@gmail.com>
---
scripts/download_rpm_package.sh | 34 +++++++++++++++++++++++++++++++++
1 file changed, 34 insertions(+)
create mode 100755 scripts/download_rpm_package.sh

diff --git a/scripts/download_rpm_package.sh b/scripts/download_rpm_package.sh
new file mode 100755
index 00000000..562b379a
--- /dev/null
+++ b/scripts/download_rpm_package.sh
@@ -0,0 +1,34 @@
+#!/bin/bash
+
+package="$1"
+release=$2
+out_dir=$3
+
+arch=aarch64
+
+letter=${package:0:1}
+
+main_repo_url="http://mirrors.kernel.org/fedora/releases/$release/Everything/$arch/os/Packages/$letter/"
+version=$(wget -t 1 -qO- $main_repo_url | grep "${package}-[0-9].*$arch" | grep -Po ">.*<" | sed -e "s/>${package}-\(.*\)\.$arch\.rpm</\1/g" | tail -l)
+
+archive_repo_url="http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/$release/Everything/$arch/os/Packages/$letter/"
+if [[ "${version}" != "" ]]; then
+ file_name="${package}-${version}.aarch64.rpm"
+ full_url="${main_repo_url}${file_name}"
+else
+ version=$(wget -t 1 -qO- $archive_repo_url | grep "${package}-[0-9].*$arch" | grep -Po "<a href.*>" | sed -e "s/<a href\=\"${package}-\(.*\)\.$arch\.rpm\".*/\1/g" | tail -l)
+ if [[ "${version}" != "" ]]; then
+ file_name="${package}-${version}.aarch64.rpm"
+ full_url="${archive_repo_url}${file_name}"
+ fi
+fi
+
+if [[ "${version}" == "" ]]; then
+ echo "Could not find $package under $main_repo_url nor $archive_repo_url!"
+ exit 1
+fi
+
+mkdir -p ${out_dir}/upstream
+wget -c -O ${out_dir}/upstream/${file_name} ${full_url}
+mkdir -p ${out_dir}/install
+rpm2cpio ${out_dir}/upstream/${file_name} | (cd ${out_dir}/install && cpio -id)
--
2.20.1

[Waldemar Kozaczuk]

References #743

Signed-off-by: Waldemar Kozaczuk <jwkoz...@gmail.com>
---

scripts/setup.py | 40 +++++++++++++++++++++++++++++++++++++++-
1 file changed, 39 insertions(+), 1 deletion(-)

diff --git a/scripts/setup.py b/scripts/setup.py
index db831f0f..946d1e4e 100755
--- a/scripts/setup.py
+++ b/scripts/setup.py
@@ -3,7 +3,7 @@
# set up a development environment for OSv. Run as root.

import sys, argparse
-import subprocess
+import subprocess, os

standard_ec2_packages = ['python-pip', 'wget']
standard_ec2_post_install = ['pip install awscli &&'
@@ -64,6 +64,40 @@ class Fedora(object):
test_packages = ['openssl-devel']
ec2_post_install = standard_ec2_post_install

+ def aarch64_download(self, version):
+ gcc_packages = ['gcc',
+ 'glibc',
+ 'glibc-devel',
+ 'libgcc',
+ 'libstdc++',
+ 'libstdc++-devel',
+ 'libstdc++-static']
+ boost_packages = ['boost-devel',
+ 'boost-static',
+ 'boost-system']
+ osv_root = os.path.join(os.path.dirname(os.path.abspath(__file__)), '..')
+ script_path = '%s/scripts/download_rpm_package.sh' % osv_root
+ destination = '%s/downloaded_packages/aarch64' % osv_root
+ ##
+ # The setup.py is typically run as root to allow yum properly install packages
+ # This however would cause all files downloaded to downloaded_packages/aarch64 directory
+ # get created and owned by the root user which in most cases is not desirable
+ # To prevent that let us compare current process user id with the owner id of osv root
+ # directory and if different run all download command with the same user as the one owning
+ # the root directory
+ current_user_id = os.getuid()
+ osv_root_owner_id = os.stat(osv_root).st_uid
+ if current_user_id != osv_root_owner_id and current_user_id == 0:
+ command_prefix = "sudo -u '#%d'" % osv_root_owner_id # Most likely setup.py is run by root so let us use sudo
+ else:
+ command_prefix = ''
+
+ install_commands = ['%s %s %s %s %s/gcc' % (command_prefix, script_path, package, version, destination) for package in gcc_packages]
+ install_commands += ['%s %s %s %s %s/boost' % (command_prefix, script_path, package, version, destination) for package in boost_packages]
+ install_commands = ['%s rm -rf %s/gcc/install' % (command_prefix, destination),
+ '%s rm -rf %s/boost/install' % (command_prefix, destination)] + install_commands
+ return ' && '.join(install_commands)
+
class Fedora_25(object):
packages = ['java-1.8.0-openjdk', 'python2-requests', 'openssl-devel', 'lua-5.3.*', 'lua-devel-5.3.*']
ec2_packages = []
@@ -361,6 +395,10 @@ for distro in distros:
if cmdargs.test:
pkg += distro.test_packages + dver.test_packages
subprocess.check_call(distro.install + ' ' + str.join(' ', pkg), shell=True)
+ if 'aarch64_download' in dir(distro):
+ print('Downloading aarch64 packages to cross-compile ARM version ...')
+ subprocess.check_call(distro.aarch64_download(dver.version), shell=True)
+ print('Downloaded all aarch64 packages!')
if cmdargs.ec2:
if distro.ec2_post_install:
subprocess.check_call(distro.ec2_post_install, shell=True)
--
2.20.1

[Waldemar Kozaczuk]

This patch eliminates aarch64 gcc and boost libraries from externals
in lieu of new mechanism to download the packages from Fedora
repository.

Fixes #743

Signed-off-by: Waldemar Kozaczuk <jwkoz...@gmail.com>
---

.gitignore | 1 +
.gitmodules | 6 ----
Makefile | 60 +++++++++++++--------------------------
external/aarch64/gcc.bin | 1 -
external/aarch64/misc.bin | 1 -
5 files changed, 21 insertions(+), 48 deletions(-)
delete mode 160000 external/aarch64/gcc.bin
delete mode 160000 external/aarch64/misc.bin

diff --git a/.gitignore b/.gitignore
index 16e192db..398c5de7 100644
--- a/.gitignore
+++ b/.gitignore
@@ -49,3 +49,4 @@ modules/libyaml/usr.manifest
modules/dl_tests/usr.manifest
.idea
compile_commands.json
+downloaded_packages
diff --git a/.gitmodules b/.gitmodules
index da6d5d62..b6ae6ac1 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -6,12 +6,6 @@
path = apps
url = ../../cloudius-systems/osv-apps
ignore = dirty
-[submodule "external/aarch64/gcc.bin"]
- path = external/aarch64/gcc.bin
- url = ../../cloudius-systems/aarch64-gcc.bin.git
-[submodule "external/aarch64/misc.bin"]
- path = external/aarch64/misc.bin
- url = ../../cloudius-systems/aarch64-misc.bin.git
[submodule "modules/httpserver/swagger-ui"]
path = modules/httpserver-html5-gui/swagger-ui
url = ../../cloudius-systems/swagger-ui.git
diff --git a/Makefile b/Makefile
index 0e5d4d32..e8e110a5 100644
--- a/Makefile
+++ b/Makefile
@@ -205,34 +205,16 @@ cscope:

###########################################################################

-
-# The user can override the build_env variable (or one or more of *_env
-# variables below) to decide if to take the host's C/C++ libraries, or
-# those from the external/ directory.
-build_env ?= $(if $(filter $(host_arch), $(arch)),host,external)
-ifeq ($(build_env), host)
- gcc_lib_env ?= host
- cxx_lib_env ?= host
- gcc_include_env ?= host
- boost_env ?= host
-else
- gcc_lib_env ?= external
- cxx_lib_env ?= external
- gcc_include_env ?= external
- boost_env ?= external
-endif
-
-
local-includes =
INCLUDES = $(local-includes) -Iarch/$(arch) -I. -Iinclude -Iarch/common
INCLUDES += -isystem include/glibc-compat

-gccbase = external/$(arch)/gcc.bin
-miscbase = external/$(arch)/misc.bin
+aarch64_gccbase = downloaded_packages/aarch64/gcc/install
+aarch64_boostbase = downloaded_packages/aarch64/boost/install

-ifeq ($(gcc_include_env), external)
- gcc-inc-base := $(dir $(shell find $(gccbase)/ -name vector | grep -v -e debug/vector$$ -e profile/vector$$))
- gcc-inc-base3 := $(dir $(shell dirname `find $(gccbase)/ -name c++config.h | grep -v /32/`))
+ifeq ($(arch),aarch64)
+ gcc-inc-base := $(dir $(shell find $(aarch64_gccbase)/ -name vector | grep -v -e debug/vector$$ -e profile/vector$$ -e experimental/vector$$))
+ gcc-inc-base3 := $(dir $(shell dirname `find $(aarch64_gccbase)/ -name c++config.h | grep -v /32/`))
INCLUDES += -isystem $(gcc-inc-base)
INCLUDES += -isystem $(gcc-inc-base3)
endif
@@ -247,7 +229,7 @@ INCLUDES += -isystem $(libfdt_base)
endif

INCLUDES += $(boost-includes)
-ifeq ($(gcc_include_env), host)
+ifeq ($(arch),x64)
# Starting in Gcc 6, the standard C++ header files (which we do not change)
# must precede in the include path the C header files (which we replace).
# This is explained in https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70722.
@@ -257,8 +239,8 @@ INCLUDES += $(shell $(CXX) -E -xc++ - -v </dev/null 2>&1 | awk '/^End/ {exit} /^
endif
INCLUDES += -isystem include/api
INCLUDES += -isystem include/api/$(arch)
-ifeq ($(gcc_include_env), external)
- gcc-inc-base2 := $(dir $(shell find $(gccbase)/ -name unwind.h))
+ifeq ($(arch),aarch64)
+ gcc-inc-base2 := $(dir $(shell find $(aarch64_gccbase)/ -name unwind.h))
# must be after include/api, since it includes some libc-style headers:
INCLUDES += -isystem $(gcc-inc-base2)
endif
@@ -289,7 +271,7 @@ $(out)/musl/%.o: source-dialects =

kernel-defines = -D_KERNEL $(source-dialects)

-gcc-sysroot = $(if $(CROSS_PREFIX), --sysroot external/$(arch)/gcc.bin) \
+gcc-sysroot = $(if $(CROSS_PREFIX), --sysroot $(aarch64_gccbase)) \

# This play the same role as "_KERNEL", but _KERNEL unfortunately is too
# overloaded. A lot of files will expect it to be set no matter what, specially
@@ -314,11 +296,9 @@ COMMON = $(autodepend) -g -Wall -Wno-pointer-arith $(CFLAGS_WERROR) -Wformat=0 -
-include compiler/include/intrinsics.hh \
$(arch-cflags) $(conf-opt) $(acpi-defines) $(tracing-flags) $(gcc-sysroot) \
$(configuration) -D__OSV__ -D__XEN_INTERFACE_VERSION__="0x00030207" -DARCH_STRING=$(ARCH_STR) $(EXTRA_FLAGS)
-ifeq ($(gcc_include_env), external)
-ifeq ($(boost_env), external)
+ifeq ($(arch),aarch64)
COMMON += -nostdinc
endif
-endif

tracing-flags-0 =
tracing-flags-1 = -finstrument-functions -finstrument-functions-exclude-file-list=c++,trace.cc,trace.hh,align.hh,mmintrin.h
@@ -1807,7 +1787,7 @@ objects += $(addprefix fs/, $(fs_objs))
objects += $(addprefix libc/, $(libc))
objects += $(addprefix musl/src/, $(musl))

-ifeq ($(cxx_lib_env), host)
+ifeq ($(arch),x64)
libstdc++.a := $(shell $(CXX) -print-file-name=libstdc++.a)
ifeq ($(filter /%,$(libstdc++.a)),)
$(error Error: libstdc++.a needs to be installed.)
@@ -1818,11 +1798,11 @@ ifeq ($(cxx_lib_env), host)
$(error Error: libsupc++.a needs to be installed.)
endif
else
- libstdc++.a := $(shell find $(gccbase)/ -name libstdc++.a)
- libsupc++.a := $(shell find $(gccbase)/ -name libsupc++.a)
+ libstdc++.a := $(shell find $(aarch64_gccbase)/ -name libstdc++.a)
+ libsupc++.a := $(shell find $(aarch64_gccbase)/ -name libsupc++.a)
endif

-ifeq ($(gcc_lib_env), host)
+ifeq ($(arch),x64)
libgcc.a := $(shell $(CC) -print-libgcc-file-name)
ifeq ($(filter /%,$(libgcc.a)),)
$(error Error: libgcc.a needs to be installed.)
@@ -1833,11 +1813,11 @@ ifeq ($(gcc_lib_env), host)
$(error Error: libgcc_eh.a needs to be installed.)
endif
else
- libgcc.a := $(shell find $(gccbase)/ -name libgcc.a | grep -v /32/)
- libgcc_eh.a := $(shell find $(gccbase)/ -name libgcc_eh.a | grep -v /32/)
+ libgcc.a := $(shell find $(aarch64_gccbase)/ -name libgcc.a | grep -v /32/)
+ libgcc_eh.a := $(shell find $(aarch64_gccbase)/ -name libgcc_eh.a | grep -v /32/)
endif

-ifeq ($(boost_env), host)
+ifeq ($(arch),x64)
# link with -mt if present, else the base version (and hope it is multithreaded)
boost-mt := -mt
boost-lib-dir := $(dir $(shell $(CC) --print-file-name libboost_system$(boost-mt).a))
@@ -1853,9 +1833,9 @@ ifeq ($(boost_env), host)
# special for Boost.
boost-includes =
else
- boost-lib-dir := $(firstword $(dir $(shell find $(miscbase)/ -name libboost_system*.a)))
+ boost-lib-dir := $(firstword $(dir $(shell find $(aarch64_boostbase)/ -name libboost_system*.a)))
boost-mt := $(if $(filter %-mt.a, $(wildcard $(boost-lib-dir)/*.a)),-mt)
- boost-includes = -isystem $(miscbase)/usr/include
+ boost-includes = -isystem $(aarch64_boostbase)/usr/include
endif

boost-libs := $(boost-lib-dir)/libboost_system$(boost-mt).a
@@ -1931,7 +1911,7 @@ $(bootfs_manifest_dep): phony
ifeq ($(arch),x64)
libgcc_s_dir := $(dir $(shell $(CC) -print-file-name=libgcc_s.so.1))
else
-libgcc_s_dir := ../../$(gccbase)/lib64
+libgcc_s_dir := ../../$(aarch64_gccbase)/lib64
endif

$(out)/bootfs.bin: scripts/mkbootfs.py $(bootfs_manifest) $(bootfs_manifest_dep) $(tools:%=$(out)/%) \
diff --git a/external/aarch64/gcc.bin b/external/aarch64/gcc.bin
deleted file mode 160000
index a85ead11..00000000
--- a/external/aarch64/gcc.bin
+++ /dev/null
@@ -1 +0,0 @@
-Subproject commit a85ead11151e18075153f44fa685757dbc6e9675
diff --git a/external/aarch64/misc.bin b/external/aarch64/misc.bin
deleted file mode 160000
index a51b8f09..00000000
--- a/external/aarch64/misc.bin
+++ /dev/null
@@ -1 +0,0 @@
-Subproject commit a51b8f093574bedc4a361da7b155515ca473774c
--
2.20.1

[Commit Bot]

From: Waldemar Kozaczuk <jwkoz...@gmail.com>
Committer: Waldemar Kozaczuk <jwkoz...@gmail.com>
Branch: master

scripts: new script to download and extract arbitrary Fedora rpm

Signed-off-by: Waldemar Kozaczuk <jwkoz...@gmail.com>

---

diff --git a/scripts/download_rpm_package.sh b/scripts/download_rpm_package.sh
--- a/scripts/download_rpm_package.sh

[Commit Bot]

From: Waldemar Kozaczuk <jwkoz...@gmail.com>
Committer: Waldemar Kozaczuk <jwkoz...@gmail.com>
Branch: master

externals: enhance setup.py to download aarch64 gcc and boost code and libraries

References #743

Signed-off-by: Waldemar Kozaczuk <jwkoz...@gmail.com>

---

diff --git a/.gitignore b/.gitignore

--- a/.gitignore
+++ b/.gitignore
@@ -49,3 +49,4 @@ modules/libyaml/usr.manifest
modules/dl_tests/usr.manifest
.idea
compile_commands.json
+downloaded_packages

diff --git a/scripts/setup.py b/scripts/setup.py

@@ -361,6 +395,10 @@ def parse_file(f):

[Waldemar Kozaczuk]

This patch eliminates aarch64 gcc and boost libraries from externals
in lieu of new mechanism to download the packages from Fedora
repository.

Fixes #743

Signed-off-by: Waldemar Kozaczuk <jwkoz...@gmail.com>
---

.gitmodules | 6 ----
Makefile | 60 +++++++++++++--------------------------
external/aarch64/gcc.bin | 1 -
external/aarch64/misc.bin | 1 -

4 files changed, 20 insertions(+), 48 deletions(-)

delete mode 160000 external/aarch64/gcc.bin
delete mode 160000 external/aarch64/misc.bin

[Nadav Har'El]

I think this may be a good indication that this should not have been part of the "setup.py" script.

Until today, the "setup.py" script was something you need to run - if at all - once. You don't need to

run it every time you check out a clean OSv directory. You don't need to re-run it after a "make clean".

Moreover, I think there are other reasons why this code should *not* be part of setup.py:

First and foremost, the aarch stuff isn't needed by everyone - people who just compile OSv on x86 won't need it at all,

I would prefer to see something like the following:

1. A separate script to download the aarch64 packages.

2. The makefile will verify, if arch=aarch64, that the aarch64 tools are installed (in the system downloaded), and if not, tell you to run the new script - and exit. (we already have a bunch of similar checks in the makefile).

+        current_user_id = os.getuid()
+        osv_root_owner_id = os.stat(osv_root).st_uid
+        if current_user_id != osv_root_owner_id and current_user_id == 0:
+            command_prefix = "sudo -u '#%d'" % osv_root_owner_id # Most likely setup.py is run by root so let us use sudo

If you suspect you're running as root, you can use "su" instead of "sudo".

But as I said above, I would prefer not to need this at all, and have a separate script.

+        else:
+            command_prefix = ''
+
+        install_commands = ['%s %s %s %s %s/gcc' % (command_prefix, script_path, package, version, destination) for package in gcc_packages]
+        install_commands += ['%s %s %s %s %s/boost' % (command_prefix, script_path, package, version, destination) for package in boost_packages]
+        install_commands = ['%s rm -rf %s/gcc/install' % (command_prefix, destination),
+                            '%s rm -rf %s/boost/install' % (command_prefix, destination)] + install_commands
+        return ' && '.join(install_commands)
+
     class Fedora_25(object):
         packages = ['java-1.8.0-openjdk', 'python2-requests', 'openssl-devel', 'lua-5.3.*', 'lua-devel-5.3.*']
         ec2_packages = []
@@ -361,6 +395,10 @@ def parse_file(f):
                 if cmdargs.test:
                     pkg += distro.test_packages + dver.test_packages
                 subprocess.check_call(distro.install + ' ' + str.join(' ', pkg), shell=True)
+                if 'aarch64_download' in dir(distro):
+                    print('Downloading aarch64 packages to cross-compile ARM version ...')
+                    subprocess.check_call(distro.aarch64_download(dver.version), shell=True)
+                    print('Downloaded all aarch64 packages!')
                 if cmdargs.ec2:
                     if distro.ec2_post_install:
                         subprocess.check_call(distro.ec2_post_install, shell=True)

--
You received this message because you are subscribed to the Google Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/00000000000059fa59059fc3cb7d%40google.com.

[Waldek Kozaczuk]

I like this approach. Shall we handle distribution upgrades (Fedora 30 to 31 for example) and if so how ? If we do not user would need to know to delete old downloaded_packages and re-run that new script. Ideally we could have makefile somehow detect that the distro version is different than of what is in downloaded_packages. 

Also I am not sure you realized that the old aarch64 packages that are part of externals are for Fedora which I think explains why we never (me and Rick I think) were never successful to build runnable aarch64 image on Ubuntu. So should makefile any check like “flavor <> Fedora”?

[Nadav Har'El]

On Sun, Mar 1, 2020 at 3:41 PM Waldek Kozaczuk <jwkoz...@gmail.com> wrote:

I like this approach. Shall we handle distribution upgrades (Fedora 30 to 31 for example) and if so how ? If we do not user would need to know to delete old downloaded_packages and re-run that new script. Ideally we could have makefile somehow detect that the distro version is different than of what is in downloaded_packages.

Don't you have the same problem with your current approach?

One thing to consider is putting this stuff in build/, so that "rm -r build" (which you'd be wise to do after a distro upgrade, otherwise things break) will delete it to.

Finally, there is something I don't understand / didn't follow:

For me, I just installed the package gcc-aarch64-linux-gnu normally (with yum) and everything just worked. Why did you need to do something else? For non-Fedora?

So for Fedora, you can still do "dnf install ...", and always get the correct version (and it gets updated on distro upgrades!). For non-Fedora, there is no issue of matching distro versions anyway - Ubuntu can't "match" with Fedora anyway.

 

Also I am not sure you realized that the old aarch64 packages that are part of externals are for Fedora which I think explains why we never (me and Rick I think) were never successful to build runnable aarch64 image on Ubuntu.

Hmm... No, I never noticed it. What was taken from there, libraries?

Is there a way on Ubuntu to check the versions of the downloaded libraries, and if it doesn't match what we need (*what* do we need) ask to run the script again?

[Waldek Kozaczuk]

On Sunday, March 1, 2020 at 8:54:21 AM UTC-5, Nadav Har'El wrote:

On Sun, Mar 1, 2020 at 3:41 PM Waldek Kozaczuk <jwkoz...@gmail.com> wrote:

I like this approach. Shall we handle distribution upgrades (Fedora 30 to 31 for example) and if so how ? If we do not user would need to know to delete old downloaded_packages and re-run that new script. Ideally we could have makefile somehow detect that the distro version is different than of what is in downloaded_packages.

Don't you have the same problem with your current approach?

One thing to consider is putting this stuff in build/, so that "rm -r build" (which you'd be wise to do after a distro upgrade, otherwise things break) will delete it to.

Finally, there is something I don't understand / didn't follow:

For me, I just installed the package gcc-aarch64-linux-gnu normally (with yum) and everything just worked. Why did you need to do something else? For non-Fedora?

So for Fedora, you can still do "dnf install ...", and always get the correct version (and it gets updated on distro upgrades!). For non-Fedora, there is no issue of matching distro versions anyway - Ubuntu can't "match" with Fedora anyway.

 

Also I am not sure you realized that the old aarch64 packages that are part of externals are for Fedora which I think explains why we never (me and Rick I think) were never successful to build runnable aarch64 image on Ubuntu.

Hmm... No, I never noticed it. What was taken from there, libraries?

Is there a way on Ubuntu to check the versions of the downloaded libraries, and if it doesn't match what we need (*what* do we need) ask to run the script again?

I should have been more clear.  On Fedora setup.py installs the gcc-c++-aarch64-linux-gnu, on Ubuntu it does not install anything equivalent. So if on my Ubuntu machine I manually install that aarch64 package for c++ (cannot remember the name, but the Ubuntu package installs the headers - besides compiler binaries - and puts them in some system directories) and I build aarch64 OSv image it hangs on boot without displaying anything. So I suspect that somehow the Ubuntu edition of aarch64 gcc compiler does NOT work well with the aarch64 headers and libraries in externals or (in future downloaded outside of externals). I suspect there is something wrong in our Makefile that makes it mix aarch64 headers from system directories on Ubuntu with whatever Fedora files in externals or downloaded. Does it make sense?    

 

To unsubscribe from this group and stop receiving emails from it, send an email to osv...@googlegroups.com.

[Waldek Kozaczuk]

On Sunday, March 1, 2020 at 8:54:21 AM UTC-5, Nadav Har'El wrote:

On Sun, Mar 1, 2020 at 3:41 PM Waldek Kozaczuk <jwkoz...@gmail.com> wrote:

I like this approach. Shall we handle distribution upgrades (Fedora 30 to 31 for example) and if so how ? If we do not user would need to know to delete old downloaded_packages and re-run that new script. Ideally we could have makefile somehow detect that the distro version is different than of what is in downloaded_packages.

Don't you have the same problem with your current approach?

One thing to consider is putting this stuff in build/, so that "rm -r build" (which you'd be wise to do after a distro upgrade, otherwise things break) will delete it to.

Finally, there is something I don't understand / didn't follow:

For me, I just installed the package gcc-aarch64-linux-gnu normally (with yum) and everything just worked. Why did you need to do something else? For non-Fedora?

So for Fedora, you can still do "dnf install ...", and always get the correct version (and it gets updated on distro upgrades!). For non-Fedora, there is no issue of matching distro versions anyway - Ubuntu can't "match" with Fedora anyway.

 

Also I am not sure you realized that the old aarch64 packages that are part of externals are for Fedora which I think explains why we never (me and Rick I think) were never successful to build runnable aarch64 image on Ubuntu.

Hmm... No, I never noticed it. What was taken from there, libraries?

Is there a way on Ubuntu to check the versions of the downloaded libraries, and if it doesn't match what we need (*what* do we need) ask to run the script again?

I should have been more clear.  On Fedora setup.py installs the gcc-c++-aarch64-linux-gnu, on Ubuntu it does not install anything equivalent. So if on my Ubuntu machine I manually install that aarch64 package for c++ (cannot remember the name, but the Ubuntu package installs the headers - besides compiler binaries - and puts them in some system directories) and I build aarch64 OSv image it hangs on boot without displaying anything. So I suspect that somehow the Ubuntu edition of aarch64 gcc compiler does NOT work well with the aarch64 headers and libraries in externals or (in future downloaded outside of externals). I suspect there is something wrong in our Makefile that makes it mix aarch64 headers from system directories on Ubuntu with whatever Fedora files in externals or downloaded. Does it make sense?    

 

So should makefile any check like “flavor <> Fedora”?

To unsubscribe from this group and stop receiving emails from it, send an email to osv...@googlegroups.com.

[Waldemar Kozaczuk]

This patch eliminates aarch64 gcc and boost libraries from externals
in lieu of new mechanism to download the packages from Fedora
repository.

Please note that Jenkins build will need to be modified
to call scripts/download_fedora_aarch64_packages.py before building
aarch64 image.

Fixes #743

Signed-off-by: Waldemar Kozaczuk <jwkoz...@gmail.com>
---

.gitmodules | 6 ----
Makefile | 69 ++++++++++++++++-----------------------

external/aarch64/gcc.bin | 1 -
external/aarch64/misc.bin | 1 -

4 files changed, 29 insertions(+), 48 deletions(-)

delete mode 160000 external/aarch64/gcc.bin
delete mode 160000 external/aarch64/misc.bin

diff --git a/.gitmodules b/.gitmodules
index da6d5d62..b6ae6ac1 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -6,12 +6,6 @@
path = apps
url = ../../cloudius-systems/osv-apps
ignore = dirty
-[submodule "external/aarch64/gcc.bin"]
- path = external/aarch64/gcc.bin
- url = ../../cloudius-systems/aarch64-gcc.bin.git
-[submodule "external/aarch64/misc.bin"]
- path = external/aarch64/misc.bin
- url = ../../cloudius-systems/aarch64-misc.bin.git
[submodule "modules/httpserver/swagger-ui"]
path = modules/httpserver-html5-gui/swagger-ui
url = ../../cloudius-systems/swagger-ui.git
diff --git a/Makefile b/Makefile

index 0e5d4d32..03a17920 100644
--- a/Makefile
+++ b/Makefile
@@ -205,34 +205,25 @@ cscope:

###########################################################################

-
-# The user can override the build_env variable (or one or more of *_env
-# variables below) to decide if to take the host's C/C++ libraries, or
-# those from the external/ directory.
-build_env ?= $(if $(filter $(host_arch), $(arch)),host,external)
-ifeq ($(build_env), host)
- gcc_lib_env ?= host
- cxx_lib_env ?= host
- gcc_include_env ?= host
- boost_env ?= host
-else
- gcc_lib_env ?= external
- cxx_lib_env ?= external
- gcc_include_env ?= external
- boost_env ?= external
-endif
-
-
local-includes =
INCLUDES = $(local-includes) -Iarch/$(arch) -I. -Iinclude -Iarch/common
INCLUDES += -isystem include/glibc-compat

-gccbase = external/$(arch)/gcc.bin
-miscbase = external/$(arch)/misc.bin

+aarch64_gccbase = build/downloaded_packages/aarch64/gcc/install
+aarch64_boostbase = build/downloaded_packages/aarch64/boost/install
+
+ifeq ($(arch),aarch64)
+ifeq (,$(wildcard $(aarch64_gccbase)))
+ $(error Missing $(aarch64_gccbase) directory. Please run "./scripts/download_fedora_aarch64_packages.py")
+endif
+ifeq (,$(wildcard $(aarch64_boostbase)))
+ $(error Missing $(aarch64_boostbase) directory. Please run "./scripts/download_fedora_aarch64_packages.py")
+endif
+endif

-ifeq ($(gcc_include_env), external)
- gcc-inc-base := $(dir $(shell find $(gccbase)/ -name vector | grep -v -e debug/vector$$ -e profile/vector$$))
- gcc-inc-base3 := $(dir $(shell dirname `find $(gccbase)/ -name c++config.h | grep -v /32/`))
+ifeq ($(arch),aarch64)
+ gcc-inc-base := $(dir $(shell find $(aarch64_gccbase)/ -name vector | grep -v -e debug/vector$$ -e profile/vector$$ -e experimental/vector$$))
+ gcc-inc-base3 := $(dir $(shell dirname `find $(aarch64_gccbase)/ -name c++config.h | grep -v /32/`))
INCLUDES += -isystem $(gcc-inc-base)
INCLUDES += -isystem $(gcc-inc-base3)
endif

@@ -247,7 +238,7 @@ INCLUDES += -isystem $(libfdt_base)

endif

INCLUDES += $(boost-includes)
-ifeq ($(gcc_include_env), host)
+ifeq ($(arch),x64)
# Starting in Gcc 6, the standard C++ header files (which we do not change)
# must precede in the include path the C header files (which we replace).
# This is explained in https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70722.

@@ -257,8 +248,8 @@ INCLUDES += $(shell $(CXX) -E -xc++ - -v </dev/null 2>&1 | awk '/^End/ {exit} /^

endif
INCLUDES += -isystem include/api
INCLUDES += -isystem include/api/$(arch)
-ifeq ($(gcc_include_env), external)
- gcc-inc-base2 := $(dir $(shell find $(gccbase)/ -name unwind.h))
+ifeq ($(arch),aarch64)
+ gcc-inc-base2 := $(dir $(shell find $(aarch64_gccbase)/ -name unwind.h))
# must be after include/api, since it includes some libc-style headers:
INCLUDES += -isystem $(gcc-inc-base2)
endif

@@ -289,7 +280,7 @@ $(out)/musl/%.o: source-dialects =

kernel-defines = -D_KERNEL $(source-dialects)

-gcc-sysroot = $(if $(CROSS_PREFIX), --sysroot external/$(arch)/gcc.bin) \
+gcc-sysroot = $(if $(CROSS_PREFIX), --sysroot $(aarch64_gccbase)) \

# This play the same role as "_KERNEL", but _KERNEL unfortunately is too
# overloaded. A lot of files will expect it to be set no matter what, specially

@@ -314,11 +305,9 @@ COMMON = $(autodepend) -g -Wall -Wno-pointer-arith $(CFLAGS_WERROR) -Wformat=0 -

-include compiler/include/intrinsics.hh \
$(arch-cflags) $(conf-opt) $(acpi-defines) $(tracing-flags) $(gcc-sysroot) \
$(configuration) -D__OSV__ -D__XEN_INTERFACE_VERSION__="0x00030207" -DARCH_STRING=$(ARCH_STR) $(EXTRA_FLAGS)
-ifeq ($(gcc_include_env), external)
-ifeq ($(boost_env), external)
+ifeq ($(arch),aarch64)
COMMON += -nostdinc
endif
-endif

tracing-flags-0 =
tracing-flags-1 = -finstrument-functions -finstrument-functions-exclude-file-list=c++,trace.cc,trace.hh,align.hh,mmintrin.h

@@ -1807,7 +1796,7 @@ objects += $(addprefix fs/, $(fs_objs))

objects += $(addprefix libc/, $(libc))
objects += $(addprefix musl/src/, $(musl))

-ifeq ($(cxx_lib_env), host)
+ifeq ($(arch),x64)
libstdc++.a := $(shell $(CXX) -print-file-name=libstdc++.a)
ifeq ($(filter /%,$(libstdc++.a)),)
$(error Error: libstdc++.a needs to be installed.)

@@ -1818,11 +1807,11 @@ ifeq ($(cxx_lib_env), host)

$(error Error: libsupc++.a needs to be installed.)
endif
else
- libstdc++.a := $(shell find $(gccbase)/ -name libstdc++.a)
- libsupc++.a := $(shell find $(gccbase)/ -name libsupc++.a)
+ libstdc++.a := $(shell find $(aarch64_gccbase)/ -name libstdc++.a)
+ libsupc++.a := $(shell find $(aarch64_gccbase)/ -name libsupc++.a)
endif

-ifeq ($(gcc_lib_env), host)
+ifeq ($(arch),x64)
libgcc.a := $(shell $(CC) -print-libgcc-file-name)
ifeq ($(filter /%,$(libgcc.a)),)
$(error Error: libgcc.a needs to be installed.)

@@ -1833,11 +1822,11 @@ ifeq ($(gcc_lib_env), host)

$(error Error: libgcc_eh.a needs to be installed.)
endif
else
- libgcc.a := $(shell find $(gccbase)/ -name libgcc.a | grep -v /32/)
- libgcc_eh.a := $(shell find $(gccbase)/ -name libgcc_eh.a | grep -v /32/)
+ libgcc.a := $(shell find $(aarch64_gccbase)/ -name libgcc.a | grep -v /32/)
+ libgcc_eh.a := $(shell find $(aarch64_gccbase)/ -name libgcc_eh.a | grep -v /32/)
endif

-ifeq ($(boost_env), host)
+ifeq ($(arch),x64)
# link with -mt if present, else the base version (and hope it is multithreaded)
boost-mt := -mt
boost-lib-dir := $(dir $(shell $(CC) --print-file-name libboost_system$(boost-mt).a))

@@ -1853,9 +1842,9 @@ ifeq ($(boost_env), host)

# special for Boost.
boost-includes =
else
- boost-lib-dir := $(firstword $(dir $(shell find $(miscbase)/ -name libboost_system*.a)))
+ boost-lib-dir := $(firstword $(dir $(shell find $(aarch64_boostbase)/ -name libboost_system*.a)))
boost-mt := $(if $(filter %-mt.a, $(wildcard $(boost-lib-dir)/*.a)),-mt)
- boost-includes = -isystem $(miscbase)/usr/include
+ boost-includes = -isystem $(aarch64_boostbase)/usr/include
endif

boost-libs := $(boost-lib-dir)/libboost_system$(boost-mt).a

@@ -1931,7 +1920,7 @@ $(bootfs_manifest_dep): phony

[Waldek Kozaczuk]

Please note I did not push this patch as first of all I would like to have it reviewed and secondly, it would have broken the build on Jenkins machine as we need to modify it to call 'cripts/download_fedora_aarch64_packages.py' for the aarch64 part of the build.

[Commit Bot]

From: Waldemar Kozaczuk <jwkoz...@gmail.com>
Committer: Waldemar Kozaczuk <jwkoz...@gmail.com>
Branch: master

externals: eliminate aarch64 gcc and boost libraries from externals

This patch eliminates aarch64 gcc and boost libraries from externals
in lieu of new mechanism to download the packages from Fedora
repository.

Please note that Jenkins build will need to be modified
to call scripts/download_fedora_aarch64_packages.py before building
aarch64 image.

Fixes #743

Signed-off-by: Waldemar Kozaczuk <jwkoz...@gmail.com>

Message-Id: <20200302131248.5...@gmail.com>

---
diff --git a/.gitmodules b/.gitmodules

--- a/.gitmodules
+++ b/.gitmodules
@@ -6,12 +6,6 @@
path = apps
url = ../../cloudius-systems/osv-apps
ignore = dirty
-[submodule "external/aarch64/gcc.bin"]
- path = external/aarch64/gcc.bin
- url = ../../cloudius-systems/aarch64-gcc.bin.git
-[submodule "external/aarch64/misc.bin"]
- path = external/aarch64/misc.bin
- url = ../../cloudius-systems/aarch64-misc.bin.git
[submodule "modules/httpserver/swagger-ui"]
path = modules/httpserver-html5-gui/swagger-ui
url = ../../cloudius-systems/swagger-ui.git
diff --git a/Makefile b/Makefile

--- a/external/aarch64/gcc.bin
+++ b/external/aarch64/gcc.bin

@@ -1 +0,0 @@
-Subproject commit a85ead11151e18075153f44fa685757dbc6e9675
diff --git a/external/aarch64/misc.bin b/external/aarch64/misc.bin

--- a/external/aarch64/misc.bin
+++ b/external/aarch64/misc.bin