[PATCH] scripts: download and build arbitrary version of boost
30 views
Skip to first unread message
Waldemar Kozaczuk
Feb 5, 2021, 4:22:33 PM2/5/21
to osv...@googlegroups.com, Waldemar Kozaczuk
On some distributions, like CentOS, the default version of boost
is really old. So the best option is to build a newer one from source.
Also even on Fedora or Ubuntu it might be useful to point to
different version of boost other than what ./scripts/setup.py installs.
This script automates process of downloading the sources of specified version
of boost and building it using default or specified toolchain.
The last CROSS_AARCH64_TOOLSET parameter (gcc or gcc-arm) is intended to cross-compile
the aarch64 version of boost on x86_64 machine.
The built version of the boost is symlinked into the
build/downloaded_packages/<arch>/boost/install/ directory and will
automatically be used where cross-compiling aarch64 OSv kernel.
In order to use x64 version of boost built using this script
we still need to make changes to the main makefile and
modules/common.gmk.
Signed-off-by: Waldemar Kozaczuk <jwkoz...@gmail.com>
---
scripts/download_and_build_boost.sh | 63 +++++++++++++++++++++++++++++
1 file changed, 63 insertions(+)
create mode 100755 scripts/download_and_build_boost.sh
diff --git a/scripts/download_and_build_boost.sh b/scripts/download_and_build_boost.sh
new file mode 100755
index 00000000..7181f5d4
--- /dev/null
+++ b/scripts/download_and_build_boost.sh
@@ -0,0 +1,63 @@
+#!/bin/bash
+#
+# Copyright (C) 2020 Waldemar Kozaczuk
+#
+# This work is open source software, licensed under the terms of the
+# BSD license as described in the LICENSE file in the top-level directory.
+#
+
+# Usage:
+# ./scripts/download_and_build_boost.sh <BOOST_VERSION> <ARCH> <CROSS_AARCH64_TOOLSET>
+#
+
+OSV_ROOT=$(dirname $(readlink -e $0))/..
+echo $OSV_ROOT
+
+BOOST_VERSION=$1
+if [[ "${BOOST_VERSION}" == "" ]]; then
+ BOOST_VERSION="1.70.0"
+fi
+BOOST_VERSION2=${BOOST_VERSION//[.]/_}
+
+ARCH=$2
+if [[ "${ARCH}" == "" ]]; then
+ ARCH=$(uname -m)
+fi
+
+#If CROSS_AARCH64_TOOLSET is set or passed we assume we are crosscompiling aarch64 on Intel
+CROSS_AARCH64_TOOLSET=$3
+
+rm -rf %s/boost/install
+
+TAR_DOWNLOAD_DIR=${OSV_ROOT}/build/downloaded_packages/${ARCH}/boost/upstream/
+mkdir -p ${TAR_DOWNLOAD_DIR}
+
+BOOST_URL=https://dl.bintray.com/boostorg/release/${BOOST_VERSION}/source/boost_${BOOST_VERSION2}.tar.gz
+wget -c -O ${TAR_DOWNLOAD_DIR}/boost_${BOOST_VERSION2}.tar.gz ${BOOST_URL}
+
+pushd ${TAR_DOWNLOAD_DIR}
+rm -rf ./boost_${BOOST_VERSION2} && tar -xf ./boost_${BOOST_VERSION2}.tar.gz
+cd ./boost_${BOOST_VERSION2}
+./bootstrap.sh --with-libraries=system,thread,test,chrono,regex,date_time,filesystem,locale,random,atomic,log,program_options
+
+BOOST_DIR=$(pwd)
+case $CROSS_AARCH64_TOOLSET in
+ gcc)
+ echo "using gcc : arm : aarch64-linux-gnu-g++ ;" > user-config.jam ;;
+ gcc-arm)
+ echo "using gcc : arm : aarch64-none-linux-gnu-g++ ;" > user-config.jam ;;
+esac
+
+B2_OPTIONS="threading=multi"
+if [[ "${CROSS_AARCH64_TOOLSET}" != "" ]]; then
+ B2_OPTIONS="${B2_OPTIONS} --user-config=${BOOST_DIR}/user-config.jam toolset=${CROSS_AARCH64_TOOLSET} architecture=arm address-model=64"
+fi
+./b2 ${B2_OPTIONS} -j$(nproc)
+
+#
+# Create symlinks to install boost and make it visible to OSv makefile
+rm -f ../../install
+ln -s upstream/boost_${BOOST_VERSION2}/stage ../../install
+mkdir -p stage/usr/include
+ln -s ../../../boost stage/usr/include/boost
+popd
--
2.29.2
is really old. So the best option is to build a newer one from source.
Also even on Fedora or Ubuntu it might be useful to point to
different version of boost other than what ./scripts/setup.py installs.
This script automates process of downloading the sources of specified version
of boost and building it using default or specified toolchain.
The last CROSS_AARCH64_TOOLSET parameter (gcc or gcc-arm) is intended to cross-compile
the aarch64 version of boost on x86_64 machine.
The built version of the boost is symlinked into the
build/downloaded_packages/<arch>/boost/install/ directory and will
automatically be used where cross-compiling aarch64 OSv kernel.
In order to use x64 version of boost built using this script
we still need to make changes to the main makefile and
modules/common.gmk.
Signed-off-by: Waldemar Kozaczuk <jwkoz...@gmail.com>
---
scripts/download_and_build_boost.sh | 63 +++++++++++++++++++++++++++++
1 file changed, 63 insertions(+)
create mode 100755 scripts/download_and_build_boost.sh
diff --git a/scripts/download_and_build_boost.sh b/scripts/download_and_build_boost.sh
new file mode 100755
index 00000000..7181f5d4
--- /dev/null
+++ b/scripts/download_and_build_boost.sh
@@ -0,0 +1,63 @@
+#!/bin/bash
+#
+# Copyright (C) 2020 Waldemar Kozaczuk
+#
+# This work is open source software, licensed under the terms of the
+# BSD license as described in the LICENSE file in the top-level directory.
+#
+
+# Usage:
+# ./scripts/download_and_build_boost.sh <BOOST_VERSION> <ARCH> <CROSS_AARCH64_TOOLSET>
+#
+
+OSV_ROOT=$(dirname $(readlink -e $0))/..
+echo $OSV_ROOT
+
+BOOST_VERSION=$1
+if [[ "${BOOST_VERSION}" == "" ]]; then
+ BOOST_VERSION="1.70.0"
+fi
+BOOST_VERSION2=${BOOST_VERSION//[.]/_}
+
+ARCH=$2
+if [[ "${ARCH}" == "" ]]; then
+ ARCH=$(uname -m)
+fi
+
+#If CROSS_AARCH64_TOOLSET is set or passed we assume we are crosscompiling aarch64 on Intel
+CROSS_AARCH64_TOOLSET=$3
+
+rm -rf %s/boost/install
+
+TAR_DOWNLOAD_DIR=${OSV_ROOT}/build/downloaded_packages/${ARCH}/boost/upstream/
+mkdir -p ${TAR_DOWNLOAD_DIR}
+
+BOOST_URL=https://dl.bintray.com/boostorg/release/${BOOST_VERSION}/source/boost_${BOOST_VERSION2}.tar.gz
+wget -c -O ${TAR_DOWNLOAD_DIR}/boost_${BOOST_VERSION2}.tar.gz ${BOOST_URL}
+
+pushd ${TAR_DOWNLOAD_DIR}
+rm -rf ./boost_${BOOST_VERSION2} && tar -xf ./boost_${BOOST_VERSION2}.tar.gz
+cd ./boost_${BOOST_VERSION2}
+./bootstrap.sh --with-libraries=system,thread,test,chrono,regex,date_time,filesystem,locale,random,atomic,log,program_options
+
+BOOST_DIR=$(pwd)
+case $CROSS_AARCH64_TOOLSET in
+ gcc)
+ echo "using gcc : arm : aarch64-linux-gnu-g++ ;" > user-config.jam ;;
+ gcc-arm)
+ echo "using gcc : arm : aarch64-none-linux-gnu-g++ ;" > user-config.jam ;;
+esac
+
+B2_OPTIONS="threading=multi"
+if [[ "${CROSS_AARCH64_TOOLSET}" != "" ]]; then
+ B2_OPTIONS="${B2_OPTIONS} --user-config=${BOOST_DIR}/user-config.jam toolset=${CROSS_AARCH64_TOOLSET} architecture=arm address-model=64"
+fi
+./b2 ${B2_OPTIONS} -j$(nproc)
+
+#
+# Create symlinks to install boost and make it visible to OSv makefile
+rm -f ../../install
+ln -s upstream/boost_${BOOST_VERSION2}/stage ../../install
+mkdir -p stage/usr/include
+ln -s ../../../boost stage/usr/include/boost
+popd
--
2.29.2
Nadav Har'El
Feb 8, 2021, 3:20:55 AM2/8/21
to Waldemar Kozaczuk, Osv Dev
Nitpicks on code which scares me (even if in this particular case it's probably benign):
1. "pushd" may fail, and if it does, the script just continues in the wrong directory. In such scripts I usually use "set -e" to make sure that any failed command stops the entire script. But you can also add checks to individual commands if you prefer, e.g., "pushd ... || echo oops; exit 1"
2. If $BOOST_VERSION2 contains a space the above command can rm -rf anything! Never try it with BOOST_VERSION2="xyz /"... If you don't believe me try the following non-harmful example in bash:
$ a="xyz /"
$ ls x${a}
ls: cannot access 'xxyz': No such file or directory
/:
boot etc lib ...
boot etc lib ...
The solution is to put quotes around the $a:
$ a = "xyz /"
$ ls "x$a"
ls: cannot access 'xxyz /': No such file or directory
Similarly below any time you use a variable.
+cd ./boost_${BOOST_VERSION2}
+./bootstrap.sh --with-libraries=system,thread,test,chrono,regex,date_time,filesystem,locale,random,atomic,log,program_options
+
+BOOST_DIR=$(pwd)
+case $CROSS_AARCH64_TOOLSET in
+ gcc)
+ echo "using gcc : arm : aarch64-linux-gnu-g++ ;" > user-config.jam ;;
+ gcc-arm)
+ echo "using gcc : arm : aarch64-none-linux-gnu-g++ ;" > user-config.jam ;;
+esac
+
+B2_OPTIONS="threading=multi"
+if [[ "${CROSS_AARCH64_TOOLSET}" != "" ]]; then
+ B2_OPTIONS="${B2_OPTIONS} --user-config=${BOOST_DIR}/user-config.jam toolset=${CROSS_AARCH64_TOOLSET} architecture=arm address-model=64"
+fi
+./b2 ${B2_OPTIONS} -j$(nproc)
+
+#
+# Create symlinks to install boost and make it visible to OSv makefile
+rm -f ../../install
+ln -s upstream/boost_${BOOST_VERSION2}/stage ../../install
+mkdir -p stage/usr/include
+ln -s ../../../boost stage/usr/include/boost
+popd
The popd at the end of the script does nothing (and also pushd instead of just cd wasn't needed), but I guess it's harmless.
--
2.29.2
--
You received this message because you are subscribed to the Google Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/20210205212222.1038474-1-jwkozaczuk%40gmail.com.
Stewart Hildebrand
Feb 8, 2021, 1:17:29 PM2/8/21
to OSv Development
This is nitpicky, but uname -m wouldn't correspond to the arch of HOST_CXX (g++) when chrooting into a aarch64 sysroot. This is a nitpick (not an objection) because it's probably an unusual situation to find yourself in. If really want to chase this down, I recommend replicating the logic from the main Makefile.
There is a possibility of many more combinations of aarch64-*-g++. aarch64-none-elf-, aarch64-xilinx-linux-, to name a couple. Instead of trying to predict all possible combinations, I recommend using CROSS_PREFIX instead.
+
+B2_OPTIONS="threading=multi"
+if [[ "${CROSS_AARCH64_TOOLSET}" != "" ]]; then
+ B2_OPTIONS="${B2_OPTIONS} --user-config=${BOOST_DIR}/user-config.jam toolset=${CROSS_AARCH64_TOOLSET} architecture=arm address-model=64"
I'm not convinced that the toolset= option should correspond to a particular CROSS_PREFIX. Wouldn't we always want toolset=gcc-arm for aarch64 builds?
Waldemar Kozaczuk
Feb 17, 2021, 12:23:21 AM2/17/21
to osv...@googlegroups.com, Waldemar Kozaczuk
On some distributions, like CentOS, the default version of boost
is really old. So the best option is to build a newer one from source.
Also even on Fedora or Ubuntu it might be useful to point to
different version of boost other than what ./scripts/setup.py installs.
This script automates process of downloading the sources of specified version
of boost and building it using default or specified toolchain.
Please set proper CROSS_PREFIX if you intend to cross-compile
is really old. So the best option is to build a newer one from source.
Also even on Fedora or Ubuntu it might be useful to point to
different version of boost other than what ./scripts/setup.py installs.
This script automates process of downloading the sources of specified version
of boost and building it using default or specified toolchain.
the aarch64 version of boost on x86_64 machine like so:
CROSS_PREFIX=aarch64-none-linux-gnu- ./scripts/download_and_build_boost.sh 1.70.0 aarch64
The built version of the boost is symlinked into the
build/downloaded_packages/<arch>/boost/install/ directory and will
automatically be used where cross-compiling aarch64 OSv kernel.
In order to use x64 version of boost built using this script
we still need to make changes to the main makefile and
modules/common.gmk.
Signed-off-by: Waldemar Kozaczuk <jwkoz...@gmail.com>
---
1 file changed, 56 insertions(+)
create mode 100755 scripts/download_and_build_boost.sh
diff --git a/scripts/download_and_build_boost.sh b/scripts/download_and_build_boost.sh
new file mode 100755
index 00000000..bf250e98
diff --git a/scripts/download_and_build_boost.sh b/scripts/download_and_build_boost.sh
new file mode 100755
--- /dev/null
+++ b/scripts/download_and_build_boost.sh
@@ -0,0 +1,56 @@
+#!/bin/bash
+#
+# Copyright (C) 2021 Waldemar Kozaczuk
+#
+# This work is open source software, licensed under the terms of the
+# BSD license as described in the LICENSE file in the top-level directory.
+#
+
+# Usage:
+# ./scripts/download_and_build_boost.sh <BOOST_VERSION> <ARCH>
+#
+# This work is open source software, licensed under the terms of the
+# BSD license as described in the LICENSE file in the top-level directory.
+#
+
+# Usage:
+# ./scripts/download_and_build_boost.sh <BOOST_VERSION> <ARCH>
+set -e
+
+OSV_ROOT=$(dirname $(readlink -e $0))/..
+
+BOOST_VERSION="$1"
+OSV_ROOT=$(dirname $(readlink -e $0))/..
+
+if [[ "${BOOST_VERSION}" == "" ]]; then
+ BOOST_VERSION="1.70.0"
+fi
+BOOST_VERSION2=${BOOST_VERSION//[.]/_}
+
+ARCH=$2
+if [[ "${ARCH}" == "" ]]; then
+ ARCH=$(uname -m)
+fi
+ BOOST_VERSION="1.70.0"
+fi
+BOOST_VERSION2=${BOOST_VERSION//[.]/_}
+
+ARCH=$2
+if [[ "${ARCH}" == "" ]]; then
+ ARCH=$(uname -m)
+
+TAR_DOWNLOAD_DIR="${OSV_ROOT}"/build/downloaded_packages/"${ARCH}"/boost/upstream/
+mkdir -p "${TAR_DOWNLOAD_DIR}"
+
+BOOST_URL=https://dl.bintray.com/boostorg/release/"${BOOST_VERSION}"/source/boost_"${BOOST_VERSION2}".tar.gz
+
+wget -c -O "${TAR_DOWNLOAD_DIR}"/boost_"${BOOST_VERSION2}".tar.gz "${BOOST_URL}"
+
+pushd "${TAR_DOWNLOAD_DIR}"
+rm -rf ./boost_"${BOOST_VERSION2}"
+tar -xf ./boost_"${BOOST_VERSION2}".tar.gz
+pushd "${TAR_DOWNLOAD_DIR}"
+rm -rf ./boost_"${BOOST_VERSION2}"
+cd ./boost_"${BOOST_VERSION2}"
+./bootstrap.sh --with-libraries=system,thread,test,chrono,regex,date_time,filesystem,locale,random,atomic,log,program_options
+
+BOOST_DIR=$(pwd)
+if [[ "$CROSS_PREFIX" == aarch64* ]]; then
+ echo "using gcc : arm : ${CROSS_PREFIX}g++ ;" > user-config.jam
+fi
+
+B2_OPTIONS="threading=multi"
+if [[ "${CROSS_PREFIX}" == aarch64* ]]; then
+ B2_OPTIONS="${B2_OPTIONS} --user-config=${BOOST_DIR}/user-config.jam toolset=gcc-arm architecture=arm address-model=64"
Waldek Kozaczuk
Feb 17, 2021, 12:31:22 AM2/17/21
to OSv Development
Disregard this one - v3 is coming :-)
Waldemar Kozaczuk
Feb 17, 2021, 12:31:59 AM2/17/21
to osv...@googlegroups.com, Waldemar Kozaczuk
On some distributions, like CentOS, the default version of boost
is really old. So the best option is to build a newer one from source.
Also even on Fedora or Ubuntu it might be useful to point to
different version of boost other than what ./scripts/setup.py installs.
This script automates process of downloading the sources of specified version
of boost and building it using default or specified toolchain.
Please set proper CROSS_PREFIX if you intend to cross-compile
the aarch64 version of boost on x86_64 machine like so:
CROSS_PREFIX=aarch64-none-linux-gnu- ./scripts/download_and_build_boost.sh 1.70.0 aarch64
The built version of the boost is symlinked into the
build/downloaded_packages/<arch>/boost/install/ directory and will
automatically be used where cross-compiling aarch64 OSv kernel.
In order to use x64 version of boost built using this script
we still need to make changes to the main makefile and
modules/common.gmk.
Signed-off-by: Waldemar Kozaczuk <jwkoz...@gmail.com>
---
scripts/download_and_build_boost.sh | 56 +++++++++++++++++++++++++++++
1 file changed, 56 insertions(+)
create mode 100755 scripts/download_and_build_boost.sh
diff --git a/scripts/download_and_build_boost.sh b/scripts/download_and_build_boost.sh
new file mode 100755
index 00000000..5431a99b
is really old. So the best option is to build a newer one from source.
Also even on Fedora or Ubuntu it might be useful to point to
different version of boost other than what ./scripts/setup.py installs.
This script automates process of downloading the sources of specified version
of boost and building it using default or specified toolchain.
Please set proper CROSS_PREFIX if you intend to cross-compile
the aarch64 version of boost on x86_64 machine like so:
CROSS_PREFIX=aarch64-none-linux-gnu- ./scripts/download_and_build_boost.sh 1.70.0 aarch64
The built version of the boost is symlinked into the
build/downloaded_packages/<arch>/boost/install/ directory and will
automatically be used where cross-compiling aarch64 OSv kernel.
In order to use x64 version of boost built using this script
we still need to make changes to the main makefile and
modules/common.gmk.
Signed-off-by: Waldemar Kozaczuk <jwkoz...@gmail.com>
---
scripts/download_and_build_boost.sh | 56 +++++++++++++++++++++++++++++
1 file changed, 56 insertions(+)
create mode 100755 scripts/download_and_build_boost.sh
diff --git a/scripts/download_and_build_boost.sh b/scripts/download_and_build_boost.sh
new file mode 100755
+mkdir -p "${TAR_DOWNLOAD_DIR}"
+
+BOOST_URL=https://dl.bintray.com/boostorg/release/"${BOOST_VERSION}"/source/boost_"${BOOST_VERSION2}".tar.gz
+wget -c -O "${TAR_DOWNLOAD_DIR}"/boost_"${BOOST_VERSION2}".tar.gz "${BOOST_URL}"
+
+pushd "${TAR_DOWNLOAD_DIR}"
+rm -rf "./boost_${BOOST_VERSION2}"
+tar -xf "./boost_${BOOST_VERSION2}.tar.gz"
+
+BOOST_URL=https://dl.bintray.com/boostorg/release/"${BOOST_VERSION}"/source/boost_"${BOOST_VERSION2}".tar.gz
+wget -c -O "${TAR_DOWNLOAD_DIR}"/boost_"${BOOST_VERSION2}".tar.gz "${BOOST_URL}"
+
+pushd "${TAR_DOWNLOAD_DIR}"
+rm -rf "./boost_${BOOST_VERSION2}"
+cd "./boost_${BOOST_VERSION2}"
Commit Bot
Feb 18, 2021, 5:57:17 PM2/18/21
to osv...@googlegroups.com, Waldemar Kozaczuk
From: Waldemar Kozaczuk <jwkoz...@gmail.com>
Committer: Waldemar Kozaczuk <jwkoz...@gmail.com>
Branch: master
scripts: download and build arbitrary version of boost
On some distributions, like CentOS, the default version of boost
is really old. So the best option is to build a newer one from source.
Also even on Fedora or Ubuntu it might be useful to point to
different version of boost other than what ./scripts/setup.py installs.
This script automates process of downloading the sources of specified version
of boost and building it using default or specified toolchain.
Please set proper CROSS_PREFIX if you intend to cross-compile
the aarch64 version of boost on x86_64 machine like so:
CROSS_PREFIX=aarch64-none-linux-gnu- ./scripts/download_and_build_boost.sh 1.70.0 aarch64
The built version of the boost is symlinked into the
build/downloaded_packages/<arch>/boost/install/ directory and will
automatically be used where cross-compiling aarch64 OSv kernel.
In order to use x64 version of boost built using this script
we still need to make changes to the main makefile and
modules/common.gmk.
Signed-off-by: Waldemar Kozaczuk <jwkoz...@gmail.com>
---
diff --git a/scripts/download_and_build_boost.sh b/scripts/download_and_build_boost.sh
--- a/scripts/download_and_build_boost.sh
Committer: Waldemar Kozaczuk <jwkoz...@gmail.com>
Branch: master
scripts: download and build arbitrary version of boost
On some distributions, like CentOS, the default version of boost
is really old. So the best option is to build a newer one from source.
Also even on Fedora or Ubuntu it might be useful to point to
different version of boost other than what ./scripts/setup.py installs.
This script automates process of downloading the sources of specified version
of boost and building it using default or specified toolchain.
Please set proper CROSS_PREFIX if you intend to cross-compile
the aarch64 version of boost on x86_64 machine like so:
CROSS_PREFIX=aarch64-none-linux-gnu- ./scripts/download_and_build_boost.sh 1.70.0 aarch64
The built version of the boost is symlinked into the
build/downloaded_packages/<arch>/boost/install/ directory and will
automatically be used where cross-compiling aarch64 OSv kernel.
In order to use x64 version of boost built using this script
we still need to make changes to the main makefile and
modules/common.gmk.
Signed-off-by: Waldemar Kozaczuk <jwkoz...@gmail.com>
---
--- a/scripts/download_and_build_boost.sh
Reply all
Reply to author
Forward
0 new messages