Installing ColumnStore on mariadb 10.5.6 from source

Saurabh Singh

unread,

Nov 11, 2020, 4:38:00 PM11/11/20

to MariaDB ColumnStore

Hi,

I tried deploying a single node mariadb server 10.5.6 from the deployment guide but got the following error:

> sudo ./mariadb_repo_setup --mariadb-server-version="mariadb-10.5"

[error] The MariaDB Repository only supports x86_64 and aarch64 (detected s390x).

[error] The MariaDB Repository supports these Linux OSs, on x86-64 only:

* RHEL/CentOS 6, 7, & 8

* Ubuntu 16.04 LTS (xenial), 18.04 LTS (bionic), & 20.04 LTS (focal)

* Debian 8 (jessie), 9 (stretch), & 10 (buster)

* SLES 12 & 15

[error] See https://mariadb.com/kb/en/mariadb/mariadb-package-repository-setup-and-usage/#platform-support

So, I build Mariadb 10.5.6 from source code on my machine:

NAME="SLES"

VERSION="15-SP1"

VERSION_ID="15.1"

PRETTY_NAME="SUSE Linux Enterprise Server 15 SP1"

ID="sles"

ID_LIKE="suse"

ANSI_COLOR="0;32"

CPE_NAME="cpe:/o:suse:sles:15:sp1"

I did not find any document to install columnstore from the source code for this version. I see the official documentation saying columnstore is included with the new Mariadb versions which I can see in my /build_directory/storage too but have been struggling to get instructions to install it from there.

Any help with the guide would be really helpful.

Thanks,

Saurabh

drrtuy

unread,

Nov 13, 2020, 2:15:37 PM11/13/20

to MariaDB ColumnStore

Greetings Saurabh,

I'm very pleased to see you interested in the topic.

There is no developer guide on how to build MCS from the source b/c we were in transition. MCS 1.2 used to use a fork of MariaDB and MCS 5.4 uses a generic MariaDB 10.5.

There is still no guide available however I can give you an automation script I use to build and install MCS from source every day and answer your questions. I would appreciate if you will contribute the how-to to community. Here is the automation script I mentioned. You just run it with either centos|focal|bionic to set up a proper file installation paths(for config files mostly). The script has a number of parameters and make a number of assumptions. You surely need to change MDB_SOURCE_PATH according to your filesystem layout. Plz be aware that MCS now is not portable so you should use a default paths so you should use either VM or container for compilation/installation.

Feel free to ask questions.

Regards,

Roman

четверг, 12 ноября 2020 г. в 00:38:00 UTC+3, saura...@gmail.com:

Message has been deleted

Saurabh Singh

unread,

Nov 16, 2020, 1:27:04 PM11/16/20

to MariaDB ColumnStore

Thank you for the steps Roman.

I could not reply in the group so I am emailing my response. Since, I am have the SUSE server with s390x and not x86_64 version, I am not sure if the automation script will work for my server. Will give it a try. But, in case it does not work I wanted to check what version of MCS can be installed on mariaDB 10.5.x as I would not want to go back to an earlier version of mariadb to install latest MCS engine.

Also, on a side note, it would be easier for me to install mariabDB from repo but when I try to do a checksum on the mariadb_repo_set from the download link it gives an error:

> sudo echo "2de6253842f230bc554d3f5ab0c0dbf717caffbf45ae6893740707961c8407b7 mariadb_repo_setup" \

> | sha256sum -c -

mariadb_repo_setup: FAILED

sha256sum: WARNING: 1 computed checksum did NOT match

The file matched an earlier version of mariadb and if I force it to install from the same repo it gives the below error:

./mariadb_repo_setup --mariadb-server-version="mariadb-10.5"

[error] The MariaDB Repository only supports x86_64 and aarch64 (detected s390x).

[error] The MariaDB Repository supports these Linux OSs, on x86-64 only:

* RHEL/CentOS 6, 7, & 8

* Ubuntu 16.04 LTS (xenial), 18.04 LTS (bionic), & 20.04 LTS (focal)

* Debian 8 (jessie), 9 (stretch), & 10 (buster)

* SLES 12 & 15

Just wanted to confirm if there is no repo for mariadb 10.5.x for my version of SLES 15 SP1 (s390x) running on IBM Z and I need to do a source build only to get MCS up and running? I will be trying the automation script today checking the parameters but since it doesn't say if it supports SLES so wanted to know if it will work for my environment as well.

Thanks,

Saurabh

Roland Noland

unread,

Nov 17, 2020, 3:31:46 AM11/17/20

to Saurabh Singh, MariaDB ColumnStore

This would be interesting to spin up MCS on s390.

There is no MCS package for s390 for sure however I can help you with the compilation and installation.

The script I shared earlier will help you to compile MDB with MCS engine given that we have all the prerequisites available for s390 version of SLES 15.
So could you go through the list of packages from this if section of the script and check that all packages are available for s390?

Then move on to cmake using the parameters from this line. You should replace -DRPM=centos7 with -DRRP=sles15 though.

I'll comment of future steps when we finish with this.

Regards,

Roman

пн, 16 нояб. 2020 г. в 21:27, Saurabh Singh <saura...@gmail.com>:

--
You received this message because you are subscribed to the Google Groups "MariaDB ColumnStore" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mariadb-columns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mariadb-columnstore/4ba2946d-3eae-4e9e-854f-5d00e190d0c4n%40googlegroups.com.

Saurabh Singh

unread,

Nov 17, 2020, 7:45:39 PM11/17/20

to MariaDB ColumnStore

Thanks for your response, Roman.

I already tried executing the script in pieces starting with the dependencies which did not produce any error. Most of the logs looked like:

--- No update candidate for 'net-tools-2.0+git20170221.479bb4a-3.11.s390x'. The highest available version is already installed.

'expect' is already installed.

OR

--- 'libboost_system-devel' not found in package names. Trying capabilities.

'libboost_system1_66_0-devel' providing 'libboost_system-devel' is already installed.

However, the CMake still gave error as below (since the output is very large, I have attached these outputs below):

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

:data/mariadb/src_root/build_mariadb # cmake ../mariadb -DCMAKE_BUILD_TYPE=debug -DWITH_SYSTEMD=yes -DPLUGIN_AUTH_GSSAPI=NO -DPLUGIN_COLUMNSTORE=YES -DPLUGIN_MROONGA=NO -DPLUGIN_ROCKSDB=NO -DPLUGIN_TOKUDB=NO -DPLUGIN_CONNECT=NO -DPLUGIN_SPIDER=NO -DPLUGIN_OQGRAPH=NO -DPLUGIN_SPHINX=NO -DMYSQL_DATADIR=/data/mariadb/data -DTMPDIR=/data/mariadb/tmp -DBUILD_CONFIG=mysql_release -DWITH_WSREP=OFF -DWITH_SSL=system -DRRP=sles15 | tee /data/mariadb/tmp/cmakeout.out

CMake Error at cmake/plugin.cmake:298 (MESSAGE):

Plugin COLUMNSTORE cannot be built

Call Stack (most recent call first):

CMakeLists.txt:433 (CONFIGURE_PLUGINS)

-- Configuring incomplete, errors occurred!

See also "/data/mariadb/src_root/build_mariadb/CMakeFiles/CMakeOutput.log".

See also "/data/mariadb/src_root/build_mariadb/CMakeFiles/CMakeError.log".

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

One of the CMake options that I did not use in my command is " -DPLUGIN_AUTH_GSSAPI=NO" not sure using it would have made any difference though. Here is one of the log output from "/data/mariadb/src_root/build_mariadb/CMakeFiles/CMakeError.log"

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Determining if files ndir.h exist failed with the following output:

Change Dir: /data/mariadb/src_root/build_mariadb/CMakeFiles/CMakeTmp

Run Build Command:"/usr/bin/gmake" "cmTC_6b158/fast"

/usr/bin/gmake -f CMakeFiles/cmTC_6b158.dir/build.make CMakeFiles/cmTC_6b158.dir/build

gmake[1]: Entering directory '/data/mariadb/src_root/build_mariadb/CMakeFiles/CMakeTmp'

Building C object CMakeFiles/cmTC_6b158.dir/CheckIncludeFiles.c.o

/usr/bin/cc -D_GNU_SOURCE=1 -pie -fPIC -fstack-protector --param=ssp-buffer-size=4 -o CMakeFiles/cmTC_6b158.dir/CheckIncludeFiles.c.o -c /data/mariadb/src_root/build_mariadb/CMakeFiles/CMakeTmp/CheckIncludeFiles.c

/data/mariadb/src_root/build_mariadb/CMakeFiles/CMakeTmp/CheckIncludeFiles.c:2:10: fatal error: ndir.h: No such file or directory

#include <ndir.h>

^~~~~~~~

compilation terminated.

gmake[1]: *** [CMakeFiles/cmTC_6b158.dir/build.make:66: CMakeFiles/cmTC_6b158.dir/CheckIncludeFiles.c.o] Error 1

gmake[1]: Leaving directory '/data/mariadb/src_root/build_mariadb/CMakeFiles/CMakeTmp'

gmake: *** [Makefile:126: cmTC_6b158/fast] Error 2

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

This might not be of much relevance but sharing the MCS directory structure from the mariadb source directory:

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

#### /data/mariadb/src_root/mariadb -- git clone here (source directory)

#### /data/mariadb/src_root/build_mariadb -- build directory

#### /build_mariadb # ls -l ../mariadb/storage/columnstore/

#### /build_mariadb # ls ../mariadb/storage/columnstore/columnstore/

Please let me know if these logs indicate anything that I am missing during the compilation or I need some further parameters. The CMake also has many "not found" and performing test failed status.

Saurabh Singh

unread,

Nov 17, 2020, 7:48:24 PM11/17/20

to Roland Noland, MariaDB ColumnStore

I am not able to attach the log in the community, so attaching them here for you guys to have a look (if required)

cmakout.out

dependency.out

Roland Noland

unread,

Nov 18, 2020, 2:58:42 AM11/18/20

to Saurabh Singh, MariaDB ColumnStore

I think the dependency related messages that you shared aren't important.
I am curious about -DRRP=sles15 argument in the cmake command. I'm not aware about this one so I would suggest to replace it with -DRPM=sles15. This argument affects the layout of the files to be installed after the compilation and its value is used as a suffix used for packages.
I believe cmake error is caused by the if condition in ./storage/columnstore/CMakeList.txt that compares CMAKE_SYSTEM_PROCESSOR with a list of predefined architectures. Could you remove the whole if condition that starts at the line 14 + related endif at the line 35?

ср, 18 нояб. 2020 г. в 03:45, Saurabh Singh <saura...@gmail.com>:

--
You received this message because you are subscribed to a topic in the Google Groups "MariaDB ColumnStore" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mariadb-columnstore/8jq1WM_Dkbg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mariadb-columns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mariadb-columnstore/52ffe6c0-94c5-48c5-9b2c-18feb183cff9n%40googlegroups.com.

Message has been deleted

Saurabh Singh

unread,

Nov 18, 2020, 1:29:09 PM11/18/20

to MariaDB ColumnStore

We noticed -DRRP from your previous response instead of -DRPM and thought it must be a typo and looked for RRP but did not found anything on it. Since the server is blank, didn't see any harm trying it but the output was same as shared before.

Today tried removing the IF condition from the ./storage/columnstore/CMakeList.txt which did not contain any processor condition for s390x but it produced the exact same output.

Roland Noland

unread,

Nov 18, 2020, 1:38:31 PM11/18/20

to Saurabh Singh, MariaDB ColumnStore

Can I take a look at your ./storage/columnstore/CMakeList.txt?

Regards,

Roman

ср, 18 нояб. 2020 г. в 21:29, Saurabh Singh <saura...@gmail.com>:

You received this message because you are subscribed to the Google Groups "MariaDB ColumnStore" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mariadb-columns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mariadb-columnstore/08c5e50a-9de9-4142-982e-ab10ff691a95n%40googlegroups.com.

Saurabh Singh

unread,

Nov 18, 2020, 2:15:33 PM11/18/20

to MariaDB ColumnStore

This is the default one in the columnstore directory:

#set(PLUGIN_COLUMNSTORE "NO" CACHE STRING "Enable ColumnStore engine")

if("NO" STREQUAL "${PLUGIN_COLUMNSTORE}")

return()

endif()

# this does everything, gets the var from the correct scope, appends new

# values, sets in the correct scope

macro(APPEND_FOR_CPACK V)

get_directory_property(var DIRECTORY columnstore DEFINITION ${V})

set(${V} "${var}${ARGN}" PARENT_SCOPE)

endmacro()

IF(CMAKE_SYSTEM_PROCESSOR STREQUAL "x86_64" OR

CMAKE_SYSTEM_PROCESSOR STREQUAL "amd64" OR

CMAKE_SYSTEM_PROCESSOR STREQUAL "i386" OR

CMAKE_SYSTEM_PROCESSOR STREQUAL "i686")

add_subdirectory(columnstore)

IF(TARGET columnstore)

# Needed to bump the component changes up to the main scope

APPEND_FOR_CPACK(CPACK_COMPONENTS_ALL)

IF (RPM)

APPEND_FOR_CPACK(CPACK_RPM_columnstore-engine_PACKAGE_REQUIRES " binutils net-tools python3")

APPEND_FOR_CPACK(CPACK_RPM_columnstore-engine_PACKAGE_RECOMMENDS " jemalloc")

APPEND_FOR_CPACK(CPACK_RPM_columnstore-engine_USER_FILELIST ";%ignore /var/lib;%ignore /var")

APPEND_FOR_CPACK(CPACK_RPM_columnstore-engine_PACKAGE_CONFLICTS " thrift MariaDB-columnstore-platform MariaDB-columnstore-libs")

# these three don't have the list semantics, so no append here

SET(CPACK_RPM_columnstore-engine_PRE_INSTALL_SCRIPT_FILE ${CMAKE_CURRENT_SOURCE_DIR}/columnstore/build/preInstall_storage_engine.sh PARENT_SCOPE)

SET(CPACK_RPM_columnstore-engine_POST_INSTALL_SCRIPT_FILE ${CMAKE_CURRENT_SOURCE_DIR}/columnstore/build/postInstall_storage_engine.sh PARENT_SCOPE)

SET(CPACK_RPM_columnstore-engine_PRE_UNINSTALL_SCRIPT_FILE ${CMAKE_CURRENT_SOURCE_DIR}/columnstore/build/preUn_storage_engine.sh PARENT_SCOPE)

ENDIF()

INSTALL_MYSQL_TEST("${CMAKE_CURRENT_SOURCE_DIR}/mysql-test/" "plugin/columnstore")

ENDIF()

I tried after removing the IF statement after line 13 through the end

Roland Noland

unread,

Nov 18, 2020, 2:21:43 PM11/18/20

to Saurabh Singh, MariaDB ColumnStore

No, you should have.