How to build Greenplum in a machine

[Hamilton]

Dear GPDB Developer Team,

I'm a developer who is trying to build Greenplum from source code. Since I don’t have the experiences of using Greenplum, I just try to run it in centos from the shell. But as I followed the instructions in README.md and went on to run the command: DATADIRS=/tmp/gpdb-cluster PORT_BASE=5555 NUM_PRIMARY_MIRROR_PAIRS=1 WITH_MIRRORS=false make create-demo-cluster, I got a failure message, which states that the server can only be started in (Pasted below in Section 1)

Because I don’t know how to establish a development environment and don’t have many servers for development, I think that running in docker may be a solution,though I don’t know how to deal with it either.

Any guide that can help on build greeplum, both in how to run a gpdb and run a cluster in docker? Thanks a lot.

Secition 1

 

…(stripped)

 

make -C gpAux/gpdemo create-demo-cluster

make[1]: Entering directory `/home/faiss/Labs/gpdb_src/gpAux/gpdemo'

======================================================================

            ______  _____  ______  _______ _______  _____

           |  ____ |_____] |     \ |______ |  |  | |     |

           |_____| |       |_____/ |______ |  |  | |_____|

 

----------------------------------------------------------------------

 

  This is a MIRRORLESS demo of the Greenplum Database system.  We will create

  a cluster installation with coordinator and 1 segment instances.

 

    GPHOME ................... : /usr/local/gpdb

    COORDINATOR_DATA_DIRECTORY : /tmp/gpdb-cluster/qddir/demoDataDir-1

 

    COORDINATOR PORT (PGPORT). : 5555

    STANDBY PORT ............. : 5556

    SEGMENT PORTS ............ : 5557

 

  NOTE(s):

 

    * The DB ports identified above must be available for use.

    * An environment file gpdemo-env.sh has been created for your use.

 

======================================================================

 

grep: clusterConfigPostgresAddonsFile: No such file or directory

 

======================================================================

CLUSTER_CONFIG_POSTGRES_ADDONS: clusterConfigPostgresAddonsFile

----------------------------------------------------------------------

fsync=off

======================================================================

 

==========================================================================================

executing:

  /usr/local/gpdb/bin/gpinitsystem -a -c clusterConfigFile -l /tmp/gpdb-cluster/gpAdminLogs -p clusterConfigPostgresAddonsFile  ""

==========================================================================================

 

20230519:10:55:43:021301 gpinitsystem:VM-12-16-centos:faiss-[INFO]:-Checking configuration parameters, please wait...

20230519:10:55:43:021301 gpinitsystem:VM-12-16-centos:faiss-[INFO]:-Reading Greenplum configuration file clusterConfigFile

20230519:10:55:44:021301 gpinitsystem:VM-12-16-centos:faiss-[INFO]:-Locale has not been set in clusterConfigFile, will set to default value

20230519:10:55:45:021301 gpinitsystem:VM-12-16-centos:faiss-[INFO]:-No DATABASE_NAME set, will exit following template1 updates

20230519:10:55:45:021301 gpinitsystem:VM-12-16-centos:faiss-[INFO]:-COORDINATOR_MAX_CONNECT not set, will set to default value 150

20230519:10:55:46:021301 gpinitsystem:VM-12-16-centos:faiss-[INFO]:-Detected a single host GPDB array build, reducing value of BATCH_DEFAULT from 60 to 4

20230519:10:55:47:021301 gpinitsystem:VM-12-16-centos:faiss-[INFO]:-Checking configuration parameters, Completed

20230519:10:55:47:021301 gpinitsystem:VM-12-16-centos:faiss-[INFO]:-Commencing multi-home checks, please wait...

.

20230519:10:55:48:021301 gpinitsystem:VM-12-16-centos:faiss-[INFO]:-Configuring build for standard array

20230519:10:55:49:021301 gpinitsystem:VM-12-16-centos:faiss-[INFO]:-Commencing multi-home checks, Completed

20230519:10:55:49:021301 gpinitsystem:VM-12-16-centos:faiss-[INFO]:-Building primary segment instance array, please wait...

.

20230519:10:55:52:021301 gpinitsystem:VM-12-16-centos:faiss-[INFO]:-Checking Coordinator host

20230519:10:55:53:021301 gpinitsystem:VM-12-16-centos:faiss-[INFO]:-Checking new segment hosts, please wait...

.

20230519:10:55:58:021301 gpinitsystem:VM-12-16-centos:faiss-[INFO]:-Checking new segment hosts, Completed

20230519:10:55:59:021301 gpinitsystem:VM-12-16-centos:faiss-[INFO]:-Building the Coordinator instance database, please wait...

20230519:10:56:14:021301 gpinitsystem:VM-12-16-centos:faiss-[INFO]:-Starting the Coordinator in admin mode

20230519:10:56:21:021301 gpinitsystem:VM-12-16-centos:faiss-[INFO]:-Commencing parallel build of primary segment instances

20230519:10:56:21:021301 gpinitsystem:VM-12-16-centos:faiss-[INFO]:-Spawning parallel processes    batch [1], please wait...

.

20230519:10:56:22:021301 gpinitsystem:VM-12-16-centos:faiss-[INFO]:-Waiting for parallel processes batch [1], please wait...

......................

20230519:10:56:48:021301 gpinitsystem:VM-12-16-centos:faiss-[INFO]:------------------------------------------------

20230519:10:56:48:021301 gpinitsystem:VM-12-16-centos:faiss-[INFO]:-Parallel process exit status

20230519:10:56:48:021301 gpinitsystem:VM-12-16-centos:faiss-[INFO]:------------------------------------------------

20230519:10:56:48:021301 gpinitsystem:VM-12-16-centos:faiss-[INFO]:-Total processes marked as completed           = 1

20230519:10:56:48:021301 gpinitsystem:VM-12-16-centos:faiss-[INFO]:-Total processes marked as killed              = 0

20230519:10:56:48:021301 gpinitsystem:VM-12-16-centos:faiss-[INFO]:-Total processes marked as failed              = 0

20230519:10:56:48:021301 gpinitsystem:VM-12-16-centos:faiss-[INFO]:------------------------------------------------

20230519:10:56:49:021301 gpinitsystem:VM-12-16-centos:faiss-[INFO]:-Removing back out file

20230519:10:56:49:021301 gpinitsystem:VM-12-16-centos:faiss-[INFO]:-No errors generated from parallel processes

20230519:10:56:49:021301 gpinitsystem:VM-12-16-centos:faiss-[INFO]:-Restarting the Greenplum instance in production mode

Error: unable to import module: No module named 'pgdb'

20230519:10:56:50:021301 gpinitsystem:VM-12-16-centos:faiss-[FATAL]:-Error from Greenplum instance shutdown, check log files Script Exiting!

========================================

gpinitsystem returned: 1

========================================

 

======================================================================

                           OPTIMIZER STATE

----------------------------------------------------------------------

  Optimizer state .. : psql: error: FATAL:  System was started in single node mode - only utility mode connections are allowed

======================================================================

 

psql: error: FATAL:  System was started in single node mode - only utility mode connections are allowed

======================================================================

 

 

make[1]: Leaving directory `/home/faiss/Labs/gpdb_src/gpAux/gpdemo'

[Ivan Novick]

This error seems relevant to the issue:

Error: unable to import module: No module named 'pgdb'

Can you check for more errors in the log files?

---

From: Hamilton <lkc.an...@gmail.com>
Sent: Wednesday, May 24, 2023 5:28 AM
To: Greenplum Developers <gpdb...@greenplum.org>
Subject: How to build Greenplum in a machine

 

!! External Email

!! External Email: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender.

[Jack Moon]

@Ivan Novick Good point!

@Hamilton I guess you might have missed some relevant and dependent packages to import pgdb module.

Can you try to run the below command to see details why pgdb could not be imported?

$ python -v -m ./gpMgmt/bin/ext/pygresql/pgdb.py
$ python -v -m ./gpMgmt/bin/pythonSrc/PyGreSQL-4.0/pgdb.py

Jack Moon ( He/His )

---

From: Hamilton <lkc.an...@gmail.com>
Sent: Wednesday, May 24, 2023 9:28 PM

To: Greenplum Developers <gpdb...@greenplum.org>
Subject: How to build Greenplum in a machine

 

!! External Email

Dear GPDB Developer Team,

[hamilton]

Thanks for reply!

The simple output is below:

[faiss@VM-12-16-centos gpdb_src]$ python -m ./gpMgmt/bin/ext/pygresql/pgdb.py
/usr/bin/python: Relative module names not supported
[faiss@VM-12-16-centos gpdb_src]$ python -m ./gpMgmt/bin/pythonSrc/PyGreSQL-4.0/pgdb.py
/usr/bin/python: Relative module names not supported

Details with `-v` is:

[faiss@VM-12-16-centos gpdb_src]$ python -v -m ./gpMgmt/bin/pythonSrc/PyGreSQL-4.0/pgdb.py
# installing zipimport hook
import zipimport # builtin
# installed zipimport hook
# /usr/lib64/python2.7/site.pyc matches /usr/lib64/python2.7/site.py
import site # precompiled from /usr/lib64/python2.7/site.pyc
# /usr/lib64/python2.7/os.pyc matches /usr/lib64/python2.7/os.py
import os # precompiled from /usr/lib64/python2.7/os.pyc
import errno # builtin
import posix # builtin
# /usr/lib64/python2.7/posixpath.pyc matches /usr/lib64/python2.7/posixpath.py
import posixpath # precompiled from /usr/lib64/python2.7/posixpath.pyc
# /usr/lib64/python2.7/stat.pyc matches /usr/lib64/python2.7/stat.py
import stat # precompiled from /usr/lib64/python2.7/stat.pyc
# /usr/lib64/python2.7/genericpath.pyc matches /usr/lib64/python2.7/genericpath.py
import genericpath # precompiled from /usr/lib64/python2.7/genericpath.pyc
# /usr/lib64/python2.7/warnings.pyc matches /usr/lib64/python2.7/warnings.py
import warnings # precompiled from /usr/lib64/python2.7/warnings.pyc
# /usr/lib64/python2.7/linecache.pyc matches /usr/lib64/python2.7/linecache.py
import linecache # precompiled from /usr/lib64/python2.7/linecache.pyc
# /usr/lib64/python2.7/types.pyc matches /usr/lib64/python2.7/types.py
import types # precompiled from /usr/lib64/python2.7/types.pyc
# /usr/lib64/python2.7/UserDict.pyc matches /usr/lib64/python2.7/UserDict.py
import UserDict # precompiled from /usr/lib64/python2.7/UserDict.pyc
# /usr/lib64/python2.7/_abcoll.pyc matches /usr/lib64/python2.7/_abcoll.py
import _abcoll # precompiled from /usr/lib64/python2.7/_abcoll.pyc
# /usr/lib64/python2.7/abc.pyc matches /usr/lib64/python2.7/abc.py
import abc # precompiled from /usr/lib64/python2.7/abc.pyc
# /usr/lib64/python2.7/_weakrefset.pyc matches /usr/lib64/python2.7/_weakrefset.py
import _weakrefset # precompiled from /usr/lib64/python2.7/_weakrefset.pyc
import _weakref # builtin
# /usr/lib64/python2.7/copy_reg.pyc matches /usr/lib64/python2.7/copy_reg.py
import copy_reg # precompiled from /usr/lib64/python2.7/copy_reg.pyc
# /usr/lib64/python2.7/traceback.pyc matches /usr/lib64/python2.7/traceback.py
import traceback # precompiled from /usr/lib64/python2.7/traceback.pyc
# /usr/lib64/python2.7/sysconfig.pyc matches /usr/lib64/python2.7/sysconfig.py
import sysconfig # precompiled from /usr/lib64/python2.7/sysconfig.pyc
# /usr/lib64/python2.7/re.pyc matches /usr/lib64/python2.7/re.py
import re # precompiled from /usr/lib64/python2.7/re.pyc
# /usr/lib64/python2.7/sre_compile.pyc matches /usr/lib64/python2.7/sre_compile.py
import sre_compile # precompiled from /usr/lib64/python2.7/sre_compile.pyc
import _sre # builtin
# /usr/lib64/python2.7/sre_parse.pyc matches /usr/lib64/python2.7/sre_parse.py
import sre_parse # precompiled from /usr/lib64/python2.7/sre_parse.pyc
# /usr/lib64/python2.7/sre_constants.pyc matches /usr/lib64/python2.7/sre_constants.py
import sre_constants # precompiled from /usr/lib64/python2.7/sre_constants.pyc
# /usr/lib64/python2.7/_sysconfigdata.pyc matches /usr/lib64/python2.7/_sysconfigdata.py
import _sysconfigdata # precompiled from /usr/lib64/python2.7/_sysconfigdata.pyc
# /usr/lib64/python2.7/site-packages/abrt_exception_handler.pyc matches /usr/lib64/python2.7/site-packages/abrt_exception_handler.py
import abrt_exception_handler # precompiled from /usr/lib64/python2.7/site-packages/abrt_exception_handler.pyc
import encodings # directory /usr/lib64/python2.7/encodings
# /usr/lib64/python2.7/encodings/__init__.pyc matches /usr/lib64/python2.7/encodings/__init__.py
import encodings # precompiled from /usr/lib64/python2.7/encodings/__init__.pyc
# /usr/lib64/python2.7/codecs.pyc matches /usr/lib64/python2.7/codecs.py
import codecs # precompiled from /usr/lib64/python2.7/codecs.pyc
import _codecs # builtin
# /usr/lib64/python2.7/encodings/aliases.pyc matches /usr/lib64/python2.7/encodings/aliases.py
import encodings.aliases # precompiled from /usr/lib64/python2.7/encodings/aliases.pyc
# /usr/lib64/python2.7/encodings/utf_8.pyc matches /usr/lib64/python2.7/encodings/utf_8.py
import encodings.utf_8 # precompiled from /usr/lib64/python2.7/encodings/utf_8.pyc
Python 2.7.5 (default, Jun 28 2022, 15:30:04)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
# /usr/lib64/python2.7/runpy.pyc matches /usr/lib64/python2.7/runpy.py
import runpy # precompiled from /usr/lib64/python2.7/runpy.pyc
import imp # builtin
# /usr/lib64/python2.7/pkgutil.pyc matches /usr/lib64/python2.7/pkgutil.py
import pkgutil # precompiled from /usr/lib64/python2.7/pkgutil.pyc
/usr/bin/python: Relative module names not supported
# clear __builtin__._
# clear sys.path
# clear sys.argv
# clear sys.ps1
# clear sys.ps2
# clear sys.exitfunc
# clear sys.exc_type
# clear sys.exc_value
# clear sys.exc_traceback
# clear sys.last_type
# clear sys.last_value
# clear sys.last_traceback
# clear sys.path_hooks
# clear sys.path_importer_cache
# clear sys.meta_path
# clear sys.flags
# clear sys.float_info
# restore sys.stdin
# restore sys.stdout
# restore sys.stderr
# cleanup __main__
# cleanup[1] encodings
# cleanup[1] site
# cleanup[1] sysconfig
# cleanup[1] abc
# cleanup[1] _weakrefset
# cleanup[1] sre_constants
# cleanup[1] re
# cleanup[1] _codecs
# cleanup[1] _warnings
# cleanup[1] _sysconfigdata
# cleanup[1] encodings.utf_8
# cleanup[1] repoze
# cleanup[1] pkgutil
# cleanup[1] signal
# cleanup[1] traceback
# cleanup[1] posix
# cleanup[1] encodings.aliases
# cleanup[1] exceptions
# cleanup[1] abrt_exception_handler
# cleanup[1] _weakref
# cleanup[1] sre_compile
# cleanup[1] _sre
# cleanup[1] codecs
# cleanup[1] zipimport
# cleanup[1] sre_parse
# cleanup[2] copy_reg
# cleanup[2] runpy
# cleanup[2] posixpath
# cleanup[2] errno
# cleanup[2] _abcoll
# cleanup[2] types
# cleanup[2] genericpath
# cleanup[2] stat
# cleanup[2] warnings
# cleanup[2] UserDict
# cleanup[2] imp
# cleanup[2] os.path
# cleanup[2] linecache
# cleanup[2] os
# cleanup sys
# cleanup __builtin__
# cleanup ints: 19 unfreed ints
# cleanup floats

[Kalen Krempely]

I would try using pip to install the python dependences (found in the repository root).

pip3 install -r python-dependencies.txt​

Where the python-dependeices.txt points to pygresql and other python dependencies needed by Greenplum.

Loosely following the README.Rhel-Rocky.bash and/or README.Ubuntu.bash might be useful. Some consideration is needed since it is not specifically for centos.

Hope this helps some!

Kalen

---

From: hamilton <lkc.an...@gmail.com>
Sent: Wednesday, May 24, 2023 11:44 PM
To: Greenplum Developers <gpdb...@greenplum.org>
Cc: Jack Moon <moo...@vmware.com>; Hamilton <lkc.an...@gmail.com>; Ivan Novick <gpdb...@greenplum.org>
Subject: Re: How to build Greenplum in a machine

 

[hamilton]

Yes, that is the point. 

Before I have runned README.Rhel-Rocky.bash but I didn't checked the result.

Also, my centos environment (in cloud) makes it a little inconvenient, since I have to:

      compile python3.9 from source code(No package python39 available).

      install libffi-devel (needed by python packages)

Now I have solved the problem and start the cluster:

faiss    22992  0.0  0.8 407524 34796 ?        Ss   May19   0:05 /usr/local/gpdb/bin/postgres -D /tmp/gpdb-cluster/qddir/demoDataDir-1 -i -p 5555 -c gp_role=utility -m

faiss    22996  0.0  0.0 231780  2240 ?        Ss   May19   0:01  \_ postgres:  5555, logger process  
faiss    22998  0.0  0.1 407940  4028 ?        Ss   May19   0:00  \_ postgres:  5555, checkpointer  
faiss    22999  0.0  0.0 407868  3200 ?        Ss   May19   0:05  \_ postgres:  5555, background writer  
faiss    23000  0.0  0.1 407688  6380 ?        Ss   May19   0:05  \_ postgres:  5555, walwriter  
faiss    23001  0.0  0.1 409316  5176 ?        Ss   May19   0:03  \_ postgres:  5555, autovacuum launcher  
faiss    23002  0.0  0.0 234032  2308 ?        Ss   May19   0:03  \_ postgres:  5555, stats collector  
faiss    23003  0.0  0.1 409272  4472 ?        Ss   May19   0:00  \_ postgres:  5555, logical replication launcher

 

Here's how I solved it:

     yum install -y libffi-devel

     build pytho39 from source code.

     ignore python39 install in README.Rhel-Rocky.bash and rerun the script.

Maybe I can try another machine with a better support for package management.

Thanks.

[Jack Moon]

May i know the version of OS, GPDB, Python are you using now with the following commands? Because there are not much things to see regarding errors and so on in the output of python -m -v command.

$ cat /etc/os-release ( or /etc/redhat-release )

$ git branch

* (detached from 6.21.1)

$ python --version

$ rpm -qa | grep python

$ alternatives --list

---

From: hamilton <lkc.an...@gmail.com>
Sent: Thursday, May 25, 2023 4:58 PM
To: Greenplum Developers <gpdb...@greenplum.org>
Cc: Kalen Krempely <kkre...@vmware.com>; Jack Moon <moo...@vmware.com>; hamilton <lkc.an...@gmail.com>

[hamilton]

Sorry to bother you. I have solved it now.

And the main reason is this machine doesn't support python3.9 so I have to compile it from source code.