compile error slurm 17.02.9 against pyslurm-17.02.0

[Cesare Delle Fratte]

Hi all, I just downloaded pyslurm version 17.02.0 and tried to compile it for my slurm 17.02.9:

all the following compile attempt

 python setup.py build --slurm-lib=/usr/lib64 --slurm-inc=/usr/include

 python setup.py build --slurm=/usr

 python setup.py build

fail with the same error stack:

INFO:root:Info: 

INFO:root:Info: Building PySlurm (17.02.0)

INFO:root:Info: ------------------------------

INFO:root:Info: 

INFO:root:Info: Cython version 0.27.3 installed

INFO:root:Info: Clean - checking for objects to clean

INFO:root:Info: Clean - removing pyslurm build temp directory ...

INFO:root:Info: Clean - removing pyslurm/pyslurm.c temp file

INFO:root:Info: Clean - removing pyslurm/bluegene.pxi temp file

INFO:root:Info: Clean - removing pyslurm/slurm_version.pxi temp file

INFO:root:Info: Clean - completed

INFO:root:Info: Build - Detected Slurm include file version - 0x110209 (17.02.09)

INFO:root:Info: Build - Found Slurm shared library in /usr/lib64

INFO:root:Info: Build - Writing Slurm version to pyslurm/slurm_version.pxi

INFO:root:Info: Build - Generating pyslurm/bluegene.pxi file

running build

running build_py

creating build

creating build/lib.linux-x86_64-2.6

creating build/lib.linux-x86_64-2.6/pyslurm

copying pyslurm/__init__.py -> build/lib.linux-x86_64-2.6/pyslurm

running build_ext

cythoning pyslurm/pyslurm.pyx to pyslurm/pyslurm.c

building 'pyslurm.pyslurm' extension

creating build/temp.linux-x86_64-2.6

creating build/temp.linux-x86_64-2.6/pyslurm

gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include -I. -I/usr/include/python2.6 -c pyslurm/pyslurm.c -o build/temp.linux-x86_64-2.6/pyslurm/pyslurm.o

pyslurm/pyslurm.c: In function ‘__pyx_f_7pyslurm_7pyslurm_6config___get’:

pyslurm/pyslurm.c:12587: error: ‘slurm_ctl_conf_t’ has no member named ‘mail_domain’

pyslurm/pyslurm.c:13392: error: ‘slurm_ctl_conf_t’ has no member named ‘sbcast_parameters’

pyslurm/pyslurm.c: In function ‘__pyx_pf_7pyslurm_7pyslurm_9partition_16get’:

pyslurm/pyslurm.c:16803: error: ‘partition_info_t’ has no member named ‘over_time_limit’

pyslurm/pyslurm.c:16803: error: ‘NO_VAL16’ undeclared (first use in this function)

pyslurm/pyslurm.c:16803: error: (Each undeclared identifier is reported only once

pyslurm/pyslurm.c:16803: error: for each function it appears in.)

pyslurm/pyslurm.c:16832: error: ‘partition_info_t’ has no member named ‘over_time_limit’

pyslurm/pyslurm.c:16862: error: ‘partition_info_t’ has no member named ‘over_time_limit’

pyslurm/pyslurm.c: In function ‘__pyx_f_7pyslurm_7pyslurm_3job_get_job_ptr’:

pyslurm/pyslurm.c:28129: error: ‘slurm_job_info_t’ has no member named ‘admin_comment’

pyslurm/pyslurm.c:28484: error: ‘slurm_job_info_t’ has no member named ‘burst_buffer_state’

pyslurm/pyslurm.c:28833: error: ‘slurm_job_info_t’ has no member named ‘fed_origin_str’

pyslurm/pyslurm.c:28845: error: ‘slurm_job_info_t’ has no member named ‘fed_siblings_str’

pyslurm/pyslurm.c: In function ‘__pyx_pf_7pyslurm_7pyslurm_4node_12get_node’:

pyslurm/pyslurm.c:35774: error: ‘node_info_t’ has no member named ‘partitions’

pyslurm/pyslurm.c: In function ‘__pyx_f_7pyslurm_7pyslurm_7jobstep___get’:

pyslurm/pyslurm.c:39917: error: ‘job_step_info_t’ has no member named ‘srun_host’

pyslurm/pyslurm.c:39929: error: ‘job_step_info_t’ has no member named ‘srun_pid’

pyslurm/pyslurm.c: In function ‘__pyx_f_7pyslurm_5slurm_int16orNone’:

pyslurm/pyslurm.c:66269: error: ‘NO_VAL16’ undeclared (first use in this function)

pyslurm/pyslurm.c: In function ‘__pyx_f_7pyslurm_5slurm_int16orUnlimited’:

pyslurm/pyslurm.c:66514: error: ‘INFINITE16’ undeclared (first use in this function)

pyslurm/pyslurm.c: In function ‘initpyslurm’:

pyslurm/pyslurm.c:72628: error: ‘JOB_OOM’ undeclared (first use in this function)

pyslurm/pyslurm.c:74848: error: ‘WAIT_FED_JOB_LOCK’ undeclared (first use in this function)

pyslurm/pyslurm.c:74860: error: ‘FAIL_OOM’ undeclared (first use in this function)

pyslurm/pyslurm.c:75364: error: ‘SELECT_NODEDATA_TRES_ALLOC_FMT_STR’ undeclared (first use in this function)

pyslurm/pyslurm.c:75376: error: ‘SELECT_NODEDATA_TRES_ALLOC_WEIGHTED’ undeclared (first use in this function)

error: command 'gcc' failed with exit status 1

Can plaese somebody help me or have an idea about a possible fix?

Regards, C.

[Giovanni]

Hi,

I am unable to replicate.  Here's what I get on a 17.02.9 cluster:

[root@ernie ~]# git clone https://github.com/pyslurm/pyslurm.git
Cloning into 'pyslurm'...
remote: Counting objects: 3714, done.
remote: Compressing objects: 100% (79/79), done.
remote: Total 3714 (delta 90), reused 124 (delta 75), pack-reused 3560
Receiving objects: 100% (3714/3714), 4.29 MiB | 7.25 MiB/s, done.
Resolving deltas: 100% (2335/2335), done.
[root@ernie ~]# cd pyslurm/
[root@ernie pyslurm]# git branch
* master
[root@ernie pyslurm]# git checkout 17.02.0
Branch 17.02.0 set up to track remote branch 17.02.0 from origin.
Switched to a new branch '17.02.0'
[root@ernie pyslurm]#
[root@ernie pyslurm]# python setup.py build

INFO:root:Info:
INFO:root:Info: Building PySlurm (17.02.0)
INFO:root:Info: ------------------------------
INFO:root:Info:
INFO:root:Info: Cython version 0.27.3 installed

INFO:root:Info: Clean - checking for objects to clean

INFO:root:Info: Clean - completed
INFO:root:Info: Build - Detected Slurm include file version - 0x110209 (17.02.09)
INFO:root:Info: Build - Found Slurm shared library in /usr/lib64
INFO:root:Info: Build - Writing Slurm version to pyslurm/slurm_version.pxi
INFO:root:Info: Build - Generating pyslurm/bluegene.pxi file
running build
running build_py
creating build

creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/pyslurm
copying pyslurm/__init__.py -> build/lib.linux-x86_64-2.7/pyslurm

running build_ext
cythoning pyslurm/pyslurm.pyx to pyslurm/pyslurm.c
building 'pyslurm.pyslurm' extension

creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/pyslurm
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include -I. -I/usr/include/python2.7 -c pyslurm/pyslurm.c -o build/temp.linux-x86_64-2.7/pyslurm/pyslurm.o
gcc -pthread -shared -Wl,-z,relro build/temp.linux-x86_64-2.7/pyslurm/pyslurm.o -L/usr/lib64 -L/usr/lib64/slurm -L/usr/lib64 -Wl,-R/usr/lib64/ -Wl,-R/usr/lib64/slurm -lslurmdb -lslurm -lpython2.7 -o build/lib.linux-x86_64-2.7/pyslurm/pyslurm.so
[root@ernie pyslurm]#

Was this a fresh clone of pyslurm?  If not, I'm wondering if you have pulled all the latest changes to that branch?

Based on the first error I saw, could you try the following grep commands:

[root@ernie pyslurm]# grep mail_domain pyslurm/pyslurm.pyx
            Ctl_dict[u'mail_domain'] = slurm.stringOrNone(self.__Config_ptr.mail_domain, '')

[root@ernie pyslurm]# grep mail_domain pyslurm/slurm.pxd  
        char *mail_domain

You should get a return for each.

Giovanni

[Cesare Delle Fratte]

Hi Giovanni, here is the output of the grep you asked for:

[root@sched pyslurm-17.02.0]# grep mail_domain pyslurm/pyslurm.pyx 

            Ctl_dict[u'mail_domain'] = slurm.stringOrNone(self.__Config_ptr.mail_domain, '')

[root@sched pyslurm-17.02.0]# grep mail_domain pyslurm/slurm.pxd   

        char *mail_domain

It doesn't look different from yours :(

Thanks, C.

--
You received this message because you are subscribed to a topic in the Google Groups "pyslurm" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pyslurm/fLO4VICxlN8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pyslurm+unsubscribe@googlegroups.com.
To post to this group, send email to pys...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pyslurm/8a442314-2add-49e2-b19e-de582d9d6af5%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

------------------------------------------------------------------------------------------------

"Greetings, Professor Falken. A strange game. The only winning move is not to play.

How about a nice game of chess? " 
------------------------------------------------------------------------------------------------

--  Cesare Delle Fratte

--                      

--  Max-Planck-Computing and Data Facilities (MPCDF)

--  Rechenzentrum Garching (RZG)

--  Gießenbachstraße 2, 85748 Garching, Germany

--

--  Max-Planck-Institut für Physik

--  Foehringer Ring 6, 80805 Munich, Germany

--

--  e-mail: cesare.de...@rzg.mpg.de

--  Phone: +49 89 3299 1157 - - Fax: +49 89 3299 1301

------------------------------------------------------------------------------------------------

[Giovanni]

On Friday, January 26, 2018 at 4:05:55 AM UTC-5, Cesare Delle Fratte wrote:

Hi Giovanni, here is the output of the grep you asked for:

[root@sched pyslurm-17.02.0]# grep mail_domain pyslurm/pyslurm.pyx 

            Ctl_dict[u'mail_domain'] = slurm.stringOrNone(self.__Config_ptr.mail_domain, '')

[root@sched pyslurm-17.02.0]# grep mail_domain pyslurm/slurm.pxd   

        char *mail_domain

It doesn't look different from yours :(

Thanks, C.

Ok.  Could you attach your pyslurm.c file?

Giovanni

[Cesare Delle Fratte]

Hi Gio, here it is the pyslurm.c

C.

--

You received this message because you are subscribed to a topic in the Google Groups "pyslurm" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pyslurm/fLO4VICxlN8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pyslurm+unsubscribe@googlegroups.com.
To post to this group, send email to pys...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pyslurm/c5946d08-c230-4e2e-84dc-74b84afdca96%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Giovanni]

On Saturday, January 27, 2018 at 1:39:34 AM UTC-6, Cesare Delle Fratte wrote:
> Hi Gio, here it is the pyslurm.c
>

Could you also post the pyslurm.pyx and slurm.pxd files as well.

Thanks.

[Cesare Delle Fratte]

Hi Giovanni, here they are...

--
You received this message because you are subscribed to a topic in the Google Groups "pyslurm" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pyslurm/fLO4VICxlN8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pyslurm+unsubscribe@googlegroups.com.
To post to this group, send email to pys...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pyslurm/df3d3245-a885-4c30-b23b-39356a689331%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Giovanni]

Hi,

I took a look at this and I still can't tell why it is not compiling for you.  All the versions appear to match up and the files you sent are identical to mine.  Those errors typically show up when struct members, for example, aren't redeclared in the slurm.pxd file, but the members it complains about are definitely there in slurm.pxd.

I did the following to manually compile pyslurm.so using the files you sent:

$ cp pyslurm.pyx slurm.pxd /tmp
$ cd /tmp

# this creates pyslurm.c
$ cython pyslurm.pyx

# this create pyslurm.so
$ gcc -Wall -shared -o pyslurm.so $(python-config --cflags) $(python-config --ldflags) -lslurm -lslurmdb pyslurm.c

In the same directory:
$ python
>>> import pyslurm
>>> pyslurm.slurm_api_version()
(17, 2, 9)


It seems like your slurm libraries are in the standard /usr/lib64 directory, so you shouldn't have to specify --slurm-lib to setup.py.  Also, if your slurm headers are in /usr/include/slurm, you shouldn't have to specify --slurm-inc either.  Could you confirm that your slurm headers are in /usr/include/slurm?

Thanks,
Giovanni


On Monday, January 29, 2018 at 3:52:42 AM UTC-5, Cesare Delle Fratte wrote:

Hi Giovanni, here they are...

[Cesare Delle Fratte]

Hi, yes I confirm

/usr/lib64 contains the slurm libraries and /usr/include/slurm has the headers.

I tried to replicate your commands and this is the result:

pyslurm.c: In function ‘__pyx_f_7pyslurm_7pyslurm_6config___get’:

pyslurm.c:12587: error: ‘slurm_ctl_conf_t’ has no member named ‘mail_domain’

pyslurm.c:13392: error: ‘slurm_ctl_conf_t’ has no member named ‘sbcast_parameters’

pyslurm.c: In function ‘__pyx_pf_7pyslurm_7pyslurm_9partition_16get’:

pyslurm.c:16803: error: ‘partition_info_t’ has no member named ‘over_time_limit’

pyslurm.c:16803: error: ‘NO_VAL16’ undeclared (first use in this function)

pyslurm.c:16803: error: (Each undeclared identifier is reported only once

pyslurm.c:16803: error: for each function it appears in.)

pyslurm.c:16832: error: ‘partition_info_t’ has no member named ‘over_time_limit’

pyslurm.c:16862: error: ‘partition_info_t’ has no member named ‘over_time_limit’

pyslurm.c: In function ‘__pyx_f_7pyslurm_7pyslurm_3job_get_job_ptr’:

pyslurm.c:28129: error: ‘slurm_job_info_t’ has no member named ‘admin_comment’

pyslurm.c:28484: error: ‘slurm_job_info_t’ has no member named ‘burst_buffer_state’

pyslurm.c:28833: error: ‘slurm_job_info_t’ has no member named ‘fed_origin_str’

pyslurm.c:28845: error: ‘slurm_job_info_t’ has no member named ‘fed_siblings_str’

pyslurm.c: In function ‘__pyx_pf_7pyslurm_7pyslurm_4node_12get_node’:

pyslurm.c:35774: error: ‘node_info_t’ has no member named ‘partitions’

pyslurm.c: In function ‘__pyx_f_7pyslurm_7pyslurm_7jobstep___get’:

pyslurm.c:39917: error: ‘job_step_info_t’ has no member named ‘srun_host’

pyslurm.c:39929: error: ‘job_step_info_t’ has no member named ‘srun_pid’

pyslurm.c: In function ‘__pyx_f_7pyslurm_5slurm_int16orNone’:

pyslurm.c:66269: error: ‘NO_VAL16’ undeclared (first use in this function)

pyslurm.c: In function ‘__pyx_f_7pyslurm_5slurm_int16orUnlimited’:

pyslurm.c:66514: error: ‘INFINITE16’ undeclared (first use in this function)

pyslurm.c: In function ‘initpyslurm’:

pyslurm.c:72628: error: ‘JOB_OOM’ undeclared (first use in this function)

pyslurm.c:74848: error: ‘WAIT_FED_JOB_LOCK’ undeclared (first use in this function)

pyslurm.c:74860: error: ‘FAIL_OOM’ undeclared (first use in this function)

pyslurm.c:75364: error: ‘SELECT_NODEDATA_TRES_ALLOC_FMT_STR’ undeclared (first use in this function)

pyslurm.c:75376: error: ‘SELECT_NODEDATA_TRES_ALLOC_WEIGHTED’ undeclared (first use in this function)

It is really a strange behavior...

Cheer, C.

--

You received this message because you are subscribed to a topic in the Google Groups "pyslurm" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pyslurm/fLO4VICxlN8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pyslurm+unsubscribe@googlegroups.com.
To post to this group, send email to pys...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pyslurm/2c750aaf-867b-4ab0-a6a1-375de80a7621%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Cesare Delle Fratte]

Hi Giovanni,

compiled, apparently there was some leftover from an old slurm installation. I just copied:

cp -r /usr/include/slurm/* /usr/local/include/slurm/

and the compile went fine. Many thank for your help, you put me in the right path ;)

But, now I have the next issue:

I can import pyslurm just when I am into /build/lib.linux-x86_64-2.6 

from outside I always get an import error.

Also if I want to run some examples I have to copy it into the same build folder. 

Do you have some tip here, where should I copy the content of the build folder in order to use pyslurm normally?

Cheer, C

--

--  e-mail: cesare.dellefratte@rzg.mpg.de

--  Phone: +49 89 3299 1157 - - Fax: +49 89 3299 1301

------------------------------------------------------------------------------------------------

[Giovanni]

Great!

Looks like you should be able to run the following to get pyslurm to install globally:

$ python setup.py build --slurm-inc=/usr/local/include/slurm
$ sudo python setup.py install

If the slurm libraries are elsewhere, you may also need to add "--slurm-lib=/usr/local/lib{64}/slurm" or wherever the proper slurm libraries are.

I think that should do it.

Giovanni

On Tuesday, January 30, 2018 at 6:15:20 AM UTC-5, Cesare Delle Fratte wrote:

Hi Giovanni,

[Cesare Delle Fratte]

Done! Many Thanks Giovanni.

--

You received this message because you are subscribed to a topic in the Google Groups "pyslurm" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pyslurm/fLO4VICxlN8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pyslurm+unsubscribe@googlegroups.com.
To post to this group, send email to pys...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pyslurm/5742cd8a-81b8-4dba-b5f4-00731c739613%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

------------------------------------------------------------------------------------------------

"Greetings, Professor Falken. A strange game. The only winning move is not to play.

How about a nice game of chess? " 
------------------------------------------------------------------------------------------------

--  Cesare Delle Fratte

--                      

--  Max-Planck-Computing and Data Facilities (MPCDF)

--  Rechenzentrum Garching (RZG)

--  Gießenbachstraße 2, 85748 Garching, Germany

--

--  Max-Planck-Institut für Physik

--  Foehringer Ring 6, 80805 Munich, Germany

--

--  e-mail: cesare.de...@rzg.mpg.de

--  Phone: +49 89 3299 1157 - - Fax: +49 89 3299 1301

------------------------------------------------------------------------------------------------