NUMA test compilation

1763 views
Skip to first unread message

Joachim Müller (JojoM)

unread,
Nov 5, 2016, 1:59:41 PM11/5/16
to FishCooking
During compilation of the current NUMA patch as taken from Thomas Zipproth, I get the following error:

g++ -Wall -Wcast-qual -fno-exceptions -fno-rtti -std=c++11 -fprofile-generate -pedantic -Wextra -Wshadow -m64 -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -flto   -c -o thread.o thread.cpp

In file included from numasf.cpp:1:0:

numasf.h:16:18: fatal error: numa.h: No such file or directory

 #include <numa.h>

                  ^

compilation terminated.

<builtin>: recipe for target 'numasf.o' failed

make[2]: *** [numasf.o] Error 1

make[2]: *** Waiting for unfinished jobs....

In file included from thread.cpp:29:0:

numasf.h:16:18: fatal error: numa.h: No such file or directory

 #include <numa.h>

                  ^

compilation terminated.


Can I do something about that?

mibere

unread,
Nov 5, 2016, 2:07:41 PM11/5/16
to FishCooking
Same on OS X 10.11.6


g++ -Wall -Wcast-qual -fno-exceptions -fno-rtti -std=c++11  -pedantic -Wextra -Wshadow -m64 -arch x86_64 -mmacosx-version-min=10.9 -DNDEBUG -O3 -mdynamic-no-pic -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -flto   -c -o movepick.o movepick.cpp
g++ -Wall -Wcast-qual -fno-exceptions -fno-rtti -std=c++11  -pedantic -Wextra -Wshadow -m64 -arch x86_64 -mmacosx-version-min=10.9 -DNDEBUG -O3 -mdynamic-no-pic -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -flto   -c -o numasf.o numasf.cpp

In file included from numasf.cpp:1:
./numasf.h:16:10: fatal error: 'numa.h' file not found
#include <numa.h>
         ^
1 error generated.
make[1]: *** [numasf.o] Error 1
make: *** [build] Error 2

Exception running games:
Traceback (most recent call last):
  File "worker.py", line 88, in worker
    run_games(worker_info, password, remote, run, task_id)
  File "/Users/xxx/fishtest/worker/games.py", line 383, in run_games
    setup_engine(new_engine, binaries_url, worker_dir, run['args']['resolved_new'], repo_url, worker_info['concurrency'])
  File "/Users/xxx/fishtest/worker/games.py", line 152, in setup_engine
    build(worker_dir, sha, repo_url, destination, concurrency)
  File "/Users/xxx/fishtest/worker/games.py", line 129, in build
    subprocess.check_call(make_cmd, shell=True)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command 'make build ARCH=osx-x86-64-modern || make build ARCH=x86-64-modern' returned non-zero exit status 2
Task exited

Ronald de Man

unread,
Nov 5, 2016, 2:19:41 PM11/5/16
to FishCooking
On Saturday, November 5, 2016 at 6:59:41 PM UTC+1, Joachim Müller (JojoM) wrote:
During compilation of the current NUMA patch as taken from Thomas Zipproth, I get the following error:

Note that this is Mohammed's patch that we have seen earlier and not something of Zipproth's own making.

In file included from thread.cpp:29:0:

numasf.h:16:18: fatal error: numa.h: No such file or directory

 #include <numa.h>

                  ^

compilation terminated.


Can I do something about that?


Same problem as you had before:
 https://groups.google.com/d/msg/fishcooking/2FG7cChRxu0/jD4eRkFiBAAJ

So the same solution would apply. But as I said then, it is going to be a very frustrating process if you cannot do this without "step-by-step fool-proof" instructions.

Thomas Zipproth

unread,
Nov 5, 2016, 2:25:30 PM11/5/16
to FishCooking
Most likely libnuma is missing.
sudo apt-get install libnuma-dev
https://www.howtoinstall.co/en/ubuntu/precise/libnuma-dev

One Remark:
That code is original Stockfish testing code with only small output modifications from me.
It seemed to have advantages both on Numa and non-Numa machines (shared History Table), so I used it in Brainfish where avoiding additional complexity is not so important as in Stockfish.



On Saturday, November 5, 2016 at 6:59:41 PM UTC+1, Joachim Müller (JojoM) wrote:

Joachim Müller (JojoM)

unread,
Nov 5, 2016, 3:02:46 PM11/5/16
to FishCooking
Thank you for trying to help and excuse me for falsely attributing the patch to Thomas instead if Mohammed. I do not have super user / admin rights on the machine in question so I am sorry I cannot participate in this test.

Ronald de Man

unread,
Nov 5, 2016, 8:44:53 PM11/5/16
to FishCooking
On Saturday, November 5, 2016 at 8:02:46 PM UTC+1, Joachim Müller (JojoM) wrote:
Thank you for trying to help and excuse me for falsely attributing the patch to Thomas instead if Mohammed. I do not have super user / admin rights on the machine in question so I am sorry I cannot participate in this test.

If at least the libnuma library is installed on the machine (and if it is a NUMA machine running Linux, how could it not have libnuma installed -- see if you have libnuma.so in /usr/lib64 or in a directory with a similar name), then that is sufficient to run the program. You could compile it on another machine that has the libnuma header files or get an executable from someone else.

Joachim Müller (JojoM)

unread,
Nov 6, 2016, 6:01:04 AM11/6/16
to FishCooking
I'll see what I can do ...

Joachim Müller (JojoM)

unread,
Nov 6, 2016, 6:27:18 AM11/6/16
to FishCooking
@Ronald: In /usr/lib64 I can indeed, giving the command ls -lh | grep numa, locate the following file:

-rwxr-xr-x  1 root root   48K Dec 29  2015 libnuma.so.1


Will that be of any help? 

Joachim Müller (JojoM)

unread,
Nov 6, 2016, 6:59:03 AM11/6/16
to FishCooking

By the way, the current numasf test has been carried out exclusively by Windows machines ;)

Ronald de Man

unread,
Nov 6, 2016, 8:13:19 AM11/6/16
to FishCooking

Yes, it means you can run executables that use libnuma.

Also this simplifies things a lot if you want to compile the numasf patch yourself: there is no need to install or even compile libnuma!

Just take the attached numa.h file and place it in your stockfish/src directory. (I think there is no other numa.h file there, only a numasf.h, otherwise you will have to rename this numa.h to numa2.h or so.)

Then replace the line
#include <numa.h>
with
#include "numa.h"
(or numa2.h)

Now compile as usual. I think this should work.

numa.h

Joachim Müller (JojoM)

unread,
Nov 6, 2016, 10:21:13 AM11/6/16
to FishCooking
Within the fishtest-master directory, what would I have to modify in order to get worker.py to "do the right thing" (and to not produce the error named above)? I cannot see a directory named stockfish/src in there. So where should I put the numa.h file you attached? And in which file (and where exactly) should the #include "numa.h" replacement be made?

Ronald de Man

unread,
Nov 6, 2016, 1:18:35 PM11/6/16
to FishCooking
On Sunday, November 6, 2016 at 4:21:13 PM UTC+1, Joachim Müller (JojoM) wrote:
Within the fishtest-master directory, what would I have to modify in order to get worker.py to "do the right thing" (and to not produce the error named above)? I cannot see a directory named stockfish/src in there. So where should I put the numa.h file you attached? And in which file (and where exactly) should the #include "numa.h" replacement be made?

You should put numa.h wherever you have numasf.cpp and numasf.h that is causing the error. And you have to change the #include in numasf.h, because that is where the line is located. See the error message you posted. And if you get more complaints about missing numa.h, then you have to change the #include in the files that cause those complaints.

Joachim Müller (JojoM)

unread,
Nov 6, 2016, 2:14:54 PM11/6/16
to FishCooking
So then it all comes down to the question: Where are this files stored after their download from Fishtest? I'll have to find out ...

pasquale....@gmail.com

unread,
Nov 6, 2016, 2:38:28 PM11/6/16
to FishCooking
fishtest download the source code in a temporary folder.
Look in games.py, from line 109

def build(worker_dir, sha, repo_url, destination, concurrency):
"""Download and build sources in a temporary directory then move exe to destination"""
tmp_dir = tempfile.mkdtemp()
os.chdir(tmp_dir)

with open('sf.gz', 'wb+') as f:
f.write(requests.get(github_api(repo_url) + '/zipball/' + sha, timeout=HTTP_TIMEOUT).content)
zip_file = ZipFile('sf.gz')
zip_file.extractall()
zip_file.close()

pasquale....@gmail.com

unread,
Nov 6, 2016, 3:02:45 PM11/6/16
to FishCooking, pasquale....@gmail.com
I'm a newbie in python but this should be useful (code not tested)

def build(worker_dir, sha, repo_url, destination, concurrency):
"""Download and build sources in a temporary directory then move exe to destination"""
tmp_dir = tempfile.mkdtemp()
os.chdir(tmp_dir)

with open('sf.gz', 'wb+') as f:
f.write(requests.get(github_api(repo_url) + '/zipball/' + sha, timeout=HTTP_TIMEOUT).content)
zip_file = ZipFile('sf.gz')
zip_file.extractall()
zip_file.close()

for name in zip_file.namelist():
if name.endswith('/src/'):
src_dir = name
os.chdir(src_dir)

custom_make = os.path.join(worker_dir, 'custom_make.txt')

# hack numasf.h to include a local numa.h

with open('tmp', 'w') as out:
with open('numasf.h') as f:
new_numasf = f.read()
new_numasf = new_numasf.replace('#include <numa.h>', '#include "/home/JojoM/SF-hack/numa.h")
out.write(new_numasf)
shutil.copyfile('tmp', 'numasf.h')

Lyudmil Antonov

unread,
Nov 6, 2016, 3:25:12 PM11/6/16
to FishCooking
In the unzipped directory, stockfish.exe, base.exe, cutechess-cli.exe and other files (2-moves and 8-moves opening books etc.) are stored in /fishtest-master/worker/testing/ directory

Joachim Müller (JojoM)

unread,
Nov 6, 2016, 4:09:40 PM11/6/16
to FishCooking, pasquale....@gmail.com
So what you mean is I should just paste those lines into games.py and put numa.h alongside it?

Joachim Müller (JojoM)

unread,
Nov 6, 2016, 4:11:29 PM11/6/16
to FishCooking
Or (see above) should I put numa.h into the testing directory?

pasquale....@gmail.com

unread,
Nov 6, 2016, 5:23:25 PM11/6/16
to FishCooking, pasquale....@gmail.com
The snippet of code is only a hint on how write a patch for games.py

It replaces in numasf.h the line '#include <numa.h>' with a local numa.h in your path (e.g. /home/JojoM/SF-hack/numa.h)


=====

# hack numasf.h to include a local numa.h

with open('tmp', 'w') as out:
with open('numasf.h') as f:
new_numasf = f.read()
new_numasf = new_numasf.replace('#include <numa.h>', '#include "/home/JojoM/SF-hack/numa.h")
out.write(new_numasf)
shutil.copyfile('tmp', 'numasf.h')

======



-pp

Joachim Müller (JojoM)

unread,
Nov 6, 2016, 5:53:09 PM11/6/16
to FishCooking, pasquale....@gmail.com
I'll give it a try when the next numa test comes up.

pasquale....@gmail.com

unread,
Nov 6, 2016, 6:03:34 PM11/6/16
to FishCooking, pasquale....@gmail.com
You can install locally a fishtest development server (it's very easy) and load a numa test already passed on fishtest

https://github.com/glinscott/fishtest/wiki/Installing-a-development-fishtest-server-on-Ubuntu

http://tests.stockfishchess.org/tests/view/581e027a0ebc5910626b9b87

Joachim Müller (JojoM)

unread,
Nov 6, 2016, 6:47:44 PM11/6/16
to FishCooking, pasquale....@gmail.com
Thanks very much, I'll give it a go when I have time ;)

Ronald de Man

unread,
Nov 7, 2016, 3:43:30 AM11/7/16
to FishCooking
On Sunday, November 6, 2016 at 8:14:54 PM UTC+1, Joachim Müller (JojoM) wrote:
So then it all comes down to the question: Where are this files stored after their download from Fishtest? I'll have to find out ...

It is also possible to store the numa.h I attached in, say, /home/jojom/include and then add /home/jojom/include to the CPATH environment variable. In that case you do not need to modify anything in the SF source at all.

But then you need to know how to set environment variables and launch fishtest with the variable correctly set.

https://gcc.gnu.org/onlinedocs/gcc/Environment-Variables.html

Joachim Müller (JojoM)

unread,
Nov 7, 2016, 4:55:23 AM11/7/16
to FishCooking
Please do me a favour and instruct me on (1) how to edit the CPath variable, and (2) how to launch the worker accordingly.

I hear, Marco is willing to start NUMA testing with multi-core hardware provided by Ipman. That being said, I might still be able to help when the results are examined in the framework later on. As I did last year, when Lazy SMP was introduced.

Ipman Chess

unread,
Nov 11, 2016, 9:41:25 AM11/11/16
to FishCooking
I'm still waiting for a reply from Marco..e-mail sent on 06-11-2016

Maybe he is working on numa-code..

Ipman.

Op maandag 7 november 2016 10:55:23 UTC+1 schreef Joachim Müller (JojoM):

Marco Costalba

unread,
Nov 11, 2016, 10:45:26 AM11/11/16
to Ipman Chess, FishCooking
Sorry I have quite limited time in these weeks but I don't forget what I have said. 

Ipman Chess

unread,
Nov 11, 2016, 10:57:07 AM11/11/16
to FishCooking, ipman...@telenet.be
No problem Marco..you know my e-mail and you can always sent me something when it's ready for testing!

Best regards,
Ipman.

Op vrijdag 11 november 2016 16:45:26 UTC+1 schreef Marco Costalba:

mcos...@gmail.com

unread,
Nov 20, 2016, 5:17:22 PM11/20/16
to FishCooking, ipman...@telenet.be
Now it's ready :-)

Please pull from

https://github.com/mcostalba/Stockfish/tree/numa

I have just fixed a possible race, so for you and people interesting to test it, please pull it again. Thanks.

Joachim Müller (JojoM)

unread,
Nov 22, 2016, 5:04:53 AM11/22/16
to FishCooking
Will NUMA be tested in the framework and if successful be merged into SF master?

peter.zs...@gmx.at

unread,
Nov 22, 2016, 2:07:19 PM11/22/16
to FishCooking, ipman...@telenet.be
Hello,

As far as I have seen, this patch should be tested only under Windows, right?
Is it only relevant for more than 64 Threads, or does it makes sense to test with 64 Threads (and less)?

br

peter.zs...@gmx.at

unread,
Nov 22, 2016, 4:08:53 PM11/22/16
to FishCooking, ipman...@telenet.be, peter.zs...@gmx.at
I posted some benchmark results in the pull request:
https://github.com/official-stockfish/Stockfish/pull/883

peter.zs...@gmx.at

unread,
Nov 22, 2016, 5:58:37 PM11/22/16
to FishCooking, ipman...@telenet.be, peter.zs...@gmx.at
Let me post something off-topic here, but it is somehow funny, at least for me :D
I bought Houdini 5. H5 has an option to enable/disable NUMA per uci option, too.

The funny thing is H5 has a bench command very similar to stockfish, too.
All I needed was to create a file with the name "benchpos.txt".
I created the bench positions from Stockfish benchmark.cpp.
Then I could run "bench <hash-size> <nr-of-threads> <depth>"

H5 fills the hash quite quickly, but does not reach high depth, so I used:
bench 2048 64 18 also 3 runs each. The differences in the individual runs were much higher than with stockfish or cfish, but for a rough comparison it should be OK:

Houdini 5 default avg: 84,9 MN/s
Houdini 5 with "setoption name NUMA Enabled value false" avg: 54,8 MN/s
so +54,9% speed gain for Houdini 5 numa awareness.

Lyudmil Antonov

unread,
Nov 23, 2016, 1:36:57 AM11/23/16
to FishCooking, ipman...@telenet.be, peter.zs...@gmx.at
The percentage gain (+54.9%) is close to what you reported for CFish (52.5%)

A. Turkoglu

unread,
Nov 24, 2016, 12:39:19 PM11/24/16
to FishCooking
Hi Peter,

Your results:

Houdini: 84 MN/s
CFish: 61 MN/s
Stockfish: 31 MN/s ??

I think there is something wrong with SF... Could you retest SF with bench 2048 32 24?

Thanks.

peter.zs...@gmx.at

unread,
Nov 24, 2016, 4:08:28 PM11/24/16
to FishCooking
Dear Atahan,

I am sorry to say, but NO.
I do not run some useless benchmarks again.
Please search in the forum, I already posted values a month ago benchmarks for the "no-Gimmicks" numa version with the same CPUs Opteron 6386SE.
Before I posted a lot of other bench-values for a similar a bit slower CPUs Opteron 6282SE (also 64 Core, but only 2.6GHz instead of 2.8GHz).

Honestly spoken, I do not understand at all what you want: Stockfish 31,9 MN/s is not possible?

Did you notice, that I measured 26.7 MN/s for official Stockfish??? 31.9 MN/s is already a good speed up. Did you expect, that I have double value like 55 MN/s for NUMA version? Or do you think that with the official (non NUMA) Stockfish version NOT all the 64 Cores are used?

Please note, that these questions were rhetorical questions, I do not need an answer to it.

The purpose of my posting was to show, that with a good NUMA implementation something like 50% speed gain on this 8-Node Machine is possible (it is the case for CFish and Houdini 5). So 26,7 * 1,5 = approx. 40 MN/s should be theoretically possible with this Hardware with official stockfish.

best regards Peter

Message has been deleted

A. Turkoglu

unread,
Nov 25, 2016, 8:11:17 AM11/25/16
to FishCooking, peter.zs...@gmx.at
Dear Peter, 

What is the difference between CFish and Official Stockfish? 
You know the difference is just programming language. (CFish: C, Stockfish: C++)

So why is CFish speed 61 MN/s but Stockfish's speed 31 MN/s? 

Is 31 MN/s normal for 64 cores? 

Why is Houdini 2.7 times faster than Stockfish on your machine? (84 MN/s vs 31 MN/s)
(TCEC Superfinal 44-core server, Houdini: 64 MN/s, Stocfish: 49 MN/s, Speed rate: 1.3)

The new Numa code has issues with 8-Socket system IMO. 

Thanks for your answer.

Best regards.



Dear Ronald de Man,

What do you think about the new "Handle Windows Processors Group" patch? 

peter.zs...@gmx.at

unread,
Nov 27, 2016, 7:17:56 AM11/27/16
to FishCooking, peter.zs...@gmx.at
Hi Atahan,

I have news for you:
On my machine official Stockfish is about twice as fast under Linux:

>>>>>
I checked this patch on my system: Opteron 64 Core, 8 Nodes:
Ubuntu 16.04.1 LTS, native gcc. I use latest master, and changed misc.cpp for the patched version.
5 Runs of bench 2048 64 24 for each version:
Master: 59,9/61,7/61,6/63,9/62,9
Patched: 63,9/64,7/65,2/65,3/63,5
=> 4% speedup
<<<<
So official Stockfish under Linux is faster than CFish under Windows :D
The patched version, which is a bit faster is
"Bind each thread to its own CPU on Linux"

best regards
Reply all
Reply to author
Forward
0 new messages