Houdini is a clone of Stockfish 8

[c9publice...@gmail.com]

(x-post of a talkchess post that I made, that is in Engine Origins, that I am currently unable to access. I am locked out of my talkchess account, so hopefully discussion can be redirected here.)

Hello,

Before I begin this, I want to make clear that I will share sources and further information with experts if asked by experts directly. I do not intend to release large segments of Houdart's original code except to e.g. actual authorities, like talkchess moderators, very well known community members (e.g. former talkchess moderators), or noted SF contributors. I'm afraid, however, that if I share the code with SF contributors, there may be a threat of tainting the SF codebase or tainting that contributor in general, as happened when Richard Vida reverse engineered Houdini 1.5, revealed it to be based on RobboLito, and was forced to stop developing Critter. If there is anyone to whom these concerns do not apply, please share your contact information (or contact me on Discord. I can be found in the TCEC Discord channel) and I can provide the necessary evidence.

Recently, I came into possession of Houdini 6's source code (experts can ask how--I already reported how to Houdart in doing my due diligence...my background is in security work. I only realized it was a clone far later) and decided to look into the code.

First, I should note that Houdart has done a good deal of original work and should be commended for this. However, Houdart's work from Houdini 5 onward comes from building on top of Stockfish. Houdini 5 is a Stockfish fork, no more and no less.

This is the Houdini 5-6 makefile: https://pastebin.com/XNZiNXc9

Line 410 is of note; it includes "to compile stockfish"

This is from util_tools.cpp, if you were curious about the version:

#if defined(CHESSKING)
#define PROGRAM_NAME "Houdini 6"
#else
#define PROGRAM_NAME "Houdini 6.03"
#endif

Houdini's roots were initially (as in for the first version(s)) in the form of being a modified RobboLito as well, but that was well known, even if Houdart would deny it.

If you translate search.cpp to the Dutch (Houdart is from a Dutch-speaking region of Belgium) zoeken.cpp, you'll find Houdini's search.

The search is based on Stockfish: https://i.imgur.com/hmAa6o0.png

The same code in Stockfish: https://i.imgur.com/ReGal5Q.png

The exact version of Stockfish that Houdini was based on can be found here: https://gitlab.com/cucumbers/stockfish-frozen

Although I believe Houdart also took incorporated newer fishtest patches and changes, so that is something worth bearing in mind.

Houdini's case appears to be somewhat similar to Rybka's, though Rybka ended up with far more innovations (and far more valuable innovations at that) than Houdini did. Houdini has taken Stockfish and made a large number of large changes. It has its own testing system, for example, completely distinct from Fishtest's. It has a custom tuner and specialized evaluation. Nonetheless, it is still Stockfish, and it is far more Stockfish than Rybka was Fruit.

I would think that this is a significant GPL violation.

[werner....@gmail.com]

On Thursday, March 26, 2020 at 2:19:46 AM UTC+1, c9publice...@gmail.com wrote:

(x-post of a talkchess post that I made, that is in Engine Origins, that I am currently unable to access. I am locked out of my talkchess account, so hopefully discussion can be redirected here.)

If you translate search.cpp to the Dutch (Houdart is from a Dutch-speaking region of Belgium) zoeken.cpp, you'll find Houdini's search.

The search is based on Stockfish: https://i.imgur.com/hmAa6o0.png

The same code in Stockfish: https://i.imgur.com/ReGal5Q.png

One remark. I had a look through the code you posted and I have a hard time believing Houdart would write his entire engine in Dutch (I'm Dutch btw).

I and every programmer I know would write commercial software in English and not the local native language (at least this is true for Western Europe I think)

[garrykli...@gmail.com]

hes saying the parts he ripped off were at least in dutch

[firstname lastname]

I don't know what to tell you. Most of the code has been translated into Dutch, but there are are lots of English bits in there as well. For example, this is uci_opties.cpp. I don't mind sharing this part of Houdini's code, because (1) it's very particularly "just SF in Dutch," and (2) I don't think Houdart has any particular attachment to Houdini's UCI interface :P 

https://pastebin.com/UpYgVFd7 is the code.

I promise, I do not speak a lick of Dutch. Anyone can decompile Houdini with software like Ghidra and search for these strings, and see that the code matches up fairly close. Just like Stockfish. Of course, my promise means very little, but that is why I am offering to share more code.

On Thursday, March 26, 2020 at 5:02:39 AM UTC-4, werner....@gmail.com wrote:

[Joseph Ellis]

It is pretty simple really.  All of whatever code this is is GPL.   Send full source to one of the top SF admins, let them compile it and compare results vs retail Houdini.

[firstname lastname]

If an admin steps up and is willing to look at the source code, and is not concerned regarding being "tainted" by seeing it or legal concerns, then I will gladly send it over.

[joost.van...@gmail.com]

Have you or anybody else with a binary copy of Houdini just asked to Houdart to provide the source code of Houdini with a GPL license... that's the first step. If that's refused (on not answered in due time) one can think about a next step.

You suggest that you contacted Houdart already, did he comment on this?

[firstname lastname]

Houdart fixed the issue that allowed me to get the source code (I still have it downloaded, of course) but never replied to the email explaining the issue to him. Because I am not a purchaser of Houdini, he is not required to provide the source code to me, so I cannot ask. If anyone owns Houdini and can try, that would be great. I would not expect a reply, though.

At this point, though, I have Houdini 6.03 compiling and running well on Linux. Based on the fastgm experiments, my binary appears to match up with the commercial release of Houdini 6(.03). It is not particularly concerning to me, then, that Houdart sends me his code, although I certainly understand why you would want it "straight from the horse's mouth."

[danny...@yahoo.co.uk]

On Thursday, March 26, 2020 at 4:59:54 PM UTC, firstname lastname wrote:

Houdart fixed the issue that allowed me to get the source code (I still have it downloaded, of course) but never replied to the email explaining the issue to him. Because I am not a purchaser of Houdini, he is not required to provide the source code to me, so I cannot ask. If anyone owns Houdini and can try, that would be great. I would not expect a reply, though.

At this point, though, I have Houdini 6.03 compiling and running well on Linux. Based on the fastgm experiments, my binary appears to match up with the commercial release of Houdini 6(.03). It is not particularly concerning to me, then, that Houdart sends me his code, although I certainly understand why you would want it "straight from the horse's mouth."

 

Not exactly remarkable, Houdini doing another disappearing act when quizzed. Magicians don't like to give away their secrets.

More deserving of note, in a TCEC interview ~ season 10, asked what he thought made Houdini unique, Houdart replied "It has good mobility". I assume he meant piece mobility, not some copy paste job. ;)

[garrykli...@gmail.com]

right, when you're assuming hes not a rip off.

[alu...@gmail.com]

I mirrored noobpwnftw's upload from talkchess https://github.com/alureon/houdini

[joost.van...@gmail.com]

I assume that's distributed under license as specified in houdini/polarssl/padlock.h ?

It is rather clear this is derived from stockfish.

It is rather sad as well... need 15% more nps?

uint64_t ThreadPool::bezochte_knopen()
{
        uint64_t knopen = 0;
        for (int i = 0; i < activeThreadCount; ++i)
                knopen += threads[i]->rootStelling->bezochte_knopen();
        knopen += knopen / 7;
        return knopen;
}

[dieter.d...@gmail.com]

Oh wow, I am speechless. Any possible doubt one could still have on the author's integrity is obliterated by this one line of code...

For people that don't speak Dutch (and couldn't deduce it from the context :p): rootStelling means rootPosition, bezochte_knopen means visited_nodes.

Op maandag 6 april 2020 08:39:11 UTC+2 schreef joost.va...@gmail.com:

[garrykli...@gmail.com]

Let's guess they will choose to 'plead the 5th no comment', so that what they've done isnt held against them. How convenient :)

[Lyudmil Antonov]

uint64_t ThreadPool::bezochte_knopen()

{
        uint64_t knopen = 0;
        for (int i = 0; i < activeThreadCount; ++i)
                knopen += threads[i]->rootStelling->bezochte_knopen();
        knopen += knopen / 7;
        return knopen;
}

 

Have we implemented this code in SF ? 

[aconn...@gmail.com]

The code just counts nodes from the threads. I think the point is more that he is cheating in the stupidest way by overreporting the nodes (knopen += knopen / 7), presumably so he can say his multithreading implementation is much better than the competition

[aconn...@gmail.com]

The code is interesting in that it never reports a node count which is 7 mod 8, which is externally observable. 

[dieter.d...@gmail.com]

That's a cool observation. Here is some evidence based on the publicly available log file of TCEC game 68 (Stoofvlees-Houdini) in current Divsion P: http://tinyurl.com/r63jr93

Number of occurrences of reported nodes mod 8:

[292, 304, 296, 295, 291, 308, 289, 0]     (Houdini)

[240, 230, 234, 231, 243, 220, 215, 210]    (Stoofvlees)

Op maandag 6 april 2020 19:46:40 UTC+2 schreef acon...@gmail.com:

[chris...@yahoo.com]

Yup. Confirmed. The nodes also alternate cyclically odd/even as is to be expected with n+=n/7

Final digit, frequency

0 400

1 342

2 408

3 384

4 427

5 364

6 425

7 353

8 440

9 355

So, that would confirm, good enough for me, that the executable matches the source with the n+=n/7 adjusted nodecount

[Михаил Чалый]

Well, okay.

Now we need to replay some TCEC finals where houdini participated and disqualify it even as a placeholder from any competition.

[Alexey Eromenko]

Here is the leaked Houdini 6 source code:

https://www.dropbox.com/s/h9w3pankltjqvy7/houdini-6-src-leaked.zip?dl=0

[Alexey Eromenko]

The question is this : is it possible or impossible to catch SF8 vs. H6 by comparing black box (binary) evaluation ? Why it wasn't caught before?

[Михаил Чалый]

Well it's still stockfish - based.

Although it changes doesn't work in current master, at least some that I tried :D

[firstname lastname]

No. 

[firstname lastname]

Might be a good idea to note what you've tried (and haven't tried) so others don't duplicate your efforts. 

[Михаил Чалый]

Razoring changes and null move changes.

[Alexey Eromenko]

So any news here? Houdini was copied from SF ? How many changes are there ? Any significant changes ?

Do we know how licensing or copy protection work in Houdini ? Is it per user or per PC ? How is it enforced ?

Normal key has a 8-core limit ? And Pro key has 128 core limit ? Why 128 cores BTW ? Where does it come from ?

What is the max core limit for Stockfish (v8 and v11) ?

[Михаил Чалый]

It's obviously a stockfish derivative.

I don't find any other questions interesting. Difference is signifficant but not enormous. Why core limit and other stuff like this is done is a way this done - honestly I give 0 cares about it.

Especially when you can take other derivative of the same program but 100 elo stronger.

[alfreds...@gmail.com]

Hello,

not being a programmer myself I tried to compile the source on Linux. I did get quite far with doing some minor modifications, but I am not able to find Linux replacements for code below. Would be nice if someone more skilled than myself could have a look at this and give a short estimate if this can be transformed to Linux easily?

g++ -Wall -Wcast-qual -fno-exceptions  -pedantic -Wextra -Wshadow -m64 -DNDEBUG -O3 -DIS_64BIT -msse -msse3 -mpopcnt -DUSE_POPCNT -flto -fno-rtti -std=c++11   -c -o zoek_smp.o zoek_smp.cpp
zoek_smp.cpp: In function ‘void* alloc_large_page_mem(size_t, bool*, bool, int)’:
zoek_smp.cpp:502:37: error: ‘MEM_COMMIT’ was not declared in this scope
  502 |   result = VirtualAlloc(NULL, size, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
      |                                     ^~~~~~~~~~
zoek_smp.cpp:502:50: error: ‘MEM_RESERVE’ was not declared in this scope
  502 |   result = VirtualAlloc(NULL, size, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
      |                                                  ^~~~~~~~~~~
zoek_smp.cpp:502:63: error: ‘PAGE_READWRITE’ was not declared in this scope
  502 |   result = VirtualAlloc(NULL, size, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
      |                                                               ^~~~~~~~~~~~~~
zoek_smp.cpp:502:12: error: ‘VirtualAlloc’ was not declared in this scope
  502 |   result = VirtualAlloc(NULL, size, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
      |            ^~~~~~~~~~~~
zoek_smp.cpp: In function ‘void free_large_page_mem(void*)’:
zoek_smp.cpp:516:20: error: ‘MEM_RELEASE’ was not declared in this scope
  516 |  VirtualFree(p, 0, MEM_RELEASE);
      |                    ^~~~~~~~~~~
zoek_smp.cpp:516:2: error: ‘VirtualFree’ was not declared in this scope
  516 |  VirtualFree(p, 0, MEM_RELEASE);
      |  ^~~~~~~~~~~
make[1]: *** [<builtin>: zoek_smp.o] Error 1

[ts.tom...@gmail.com]

this could be simply replaced by normal allocations (new/delete or std::unique_ptr<T[]>) if a small performance penalty with large tables is ok

[Andrea Manzo]

Did you forget the licentie.cpp file ?

It doesnt't compile.

[alfreds...@gmail.com]

Nice to hear that replacing these functions is no problem for a somewhat skilled person...

You don't need the file licentie.cpp to successfully build the source. Just remove it from the Makefile.