The Evidence against Rybka

[BB+]

Perhaps it's not a coincidence we are both ex-commercials and therefore have different weight-factors.

So now we have three groups: "academics" who mostly value "originality", "commercials" who balance Elo and originality, and "end-users" who only want Elo.

Perhaps since I am mathematician, I personally think that "tuning parameters" is of less value than "choosing/combining features", and indeed, I get the impression that some of the Fruit 2.1 values are simply "placeholders" before the tuning was done [FL carried out (in part) such tuning over the next few months, before the 2005 WCCC, and then after it prior to Fruit commercialisation]. Back in 2004, VR had a comment regarding the scope of "simplistic evaluation" with Fruit 1.0:

From: Vasik Rajlich
Message Number: 354659
Date: March 15, 2004 at 09:25:31

Fabien,

your project is interesting, because I think a lot of the programmers are wondering how many of
their own evaluation terms are actually useful, and how many just slow things down, introduce bugs, etc.

Are there any "final" numbers regarding the strength of your engine?

Also, I would be curious to see the results of a reasonably strong mature engine, with its
evaluation function greatly condensed to some very simple, but still logical, piece-square tables,
and some necessary material eval & passed pawn stuff.

Approaching 9 months after this, Rybka's eval appears still to be in a nascent form.

From: Vasik Rajlich
Message Number: 398692
Date: December 03, 2004 at 07:22:10
[...]
Unfortunately at the moment Rybka follows an even more basic law: if your evaluation is bad,
at least keep it cheap  There's nothing worse than a big mess of untested eval terms ...

I would guess that [contrary to LK's first guess] it would take more than just a couple of weeks to spruce out a suitable set of tested eval terms. I would venture about 6 months (it took FL around 15 months on a part-time basis).

Incidentally, Fruit 1.0 was initially designed on paper -- FL lacked a computer for the first month or so upon moving to Cambridge.

Taking 4.1 - 4.3 into consideration I can't really say this makes a convincing case for copying, neither semantics.

I think the case for "copyright infringement" (or plagiarism) of the evaluation function as a whole is quite weighty, particularly when combined with the various other Fruit 2.1 bits that appear here-and-there in Rybka 1.0 Beta. Whether or not such "copyright infringement" (as I see it) was done by "copying" [as you seem to use the term, giving it a more literal sense] is a different matter. I look at the end result, and see (as per Fabien's comment about Strelka "re-writing the same thing with different words") what I consider to fall under the umbrella of "substantial similarity". Whether or not it's 10% or 90% of the totality of Rybka version X is a different (and difficult) matter. The ICGA Board verdict concluded that Rajlich had exceeded their "originality" boundary, though some of the Panel (such as Gerd Isenberg, and also Bob Hyatt and myself in part) counselled that a more appropriate punishment would be to simply change the winners for the appropriate years to be a "Rybka/Fruit" hybrid [this would seem much more delicate, perhaps even impossible, w/o VR's agreement on the matter].

Fruit's EVAL is nothing special, it's just good. Rebel has everything Fruit has and more AND already in the 80's. From the early accusations back in 2006 till now only the 0.0 still stands as strong evidence of COPYING. I always felt EVAL (even during the period I held the VIG view) to be the weakest part of the of the allegations. What you find in Fruit and Rybka are the basic ingredients of normal general and public available knowledge. There is no copyright on knowledge.

I agree, that Fruit has little "unique" to itself in evaluation. But what is of value is its rendition on what knowledge to use/combine. As can be noted from the above, VR was quite interested in how strong one could get an engine with the (essentially empty) Fruit 1.0 eval [I could give other 2004 posts, where he concludes that eval is by far the most important aspect -- by 2005, perhaps in part due to Fruit influence, I think he had switched his opinion of what was of most import]. I would call it "taking a shortcut" if one were to use FL's choice of eval components, even if none is spectacular by itself. As stated above, I personally find "tuning the numbers" to be of a more rote nature, with choosing features (as eventually done by LK for Rybka 3) to have more creative content.

To adduce another example about "collation of knowledge" being protected expression, I happened to stop at the library this morning, and picked up Sacrifices in the Sicilian by no less than DNL Levy. In his Preface, he even notes that not much is "new" in this book, other than his collation aspect (emphasis added).

In collecting the material for this book I have relied almost entirely on my own library and that of R. G. Wade who I would like to thank for his ever helpful assistance. I should also like to thank P. Poutiainen, C. W. Pritchett and H. Westerinen who readily contributed analyses, The Chess Player for permission to reproduce the Padevsky-Botvinnik example from Botvinnik's Best Games 1947-70, Chess for permission to use a translation that appeared in some of its 1963 issues, D. N. L. Levy for permission to quote from his excellent monograph The Sicilian Dragon, and lastly K. J. O'Connell for preparing the indexes and reading the proofs. Much of the analysis contained in this book stems from Soviet and Yugoslav sources. I would like to mention all those commentators whose notes I have used but there are too many of them. Perhaps one should also acknowledge the players who produced these sacrificial examples and games. While creativity of this kind exists in master chess the game will continue to appeal to an ever increasing number.

You can probably guess my comment on this, that although not much is "novel" or "unique" to the contents of the book, DNL Levy would indeed retain copyright on the structural aspects of this collation (such as which games to include). A modern version of this could certainly re-use the overall chapter division (Rxc3, Nf5, Rxf6, Bxb5, Nxb5, Nd5, Bd5, Ne6, Bxe6, Nxe6, d5), and could use the older classical/historical games where appropriate (Tal-Larsen and Velimirovic-Ljubojevic for Nd5, Bronstein-Najdorf and Vasyukov-Averbakh for Bxb5), but no one should expect the overall game selection to have great resemblance with that chosen by Levy in 1973.

[Rebel]

Taking 4.1 - 4.3 into consideration I can't really say this makes a convincing case for copying, neither semantics.

I think the case for "copyright infringement" (or plagiarism) of the evaluation function as a whole is quite weighty, particularly when combined with the various other Fruit 2.1 bits that appear here-and-there in Rybka 1.0 Beta.

I don't feel that addresses the point I made.

1. In science, unfolding a constant "A" into "B - C" just because you assume it is justified, without any explanation, is literally, academic dishonesty. In science you are supposed to present the data as it is, transparently. The fact that you do not realize how bad this is I consider problematic.

2. The other denial of unburden evidence, that adding to score is principally different.

as a whole

The rook evaluation example I gave could be tolerated if it were an isolated case, but it isn't, it's a pattern.

Do you want me to post more EVAL examples ?

-------

I will comment on your other points in a separate post just because for the fun of it.

[Rebel]

Perhaps it's not a coincidence we are both ex-commercials and therefore have different weight-factors.

So now we have three groups: "academics" who mostly value "originality", "commercials" who balance Elo and originality, and "end-users" who only want Elo.

Something like that yes

And probably 4 groups. Active commercials and ex-commercials. Note the Panel voting.

Perhaps since I am mathematician, I personally think that "tuning parameters" is of less value than "choosing/combining features", and indeed, I get the impression that some of the Fruit 2.1 values are simply "placeholders" before the tuning was done [FL carried out (in part) such tuning over the next few months, before the 2005 WCCC, and then after it prior to Fruit commercialisation]. Back in 2004, VR had a comment regarding the scope of "simplistic evaluation" with Fruit 1.0:

If you compare all those Fruit <> Rybka evaluation values you can obviously see the hand of the better chess player.

I agree, that Fruit has little "unique" to itself in evaluation. But what is of value is its rendition on what knowledge to use/combine. As can be noted from the above, VR was quite interested in how strong one could get an engine with the (essentially empty) Fruit 1.0 eval [I could give other 2004 posts, where he concludes that eval is by far the most important aspect -- by 2005, perhaps in part due to Fruit influence, I think he had switched his opinion of what was of most import].

Absolutely. Every chess programmer by experience eventually will figure it out that after having a good EVAL search rules and the quality of your search will be decisive about your place in the rating lists. And that adding to EVAL after that gives very little.

To adduce another example about "collation of knowledge" being protected expression, I happened to stop at the library this morning, and picked up Sacrifices in the Sicilian by no less than DNL Levy. In his Preface, he even notes that not much is "new" in this book, other than his collation aspect (emphasis added).

In collecting the material for this book I have relied almost entirely on my own library and that of R. G. Wade who I would like to thank for his ever helpful assistance. I should also like to thank P. Poutiainen, C. W. Pritchett and H. Westerinen who readily contributed analyses, The Chess Player for permission to reproduce the Padevsky-Botvinnik example from Botvinnik's Best Games 1947-70, Chess for permission to use a translation that appeared in some of its 1963 issues, D. N. L. Levy for permission to quote from his excellent monograph The Sicilian Dragon, and lastly K. J. O'Connell for preparing the indexes and reading the proofs. Much of the analysis contained in this book stems from Soviet and Yugoslav sources. I would like to mention all those commentators whose notes I have used but there are too many of them. Perhaps one should also acknowledge the players who produced these sacrificial examples and games. While creativity of this kind exists in master chess the game will continue to appeal to an ever increasing number.

You can probably guess my comment on this, that although not much is "novel" or "unique" to the contents of the book, DNL Levy would indeed retain copyright on the structural aspects of this collation (such as which games to include). A modern version of this could certainly re-use the overall chapter division (Rxc3, Nf5, Rxf6, Bxb5, Nxb5, Nd5, Bd5, Ne6, Bxe6, Nxe6, d5), and could use the older classical/historical games where appropriate (Tal-Larsen and Velimirovic-Ljubojevic for Nd5, Bronstein-Najdorf and Vasyukov-Averbakh for Bxb5), but no one should expect the overall game selection to have great resemblance with that chosen by Levy in 1973.

I fail to see the link between me going forwards and backwards through the Max Euwe books taking many things and the 200x period when the new modern programmers started to take from freely downloadable sources writing their own code. Are you saying that I should have expressed my thanks to the late great Max Euwe in every version ?

From a commercial point of view that's a dumb thing to do, your customers don't want to hear your software relies the idea's of others, they want to hear that your software is the best buy on the Planet.

Classic academic <> commercial type of discussion

[hyatt]

In science, unfolding a constant "A" into "B - C" just because you assume it is justified, without any explanation, is literally, academic dishonesty. In science you are supposed to present the data as it is, transparently. The fact that you do not realize how bad this is I consider problematic.

That STILL shows a complete lack of understanding dealing with semantic equivalence and what one has to do to compare a binary (assembly language after disassembly) with an existing source program. The binary can NOT be changed in any way that alters semantics. Neither can the C source. The ultimate goal is to match up the C and the asm, leaving nothing out, adding nothing that is not already there.

If one sees this in the C source:

score = score + term 1 - term 2;

where term 1 and 2 are local constants that can not be changed anywhere, then one could expect the compiler to fold those two constants into one, since constant - constant == constant.

When one sees this in the asm at the same point:

addl $value, %rax

one can assume $value == constant == term1 - term2

This is NOT "academic dishonesty". It is a total lack of understanding about the RE process on your part. For example, I compile Crafty and produce a binary. Someone somehow copies that binary and I want to prove it is mine. How do I do that? Just like we did with pieces of Fruit/Rybka. You see if the asm given matches the C given, without leaving any asm out, or having to add any C (one can add C that does not change semantics, of course, but there is no reason to do so).

This entire line of argument is one more "last gasp" effort to discredit the ICGA report. It is based on nonsense, is not supported by any type of "authority" on the planet (did you ever try posing your question to someplace like "comp.compilers" in usenet news. You might get a REAL education if you'd try. Warning: do NOT introduce any of your nonsense arguments there, things will turn real ugly, real quick, when you try to tell people that know what they are doing, that what they are doing is nonsense and "academic dishonesty."

It is time to move on to some other nonsensical argument, this one has been taken off life support, and is dead.

There is no point 2 at all...

[hyatt]

If you compare all those Fruit <> Rybka evaluation values you can obviously see the hand of the better chess player.

I've said this many times, it is much less important which weight you choose for a term, compared to choosing to implement the term in the first place. "square of the king" for passed pawn races is a good example. Implementing it is important. Does it matter whether you use +5.0 or +7.0 as the score? Hardly at all. Taking it out completely will make a difference. Ditto for trapped bishop. Just because Vas optimized the weights, does not excuse him for copying the code that does the scoring in the first place.

[mjlef]

Have you looked at Fruit source? Understand the "[me]" idea? For example, in the passed pawn code (eval_passer()) you find that op[white] and op[black] (or op[me] where there is a loop over white and black, with "me = color" at the top of the loop.

Then, at the end, you find this (again, eval_passer() for just one example:

*opening += ((op[White] - op[Black]) * PassedPawnWeight) / 256;
*endgame += ((eg[White] - eg[Black]) * PassedPawnWeight) / 256;

Which is now back to the usual opening/endgame scores. Why fabien did that I do not know. Perhaps he thought it would be useful to have the white/black scores separated for some other use. It is certainly a simple enough change with an editor...

I did something similar at one point, because I thought it would be useful to have pawn scores separate rather than just a single composite score, when trying to determine if I wanted to drag the eval toward draw or not in endgames, where pawn scoring is important.

My understanding was this was on purpose. Fabien wrote the code so it would work from either side, so he just looped over the to sides and correct for black by subtracting. A classic speedup trick is to unroll the loop, and this is what was done in Rybka. Save a little time since the two score do not need to be combined.

[hyatt]

Very possible. The primary point, however, is that we have a very small group (2 actually) that want to claim dishonesty, misrepresentation, and a host of other things, since they have, apparently, given up on discrediting the evidence itself. So now we are entering state III of this process (the first was denial, tne second was the evidence is flawed) where the people involved are simply trying to deceive everyone by pointing out flaws that do not exist and which do not make any sense whatsoever.

Hopefully, this will be a "short stage".

[Rebel]

My understanding was this was on purpose. Fabien wrote the code so it would work from either side, so he just looped over the to sides and correct for black by subtracting. A classic speedup trick is to unroll the loop, and this is what was done in Rybka. Save a little time since the two score do not need to be combined.

Assumption on assumption on assumption is good science ?

[hyatt]

No, because we don't need an "assumption" there. The bottom line is "who cares WHY he did it?" The investigation was not about "why" something was done, but instead was all about "what/how something was done."

[mjlef]

My understanding was this was on purpose. Fabien wrote the code so it would work from either side, so he just looped over the to sides and correct for black by subtracting. A classic speedup trick is to unroll the loop, and this is what was done in Rybka. Save a little time since the two score do not need to be combined.

Assumption on assumption on assumption is good science ?

No, not an assumption. My understanding. Fabien discussed his programming philosophy either on one of these forums or in a private email. I will try to find a quote. But the basic idea was to keep things simple. By coding things once but having the code aware for each side, you avoid silly errors like forgetting to add a new term in two different places for each side. Granted, it might be a bit slower, but it is easier to understand and less prone to programming errors since the code is about half the size. Fabien preferred clarity over speed. I recall in some Fruit release notes he said "bear in mind mobility is a costly feature, when implemented in a straightforward way as I did" making it clear he likes clear, straight forward code over clever, faster and more confusing code.