ChessBase: A Gross Miscarriage of Justice in Computer Chess

[kingliveson]

ChessBase on the Rybka/Fruit controversy: http://www.chessbase.com/newsdetail.asp?newsid=7791

As Jeremy said, "In the new article, they should have mentioned that a) Søren is a moderator on the Rybka Forum and b) ChessBase is a distributor of Rybka and in no way impartial."

[Harvey Williamson]

ChessBase on the Rybka/Fruit controversy: http://www.chessbase.com/newsdetail.asp?newsid=7791

As Jeremy said, "In the new article, they should have mentioned that a) Søren is a moderator on the Rybka Forum and b) ChessBase is a distributor of Rybka and in no way impartial."

Soren sent me an email a few months ago where he says, 'It looks like Vas stole his evaluation from Fruit.' It seems for whatever reason he has changed his opinion although I see no new evidence in his article.

[hyatt]

"changed his mind"??? Based on his comments on the Rybka forum, I sometimes wonder if he has one to change... He is repeating past false statements, and referring to the "king of false statements" by referencing Ed's web page which is intentionally misleading and unsound.

More on this once we get to see the "rest of the story" from him.

[kingliveson]

ChessBase on the Rybka/Fruit controversy: http://www.chessbase.com/newsdetail.asp?newsid=7791

As Jeremy said, "In the new article, they should have mentioned that a) Søren is a moderator on the Rybka Forum and b) ChessBase is a distributor of Rybka and in no way impartial."

Soren sent me an email a few months ago where he says, 'It looks like Vas stole his evaluation from Fruit.' It seems for whatever reason he has changed his opinion although I see no new evidence in his article.

This is telling given his affiliation with Rybka. And true, there was no new evidence presented in the opinion piece, or by Mr. Schröder. At this point, nothing short of releasing the source code could exoneration Rybka.

[Harvey Williamson]

ChessBase on the Rybka/Fruit controversy: http://www.chessbase.com/newsdetail.asp?newsid=7791

As Jeremy said, "In the new article, they should have mentioned that a) Søren is a moderator on the Rybka Forum and b) ChessBase is a distributor of Rybka and in no way impartial."

Soren sent me an email a few months ago where he says, 'It looks like Vas stole his evaluation from Fruit.' It seems for whatever reason he has changed his opinion although I see no new evidence in his article.

This is telling given his affiliation with Rybka. And true, there was no new evidence presented in the opinion piece, or by Mr. Schröder. At this point, nothing short of releasing the source code could exoneration Rybka.

No need to release it but it could be sent to a trusted 3rd party - maybe Ken Thompson.

[Nelson Hernandez]

Hold your fire, guys. Sounds to me like Soren is just clearing his throat in Part I.

[Uly]

He is repeating past false statements, and referring to the "king of false statements" by referencing Ed's web page which is intentionally misleading and unsound.

Can you elaborate? I didn't find anything false in Ed's pages.

[Rebel]

This is telling given his affiliation with Rybka. And true, there was no new evidence presented in the opinion piece, or by Mr. Schröder. At this point, nothing short of releasing the source code could exoneration Rybka.

Hi,

http://www.top-5000.nl/evidence.htm#C6

I think that:

Code: Select all

Compile the following code with any of the MS compilers.

static int T=1;
void main() { if (T >= 0.) T=5; }

It will produce the same code as 0.0

is quite interesting.

Also the PST's

http://www.top-5000.nl/pst.htm

For starters

[BB+]

I respond to the ChessBase part 1 of Dr. Riis. Among various extraneous parts, he brings up two things of import. Firstly, he claims that Rybka implements concepts and algorithms learned from Fruit” appears to be the most correct and accurate formulation -- this opinion was essentially considered and rejected by the ICGA process. The conclusion was that Rajlich did more than just "implement concepts and algorithms learned from Fruit". Secondly, he quotes Rajlich as saying: When two modern top chess programs play against each other maybe 95% of the programs are algorithmically the same. What is classing is the other 5%. I find these numbers almost laughable, unless the word "algorithmically" is couched quite carefully. Dr. Riis seems to assume that the evaluation function is part of this "algorithmic" quantity, whereas the ICGA process found that there was wide scope for creativity therein. Quoting Rajlich himself: As far as I know, Rybka has a very original search and evaluation framework. The juxtaposition of this with the 95% number seems obscure.

Finally, after giving a selection of Rybka highlights over the years (his article is in part to defend the reputation of Rajlich, and thus does not deal only with the ICGA process), Dr. Riis plays the role of lawyer, essentially asking how the ICGA can interpret its own rules, how exactly this should be done, etc. One might as well ask how any quasi-judicial process might occur if it relies on "subjective" or "interpretative" aspects. The typical legal adage here usually relies on the fact that the law need not over-specify every offense, as then the capacity of man for mischief would necessitate legal manuals that would inundate the world. As an example, I refer him to the information page regarding "plagiarism" at Queen Mary (his home institution) -- do they define this any better than the ICGA? [There is presumably an accompanying body of regulations and procedures, but I doubt they shed any light on the initial definition]. I particularly note #10: If in doubt, ask! If in doubt always speak to a member of staff before submitting your work – it’s too late after the work has been submitted. Oh how such a norm of behaviour could have prevented this mess!

Here is a more itemised list of quibbles, some of picayune import:

The latest public edition of Rybka (Rybka 4.1) is more than 400 Elo points stronger than the top competitors that existed in late 2005 on comparable hardware.

I'm not quite sure what the comparison is (or its basis), as the "top competitors" are not mentioned. The "self-Rybka" comparison is closer to a 250-300 Elo gain on a 1cpu 64-bit basis. I can't find enough records of whether R1 was already 100-150 Elo ahead of everyone (and 32-bit might be a better comparison standard). As noted elsewhere, of the 250-300 Elo gained by Rybka since R1, approximately 100 of this was between 1.0 and 1.2f (May 2006), then 50 to R2.3.2a (SMP was also added), then 100 in R3 (when LK re-wrote the eval), and the rest with R4.

Rybka went on to become a commercial engine in 2006.

Rybka was already a commercial engine in 2005. See CCC posts from Dec 2005 (e.g. this link).

[Rajlich] refused to be drawn into a protracted dispute with his accusers or mount a comprehensive defense.

Indeed. Calling the subsequent result a "gross miscarriage of justice" when the accused fails to defend himself is, well, a gross misuse of the English language.

Judging from the Wiki which they used during their investigation, approximately seven of these 34 experts actively participated in the discussions.

I am not aware that Dr. Riis had access to said the internal part of said wiki, nor is quantity a measure of much (posting is more of an indicator of "free time" than anything else ). In any event, it was the warrant of the Board to make any decision in the end.

None of the actual Rybka versions that participated in the four WCCC tournaments were investigated, although a very close version (Rybka 2.3.2a) was examined following a laborious process of reverse-engineering.

The former was the choice of Rajlich (possibly by necessity, in that they no longer exist). Dr. Riis slightly errs when he says "four" WCCC tournaments, as 2006 was also considered.

It is argued within the paper that this rule is vague on key points and become obsolete for several reasons, and that the overarching reason it has passed into obsolescence is that there has been a paradigm shift in computer chess programming in the past decade which the rule does not take into account.

If the rule is vague, it is at least partially the responsibility of the entrant to ask for guidance. Furthermore, for an outsider(?) such as Dr. Riis to declare an ICGA rule to be "obsolete" is odd to say the least. The ICGA has no compulsion to follow "paradigm shifts", in its interpretation of rules, or otherwise.

Based on the evidence I will present, a person can form a very credible alternative conclusion: that the implementation of similar evaluation concepts and algorithms in a computer chess program will inherently lead to code similarities even if no code is copied. Thus, the statement “Rybka implements concepts and algorithms learned from Fruit” appears to be the most correct and accurate formulation—a semantically subtle difference that nonetheless completely overturns the ICGA’s conclusion. It is important to understand that the implementation of ideas and algorithms learned from other programs is universal practice in chess programming as well as many other types of programming. The issue in contention in this case is whether source code was copied from one program to the next.

OK, we finally get to something of merit. Dr. Riis is quite right to make this distinction. However, the ICGA process also realised the crucial nature of it, and the conclusion was that Rybka did more than just "implement concepts and algorithms learned from Fruit".

Here’s the main point: to convict and sentence a man due to his presumed ethical failings and then attempt to ruin him on a world stage you need a very high standard of evidence. The ICGA claims its evidence is of a high standard. We shall explore the veracity of that claim.

To "convict and sentence" a man who decides not to defend himself is hardly a great leap. Furthermore, the implication that the ICGA "attempt[ed] to ruin [Rajlich] on a world stage" seems unfounded -- the media will do what they want (such as ignore it, if convenient). Finally, the phrase "its evidence" is odd (meaning the ICGA's), as the evidence doesn't exactly belong to anyone. If anything, the evidence is support of Letouzey's complaint.

In early 2011 sixteen chess programmers, many of whose programs were in direct competition with Rybka, signed a letter wherein they asserted that Rajlich copied programming code from another engine [...]

Dr. Riis makes it sound like only non-competitors should have signed such a letter?!

Rajlich "When two modern top chess programs play against each other maybe 95% of the programs are algorithmically the same. What is classing is the other 5%."

This is simply not true. Maybe if you render "algorithmically" to mean the generic alpha-beta search framework, ignore evaluation, etc., you might justify this number, but even then the amount of pruning concepts that have been introduced are quite large. I seriously doubt that (say) Komodo and R3 are "95% the same". Given that Dr. Riis then uses this quotation to conclude (bluntly) that Rule #2 is obsolete, I find his argument essentially baseless.

As an additional rebuttal to this, I will simply quote Rajlich, who writes: "As far as I know, Rybka has a very original search and evaluation framework." To claim that that all engines are "95% algorithmically the same" while Rybka is "very original" is a bit undecipherable (and, as with many things, Rajlich seems undesirous of any clarification).

I have yet to decide whether a PDF is a better way to address Mr. Schröder's website. [In reality, I am tempted to simply let it speak for itself, but as Dr. Riis quotes it as "extensive", a response is perhaps due].

Finally, Dr. Riis attempts to show that ICGA's Rule #2 is "absurd", though he seems to be creating as many obstacles as possible.

To make Rule 2’s absurdity as clear as possible, let me pose some straightforward questions:

* Given the great algorithmic overlap between modern chess programs, what is the definitional distinction between “original” and “non-original” work?

The ICGA found that the evaluation overlap between Rybka 1.0 Beta and Fruit 2.1 more than sufficed in the case at hand. The evaluation function is not an "algorithm" in the strict sense of input->output, but rather there is a large capacity for choice by the implementor. This is fairly well adumbrated in (say) the EVAL_COMP write-up. Dr. Riis also seems to accept Rajlich's claim of "95% algorithmically the same", while I find it nearly laughable (unless codified more).

I might add that I think this is the main basis for his erroneous conclusion -- he accepts that most computer chess programs are "essentially the same", while the ICGA programmers, Panel, and Board, all appear to disagree.

A modern computer chess program can consist of tens of thousands of lines of code. Which of these lines can a programmer feel certain are in public domain and therefore exempt from Rule 2, and which are not?

The programmer should ask the ICGA for clarification. The ICGA might canvass its members, make a decision based upon precedent, etc.

What exactly is meant by “game playing code” and on what basis does the ICGA make its distinction?

What exactly is the definition of a “close derivative”? Is this phrase entirely a “we know it when we see it” construct, and if not, then what sensible, consistent, well-defined and articulated principles is such a determination based upon?

Dr. Riis is essentially trying to argue that the ICGA is not capable of interpreting its own rules. He might as well argue that all law itself is de facto moot, as it depends on interpretation. Again, one can note that a competitor typically has the responsibility to ask for clarification when there is an issue. In his piece in the ICGA Journal, Levy essentially noted this, recalling that Rajlich was present in Turin with the LION++ disqualification, and certainly was not unaware of the ICGA interpretation. [The LION++ team could similarly argue that the ICGA rules were "vague" or "unclear", and what do they exactly mean].

Does Rule 2 require all competitors to maintain a copy of any source code they used in competition for an indefinite number of years?

Can Rule 2 be invoked after tournaments are completed without any time limitation whatsoever? (In law, there is a defense called laches, which certainly applies to the Rajlich/Rybka case.)

The start of the "clock" for a statute of limitations usually only starts when the offense comes to light. For instance, Letouzey will have 3 (or 4 years) from Jan 2011 [the date of discovery] to pursue Rajlich in court regarding copyright infringement. The alternative, where entrants tried to hide their misdeeds until enough time passed, would be "absurd" to say the least.

Finally, what safeguards exist to prevent ex post facto interpretations of rules which are not fully consistent with what competitors understood at the time the tournament took place?

I think most of the other competitors (but wait -- I thought their opinions were to be excluded, due to "bias"?!) in, say, 2007, understood the rules. If Rajlich had a different understanding, he should, again, have asked for a clarification.

It seems to me rather imperative that a tournament billing itself a “world championship” have crystal-clear rules.

No matter "clear" they are, people will find exceptions. See: Elista, toilet.

These rules should evolve in response to circumstances, contain well-defined procedures and credible enforcement mechanisms, and be designed to protect the integrity of the competition and the title.

I think the disqualfication of Rajlich, at least for 2006 and 2007, did exactly this. Admittedly, I think it would have been (much) better to simply change the entry name to "Fruit/Rybka" with similar "Rajlich/Letouzey" attribution, but this did not seem to be a possible endpoint, given the circumstances.

[BB+]

approximately 100 of this was between 1.0 and 1.2f (May 2006)

I think I meant between 1.0 and 1.1 of March 16 2006, though that uses CEGT numbers I think, and CCRL is a bit different