value of LMR and null-move

[Sentinel]

OK, test starts off weird. I first started a test between two versions of stockfish 1.8. I am calling them 1.8 and 1.8a. In 1.8a, at every "if (cond)" that causes the LMR reduction to be done, I just added a simple "0 &&" at the front which causes that code to not execute and actually be optimized out (I looked at the .s file to verify).

Maybe you have missed a spot:
http://www.open-chess.org/viewtopic.php?p=3647#p3647

[hyatt]

For the record, you have a _TON_ of warnings that should be cleaned up. I much prefer to eliminate all warnings by fixing the source rather than telling the compiler to just not display a set of warnings...

So do we.

What compiler are you using? I don't get a single compiler warning (when compiling with -Wall) with GCC 4.2 in Mac OS X 10.6.

I use Intel's compiler almost exclusively. Here's a couple for reference:


add<EvaluationFunction<KRKP> >("KRKP");
^
material.cpp(366): remark #383: value copied to temporary, reference to temporary used

material.cpp(421): remark #383: value copied to temporary, reference to temporary used
get<F>().insert(pair<Key, F*>(buildKey(keyCode), new T(WHITE)));
^
detected during instantiation of "void EndgameFunctions::add<T>(const std::string &) [with
T=EvaluationFunction<KNNK>]"

return rank_distance(square_rank(s1), square_rank(s2));
^
square.h(159): remark #981: operands are evaluated in unspecified order

[hyatt]

OK, test starts off weird. I first started a test between two versions of stockfish 1.8. I am calling them 1.8 and 1.8a. In 1.8a, at every "if (cond)" that causes the LMR reduction to be done, I just added a simple "0 &&" at the front which causes that code to not execute and actually be optimized out (I looked at the .s file to verify).

Maybe you have missed a spot:
http://www.open-chess.org/viewtopic.php?p=3647#p3647

Nope. This code is not that complicated. All in search.c, and all I am interested in turning off is specifically LMR, not other things like null-move or futility pruning...

[hyatt]

OK, clusters are idle so I can run over 500 games at a time. Results so far:

Code: Select all

Rank Name                  Elo    +    - games score oppo. draws
   1 Stockfish 1.8 64bit   2848   15   15  1763   83%  2552   19% 
   2 Stockfish 1.8a 64bit  2810   14   14  1754   78%  2561   22% 
   3 Glaurung 2.2          2556   13   13  1638   43%  2619   21% 
   4 Toga2                 2550   14   14  1595   42%  2618   19% 
   5 Fruit 2.1             2448   14   14  1625   28%  2639   20% 
   6 Glaurung 1.1 SMP      2387   15   15  1649   21%  2653   16% 

1.8a stockfish has LMR completely removed. Note that 1.8 and 1.8a will have 6,000 games between them by the time this is finished, along with 6000 games against each of the other opponents (a real RR that will waste some time since the others are also playing each other in addition to the two stockfish versions. So far, only +38 against a group, vs +50 head-to-head.

[hyatt]

The above is looking pretty close to "the truth".

After each version of stockfish has played 12,000 games out of 30,000 total, stockfish + LMR is +43 elo ahead of stockfish - LMR. May be a result of the TT "pseudo-singular extension" or something, not sure yet.

[hyatt]

Probably last update tonight:

Code: Select all

Rank Name                  Elo    +    - games score oppo. draws
   1 Stockfish 1.8 64bit   2856    5    5 20549   84%  2549   19% 
   2 Stockfish 1.8a 64bit  2812    4    4 20567   78%  2558   20% 
   3 Glaurung 2.2          2550    4    4 20483   43%  2610   20% 
   4 Toga2                 2543    4    4 20411   43%  2612   20% 
   5 Fruit 2.1             2447    4    4 20453   29%  2631   19% 
   6 Glaurung 1.1 SMP      2391    4    4 20495   23%  2643   15% 

Still looks like +44 Elo from LMR.

With the difference in gain between stockfish and Crafty, clearly YMMV. But don't expect those 200-300-400 jumps... just not there.

[BB+]

I will reiterate what I said in the other thread, that futility parameters can have some overlap with LMR. Here is a relevant bit of Stockfish code (though there is both move-count and value-based pruning)

Code: Select all

// Step 12. Futility pruning
// Futility margin for quiescence search
const Value FutilityMarginQS = Value(0x80);

// Futility lookup tables (initialized at startup) and their getter functions
int32_t FutilityMarginsMatrix[16][64]; // [depth][moveNumber]
inline Value futility_margin(Depth d, int mn)
 {return Value(d < 7*OnePly ? FutilityMarginsMatrix[Max(d,0)][Min(mn,63)] : 2 * VALUE_INFINITE);}
// FutilityMarginsMatrix[d][mc] = 112 * int(log(double(d*d) / 2) / log(2.0) + 1.001) - 8*mc + 45;

int FutilityMoveCountArray[32]; // [depth]
inline int futility_move_count(Depth d) {return d < 16*OnePly ? FutilityMoveCountArray[d] : 512;}
// FutilityMoveCountArray[d] = 3 + (1 << (3 * d / 8));

Just as an example, at a Depth of 10 half-ply and move count of 5, then FutilityMarginsMatrix is 112*6 - 40 + 45 = 677 hexa-pawns (I think). If you have a lessened concept of futility, you will have more moves to which to apply LMR, thus (likely) making it seem to be worth a greater amount. The FutilityMoveCountArray is 11 at this point, and all moves beyond that move-count are skipped unless they have some "threat" it seems. This again means that fewer moves are available to which to apply LMR.

[mcostalba]

Probably last update tonight:

Code: Select all

Rank Name                  Elo    +    - games score oppo. draws
   1 Stockfish 1.8 64bit   2856    5    5 20549   84%  2549   19% 
   2 Stockfish 1.8a 64bit  2812    4    4 20567   78%  2558   20% 
   3 Glaurung 2.2          2550    4    4 20483   43%  2610   20% 
   4 Toga2                 2543    4    4 20411   43%  2612   20% 
   5 Fruit 2.1             2447    4    4 20453   29%  2631   19% 
   6 Glaurung 1.1 SMP      2391    4    4 20495   23%  2643   15% 

Still looks like +44 Elo from LMR.

With the difference in gain between stockfish and Crafty, clearly YMMV. But don't expect those 200-300-400 jumps... just not there.

Thanks ! Very interesting results.

You may want to try with longer TC, LMR really kicks in at deep searches.

It would be interesting how much it gains going from 10"+0.1 to some thing like 30"+0.1, you just need to test 1.8 vs 1.8a because we have already seen that more or less is comparable to gauntlet result.

[ThinkingALot]

Removing LMR increases the brenching factor. Hence TM & IID must also be tuned for the testing to be accurate .

[Rebel]

You may want to try with longer TC, LMR really kicks in at deep searches.

Agree.

The deeper the better recursive-null-move rocks also

BB also made a valid point regarding futility pruning, on a very short TC it favors the no-LMR version. And that should be avoided.

So a reasonable time-control is needed.

I chose for Blitz 5m. Still debatable of course.

My first try to run 100 blitz 5m games failed due to a wrong parameter (early draws) in the Fritz GUI and I had to abort the match. As a result many games were terminated after 15-20 moves with a draw result. However not counting the draws (35) SF 1.8 had 17 wins and SF 1.8 (no LMR) only 3. Telling IMO.

I am running it again now (draw=never) and with Blitz 5m SF 1.8 typically hits 20-22 plies in the middle-game, SF 1.8 (no LMR) typically 15-16. I am using 4 cores for each version, no PB. This in order to get a reasonable depth.

Bob of course will start to hammer on the issue of 100 games but if the results turns to be 60/30/10 it's clear we are not talking about a 50 elo improvement

BTW, I made the changes as you suggested. Reading Bob I get the impression he made different ones?

Ed