Page 2 of 7
Re: value of LMR and null-move
Posted: Mon Jul 12, 2010 10:12 pm
by Sentinel
hyatt wrote:OK, test starts off weird. I first started a test between two versions of stockfish 1.8. I am calling them 1.8 and 1.8a. In 1.8a, at every "if (cond)" that causes the LMR reduction to be done, I just added a simple "0 &&" at the front which causes that code to not execute and actually be optimized out (I looked at the .s file to verify).
Maybe you have missed a spot:
http://www.open-chess.org/viewtopic.php?p=3647#p3647
Re: value of LMR and null-move
Posted: Mon Jul 12, 2010 10:19 pm
by hyatt
Tord wrote:hyatt wrote:For the record, you have a _TON_ of warnings that should be cleaned up. I much prefer to eliminate all warnings by fixing the source rather than telling the compiler to just not display a set of warnings...
So do we.
What compiler are you using? I don't get a single compiler warning (when compiling with -Wall) with GCC 4.2 in Mac OS X 10.6.
I use Intel's compiler almost exclusively. Here's a couple for reference:
add<EvaluationFunction<KRKP> >("KRKP");
^
material.cpp(366): remark #383: value copied to temporary, reference to temporary used
material.cpp(421): remark #383: value copied to temporary, reference to temporary used
get<F>().insert(pair<Key, F*>(buildKey(keyCode), new T(WHITE)));
^
detected during instantiation of "void EndgameFunctions::add<T>(const std::string &) [with
T=EvaluationFunction<KNNK>]"
return rank_distance(square_rank(s1), square_rank(s2));
^
square.h(159): remark #981: operands are evaluated in unspecified order
Re: value of LMR and null-move
Posted: Mon Jul 12, 2010 10:20 pm
by hyatt
Sentinel wrote:hyatt wrote:OK, test starts off weird. I first started a test between two versions of stockfish 1.8. I am calling them 1.8 and 1.8a. In 1.8a, at every "if (cond)" that causes the LMR reduction to be done, I just added a simple "0 &&" at the front which causes that code to not execute and actually be optimized out (I looked at the .s file to verify).
Maybe you have missed a spot:
http://www.open-chess.org/viewtopic.php?p=3647#p3647
Nope. This code is not that complicated. All in search.c, and all I am interested in turning off is specifically LMR, not other things like null-move or futility pruning...
Re: value of LMR and null-move
Posted: Mon Jul 12, 2010 10:28 pm
by hyatt
OK, clusters are idle so I can run over 500 games at a time. Results so far:
Code: Select all
Rank Name Elo + - games score oppo. draws
1 Stockfish 1.8 64bit 2848 15 15 1763 83% 2552 19%
2 Stockfish 1.8a 64bit 2810 14 14 1754 78% 2561 22%
3 Glaurung 2.2 2556 13 13 1638 43% 2619 21%
4 Toga2 2550 14 14 1595 42% 2618 19%
5 Fruit 2.1 2448 14 14 1625 28% 2639 20%
6 Glaurung 1.1 SMP 2387 15 15 1649 21% 2653 16%
1.8a stockfish has LMR completely removed. Note that 1.8 and 1.8a will have 6,000 games between them by the time this is finished, along with 6000 games against each of the other opponents (a real RR that will waste some time since the others are also playing each other in addition to the two stockfish versions. So far, only +38 against a group, vs +50 head-to-head.
Re: value of LMR and null-move
Posted: Tue Jul 13, 2010 1:26 am
by hyatt
The above is looking pretty close to "the truth".
After each version of stockfish has played 12,000 games out of 30,000 total, stockfish + LMR is +43 elo ahead of stockfish - LMR. May be a result of the TT "pseudo-singular extension" or something, not sure yet.
Re: value of LMR and null-move
Posted: Tue Jul 13, 2010 3:07 am
by hyatt
Probably last update tonight:
Code: Select all
Rank Name Elo + - games score oppo. draws
1 Stockfish 1.8 64bit 2856 5 5 20549 84% 2549 19%
2 Stockfish 1.8a 64bit 2812 4 4 20567 78% 2558 20%
3 Glaurung 2.2 2550 4 4 20483 43% 2610 20%
4 Toga2 2543 4 4 20411 43% 2612 20%
5 Fruit 2.1 2447 4 4 20453 29% 2631 19%
6 Glaurung 1.1 SMP 2391 4 4 20495 23% 2643 15%
Still looks like +44 Elo from LMR.
With the difference in gain between stockfish and Crafty, clearly YMMV. But don't expect those 200-300-400 jumps... just not there.
Re: value of LMR and null-move
Posted: Tue Jul 13, 2010 3:31 am
by BB+
I will reiterate what I said in the other thread, that futility parameters can have some overlap with LMR. Here is a relevant bit of Stockfish code (though there is both move-count and value-based pruning)
Code: Select all
// Step 12. Futility pruning
// Futility margin for quiescence search
const Value FutilityMarginQS = Value(0x80);
// Futility lookup tables (initialized at startup) and their getter functions
int32_t FutilityMarginsMatrix[16][64]; // [depth][moveNumber]
inline Value futility_margin(Depth d, int mn)
{return Value(d < 7*OnePly ? FutilityMarginsMatrix[Max(d,0)][Min(mn,63)] : 2 * VALUE_INFINITE);}
// FutilityMarginsMatrix[d][mc] = 112 * int(log(double(d*d) / 2) / log(2.0) + 1.001) - 8*mc + 45;
int FutilityMoveCountArray[32]; // [depth]
inline int futility_move_count(Depth d) {return d < 16*OnePly ? FutilityMoveCountArray[d] : 512;}
// FutilityMoveCountArray[d] = 3 + (1 << (3 * d / 8));
Just as an example, at a Depth of 10 half-ply and move count of 5, then FutilityMarginsMatrix is 112*6 - 40 + 45 = 677 hexa-pawns (I think). If you have a lessened concept of futility, you will have more moves to which to apply LMR, thus (likely) making it seem to be worth a greater amount. The FutilityMoveCountArray is 11 at this point, and all moves beyond that move-count are skipped unless they have some "threat" it seems. This again means that fewer moves are available to which to apply LMR.
Re: value of LMR and null-move
Posted: Tue Jul 13, 2010 6:52 am
by mcostalba
hyatt wrote:Probably last update tonight:
Code: Select all
Rank Name Elo + - games score oppo. draws
1 Stockfish 1.8 64bit 2856 5 5 20549 84% 2549 19%
2 Stockfish 1.8a 64bit 2812 4 4 20567 78% 2558 20%
3 Glaurung 2.2 2550 4 4 20483 43% 2610 20%
4 Toga2 2543 4 4 20411 43% 2612 20%
5 Fruit 2.1 2447 4 4 20453 29% 2631 19%
6 Glaurung 1.1 SMP 2391 4 4 20495 23% 2643 15%
Still looks like +44 Elo from LMR.
With the difference in gain between stockfish and Crafty, clearly YMMV. But don't expect those 200-300-400 jumps... just not there.
Thanks ! Very interesting results.
You may want to try with longer TC, LMR really kicks in at deep searches.
It would be interesting how much it gains going from 10"+0.1 to some thing like 30"+0.1, you just need to test 1.8 vs 1.8a because we have already seen that more or less is comparable to gauntlet result.
Re: value of LMR and null-move
Posted: Tue Jul 13, 2010 8:13 am
by ThinkingALot
Removing LMR increases the brenching factor. Hence TM & IID must also be tuned for the testing to be accurate .
Re: value of LMR and null-move
Posted: Tue Jul 13, 2010 10:13 am
by Rebel
mcostalba wrote:You may want to try with longer TC, LMR really kicks in at deep searches.
Agree.
The deeper the better recursive-null-move rocks also
BB also made a valid point regarding futility pruning, on a very short TC it favors the no-LMR version. And that should be avoided.
So a reasonable time-control is needed.
I chose for Blitz 5m. Still debatable of course.
My first try to run 100 blitz 5m games failed due to a wrong parameter (early draws) in the Fritz GUI and I had to abort the match. As a result many games were terminated after 15-20 moves with a draw result. However not counting the draws (35) SF 1.8 had 17 wins and SF 1.8 (no LMR) only 3. Telling IMO.
I am running it again now (draw=never) and with Blitz 5m SF 1.8 typically hits 20-22 plies in the middle-game, SF 1.8 (no LMR) typically 15-16. I am using 4 cores for each version, no PB. This in order to get a reasonable depth.
Bob of course will start to hammer on the issue of 100 games but if the results turns to be 60/30/10 it's clear we are not talking about a 50 elo improvement
BTW, I made the changes as you suggested. Reading Bob I get the impression he made different ones?
Ed