Page 13 of 37
Re: Designing an analysis friendly Stockfish?
Posted: Sat Feb 05, 2011 2:18 pm
by Jeremy Bernstein
New builds, based on PA_J + GTB, attached. github updated.
Testing so far seems good. I chucked the tournament I was running in favor of a very fast tournament when I discovered an issue with the opening books (I was using Stockfish's own book for all 3 engines. Apparently Stockfish only knows how to play the Najdorf.). I guess that this is why I don't generally run tests -- too dumb. Stockfish_64 and Stockfish_PA_I_GTB_64 were neck in neck, with PA_J_64 about 6 games behind. With a round-robin tourney of 75 games, that could mean anything.
In general, very drawish play between these 3 engines, which is to be expected. I don't think the modifications have much, if any effect on strength, time management, etc. whatsoever, at least not at these time controls.
BTW, I increased the number of cores on my VM to 2, so we're seeing multicore runs here.
New tourney. 5 seconds + 100ms, 300 games total which makes for a better sample size. I suspect that the PA_J is a tiny bit weaker than PA_I based on what I was seeing in the 1/1 games, but let's see how it fares at these time controls.
Re: Designing an analysis friendly Stockfish?
Posted: Sat Feb 05, 2011 4:36 pm
by Jeremy Bernstein
In case anyone else wants to run some tests, here are the 3 engines I'm working with, all compiled with exactly the same settings.
Re: Designing an analysis friendly Stockfish?
Posted: Sat Feb 05, 2011 5:29 pm
by Jeremy Bernstein
Here's a game:
Note how the PA_J+GTB finds a forced mate around move 80 or so and then loses its way by move 99. What happened?
Re: Designing an analysis friendly Stockfish?
Posted: Sat Feb 05, 2011 7:51 pm
by gaard
fruity wrote:Uly wrote:Thanks fruity and gaard, here's a quick comparison of Stockfish PA_G and PA_H at 1CPU in the opening position. It seemed mostly luck, but PA_H found 1...e6 much earlier and reached depth 24 much faster, PA_G caught up at depth 25 and finished one minute faster in the end, though.
Engine: Stockfish 2.0.1 PA_G (128 MB)
by Tord Romstad, Marco Costalba, Joona Kii
15/14 0:00 +0.16-- 1.Nf3 Nf6 2.e3 d5 3.d4 e6 4.Bd3 c5 5.O-O c4 6.Be2 Nc6 7.Nc3 Bd6 (509.807) 679
15/09 0:01 +0.32++ 1.e4 Nf6 2.e5 Nd5 3.Nf3 Nc6 4.d4 e6 5.Bd3 (928.123) 715
15/09 0:01 +0.40++ 1.e4 Nf6 2.e5 Nd5 3.Nf3 Nc6 4.d4 e6 5.Bd3 (1.152.385) 730
15/09 0:01 +0.56++ 1.e4 Nf6 2.e5 Nd5 3.Nf3 Nc6 4.d4 e6 5.Bd3 (1.424.940) 741
15/16 0:03 +0.44 1.e4 e5 2.Nf3 Bd6 3.Nc3 Nf6 4.Be2 Nc6 5.O-O O-O 6.d3 b6 7.d4 exd4 8.Nxd4 Bb7 (2.247.548) 749
16/18 0:04 +0.24-- 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Bd3 Nf6 6.O-O Be7 7.Nc3 O-O 8.Re1 Nc6 9.Ng5 d5 (3.151.485) 746
16/26 0:05 +0.32 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Qe2 Qe7 6.Nc3 Nxc3 7.dxc3 Qxe2+ 8.Bxe2 Nc6 9.Be3 Be7 10.O-O-O O-O 11.Kb1 Be6 12.Ng5 a6 13.Nxe6 fxe6 (4.052.349) 747
17/26 0:05 +0.32 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Qe2 Qe7 6.Nc3 Nxc3 7.dxc3 Qxe2+ 8.Bxe2 Nc6 9.Be3 Be7 10.O-O-O O-O 11.Kb1 Be6 12.Ng5 a6 13.Nxe6 fxe6 (4.252.062) 745
18/26 0:06 +0.32 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Qe2 Qe7 6.Nc3 Nxc3 7.dxc3 Qxe2+ 8.Bxe2 Nc6 9.Be3 Be7 10.O-O-O O-O 11.Kb1 Be6 12.Ng5 a6 13.Nxe6 fxe6 (4.601.681) 747
19/25 0:07 +0.40++ 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Qe2 Qe7 6.Nc3 Nxc3 7.dxc3 Qxe2+ 8.Bxe2 Nc6 9.Be3 Be7 10.O-O-O O-O 11.Kb1 Be6 12.Ng5 a6 13.h4 (5.619.232) 750
19/19 0:09 +0.48++ 1.e4 e5 2.Nf3 Nf6 3.d4 exd4 4.e5 Qe7 5.Be2 Ng4 6.Qxd4 d6 7.exd6 cxd6 8.Nc3 Nc6 9.Qa4 Qe6 10.O-O (6.811.131) 754
19/26 0:11 +0.32 1.e4 e5 2.Nf3 Nf6 3.d4 exd4 4.e5 Qe7 5.Be2 Ng4 6.Qxd4 h5 7.Nc3 Nc6 8.Qf4 Ncxe5 9.O-O c6 10.Re1 d6 11.h3 Nxf3+ 12.Qxf3 Ne5 13.Qf4 Be6 (8.935.601) 753
20/26 0:13 +0.32 1.e4 e5 2.Nf3 Nf6 3.d4 exd4 4.e5 Qe7 5.Be2 Ng4 6.Qxd4 h5 7.Nc3 Nc6 8.Qf4 Ncxe5 9.O-O c6 10.Re1 d6 11.h3 Nxf3+ 12.Qxf3 Ne5 13.Qf4 Be6 (10.320.940) 752
21/26 0:17 +0.32 1.e4 e5 2.Nf3 Nf6 3.d4 exd4 4.e5 Qe7 5.Be2 Ng4 6.Qxd4 h5 7.Nc3 Nc6 8.Qf4 Ncxe5 9.O-O c6 10.Re1 d6 11.h3 Nxf3+ 12.Qxf3 Ne5 13.Qf4 Be6 (13.161.101) 750
22/17 0:28 +0.40++ 1.e4 e5 2.Nf3 Nf6 3.d4 exd4 4.e5 Qe7 5.Be2 Ng4 6.Qxd4 h5 7.Nc3 Nc6 8.Qd1 Nxf2 9.Kxf2 (20.919.656) 744
22/26 0:35 +0.24-- 1.e4 e5 2.Nf3 Nf6 3.d4 exd4 4.e5 Ne4 5.Qxd4 f5 6.exf6 Nxf6 7.Nc3 Nc6 8.Qe3+ Be7 9.Bd3 O-O 10.O-O d5 11.Qg5 d4 12.Bc4+ Kh8 13.Rd1 Nd7 (26.047.732) 742
22/25 0:48 +0.44 1.e4 e5 2.Nf3 Nc6 3.Bc4 Be7 4.Nc3 Nf6 5.O-O O-O 6.d4 exd4 7.Nxd4 Bb4 8.Bg5 Bxc3 9.bxc3 Qe7 10.Bd3 d6 11.Rb1 h6 12.Bh4 Re8 13.f3 (35.672.600) 742
23/26 2:12 +0.36 1.e4 e5 2.Nf3 d6 3.d4 exd4 4.Nxd4 Nf6 5.Nc3 Be7 6.Be2 O-O 7.Bf4 Bd7 8.O-O Nc6 9.Qd2 Ne5 10.Rad1 Re8 11.Bg3 a6 12.f4 Nc6 13.Bf2 Nxd4 (96.852.976) 733
24/16 3:09 +0.20-- 1.e4 e6 2.Nf3 d5 3.e5 Ne7 4.Bb5+ Bd7 5.Bxd7+ Nxd7 6.O-O Nc6 7.d4 Be7 8.c3 O-O (138.809.221) 731
24/28 3:51 +0.24 1.e4 e6 2.Nf3 d5 3.exd5 exd5 4.d4 Nf6 5.Bb5+ c6 6.Bd3 Be7 7.O-O O-O 8.Nc3 Bd6 9.a3 Re8 10.Bg5 Bg4 11.h3 Bh5 12.g4 Bg6 13.Bxg6 hxg6 14.Re1 (168.932.447) 730
25/31 4:50 +0.20 1.e4 e6 2.Nf3 d5 3.exd5 exd5 4.d4 Nf6 5.Bb5+ c6 6.Bd3 Be7 7.O-O O-O 8.Nc3 Bd6 9.a3 Re8 10.Re1 Bg4 11.Rxe8+ Qxe8 12.Bg5 Nbd7 13.h3 Bh5 14.g4 (212.238.135) 730
best move: e2-e4 time: 5:06.000 min
Engine: Stockfish 2.0.1 PA_H (128 MB)
by Tord Romstad, Marco Costalba, Joona Kii
16/16 0:00 +0.28-- 1.Nf3 Nf6 2.e3 e6 3.d4 d5 4.Bd3 Be7 5.O-O Nc6 6.c4 O-O 7.cxd5 exd5 8.Nc3 Be6 (497.513) 740
16/13 0:01 +0.44++ 1.e4 Nc6 2.d4 Nf6 3.e5 Nd5 4.c4 Nb6 5.Nf3 d6 6.exd6 cxd6 7.Nc3 (1.211.455) 760
16/13 0:02 +0.52++ 1.e4 Nc6 2.d4 Nf6 3.e5 Nd5 4.c4 Nb6 5.Nf3 d6 6.exd6 cxd6 7.Nc3 (1.549.036) 768
16/18 0:03 +0.44 1.e4 e6 2.Nf3 d5 3.e5 Nd7 4.Nc3 f6 5.d4 Bb4 6.Bd3 c5 7.O-O cxd4 8.Nxd4 Bxc3 9.bxc3 Nxe5 (2.323.010) 770
17/19 0:04 +0.52 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Nc3 Nxc3 6.dxc3 Be7 7.Bc4 c6 8.O-O d5 9.Bb3 O-O 10.Bf4 (3.581.756) 764
18/20 0:05 +0.40-- 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Nc3 Nxc3 6.dxc3 Be7 7.Bc4 c6 8.O-O d5 9.Bb3 O-O 10.Bf4 Be6 (4.032.770) 763
18/18 0:05 +0.28-- 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Nc3 Nxc3 6.dxc3 Be7 7.Bc4 Nc6 8.O-O O-O 9.Qd5 Be6 (4.541.806) 764
18/24 0:08 +0.32 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Nc3 Nxc3 6.dxc3 Be7 7.Bc4 Nc6 8.O-O O-O 9.Re1 Bf5 10.Bf4 Be6 11.Bd5 Bh4 12.Qd3 Qf6 (6.262.735) 761
19/24 0:08 +0.32 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Nc3 Nxc3 6.dxc3 Be7 7.Bc4 Nc6 8.O-O O-O 9.Re1 Bf5 10.Bf4 Be6 11.Bd5 Bh4 12.Qd3 Qf6 (6.726.012) 764
20/24 0:10 +0.32 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Nc3 Nxc3 6.dxc3 Be7 7.Bc4 Nc6 8.O-O O-O 9.Re1 Bf5 10.Bf4 Be6 11.Bd5 Bh4 12.Qd3 Qf6 (8.026.327) 764
21/24 0:14 +0.32 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Nc3 Nxc3 6.dxc3 Be7 7.Bc4 Nc6 8.O-O O-O 9.Re1 Bf5 10.Bf4 Be6 11.Bd5 Bh4 12.Qd3 Qf6 (10.885.253) 761
22/24 0:21 +0.24-- 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Nc3 Nxc3 6.dxc3 Be7 7.Bc4 Nc6 8.O-O O-O 9.Re1 Bf5 10.Bf4 Bf6 11.Nd4 Bxd4 12.cxd4 Nb4 (16.389.829) 755
22/19 0:31 +0.40++ 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.d4 d5 6.Bd3 Be7 7.O-O O-O 8.Re1 Nc6 9.c4 Bb4 10.cxd5 (23.765.595) 751
22/24 1:03 +0.20 1.e4 e6 2.Nf3 d5 3.exd5 exd5 4.d4 Nf6 5.Bd3 Nc6 6.O-O Be7 7.Nc3 Bg4 8.Be3 O-O 9.h3 Bh5 10.Re1 Re8 11.a3 Bd6 12.Bf5 Bg6 (47.007.795) 744
23/25 1:17 +0.32++ 1.e4 e6 2.Nf3 d5 3.exd5 exd5 4.d4 Nf6 5.Bd3 Nc6 6.O-O Be7 7.c3 O-O 8.Bf4 Bd6 9.Bxd6 Qxd6 10.Nbd2 Re8 11.Qc2 Qf4 12.Rfe1 Bg4 13.Qb3 (57.654.660) 741
23/22 1:28 +0.20 1.e4 e6 2.Nf3 d5 3.exd5 exd5 4.d4 Nf6 5.Bd3 Bd6 6.Nc3 Nc6 7.O-O O-O 8.Bg5 Nb4 9.Nb5 Bg4 10.Re1 h6 11.Nxd6 Qxd6 (65.272.603) 739
24/10 1:45 +0.28++ 1.e4 e6 2.Nf3 d5 3.exd5 exd5 4.d4 Nf6 5.Bd3 Bd6 (77.754.590) 736
24/26 1:57 +0.28 1.e4 e6 2.Nf3 d5 3.exd5 exd5 4.d4 Nf6 5.Bd3 Nc6 6.O-O Be7 7.Bf4 Bg4 8.c3 O-O 9.Nbd2 Bd6 10.Bxd6 Qxd6 11.Qb3 b6 12.Rae1 Rae8 13.h3 Bh5 (86.755.475) 735
25/20 2:44 +0.20-- 1.e4 e6 2.Nf3 d5 3.exd5 exd5 4.d4 Nf6 5.Bd3 Nc6 6.O-O Be7 7.Bf4 O-O 8.Nc3 Bg4 9.Nb5 a6 10.Nxc7 Rc8 (120.464.810) 733
25/18 3:49 +0.12-- 1.e4 e6 2.Nf3 d5 3.exd5 exd5 4.Be2 Nf6 5.O-O Bd6 6.d4 O-O 7.Nc3 Re8 8.Bg5 c6 9.Be3 Qe7 (168.546.917) 733
25/24 6:10 +0.20 1.e4 e6 2.Nf3 d5 3.exd5 exd5 4.Nc3 Nf6 5.d4 Bb4 6.Bd3 Qe7+ 7.Be3 Ng4 8.Qe2 Nxe3 9.Qxe3 Nc6 10.O-O Bxc3 11.bxc3 Qxe3 12.fxe3 O-O (272.208.066) 734
best move: e2-e4 time: 6:12.578 min
The change regarding history and killer moves seems to work well only at lower depths. I'm actually running a 1000 games 20 sec. match, 6 Threads. SF default is clearly in front after 800 games. I'll test SF PA_G against SF default at longer tc tonight.
Any results with the longer time control game for this fix? These updates to history/killers are in version I. Against the default with the new ok_to_use_tt() SF I have a score of +235 -140 =156 from the perspective of the default version at g/24"
Re: Designing an analysis friendly Stockfish?
Posted: Sat Feb 05, 2011 8:33 pm
by Jeremy Bernstein
I would imagine that fast time controls would amplify any slowdown due to additional tablebase or hashing activity. In which case, all seems well. I'm running a 450 game round-robin at 5 minutes per game. I'll post about that tomorrow.
Engine Score St St St S-B
Stockfish_64 57,5/108 ······················································ 11=111101=1111==0001100=1110000000011100=100100010000= 010=1101001000=011101101100111111111101=1001100=101011 3002,7
Stockfish_PA_J_GTB_64 52,5/108 00=000010=0000==1110011=0001111111100011=011011101111= ······················································ 111011==100011=01000110100000=11==10110010000010110000 2886,7
Stockfish_PA_I_GTB_64 52,0/108 101=0010110111=100010010011000000000010=0110011=010100 000100==011100=10111001011111=00==01001101111101001111 ······················································ 2840,0
162 of 300 games played
Level: Blitz 0:05/0,1
Hardware: Intel(R) Xeon(R) CPU W3520 @ 2.67GHz 2659 MHz with 1,0 GB Memory
Operating system: Windows 7 Professional Professional (Build 7600) 64 bit
Re: Designing an analysis friendly Stockfish?
Posted: Sat Feb 05, 2011 9:15 pm
by gaard
Uly wrote:What time control are you using? This may be exclusive to incremental time controls (or nothing at all, but Rybka's time management specially sucks at incremental TC, so one is used to seeing patterns).
Bozo the Clown at RF suggested to make a new compile of Stockfish Default and use it for the testing, instead of JA's compile, to compare apples to apples (speed of the compile would be a non factor, clear improvements would stand as better code.)
http://dl.dropbox.com/u/11904592/stockfish_def_G_J.7z
Good idea. These are all compiled under the same conditions and are the most promising versions, IMO.
Re: Designing an analysis friendly Stockfish?
Posted: Sat Feb 05, 2011 10:53 pm
by Jeremy Bernstein
Engine Score St St St S-B
Stockfish_PA_I_GTB_64 103,5/200 ···································································································· 000100==011100=10111001011111=00==0100110111110100111101110101010111=11101011==11111111=0=1111101111 101=0010110111=100010010011000000000010=0110011=0101000==000=0=110000011000010=1==00100101=100101010 10175,
Stockfish_PA_J_GTB_64 98,5/200 111011==100011=01000110100000=11==1011001000001011000010001010101000=00010100==00000000=1=0000010000 ···································································································· 00=000010=0000==1110011=0001111111100011=011011101111=110=111=1110111111001=00111111=1110001=1111111 9848,2
Stockfish_64 98,0/200 010=1101001000=011101101100111111111101=1001100=1010111==111=1=001111100111101=0==11011010=011010101 11=111101=1111==0001100=1110000000011100=100100010000=001=000=0001000000110=11000000=0001110=0000000 ···································································································· 9958,0
300 games played / Tournament is finished
PGNs attached.
Re: Designing an analysis friendly Stockfish?
Posted: Sat Feb 05, 2011 10:58 pm
by fruity
gaard wrote:
Any results with the longer time control game for this fix? These updates to history/killers are in version I. Against the default with the new ok_to_use_tt() SF I have a score of +235 -140 =156 from the perspective of the default version at g/24"
I'm still fiddling around. According to my tests at 20'' all the versions so far are 10-15 Elo below SF default. Best result I've found so far is
(1) remove the history/killer tweak. It seems to work well only for low depths. Could be reintroduced later with tweaking for low depths only, but at the moment it doesn't help to find the best PV TT code, so leave it away for now.
(2) use template and inline for ok_to_use_TT() and the following code
Code: Select all
template <NodeType PvNode>
inline bool ok_to_use_TT(const TTEntry* tte, Depth depth, Value alpha, Value beta, int ply) {
Value v = value_from_tt(tte->value(), ply);
return
PvNode ? tte->depth() >= depth && tte->type() == VALUE_TYPE_EXACT && v < beta && v > alpha
: ( tte->depth() >= depth
|| v >= Max(value_mate_in(PLY_MAX), beta)
|| v <= Min(value_mated_in(PLY_MAX), alpha)
)
&&
(
((tte->type() & VALUE_TYPE_LOWER) && v >= beta)
|| ((tte->type() & VALUE_TYPE_UPPER) && v <= alpha)
);
}
This seems to be about -5 ELO compared to SF default at 20'', but needs more testing, also at longer tc.
It seems there is still work left to find a real neat solution regarding the playing strength.
Re: Designing an analysis friendly Stockfish?
Posted: Sun Feb 06, 2011 12:01 am
by Jeremy Bernstein
Re: Designing an analysis friendly Stockfish?
Posted: Sun Feb 06, 2011 12:08 am
by Peter C
Jeremy Bernstein wrote:Here's a game:
...
Note how the PA_J+GTB finds a forced mate around move 80 or so and then loses its way by move 99. What happened?
I'd guess it's probably a bug in the GTB probing. Is the Decemberists' patch applied?
fruity wrote:(1) remove the history/killer tweak. It seems to work well only for low depths. Could be reintroduced later with tweaking for low depths only, but at the moment it doesn't help to find the best PV TT code, so leave it away for now.
I added a UCI option to control the depth in ply it's used up to. With my default setting of 20 (which could/should be tweaked a bit), it's tad better than SF default at 40 moves/1 second.
I've noticed that our SF tends to lose on time more often than the default at hyperbullet (40 moves/1 second), especially in the first 4-5 games. Has anyone else noticed this behavior?
Peter