50 Gb 6-men scorpio bitbases available

[stackOVERFLOW]

Six men scorpio bitbases are available for download free of charge.

Some of the advantages are:
1) Very compact about 48.1G. So you can have them on 50gb flash drives for instance.
No need for super expensive SSD drives

2) Have been around the block for over 10 years. No need to over-hype them, the number
of engines using them speaks for itself.

Enjoy the very compact Scorpio 6-men bitbases in 2014

http://oics.olympuschess.com/tracker/index.php

regards,
OVERFLOW, stack OVERFLOW

[pgn4web]

Do you have those bitbases available on your olympuschess.com server?

It would be interesting to have a web service that could access a query like

http://olympuschess.com/fen=1k6/2pp4/8/ ... 0-%200%201

and return somehow (html page, text info, xml data) the evaluation of the position and the PV line.

[BB+]

Is there a way to generate them? Downloading is next to useless to me.

Also, I'm not your typical user, but the size-tradeoff versus (say) access time with bitbases is not really an issue for me. Indeed, in maybe another 3 or 4 years, the 50 or 70 or 90 GB will fit into RAM for 6 piece for average users, and it really won't make much difference whether one can save 30% in size. I recall when the 5-piece Shredderbases boasted that they fit into 157MB (1-sided), or power users could load the whole 441MB into RAM and have it be much faster. The comparison of sizes is also a bit dodgy, for instance the 90GB 6-piece RobboBases don't use a final Huffmann layer as Syzygy does (and thus might be marginally faster) and if this were included the size would reduce to around 50GB (I base this on the bzip2 size), but on the other hand the 70GB Syzygy has some support of the FIDE 50-move rule [since I dislike this rule, I don't care much about that addition].

Probably, if I cared merely about minimal-size bitbases, some sort of 1-sided ID3 or OBDD would likely be best, though the access could be painfully slow. In the same regard, the Lomonosov gang has spoken about their difficulties in making 7-piece bitbases, such as whether don't-caring promotions is reasonable given the access patterns.

More interesting to me is extensibility, such as the BlockedPawn with the RobboBases. Unfortunately, though it seems feasible to me, no one seems to be interested in extending this to more general same-file pawn configurations (which would be quite useful IMO). Here again one has that N-unit endgames will often descend to another N-unit endgame, so don't-caring all such transformations might not be wise.

[BB+]

If I go to the Scorpio page (https://github.com/dshawul/Scorpio), I read:

"You can also load the 5 men egbb's to RAM but they require too much RAM. So you might prefer to probe them on disk like i do. But make sure that you don't call them near the leaves of the search tree if they are not loaded on RAM."

Is there a soft-probe option, or maybe even a background loader like with RobboBases (for when soft probes fail)?

Another point: my recollection was that Scorpio Bitbases, at least in a previous version, actually expanded the compressed data à la Nalimov (unlike Syzygy and Robbo which use pattern encoding [Robbo is just RLE I think] and just scan over these), which means that all 5 piece endgames would already take a lot of space if they were all sitting in immediately usable form [i.e. uncompressed] at the same time (there are something like 35 billion 5-piece positions already, and at say 2 bits per position this is nearly 10GB just for the 5s, with the 6s maybe 150x more).

[BB+]

Just to elaborate on this, it seems that one could have 1-sided 6-piece bitbases in about 18GB, and if in-check don't caring was included about 14GB. With pawn-push don't caring (is this wise with 1-sided?!), the goal might be to make it under 10GB. But as I say, the access will be slow, even if the whole lot were in RAM, as the resolution procedure is now computation-heavy.

[syzygy]

In the same regard, the Lomonosov gang has spoken about their difficulties in making 7-piece bitbases, such as whether don't-caring promotions is reasonable given the access patterns.

Did you get that from the "7-man TB" Google+ page, or from some other source?

More interesting to me is extensibility, such as the BlockedPawn with the RobboBases. Unfortunately, though it seems feasible to me, no one seems to be interested in extending this to more general same-file pawn configurations (which would be quite useful IMO). Here again one has that N-unit endgames will often descend to another N-unit endgame, so don't-caring all such transformations might not be wise.

I'm interested, but haven't worked on it yet. I'm currently generating 6-piece suicide tables (with FICS stalemate rule...).

[BB+]

Is there a way to copy a specific post link from Google+? Plus the scrolling there is lousy (top 1/3 of screen is eaten), when I clicked on a link (hoping for a useful URL) it gave me a "sign-in" option with no "Cancel"...

So I have to copy/paste:
EDIT: https://plus.google.com/100454521496393 ... G6QQgPxeN1

Sep 2, 2013
You can think that after achieving of main goals TB7 project is frozen. Partially it is true. It went to background. But it is not frozen.
[...]
3) We are working for making ternary tables more compact by using "don't care" compression technique described by Ronald de Man (known also as syzygy on Kirill-Kryukov Endgame Tablebases forum). Looks like ternary tables can be 2 times more compact with this technique. Payment for this is extra probes to minor tables.

Then in a comment to the same (Sep 13, 2013):

Well, with "don't care for winning capures" compression technique introduced by Ronald de Man for WDL (Win/Draw/Loss) tables
5-man win/draw/loss DTM tables occupy 384 MB (without 4+1 tables),
6-man win/draw/loss DTM tables take 64.2 GB. Extrapolating the results we can guess that all 7-man tables will take somewhere 8 TB of disk space (5-6% of size of full tables on PC that is about 140TB).

But compression ratio of most popular tables is worse .
If we will look to the most popular KRPP-KRP it takes:
111 GB in generation format (2 MB blocks)
166 GB in PC format (16 KB blocks)
30.5 GB in WDL format
23 GB in WDL "don't care" format (we will name it DL to be short). It is 14 % of PC format.

The worst value is for KRBPP-KQ for a while: 16% (180 GB via 28.3 GB).
The best is for KBPPP-KN: 1.7% (44.5 GB via 0.75 GB)

We need to compare carefully the speed of access to WDL and DL tables. A serious drawback of DL-tables is not only in extra probes but is in the need to have in hands all minor tables also. It was noticed earlier that it is not usefull for speed to use tables with evident results (like KQQ-KNN). Evaluation function returns the results faster than probes to the tables.

I had remembered something more specific on the matter (specifically with some word like "promote"), but if it is on the Google+ page I can't find it.

I find Google+ to be the least navigable thing I have seen in a loooonnnnnggg time. Why can't I just get posts/comments in their own web page (and with their own links), like most blogs on the web??? And then constant open/closing with Read More.... This is AWFUL!!!

Incidentally, does anyone know anything about the 7-piece tables Kasparov received from Elmar Magerramov (see page 20 [near end of Chapter 1] of Timman's book The Art of the Endgame, he claims Kasparov sent him this at the beginning of 2009). Maybe he got these from Konoval/Bourzutschky?

I'm currently generating 6-piece suicide tables (with FICS stalemate rule...).

I finally clusterised my code, and am working on 1. e3 e6 [on 96 cores at the moment]. I have some positions where 6-piece results might aid (but I think I have more where blocked pawns would), though my general opinion is that I can almost always avoid tedious endgames via alternative lines. The pn-searches only use 4-piece themselves (though I could probably make this 5 w/o too much a problem), but I do find that having the 5s in the pn-gluer (into a second level tree) is at least useful sometimes for killing off lines [due to White not winning] at an earlier stage than otherwise.

[syzygy]

I also didn't find more than this on their Google+ page.

I finally clusterised my code, and am working on 1. e3 e6 [on 96 cores at the moment]. I have some positions where 6-piece results might aid (but I think I have more where blocked pawns would), though my general opinion is that I can almost always avoid tedious endgames via alternative lines. The pn-searches only use 4-piece themselves (though I could probably make this 5 w/o too much a problem), but I do find that having the 5s in the pn-gluer (into a second level tree) is at least useful sometimes for killing off lines [due to White not winning] at an earlier stage than otherwise.

Ok, so you haven't solved the game... yet.

My 5-piece tables are about 8.8 GB (WDL50+ still). I can cut that size in about half if I set winning and drawing "capture-threats" (i.e. a move forcing a capture by the other side) to don't care, but that would slow down the generation of 6-piece too much and 8.8 GB is anyway quite manageable nowadays. My 6-piece tables do use the capture-threat compression.

I think I took capture-threats from this Chinook paper, but the description is a bit vague:

Further, positions where one side could capture if the side-to-move were switched comprise an additional 25% of the positions (capture-threat). Again, by removing these positions from the database, additional compression can be achieved, at the cost of a more complicated search algorithm for resolving values.

From 4 to 5-piece takes slightly less than a factor 100. If the same applies from 5 to 6 and I get a 50% reduction due to improved compression, the 6-piece would stay below 500 GB and putting them on SSD would be an option.