Headsup Strategies Convergence issue with Default Set up

[andyafter]

I have been playing around with the code for quite a while now. I got the API ready to use for a headsup set up.

However I noticed several things that has led to me believing that the strategy has not been configured correctly:

1. AA preflop has 8% folding frequency.
2. The convergence of the MCCFR took around 2 hours to complete, which is rather quick, however the saving of the blueprint strategy then took around 5 days to finish.

I have been using the default parameters as provided in the lib.rs file. So my question is:

1. Is this because that my strategy didnt really converge?
2. Why does it take so little time to train and so much time to write the blueprint file.

Would it be ok if you could share with us the configs/parameters that you used for a proper headsup set up, for example, what is the proper number of MAX_DEPTH_SUBGAME, CFR_BATCH_SIZE_NLHE, etc.

if there are a proper set up for multiple people table that would also be great!

[krukah]

so the default parameters won't actually get a good solution for a few reasons

1. the number of iterations needs to be on the order of 10^9 or so to get a playable blueprint solution. at least, in most of the literature, they describe the "length" of training in terms of "nodes touched", and we touch a few hundred nodes under the current exploration parameters. regardless, you'd need a lot more training to converge to anything interesting
2. there is maybe something to be said about not defaulting to 100% uniform policy over possible actions. probably biasing toward foldnig for example, or allocating a fixed amount of probability toward all raises, rather than each size of raise getting equal share as other actions. so most of the policies probably won't stray much from being close to uniform in the "low N training iterations" regime, just because their infosets won't have been visited much.
3. writing the blueprint file probably does a ton of thrashing and unoptimized IO. we're basicaly taking a massive BTreeMap and then writing it sequentially to disk, but the internals of this process aren't something i've looked into optimizing. something like the Disk trait would be where to look, implemented for Profile
4. multi-way decisions aren't supported but it might be as simple as changing the constant N_PLAYER in lib.rs? not sure, but if there are any assumptions about 2player game, there will probably be a corresponding assertion that would prevent the code from reaching an invalid state.

[andyafter]

One more question, will 128 clusters be sufficient for a super human level AI for heads-up? Or is the real clustering number much more than that

[andyafter]

Also, by increasing the iteration here you mean the sinkhorn iteration?

[krukah]

• sinkhorn iteration is only relevant for the kmeans step, it's a hyperparameter in calculating the optimal transport / wasserstein distance between two arbitrary distributions. see sinkhorn.rs if curious. but you can largely ignore this, since it sounds like you've already completed the kmeans step (maybe not though)
• if i had to guess, i would guess that "number of kmeans clusters on ___ stret" is very far from being the most important hyperparameter to tune. the Pluribus paper describes a kmeans learning process that uses k = 200 so i figured 128 would be same order of magnitude. intuitively, i would also think there are more "classes" of hands in earlier streets than later streets? so it might be that k should be a bit larger in earlier streets. i recall reading some literature about how you only need "cluster resolution of 2% on the river" i.e. there's not that much granularity needed for river equity calculations. but i don't know how this translates to earlier streets, where granularity in the clusters might have compounding effects on how well earlier streets can cluster, no idea.
• i would also guess that the number of "nodes touched" or roughly equivalently, number of "trees explored" would be the most important hyperparameter. the "Bitter Lesson" tells us that scale is all we need. it's probably true that our convergence could be more efficient if we used strategies like discounting and pruning (see references to papers in README/main.rs) , but , the literature (pluribus paper) yields superhuman performance after touching 10^12 nodes, which is probably feasible for a few thousands of dollars of compute? not sure though.
• tldr, the training iterations is the most important parameter bc it allows the algo to actually explore the tree. if you only explore a few million trees, i expect the blueprint will not explore enough infosets and even those infosets that it does explore, it will not have visited enough times to converge to anything interesting

[andyafter]

So in summary:

1. 128 ~ 200 clusters will be enough for a superhuman performance
2. 128 ~ 200 clusters would possibly be sufficient for multiple people table as well
3. "Nodes touched" can be set by configuring "CFR Tree size" to more than 10^9

Is that understanding accurate?

Forgive me for my lots of following up questions:

1. By training iterations, does it mean the k-means iterations like KMEANS_FLOP_TRAINING_ITERATIONS?
2. What does CFR_BATCH_SIZE_NLHE do and what does it affect?
3. Bit ambitious, but would it be possible to adjust the code for it to run on an on-demand instance or stuff like Lambda, where the resources are cheap and the workload can be scattered. In a word, making the infra scalable.
4. Would it be useful if I share the clustered data for people to use?

[krukah]

1. there are two learning loops: one is for clustering abstractions (KMEANS) the other is for learning blueprint strategy (CFR)
2. this is the size of batches used to update counterfactual regrets and policy weights in blueprint training
3. this is a very interesting idea. to first order, this would basically mean making all the fn's of the Policy and Blueprint and other CFR traits async, and then figuring out how to effectively parallelize using postgres locks rather than rayon::iter. it is indeed ambitious, but i don't think it's fundamentally impossible at all.
4. sure! i will say that i discovered a subtle bug in the Evaluator about a month ago, so any cluster abstractions learned prior to that will be slightly incorrect, particularly around any cards with flush potential. but given that this kmeans clustering is in fact deterministic, i imagine this would indeed be a useful contribution!