Set max from/size for ES

[gibrown]

In ES 2.x there is now a maximum value for from+size in order to prevent deep paging which can cause performance problems.

We've also added this same limit when querying WP.com 1.x indices (after seeing some performance problems). It seems that es-wp-query should handle this case cleanly. For example, what happens when a bot tries to walk all archive pages.

[mboynes]

For example, what happens when a bot tries to walk all archive pages.

That's a problem in MySQL too. I wonder which handles it better.

[gibrown]

BTW, there are ways to get around this sometimes with filters. May be worth implementing. For inf scroll of calypso posts, sorted by date across all of a user's blogs we use encoded page handles. Something like:

$args = wp_parse_args( “blog=1234&post=12&value=2015-01-12T12:00:21” )
{ “bool” : { “should” : [
  { “range” : { “date” : { “lt” : $args[‘date’] },
  { “and” : [
    { “term” : { “date” : $args[‘date’] },
    { “range” : { “blog_id” : { “gte” : $args[‘blog’] },
    { “range” : { “post_id” : { “gt” : $args[‘post’] },
  ] }
],
“sort” : [
  { “date” : { “order” : “desc” } },
  { “blog_id” : { “order” : “asc” } },
  { “post_id” : { “order” : “asc” } },
]
} }

Though the assumption here is that you need to pass around info about the last item that was shown rather than just a page number. It also has the nice property of not showing duplicate content when new content is posted. I suspect though that this would require a change to Core first. But mentioning in case there is another way.