Paging improvements

[cholmes]

So I believe paging as it stands right now is a bit problematic. Hopefully a developer who has actually worked with paging on restful api's with elastic search backends sounds in and can explain more.

But in Planet's data API we don't have 'startIndex' or anything like that, since it's a lot more complexity to enable clients to request arbitrary locations in an index. Indeed if the index changes, with data added or deleted then that can throw off the results.

Planet solves this by requiring users to create a saved search, or there's a shortcut 'quicksearch' to get results right away. But both create an index on the server side that's set in time, that the user can page through. DigitalGlobe actually doesn't enable paging on their Catalog results, and the underlying vector data store seems to require a similar creation of a paging ID - https://gbdxdocs.digitalglobe.com/v1/docs/vs-retrieve-page-of-vector-items

I don't think we need to specify that level of paging / search in the core of the spec (though possibly an extension / best practice). But I think we should remove 'startIndex', in favor of just having results supply their 'next' link, which can be generated by the service. Planet's 'next' links look like https://api.planet.com/data/v1/searches/f8abab5007a14b31b5ccfb8a3d3f02d1/results?_page=eyJxdWVyeV9wYXJhbXMiOiB7fSwgInNvcnRfcHJldiI6IGZhbHNlLCAicGFnZV9zaXplIjogMjUwLCAic29ydF9ieSI6ICJwdWJ

(with a string that is actually 3 times that long).

In the STAC spec we just have an optional nextPageToken, and then the 'links' section uses that token.

Services wouldn't be required to use an obscure string for paging through results. Though we should make a recommendation on if services are expected to return consistent results, or if it is ok that they are returning like less than the 'count'. See like https://developers.facebook.com/blog/post/478/ for an api that doesn't guarantee the number of results.

Additional discussion on paging best practices at https://stackoverflow.com/questions/13872273/api-pagination-best-practices - we should perhaps recommend some default ordering of results that paging can be driven off of.

The other thing that seems a bit arbitrary is the 10000 maximum limit on the 'count' parameter. And I couldn't figure out in the spec if that is something that implementations can change, or is like hard coded in. At Planet we enable dynamic setting of the page size, but the limit is 250 results per page. And the operations team resists any increases to that, as it introduces more complexity to support.

[pvretano]

Review this from WFS 2.0: http://docs.opengeospatial.org/is/09-025r2/09-025r2.html#76 Sounds pretty close to what you are describing no? ... ignoring all the references to capabilities documents which would no longer be relevant in WFS 3.0 anyway.

WFS 2.0 supported two types of paging. The first used startIndex and count and was there for backward compatibility with WFS 1.X. The second method used count and prev/next links in the response.

I always preferred that latter so I would be good with tossing the legacy and sticking with count/prev/next.

The draft says that the 10000 value is only an example and may be changed (see Req 17 in the core).

[cholmes]

Yeah, that's definitely right in line. Count & prev/next links. Would be happy to toss the legacy and do the count/prev/next.

Ah, and missed that the 10000 is an example. I think that is the challenge with the current format, how to get across what can be tweaked and what must be there - though I've got no great ideas on how to do it better, as the current spec does a decent job, particularly with the sample openapi document.

[cportele]

Tossing the startIndex legacy would be fine with me, too. So we would "just" keep count and it would be up to the implementation how it creates the prev/next links (which could use a parameter like startIndex).

To make it clearer that the 10 and 10000 are just examples, maybe we should add an explicit sentence in the count requirement that default and maximum count values have to be declared in the API definition, but the values are determined by the server.

[pvretano]

That is correct. The prev/next links would be opaque. An implementation could generate whatever links it wants including using the startIndex parameter (which would now be reclassified as a vendor-specific parameter).
About the default and max count values, yes we should add some clarification text.

[cholmes]

Agreed. And yeah, an explicit sentence on the server determining values makes good sense.

[rcoup]

About the default and max count values, yes we should add some clarification text.

I think count should be a suggestion to the server ("I only want 10", or "I'd like 1M"), but at the end of the day it can send back what it likes. You can ask for 1M but I'll only send 250, or you can ask for 10 but I might send you 100. As long as the next link is present you can eventually get the number you're after.

Should prev be optional?

[cholmes]

Agreed on both counts. Better to return records, even less than asked for, than return an error.

And I'd missed 'prev' in there - definitely should make it optional. We don't do it in the Planet data API, as it's definitely more complex to support it, and we didn't see any win from doing it.

Maybe we should put that one in its own ticket?

[pvretano]

By "suggestion" do we mean a maximum per fetch? count=100, I want 100 but the server can return any number up to 100 ... and provide a next link for next set of features.
As for prev being optional ... hmmm ... I don't think that I am opposed to that.

[rcoup]

I think it probably boils down to:

• the client may request a count it's interested in
• the server likely has a default value for count, and a maximum limit
• if the server has any more results available than it returns (whether the number it returns is less than, equal to, or greater than either the requested/limited/default count) then the server should include a link to the next set of results.

So:

• If you ask for 50, you might get 50 and a next link if there are more. (requested)
• If you ask for 50000, you might get 1000 and a next link if there are more (server-limited)
• If you ask for 1, you might get 100 and a next link if there are more. (ignored your request)
• If you don't specify a count, you might get 100 and a next link if there are more. (default)

[pvretano]

Don't think that the server should ever return more than requested ... If client requests 1 gets 1 or zero and possibly a next link.

[cportele]

I agree with making prev optional and count as a hint with the constraint that the server must not return more entities than the count value.

[cportele]

To summarize, the conclusion seems to be:

• Remove startIndex (it could be one example for how to implement next links, but opaque links are fine, too)
• Clarify that servers may provide prev, but it is always optional (remove from the requirement)
• For count (reusing @rcoup's summary, with the change discussed above)
  • the client may request a count it is interested in
  • the server likely has a default value for count, and a maximum limit
  • if the server has any more results available than it returns (whether the number it returns is less than or equal to the requested/limited/default count) then the server should include a link to the next set of results.
  • So:
    • If you ask for 50, you will get 0 to 50 and a next link if there are more. (requested)
    • If you ask for 50000, you might get 1000 and a next link if there are more (server-limited)
    • If you don't specify a count, you might get 100 and a next link if there are more. (default)

Is everyone ok with this or do we need more discussion?