OpenAPI: Add min-rows-requested field to PlanTableScanRequest

[ajreid21]

Proposal to add to the spec a way for the client to give the server a hint/indicator for the amount of rows being requested so that the server does not need to return more results than is necessary. Just to be clear, this is not meant to mean the server must return X number of rows or that the server should return at most X of number of rows.

ML Discussion: https://lists.apache.org/thread/m51fxlsbt5yk219ypk2dhj07tlk3407b

[singhpk234]

[doubt] how is the server supposed to fulfill this request, when there are equality deletes present ?

Additional questions -

• may be we should be explicit that its an hint ?
• what would server do if its certain table doesn't have minimum number of row, 400 bad request ?
• we might need to define iceberg scan api like we do for projection and filter so that E2E plumbing works

[ajreid21]

Yes, this is intended as a "hint" or "indicator" from the client to help the server not have to return more than is necessary. It is not required for the server to return that many results (as the result of the scan may not have that many rows, that's why it's named "min-rows-requested" and not "min-rows-required"). Also, it's not a max limit either and the server can return more than the requested number.

I'm open to different name and better description for this

[nastra]

What about The minimum number of rows requested for the scan. This is used as a hint to the server to not have to return more rows than necessary. It is not required for the server to return that many rows since the scan may not have that many rows. The server can also return more rows than requested

This should make it easier to understand that this is a hint and what the expectations are

[amogh-jahagirdar]

I think @nastra description is pretty reasonable, +1. This is fundamentally a hint to the server, and I would almost certainly not want to fail if a table doesn't have enough records.

[singhpk234]

+1 from my end too ! it addresses my questions above

[ajreid21]

Thanks, I updated the description.

[Fokko]

I find the name and description a bit confusing. For me, it is obvious that it might not return exactly the number of rows that are requested. For example, the table might be empty. I would much rather go with something that's analogue to SQL:

Suggested change

	min_rows_requested: Optional[int] = Field(
	limit: Optional[int] = Field(

Of course, it might still return more rows since the Parquet file contains more rows.

[amogh-jahagirdar]

I guess I look at things the other way since I feel like limit connotes a strict upper-bound (that is if there are sufficient matching rows) as how it means in SQL. Another idea: limit-hint?

[raulcd]

I agree with @Fokko , what about soft_limit?

[emkornfield]

just to jump in on bike shedding how about max_rows_required I think it lacks the ambiguity if someone isn't familiar with SQL and I think my issue with and hopefully makes it clearer that this is in fact an upper bound?

[stevenzwu]

just to pile on. target-scan-task-number

[stevenzwu]

is it actually data row count or scan task count? I thought it is later. but if it is former, this config can be target-row-count.

This is similar to the table properties for write.target-file-size-bytes or read.split.target-size.

[stevenzwu]

if we are talking about fine grained control, maybe target-plan-size-bytes

[emkornfield]

is it actually data row count or scan task count? I thought it is later. but if it is former, this config can be target-row-count.

I'll let @nastra confirm but I think it is a row count (as the original name suggests). target-row-count seems more ambiguous to me in terms of semantics.

[nastra]

this is for rows and we're looking for a lower bound, hence why we're having the min in the name. We're open to better naming suggestions, but using something with limit in the name rather implies an upper bound, which wouldn't be correct here

[stevenzwu]

@nastra lower bound may not always be valid. e.g. maybe the table just don't have enough number of rows to satisfy the lower bound.

target is probably a good name for the desired plan size. Wondering why not using bytes (instead of rows) to express the target, similar to read.split.target-size?

Anyway, I would suggest target-plan-size-rows or target-plan-size-bytes.