[vdsql] Do sort/search/etc only work on the 500 returned rows?

[massyah]

Hi everyone,
I am currently evaluating visidata / vdsql to browse a duckdb database, that I load directly with vdsql db.duckdb

By default only 500 rows are loaded from a table, and table wise operations (sort, search, filtering) only apply to those 500 rows.

I’d have expected that such operations apply to the whole dataset, even if only 500 results are displayed.
I cannot find any settings to work with the whole dataset nor any discussion related to this behaviour.

Am I missing something?

Thanks a lot for your help

[saulpw]

Those operations should extend the query and not only operate on the returned rows. That said, I haven't worked on vdsql in awhile, and it takes effort to keep up with its Ibis dependency, so if it doesn't work for you out of the box, it might not be a good fit for your needs.

[midichef]

@saulpw has put so much excellent work into Visidata, that after some time it's hard to remember it all!

I was able to reproduce the 500 row limit, and discovered that it's controlled by Visidata's ibis_limit option. You can remove the limit by setting it to 0. Either add a line to your .visidatarc to configure it permanently:
options.ibis_limit = 0
or set the option in the command line:
vdsql --ibis_limit=0 test.duckdb or vd -f vdsql --ibis_limit=0 test.duckdb.

You can look for configurable options inside visidata/vdsql by pressing O (options-global), and searching for 500 with g/ (search-cols). Or if you'd like to look inside the source code, you can search the Visidata repo using a regex

[saulpw]

Thanks @midichef :) But @massyah is right, doing a 'sort' on a returned query should change the underlying query, and not just sort on those 500 rows. You should be able to verify this when you see the query in the sidebar. Possibly it requires Ctrl+R to reload the new query, before which it only deals with the in-memory data? Sorry I'm not more use on this one at the moment.

[midichef]

Oh, now I understand the goal.