v2.9: windows, windows, and more windows

This release has about 6 months of development in it, including many bugfixes, improvements, and a few experimental features. Here are the highlights:

• new confirm dialog window (press y to accept; Enter no longer required)
• new command addcol-window bound to w to create a new column with a list of values "around" the current row
• new formatter attribute on column (and options.disp_formatter) to set the format for both display and text saving. Can be generic, python, json (and it's also extensible)
• support for standard paths for config file and support dirs (XDG and otherwise)
• new loaders for Apache Arrow (IPC and streaming) and parquet formats
• browse remote .zip URLs and extract files from them without downloading the whole archive
• by popular demand, we've decided to invert the default scroll wheel direction! If you liked how it was before, you can set options.scroll_incr = 3
• and a couple things to make it easier for Windows users

There's a lot to love in this release of VisiData; the above is only a partial list. See the official CHANGELOG for more details.

v2.8: Python 3.10 compatibility

This is a pretty small release of mostly bugfixes, which still has a few changes that merit a minor version bump.

The all-stars of this release are @ajkerrigan and @geekscrapy.

@ajkerrigan found, debugged, and fixed a Python 3.10 compatibility bug. Before v2.8, you could see many warnings related to COLOR-1 in the status message.

@geekscrapy contributed 7 PRs for v2.8. Thanks to his PRs, VisiData now gracefully skips blank lines in json files, and will not attempt to load plugins which failed to install.

v2.7.1 Hotfix

The ever-vigilant @geekscrapy found a bug with Enter on the helpmenu #1196.

This is a small enough fix for a very important feature.

[v2.7] new loaders: odf and lsv

A bunch of improvements, bugfixes, and a couple of new loaders.

Improvements

• [movement] bind Home/End to go-top/go-bottom (thanks @geekscrapy #1161)
• [api] add vd.urlcache as alias for urlcache global (thanks @geekscrapy for PR #1164)
• [plugins] do not continue installation if main package fails pip install (thanks @geekscrapy for PR #1194)
• [plugins] allow for plugin records without SHA256; warn if absent (thanks @geekscrapy for PR #1183)
• [load_lazy] do not load subsheets, if sheet.options.load_lazy is True (thanks @geekscrapy for PR #1193)
• [save] confirm when save_foo function does not exist and saver fallsback to options.save_filetype (reported by @geekscrapy #1180)
• [save] options.save_filetype default now 'tsv'
• several cosmetic improvements

Loaders

• [lsv] add lsv filetype for simple awk-like records (requested by @fourjay #1179)
• [ods] add odf filetype for Open Document Format spreadsheets
• [xlsx] add extra columns (cellobject, fontcolor, fillcolor) if options.xlsx_meta_columns (default False) (thanks @hoclun-rigsep for PR #1098)
• [sqlite] allow query/insert (no modify/delete yet) for WITHOUT ROWID tables (requested by @stephancb #1111)

Bugfixes

• [savers compression formats] fix corruption when saving to compression formats (#1159)
• fix "ModuleNotFoundError: no module named 'plugins'" error on startup (#1131 #1152)
• [windows] fix issue with Enter key on Windows (reported by @hossam-houssien #1154)
• [draw] fix multiline rows by making height fixed for all rows (reported by @geekscrapy #916)
• [DirSheet] fix bug where fix key column sheets (e.g. DirSheet, SqliteIndexSheet) keycols were not being saved in batchmode (reported by @geekscrapy #1181)
• [async] make sure all threads started on sheet are cancelable (reported by @geekscrapy #1136)
• [AttrDict] fix bug with setting value on nested AttrDict
• [dup-X-deep] fix error with async_deepcopy (thanks @pstuifzand for fix)
• [join] fix 'inconsistent-keys' issue when joining between XlsxSheet with typed columns and CsvSheet with untyped columns (reported by @DAVIDWALES #1124)
• [sqlite] handle sqlite column names with spaces (thanks @davidskeck for PR #1157)
• [sqlite] use options.encoding and options.encoding_errors for decoding of sqlite db text (reported by @WesleyAC #1156)
• [xlsx] add handling for EmptyCell instances (thanks @hoclun-rigsep for PR #1121)
• [xlsx] gate sheet name cleaning on options.clean_names (reported by @DAVIDWALES #1122)
• [macos] fix bindings for Option+key
• [random-rows] fix import (reported by @geekscrapy #1162)
• [save-selected] better default save filename (reported by @geekscrapy #1180)
• [save] fix bug where saving multiple sheets to a single non-embeddable format did not result in fail (reported by @geekscrapy #1180)
• [slide] fix Shift slide-down and Shift slide-up with arrow keys (reported by @a-y-u-s-h #1137)
• [replay] fix replay where join-sheets operation hangs (reported by @agjohnson #1141)
• [undo] no more KeyError when Undoing modifications (reported by @geekscrapy #1133)
• [unfurl-col] fix unfurl-col on cells containing exceptions (reported by @jsvine #1171)

VisiData v2.6.1: minor bugfixes!

In preparation for the VisiData workshop at Strangeloop 2021, this patch was released with several minor bugfixes.

Features with fixes include:

• sysopen-row
• fixed-format saver
• DirSheet's copy-files
• sqlite loader's exception handling

On top of this, we have updated the homebrew and personal debian repositories to contain the most recent version of VisiData.

We appreciate everyone who has mentioned VisiData in several threads on HackerNews and reddit, recently. Talking about tools you love in public spaces, means a lot to everyone who dedicates time to work on them.

Happy VisiDating!

[v2.6] now with menus

v2.6 (2021-09-19)

VisiData v2.6: now with menus!

This release introduces a new[0] hierachical menu system for discovering and executing commands.

Press Ctrl+H to open the help menu, arrows and Enter to move around. Or use the mouse.

Here's a video demo of this new menu system, including a sample workflow within VisiData for finding which commands aren't in the menu (using the mouse almost entirely).

Also in this release are several performance improvements, feature enhancements, and bugfixes. See the CHANGELOG for full details.

[0] Back in v1.1 (2018), VisiData had an experimental one-line menu system, but it was removed in v1.3 (6 months later), as it wasn't feeling right.

[v2.5] spring cleaning

v2.5 (2021-07-08)

First off, some non-technical notices:

Since 2017 when VisiData was started, I've created a number of plugins and tools that take advantage of the VisiData "platform".
These have historically been separate from VisiData, but it became a fair amount of mental overhead tracking a dozen different repos when making changes to the VisiData core repo. So a little while ago I migrated some of my older projects, like vsh and vgit, into the saulpw/visidata repo, aiming at a "mono-repo". But they never really got integrated, because they didn't feel right either: some of them were wild experiments, some I don't want to release as open source, some I don't want to provide the level of support for, and some simply don't belong in the visidata repo (like galcon).

So now I'm migrating towards a "duo-repo" approach. The saulpw/visidata repo can remain focused on providing a reliable and flexible platform, and a separate repo, "vdplus", is where I put all my VisiData plugins and extras. (This vdplus repo is private and not open-source, but Patreon subscribers can get access to it and use it if they want.)

Secondly, the core VisiData repo now requires submitters to e-sign a Copyright Assignment Agreement (CAA). The core repo will always be open-source, but as the author I should have the legal authority to release it under a different license if I choose.

Finally, our IRC channel has moved from Freenode to libera.chat; come visit us on the new #visidata!

And now for the actual software changes:

Interface Changes

• quitguard ux improvement

  • [quitguard] confirm quit/reload only if sheet modified (references #955, #844, #483; thanks @jvns, @frosencrantz)
  • not enabled by default yet

• [quit] add Shift+Q/quit-sheet-free to quit and free associated memory (thanks @cwarden)

• New major keybinding!

  • x is now "cut" (copy and delete), like in vim.
  • In v2.3 the delete family was changed to delete without copying to the clipboard (previously it acted like "cut").

• [options] save to foo.visidatarc from OptionsSheet (thanks @njthomas #958)

  • note that you can't save directly to .visidatarc (call it a security feature).

Minor Improvements

• [splitwin] push sheet in empty pane iff splitwin
• [movement] bind Ctrl+Left/Right to go-left/right-page (thanks @DAVIDWALES #1002)
• [unfurl] add options.unfurl_empty to include row for empty list/dict (thanks @frosencrantz #898)
• [regex expand] deprecate expand_col_scanrows; standardize on options.default_sample_size (thanks @jsvine)
• [canvas] add options.disp_canvas_charset to change displayed chars (thanks @albert-ying #963)
  • @albert-ying convinced me to make the canvas character set configurable. With options.disp_canvas_charset = ' o', any points on the graph become an o, which is hopefully big enough to see. You can set this to use different characters, just make sure that the first character is a space. This option can also be configured more precisely if you want to vary the character by how many/which points are present (like the braille characters).
• [join] extend join columns now writable (and correct)

Loader-specific Improvements

• [sqlite] rename or drop tables from the sqlite index sheet (edit the name or delete the row and then z Ctrl+S to commit as usual)
• [DirSheet] add y/gy to copy file(s) to given directory
• [zstd] support loading zstd-compressed files (thanks @lxcode #971)

Bugs fixed (34): #758 #895 #896 #898 #946 #952 #955 #958 #959 #961 #962 #963 #964 #966 #969 #971 #974 #975 #979 #982 #983 #987 #991 #1002 #1003 #1010 #1011 #1011 #1012 #1016 #1018 #1029 #1032 #1033

See the CHANGELOG for the comprehensive list of bugfixes and changes.

---

[v2.4] bugfixes for default colors, splitwindow, sqlite, and more

v2.4 (2021-04-11)

This release, hot on the heels of the 2.3 release last week, is primarily to address some bugs that were found in 2.3.
Originally it was slated to be a patch release (2.3.1), but the removal of options.use_default_colors with the fix for #939 and the semantics of splitwindow felt sufficient to warrant a minor version bump, and so we tossed the few other improvements that were sitting on the develop branch.

Improvements

• [color] remove options.use_default_colors (thanks @lxcode #939)

  • options.color_default can now have both fg and bg
  • other color options which do not specify fg or bg will use the missing component from color_default
  • to use terminal default colors, set options.color_default=""

• [splitwindow] stabilize sheet stack associations

  • Shift+Z pushes 'under sheet' (if any) onto other stack
  • Shift+Z does not swap panes anymore
  • g Tab changes which stack is in the top vs bottom panes
  • options.disp_splitwin_pct cannot be sheet-specific

• [status] show nSelectedRows on rstatus

Bugfixes

• [pyobj-show-hidden] grab visibility lvl from sheet specific option (thanks @frosencrantz #947)
• [splitwin] prevent flickering-on-full-window
• [splitwin] if top sheet quit, keep bottom sheet in bottom pane

Loader specific bugfixes

• [gzip] fix progress bar when opening gzip (thanks @geekscrapy #925)
• [http] fix loading files from url without specifying filetype
• [sqlite] use TABLE_XINFO for hidden/virtual columns (thanks @dotcs #945)
• [sqlite] reload perf: load without pre-counting rows (which required 2nd full table scan)
• [vds] save typed values instead of formatted display values (thanks @frosencrantz #885)
• [xlsx] stringify "header" row values for column names (thanks @DAVIDWALES #921)

v2.3: memory sheet; splitpane and other improvements

This release adds a memory sheet and improves on several other features. Major changes are listed below. Please see the CHANGELOG for the full list of changes.

1. Memory sheet, for recording values usable in expressions. View the memory sheet with open-memory (Alt+Shift+M), add current cell with Alt+M, add single aggregated value with z+. Also reference the current clipboard value as clipval. #912

2. The split pane feature has been revamped. Now there is one sheet stack per pane, and sheets do not jump between panes. New sheets open in the current pane. Send the current sheet to the other pane with Shift+Z (same command as invoking splitpane in the first place). Ctrl+^ swaps sheets in the current pane. See the pane column on the Sheets Sheet (pane 0 means "not on any sheet stack", i.e. it's been quit). #894

3. Colors now may have a background color in addition to the fg color and attributes. Add e.g. "on blue" or "on 214" to a color to set the background color. There is still the (curses) limit of 256 active colors shown at one time. (This is one of several changes in 2.3 necessary to use the new darkdraw plugin.) #435

4. minor additions

• new command: jump-first (bound to g Ctrl+^) to push the bottom sheet (used repeatedly will cycle through sheets in the current pane).
• Ctrl+X within Ctrl+X (pyobj-expr) drops into a Python REPL.

5. changes to options and commands

• options.use_default_colors now defaults to True (was False).
• delete commands no longer move deleted values to clipboard #895
• zd and gzd (delete-cells) set the value to options.null_value instead of always using None.

v2.2.1

This is a small patch.

There are two bugfixes for setcol-fill (see #884), and a typo fix in the manpage.

Additionally, there is now a packaged visidata.1, so that man visidata will also bring up the VisiData manpage, when the manpage is successfully installed.

VisiData v2.2 has been packaged for Debian, and is slated to be part of the bullseye distribution. I am hoping I can sneak v2.2.1 in there, also.

Happy data-sleuthing!