Merge pull request #1178 from saulpw/develop

v2.7

Author: anjakefala

CHANGELOG.md

@@ -1,5 +1,49 @@
 # VisiData version history
 
+# 2.7 (2021-11-14)
+
+## Improvements
+
+- [movement] bind Home/End to go-top/go-bottom (thanks @geekscrapy #1161)
+- [api] add vd.urlcache as alias for urlcache global (thanks @geekscrapy for PR #1164)
+- [plugins] do not continue installation if main package fails pip install (thanks @geekscrapy for PR #1194)
+- [plugins] allow for plugin records without SHA256; warn if absent (thanks @geekscrapy for PR #1183)
+- [load_lazy] do not load subsheets, if `sheet.options.load_lazy` is True (thanks @geekscrapy for PR #1193)
+- [save] confirm when `save_foo` function does not exist and saver fallsback to `options.save_filetype` (reported by @geekscrapy #1180)
+- [save] `options.save_filetype` default now 'tsv'
+- several cosmetic improvements
+
+## Loaders
+
+- [lsv] add `lsv` filetype for simple awk-like records (requested by @fourjay #1179)
+- [ods] add `odf` filetype for Open Document Format spreadsheets
+- [xlsx] add extra columns (`cellobject`, `fontcolor`, `fillcolor`) if `options.xlsx_meta_columns` (default False) (thanks @hoclun-rigsep for PR #1098)
+- [sqlite] allow query/insert (no modify/delete yet) for `WITHOUT ROWID` tables (requested by @stephancb #1111)
+
+## Bugfixes
+
+- [savers compression formats] fix corruption when saving to compression formats (#1159)
+- fix "ModuleNotFoundError: no module named 'plugins'" error on startup (#1131 #1152)
+- [windows] fix issue with Enter key on Windows (reported by @hossam-houssien #1154)
+- [draw] fix multiline rows by making height fixed for all rows (reported by @geekscrapy #916)
+- [DirSheet] fix bug where fix key column sheets (e.g. DirSheet, SqliteIndexSheet) keycols were not being saved in batchmode (reported by @geekscrapy #1181)
+- [async] make sure all threads started on sheet are cancelable (reported by @geekscrapy #1136)
+- [AttrDict] fix bug with setting value on nested AttrDict
+- [dup-X-deep] fix error with async_deepcopy (thanks @pstuifzand for fix)
+- [join] fix 'inconsistent-keys' issue when joining between XlsxSheet with typed columns and CsvSheet with untyped columns (reported by @davidwales #1124)
+- [sqlite] handle sqlite column names with spaces (thanks @davidskeck for PR #1157)
+- [sqlite] use `options.encoding` and `options.encoding_errors` for decoding of sqlite db text (reported by @WesleyAC #1156)
+- [xlsx] add handling for EmptyCell instances (thanks @hoclun-rigsep for PR #1121)
+- [xlsx] gate sheet name cleaning on `options.clean_names` (reported by @davidwales #1122)
+- [macos] fix bindings for `Option`+key
+- [random-rows] fix import (reported by @geekscrapy #1162)
+- [save-selected] better default save filename (reported by @geekscrapy #1180)
+- [save] fix bug where saving multiple sheets to a single non-embeddable format did not result in fail (reported by @geekscrapy #1180)
+- [slide] fix Shift slide-down and Shift slide-up with arrow keys (reported by @a-y-u-s-h #1137)
+- [replay] fix replay where `join-sheets` operation hangs (reported by @agjohnson #1141)
+- [undo] no more KeyError when Undoing modifications (reported by @geekscrapy #1133)
+- [unfurl-col] fix unfurl-col on cells containing exceptions (reported by @jsvine #1171)
+
 # 2.6.1 (2021-09-28)
 
 ## Bugfixes

MANIFEST.in

@@ -3,3 +3,4 @@ include LICENSE.gpl3
 include visidata/man/vd.1
 include visidata/man/vd.txt
 include visidata/man/visidata.1
+include visidata/ddw/input.ddw

README.md

@@ -1,5 +1,5 @@
 
-# VisiData v2.6.1 [![twitter @VisiData][1.1]][1] [![CircleCI](https://circleci.com/gh/saulpw/visidata/tree/stable.svg?style=svg)](https://circleci.com/gh/saulpw/visidata/tree/stable) [![Gitpod ready-to-code](https://img.shields.io/badge/Gitpod-ready--to--code-blue?logo=gitpod)](https://gitpod.io/#https://github.com/saulpw/visidata)
+# VisiData v2.7 [![twitter @VisiData][1.1]][1] [![CircleCI](https://circleci.com/gh/saulpw/visidata/tree/stable.svg?style=svg)](https://circleci.com/gh/saulpw/visidata/tree/stable) [![Gitpod ready-to-code](https://img.shields.io/badge/Gitpod-ready--to--code-blue?logo=gitpod)](https://gitpod.io/#https://github.com/saulpw/visidata)
 
 A terminal interface for exploring and arranging tabular data.
 

dev/checklists/manual-tests.md

@@ -104,3 +104,10 @@
         - test both panes
 17. Anything new in this release (should it have its own automated test?)
 18. `edit-cell` and then `Ctrl+O` to launch editor.
+19. Save to a non-existent format.
+    - Saves to tsv by default
+    - Save to it a second time, asks for confirmation, if confirm_overwrite=True
+20. Save multiple sheets to a single non-embeddable format
+    - save name makes sense
+    - fails if not offered a directory
+    - succeeds if offered a directory

dev/checklists/release.md

@@ -2,10 +2,6 @@
 
 1. Merge `stable` to `develop` (if necessary)
 
-2. Verify that VisiData is up-to-date:
-    * help menu
-    * plugins.jsonl
-
 2. Verify that documentation/docstrings are up-to-date on features and functionality
 
     a. CHANGELOG;
@@ -14,6 +10,10 @@
 
     c. visidata.org; (formats?)
 
+3. Verify that VisiData is up-to-date:
+    * help menu
+    * plugins.jsonl
+
 3. Ensure `develop` automated tests run correctly with dev/test.sh
 
 4. Go through the manual tests checklist

dev/debian/changelog

@@ -1,3 +1,9 @@
+visidata (2.6.1-1) unstable; urgency=low
+
+  * Update package to 2.6.1
+
+ -- Anja Boskovic <[email protected]>  Tue, 28 Sep 2021 23:20:47 -0700
+
 visidata (2.4-1) unstable; urgency=low
 
   * Update package to 2.4

dev/mkman.sh

@@ -23,3 +23,17 @@ soelim -rt -I $BUILD $BUILD/vd.inc > $BUILD/vd-pre.1
 preconv -r -e utf8 $BUILD/vd-pre.1 > $MAN/vd.1
 preconv -r -e utf8 $BUILD/vd-pre.1 > $MAN/visidata.1
 MANWIDTH=80 man $MAN/vd.1 > $MAN/vd.txt
+
+# build docs/man.md
+
+manhtml="$VD"/docs/man.md
+echo '---' > "$manhtml"
+echo 'eleventyNavigation:' >> "$manhtml"
+echo '  key: Quick Reference Guide' >> "$manhtml"
+echo '  order: 2' >> "$manhtml"
+echo 'permalink: /man/' >> "$manhtml"
+echo '---' >> "$manhtml"
+echo '<section><pre id="manpage" class="whitespace-pre-wrap text-xs">' >> "$manhtml"
+
+MAN_KEEP_FORMATTING=1 COLUMNS=1000 man "$MAN"/vd.1 | ul | aha --no-header >> "$manhtml"
+echo '</pre></section>' >> "$manhtml"

docs/architecture.md

@@ -2,8 +2,8 @@
 eleventyNavigation:
     key: Columns
     order: 5
-update: 2021-06-16
-version: VisiData 2.4
+update: 2021-11-01
+version: VisiData 2.6
 ---
 
 ## How to manipulate columns
@@ -173,8 +173,8 @@ uses the commands for column splitting and transformation with [xd/puzzles.tsv](
 
 ###
 
-- `:` adds new columns derived from splitting the current column at positions defined by a *regex pattern*. The current row will be used to infer the number of columns that will be created.
-- `;` adds new columns derived from pulling the contents of the current column which match the *regex within capture groups*. The new columns are named using the capture group index, or if named capture groups are used, the capture group names. This command also requires an example row.
+- `:` adds new columns derived from splitting the current column at positions defined by a *regex pattern*. `options.default_sample_size` (default: 100) rows around the cursor will be used to determine the number of columns that will be created.
+- `;` adds new columns derived from pulling the contents of the current column which match the *regex within capture groups*. The new columns are named using the capture group index, or if named capture groups are used, the capture group names. This command uses the `options.default_sample_size` (default:100) rows around the cursor as sample rows.
 - `*` followed by *regex*`/`*substring* replaces the text which matches the capture groups in *regex* with the contents of *substring*. *substring* may include backreferences (*\1* etc).
 
 ## [How do I substitute text in my column]
@@ -221,7 +221,7 @@ The following demo shows `(` commands applied to this data:
     <asciinema-player id="player-expand-cols" poster="npt:0:20" rows=13 src="../casts/expand-cols.cast"></asciinema-player>
 </div>
 
-Note that by default the expansion logic will look for nested columns in **up to 1,000 rows surrounding the cursor**. This behavior can be controlled by adjusting `expand_col_scanrows` in the **Options Sheet**, or setting `options.expand_col_scanrows` in the `~/.visidatarc` file.
+Note that by default the expansion logic will look for nested columns in **up to** `options.default_sample_size` (Default: 100) **rows surrounding the cursor**. This behavior can be controlled by adjusting `default_sample_size` in the **Options Sheet**, or setting `options.default_sample_size` in the `~/.visidatarc` file.
 
 ---
 

docs/columns.md

@@ -49,6 +49,8 @@ eleventyNavigation:
 |vcf           |Virtual Contact File \(vCard\)|2\.0             |                |  |1995     |Versit Consortium   |https://tools\.ietf\.org/html/rfc6350|                    |
 |rec           |recutils database file|2\.0             |displayed text  |  |2010     |Jose E\. Marchesi   |https://www\.gnu\.org/software/recutils/|                    |
 |eml           |Multipurpose Internet Mail Extensions \(MIME\)|2\.0             |                |  |1996     |Nathaniel Borenstein and Ned Freed|https://tools\.ietf\.org/html/rfc2045|                    |
+|ods           |OpenDocument Spreadsheet|2\.7             |                |  |2006     |[OASIS](https://en.wikipedia.org/wiki/OASIS_(organization))|https://docs.oasis-open.org/office/v1.1/|odfpy|
+|lsv           |awk-like key-value line-separated values|2\.7             |v2\.7                |  |     |   |   |   |
 
 
 # Extra notes about formats
@@ -97,6 +99,17 @@ eleventyNavigation:
     - `postgres_schema` (default: 'public') the desired schema for the Postgres database
 - `vd postgres://`*username*`:`*password*`@`*hostname*`:`*port*`/`*database* opens a connection to the given postgres database.
 
+### using VisiData as a pager within psql
+
+In psql:
+
+~~~
+\pset format csv
+\pset pager always
+\setenv PSQL_PAGER 'vd -f csv'
+\pset pager_min_lines
+~~~
+
 ## sqlite {#sqlite}
 - supports saving for CREATE/INSERT (not wholesale updates)
 - `z Ctrl+S` to commit any `add-row`/`edit-cell`/`delete-row`
@@ -138,6 +151,20 @@ vd -f pandas data.parquet
 
 loads a parquet file. When using the **pandas** loader, the `.fileformat` file extension is mandatory.
 
+To load a hierarchy of parquet files located in folder `data/`, run
+
+~~~
+vd -f parquet data/
+~~~
+
+or rename the directory to `data.parquet` and run
+
+~~~
+vd data.parquet -f pandas
+~~~
+
+This should similarly work for any format that has a `pandas.read_format()` function.
+
 ## vd {#vd}
 - Command history log format for a VisiData session.
 - `Ctrl+D` to save the current session's CommandLog.