Merge pull request #1127 from saulpw/develop

anjakefala · web-flow · commit dc8cef4980bd · 2021-09-28T22:13:30.000-07:00
v2.6.1 release
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,14 @@
 # VisiData version history
 
+# 2.6.1 (2021-09-28)
+
+## Bugfixes
+
+- [editor] fix sysopen-row (thanks @frosencrantz #1116)
+- [loaders fixed] fix saver (#1123)
+- [loaders shell] fix copy-files
+- [loaders sqlite] fix import error on exception (thanks @jsvine #1125)
+
 # 2.6 (2021-09-19)
 
 ## Major feature
diff --git a/README.md b/README.md
@@ -1,5 +1,5 @@
 
-# VisiData v2.6 [![twitter @VisiData][1.1]][1] [![CircleCI](https://circleci.com/gh/saulpw/visidata/tree/stable.svg?style=svg)](https://circleci.com/gh/saulpw/visidata/tree/stable) [![Gitpod ready-to-code](https://img.shields.io/badge/Gitpod-ready--to--code-blue?logo=gitpod)](https://gitpod.io/#https://github.com/saulpw/visidata)
+# VisiData v2.6.1 [![twitter @VisiData][1.1]][1] [![CircleCI](https://circleci.com/gh/saulpw/visidata/tree/stable.svg?style=svg)](https://circleci.com/gh/saulpw/visidata/tree/stable) [![Gitpod ready-to-code](https://img.shields.io/badge/Gitpod-ready--to--code-blue?logo=gitpod)](https://gitpod.io/#https://github.com/saulpw/visidata)
 
 A terminal interface for exploring and arranging tabular data.
 
diff --git a/dev/checklists/manual-tests.md b/dev/checklists/manual-tests.md
@@ -2,6 +2,7 @@
 1. cmdlog + replay
     - logging of options
     - logging of rows and columns
+    - if empty sheet, row or column on cmdlog, executes command on current sheet, row, column
     - pause, continue, abort
     - test batch mode
         - bin/vd -b -p tests/append.vd
@@ -11,12 +12,14 @@
 5. syscopy
 6. plots and image-loaders (like png)
     - relationship between plots and mice
-7. savers and gSavers
 8. .visidatarc
     - numerical, boolean and string option
     - sheet-specific and global
     - motd_url
 9. using the pandas loader
+    - select
+    - edit
+    - frequency table
 10. large dataset (311)
 11. multiline scrolling tests
     - test case 1
@@ -45,7 +48,7 @@
     - -w and others should be set "globally" (work without -g option)
     - bin/vd -f xlsx sample_data/sample-sales-reps.xlsx -f json sample_data/y77d-th95.json.gz
         - the xlsx sheet should have filetype 'xlsx'
-    - bin/vd sample_data/sample-sales-reps.xlsx -f xlsx
+    - bin/vd sample_data/sample-sales-reps.xlsx -n -f xlsx
         - `o` another file
         - check that it loads
         - check that it does not show 'xlsx' on its sheet-specific options
@@ -55,6 +58,7 @@
     - -f should apply to inner file for zipped filetypes
         - bin/vd -f txt sample_data/y77d-th95.json.gz
 14. Testing the starting position syntax
+    - `bin/vd +:sample-salesv4:: sample_data/sample-sales-reps.xlsx`
 15. Test loading url
 16. Split window
     - make sure that if you exit split window, all the sheets from both panes can be accessible on the resulting stack
@@ -98,3 +102,5 @@
                 - if the second window is a smaller size, should it orient around the cursor position
     - using cursor in active pane
         - test both panes
+17. Anything new in this release (should it have its own automated test?)
+18. `edit-cell` and then `Ctrl+O` to launch editor.
diff --git a/dev/checklists/release.md b/dev/checklists/release.md
@@ -3,7 +3,7 @@
 1. Merge `stable` to `develop` (if necessary)
 
 2. Verify that VisiData is up-to-date:
-    * vdmenu
+    * help menu
     * plugins.jsonl
 
 2. Verify that documentation/docstrings are up-to-date on features and functionality
@@ -14,10 +14,6 @@
 
     c. visidata.org; (formats?)
 
-    d. patrons.tsv;
-
-    e. contributors.tsv
-
 3. Ensure `develop` automated tests run correctly with dev/test.sh
 
 4. Go through the manual tests checklist
@@ -39,6 +35,7 @@
    d. bump version in `__version__` in source code (visidata/main.py, visidata/__init__.py) and setup.py;
 
 6. Run dev/mkman.sh to build the manpage and updated website
+    - Run ./mkmanhtml.sh, and move that to visidata.org:site/docs/man, and to visidata:docs/man.md
 
 7. Push `develop` to testpypi
 
@@ -116,6 +113,7 @@ git push --tags
 16. Upload new motd
 
 17. Update the website by pushing to master. Update with new manpage. Update redirect to point to new manpage.
+    - release notes
 
 18. Comb through issues and close the ones that have been solved, referencing the version number
 
diff --git a/docs/man.md b/docs/man.md
diff --git a/plugins/plugins.jsonl b/plugins/plugins.jsonl
@@ -1,12 +1,12 @@
 {"name": "vfake", "description": "replace column with fake values", "maintainer": "Saul Pwanson @saulpw", "latest_release": "2020-10-06", "url": "https://raw.githubusercontent.com/saulpw/visidata/0d8af5a226818aa2f99a2f74e08543152b22a36d/plugins/vfake.py", "latest_ver": "1.1", "visidata_ver": "2.0", "pydeps": "Faker", "vdplugindeps": "", "sha256": "8153b04a364ac3db2ee310503f762924c4dc472445477af9e027c4d0e921af7f"}
-{"name": "dedupe", "description": "adds commands for selection and removal of rows which are duplicates of prior rows", "maintainer": "Jeremy Singer-Vine @jsvine", "latest_release": "2020-10-11", "url": "https://raw.githubusercontent.com/jsvine/visidata-plugins/37cba82ed2cc49aedc1a377af984acfe1ef3d5cd/plugins/dedupe.py", "latest_ver": "0.1.0", "visidata_ver": "2.0", "pydeps": "", "vdplugindeps": "", "sha256": "2e077b8d62bc8ec2235d22d7d9711d99e7366c8c0ff20405f70647044cca67a1"}
+{"name": "dedupe", "description": "adds commands for selection and removal of rows which are duplicates of prior rows", "maintainer": "Jeremy Singer-Vine @jsvine", "latest_release": "2021-09-22", "url": "https://raw.githubusercontent.com/jsvine/visidata-plugins/0b82e6b7662320a291a846eb4a19899ef8d8b148/plugins/dedupe.py", "latest_ver": "0.2.0", "visidata_ver": "2.0", "pydeps": "", "vdplugindeps": "", "sha256": "d94adcc770349c6ae1d1a63628520558de74bc5e1022b801e29879fc580e571a"}
 {"name": "normcol", "description": "normalizes column names in any given sheet so that the names are unique, valid Python identifiers, and only composed of lowercase letters, numbers, and underscores", "maintainer": "Jeremy Singer-Vine @jsvine", "latest_release": "2020-10-11", "url": "https://raw.githubusercontent.com/jsvine/visidata-plugins/37cba82ed2cc49aedc1a377af984acfe1ef3d5cd/plugins/normcol.py", "latest_ver": "0.1.0", "visidata_ver": "2.0", "pydeps": "", "vdplugindeps": "", "sha256": "ede6b9e508e1ce7842e00d308c372837c9b4d4906b88e0ba62efd5b481f816f6"}
 {"name": "fec", "description": "loader for .fec files from the Federal Election Commission", "maintainer": "Jeremy Singer-Vine @jsvine", "latest_release": "2019-04-21", "url": "https://raw.githubusercontent.com/jsvine/visidata-plugins/37cba82ed2cc49aedc1a377af984acfe1ef3d5cd/plugins/fec.py", "latest_ver": "0.0.0", "visidata_ver": "1.5.2", "pydeps": "fecfile", "vdplugindeps": "", "sha256": "e917170e5bb74ac6242f81113aa48ba7e36763ad32cb190420acb93c3f62765c"}
 {"name": "sparkline", "description": "add a sparkline column to visualise trends of numeric cells in a row", "maintainer": "Lucas Messenger @layertwo", "latest_release": "2020-09-13", "url": "https://raw.githubusercontent.com/saulpw/visidata/e4a373cd74b49b2b91876b17d7a1cd0cd0303848/plugins/sparkline.py", "latest_ver": "0.1", "visidata_ver": "2.0", "pydeps": "", "vdplugindeps": "", "sha256": "4256bea5535405a2478cb6827df4be86f28c69b3cb7f30cc08e2cd655e48b592"}
 {"name": "rownum", "description": "add column of original row ordering", "maintainer": "Saul Pwanson @saulpw", "latest_release": "2019-11-07", "url": "https://raw.githubusercontent.com/saulpw/visidata/dd3aaa4a01e5116c8cd25a30a36bf39756a719b6/plugins/rownum.py", "latest_ver": "0.9", "visidata_ver": "2.0", "pydeps": "", "vdplugindeps": "", "sha256": "41dd84ce3b62504e9c7f40e3ace2285330431d6b46f2f17498ee7e00cb9d8907"}
 {"name": "vmailcap", "description": "add mailcap-view(-selected) commands to DirSheet", "maintainer": "Saul Pwanson @saulpw", "latest_release": "2020-10-06", "url": "https://raw.githubusercontent.com/saulpw/visidata/0d8af5a226818aa2f99a2f74e08543152b22a36d/plugins/vmailcap.py", "latest_ver": "0.9", "visidata_ver": "2.0", "pydeps": "", "vdplugindeps": "", "sha256": "b8aa66821dfa38107ee45fe49a5262e766cc23a0ff362ce5aea5aa6e9b8e6fb0"}
 {"name": "vmutagen", "description": "add mp3 metadata columns to DirSheet", "maintainer": "Saul Pwanson @saulpw", "latest_release": "2019-09-25", "url": "https://raw.githubusercontent.com/saulpw/visidata/v2.-2/plugins/vmutagen.py", "latest_ver": "0.1", "visidata_ver": "2.0", "pydeps": "mutagen", "vdplugindeps": "", "sha256": "de9429e6de46d130f1149d2d8834d8b3bbb55e523acefd735bf101384ebfab07"}
-{"name": "vds3", "description": "add support for Amazon S3 paths", "maintainer": "AJ Kerrigan @ajkerrigan", "latest_release": "2021-08-20", "url": "https://raw.githubusercontent.com/ajkerrigan/visidata-plugins/vds3-0.7/plugins/vds3.py", "latest_ver": "0.7", "visidata_ver": "2.0", "pydeps": "s3fs", "vdplugindeps": "", "sha256": "a8d561fb8569da0b3eb64c795e14293ee6e101833489950099b58e148d0e6d6c"}
+{"name": "vds3", "description": "add support for Amazon S3 paths", "maintainer": "AJ Kerrigan @ajkerrigan", "latest_release": "2021-09-23", "url": "https://raw.githubusercontent.com/ajkerrigan/visidata-plugins/vds3-0.8/plugins/vds3.py", "latest_ver": "0.8", "visidata_ver": "2.0", "pydeps": "s3fs", "vdplugindeps": "", "sha256": "1a52327be9edc87e2b29df5c2cf57903ac0d95625c0c5ccf9cc194f44d3d0325"}
 {"name": "genericSQL", "description": "add loaders for SQL databases (Oracle, MySQL)", "maintainer": "Andrew Swanson @aswan89", "latest_release": "2020-04-15", "url": "https://raw.githubusercontent.com/aswan89/visidata_plugin_genericSQL/1.0.0/generic_sql.py", "latest_ver": "1.0.0", "visidata_ver": "2.-3", "pydeps": "sqlalchemy cx_oracle mysqlclient pyodbc", "vdplugindeps": "", "sha256": "71bbaabaa4ffb973ef7745dc3eadca108b34a7e0f0bf6cf6631eca3fa4532513"}
 {"name": "diff", "description": "adds command to create diff sheets", "maintainer": "Anja Kefala @anjakefala", "latest_release": "2020-10-09", "url": "https://raw.githubusercontent.com/saulpw/visidata/492d558ff4aba55a192965a27ae6499f8d219072/plugins/diff.py", "latest_ver": "0.9", "visidata_ver": "2.0", "sha256": "0085bf8afdc9abf74b7dd52c251f7b7f7befc507aae262bbb656c2ab7379ddd1"}
 {"name": "marks", "description": "adds commands for marking selected rows, and selecting + viewing marked rows", "maintainer": "Saul Pwanson @saulpw", "latest_release": "2020-10-09", "url": "https://raw.githubusercontent.com/saulpw/visidata/54e49feee97a607cce5355bf0e3a79b2466c3d8a/plugins/marks.py", "latest_ver": "0.1", "visidata_ver": "2.0", "sha256": "f15977f327ccc387be208922b66afe4b65666912e8066d02352c53b1aaff941f"}
diff --git a/setup.py b/setup.py
@@ -3,7 +3,7 @@
 from setuptools import setup
 # tox can't actually run python3 setup.py: https://github.com/tox-dev/tox/issues/96
 #from visidata import __version__
-__version__ = '2.6'
+__version__ = '2.6.1'
 
 setup(name='visidata',
       version=__version__,
diff --git a/visidata/__init__.py b/visidata/__init__.py
@@ -1,6 +1,6 @@
 'VisiData: a curses interface for exploring and arranging tabular data'
 
-__version__ = '2.6'
+__version__ = '2.6.1'
 __version_info__ = 'VisiData v' + __version__
 __author__ = 'Saul Pwanson <vd@saul.pw>'
 __status__ = 'Production/Stable'
diff --git a/visidata/_input.py b/visidata/_input.py
@@ -6,7 +6,7 @@
 
 from visidata import EscapeException, ExpectedException, clipdraw, Sheet, VisiData
 from visidata import vd, options, colors
-from visidata import launchExternalEditor, suspend, ColumnItem, AttrDict
+from visidata import suspend, ColumnItem, AttrDict
 
 
 vd.option('color_edit_cell', 'white', 'cell color to use when editing cell')
@@ -215,7 +215,7 @@ def find_nonword(s, a, b, incr):
         elif ch == 'KEY_BTAB':                     v, i = complete_state.complete(v, i, -1)
         elif ch in ['^J', '^M']:                   break  # ENTER to accept value
         elif ch == '^K':                           v = v[:i]  # ^Kill to end-of-line
-        elif ch == '^O':                           v = launchExternalEditor(v)
+        elif ch == '^O':                           v = vd.launchExternalEditor(v)
         elif ch == '^R':                           v = str(value)  # ^Reload initial value
         elif ch == '^T':                           v = delchar(splice(v, i-2, v[i-1]), i)  # swap chars
         elif ch == '^U':                           v = v[i:]; i = 0  # clear to beginning
diff --git a/visidata/editor.py b/visidata/editor.py
@@ -51,9 +51,9 @@ def launchExternalEditor(vd, v, linenum=0):
 def launchExternalEditorPath(path, linenum=0):
         'Launch $EDITOR to edit *path* starting on line *linenum*.'
         if linenum:
-            launchEditor(path, '+%s' % linenum)
+            visidata.vd.launchEditor(path, '+%s' % linenum)
         else:
-            launchEditor(path)
+            visidata.vd.launchEditor(path)
 
         with open(path, 'r') as fp:
             try:
diff --git a/visidata/loaders/fixed_width.py b/visidata/loaders/fixed_width.py
@@ -80,16 +80,18 @@ def save_fixed(vd, p, *vsheets):
             if len(vsheets) > 1:
                 fp.write('%s\n\n' % vs.name)
 
+            widths = {}  # Column -> width:int
             # headers
-            for col in sheet.visibleCols:
-                fp.write('{0:{width}}'.format(col.name, width=col.width))
+            for col in Progress(sheet.visibleCols, gerund='sizing'):
+                widths[col] = col.width or sheet.options.default_width or col.getMaxWidth(sheet.rows)
+                fp.write(('{0:%s} ' % widths[col]).format(col.name))
             fp.write('\n')
 
             # rows
             with Progress(gerund='saving'):
                 for dispvals in sheet.iterdispvals(format=True):
                     for col, val in dispvals.items():
-                        fp.write('{0:{align}{width}}'.format(val, width=col.width, align='>' if vd.isNumeric(col) else '<'))
+                        fp.write(('{0:%s%s.%s} ' % ('>' if vd.isNumeric(col) else '<', widths[col], widths[col])).format(val))
                     fp.write('\n')
 
             vd.status('%s save finished' % p)
diff --git a/visidata/loaders/sqlite.py b/visidata/loaders/sqlite.py
@@ -29,6 +29,8 @@ def execute(self, conn, sql, parms=None):
         return conn.execute(sql, parms)
 
     def iterload(self):
+        import sqlite3
+
         def parse_sqlite_type(t):
             m = re.match(r'(\w+)(\((\d+)(,(\d+))?\))?', t.upper())
             if not m: return anytype
diff --git a/visidata/main.py b/visidata/main.py
@@ -2,7 +2,7 @@
 # Usage: $0 [<options>] [<input> ...]
 #        $0 [<options>] --play <cmdlog> [--batch] [-w <waitsecs>] [-o <output>] [field=value ...]
 
-__version__ = '2.6'
+__version__ = '2.6.1'
 __version_info__ = 'saul.pw/VisiData v' + __version__
 
 from copy import copy
diff --git a/visidata/shell.py b/visidata/shell.py
@@ -236,14 +236,14 @@ def copy_files(sheet, paths, dest):
     destdir = Path(dest)
     destdir.is_dir() or vd.fail('target must be directory')
     vd.status('copying %s %s to %s' % (len(paths), sheet.rowtype, destdir))
+    os.makedirs(destdir, exist_ok=True)
     for srcpath in Progress(paths, gerund='copying'):
         try:
-            if not srcpath.is_dir():
-                destpath = destdir/str(srcpath)
-                os.makedirs(destpath.parent, exist_ok=True)
-                shutil.copyfile(srcpath, destpath)
+            destpath = destdir/str(srcpath._path.name)
+            if srcpath.is_dir():
+                shutil.copy_tree(srcpath, destpath)
             else:
-                os.makedirs(p, exist_ok=True)
+                shutil.copyfile(srcpath, destpath)
         except Exception as e:
             vd.exceptionCaught(e)
 

Original file line number	Diff line number	Diff line change
`@@ -1,5 +1,5 @@`
`1`	`1`
`2`		`-# VisiData v2.6 [![twitter @VisiData][1.1]][1] [![CircleCI](https://circleci.com/gh/saulpw/visidata/tree/stable.svg?style=svg)](https://circleci.com/gh/saulpw/visidata/tree/stable) [![Gitpod ready-to-code](https://img.shields.io/badge/Gitpod-ready--to--code-blue?logo=gitpod)](https://gitpod.io/#https://github.com/saulpw/visidata)`
	`2`	`+# VisiData v2.6.1 [![twitter @VisiData][1.1]][1] [![CircleCI](https://circleci.com/gh/saulpw/visidata/tree/stable.svg?style=svg)](https://circleci.com/gh/saulpw/visidata/tree/stable) [![Gitpod ready-to-code](https://img.shields.io/badge/Gitpod-ready--to--code-blue?logo=gitpod)](https://gitpod.io/#https://github.com/saulpw/visidata)`
`3`	`3`
`4`	`4`	`A terminal interface for exploring and arranging tabular data.`
`5`	`5`