Merge pull request #1565 from nextstrain/curate-internal-quotes

Fix curate internal quotes take 2

Author: joverlee521

CHANGES.md

@@ -2,6 +2,13 @@
 
 ## __NEXT__
 
+### Features
+
+- curate: change output metadata to [RFC 4180 CSV-like TSVs][] to match the TSV format output by other Augur subcommands and the Nextstrain ecosystem as discussed in [#1566][]. [#1565][] (@joverlee521)
+
+[#1565]: https://github.com/nextstrain/augur/pull/1565
+[#1566]: https://github.com/nextstrain/augur/issues/1566
+[RFC 4180 CSV-like TSVs]: https://datatracker.ietf.org/doc/html/rfc4180
 
 ## 26.1.0 (12 November 2024)
 

augur/io/metadata.py

@@ -273,7 +273,8 @@ def visible_worksheet(s: calamine.SheetMetadata) -> bool:
                 # change in a future Python version.
                 raise InvalidDelimiter from error
 
-            metadata_reader = csv.DictReader(handle, dialect=dialect)
+            # Only use the dialect delimiter and keep all other default format params
+            metadata_reader = csv.DictReader(handle, delimiter=dialect.delimiter)
 
             columns, records = metadata_reader.fieldnames, iter(metadata_reader)
 
@@ -549,8 +550,6 @@ def write_records_to_tsv(records, output_file):
             extrasaction='ignore',
             delimiter='\t',
             lineterminator='\n',
-            quoting=csv.QUOTE_NONE,
-            quotechar=None,
         )
         tsv_writer.writeheader()
         tsv_writer.writerow(first_record)

tests/functional/curate/cram/metadata-output-with-internal-quotes.t

@@ -12,18 +12,18 @@ Create NDJSON with internal quotes
   > ~~
 
 Test passthru with output to TSV.
-This should not add any quotes around the field with internal quotes.
+This should add double quotes around the internal quotes to match CSV-like quoting.
 
   $ cat records.ndjson \
   >   | ${AUGUR} curate passthru \
   >     --output-metadata output-metadata.tsv
 
   $ cat output-metadata.tsv
   strain\tsubmitting_lab (esc)
-  sequence_A\tSRC VB "Vector", Molecular Biology of Genomes (esc)
+  sequence_A\t"SRC VB ""Vector"", Molecular Biology of Genomes" (esc)
 
 Run the output TSV through augur curate passthru again.
-The new output should still be identical to the first output.
+The new output should still be identical to the first output because it is already double quoted.
 
   $ ${AUGUR} curate passthru \
   > --metadata output-metadata.tsv \

tests/io/test_metadata.py

@@ -458,7 +458,7 @@ def output_records():
 def expected_output_tsv():
     return (
         "strain\tcountry\tdate\n"
-        'SEQ_A\t"USA"\t2020-10-01\n'
+        'SEQ_A\t"""USA"""\t2020-10-01\n'
         "SEQ_T\tUSA\t2020-10-02\n"
     )