Skip to content

Conversation

@khaeru
Copy link
Owner

@khaeru khaeru commented Sep 22, 2025

As discussed at #242 (comment), this PR adjusts the code layout for conversion to pandas objects and producing SDMX-CSV.

  • Move .writer.pandas to .convert.pandas; refactor to reduce complexity.
    • Use a similar column-handler pattern to .reader.csv. This could later be unified.
  • Simplify .writer.csv.
  • Add new classes to describe the different SDMX-CSV formats and their respective options.
  • The new PandasConverter class has a format=… attribute/argument.
    • If this is set to one of the CSV formats, then valid SDMX-CSV is produced.
    • Otherwise, a variety of pandas types and layouts are produced, according to the other/existing options.
  • Deprecate the sdmx.to_pandas(…, rtype="compat") argument (though still works). Add tests.
  • Deprecate the sdmx.to_pandas(…, datetime=…) argument (though still works); instead of 1 object that takes a variety of complex values, 3 separate arguments are exposed directly.

So making use of the above changes:

  • SDMX-CSV 2.0.0 can now be written.
  • The labels="both" parameter is supported when converting to SDMX-CSV 1.0 or SDMX-CSV 2.0.0.

PR checklist

  • Checks all ✅
  • Update documentation
  • Update doc/whatsnew.rst

- Make .writer.base.BaseWriter a subclass of Converter.
@khaeru khaeru self-assigned this Sep 22, 2025
@khaeru khaeru added the enh Enhancements & new features label Sep 22, 2025
@codecov
Copy link

codecov bot commented Sep 22, 2025

Codecov Report

❌ Patch coverage is 99.58735% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 98.24%. Comparing base (59a90ec) to head (d7a0975).
⚠️ Report is 24 commits behind head on main.

Files with missing lines Patch % Lines
sdmx/tools.py 91.30% 2 Missing ⚠️
sdmx/convert/pandas.py 99.73% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #243      +/-   ##
==========================================
- Coverage   98.91%   98.24%   -0.67%     
==========================================
  Files         105      113       +8     
  Lines        8910     9297     +387     
==========================================
+ Hits         8813     9134     +321     
- Misses         97      163      +66     
Files with missing lines Coverage Δ
sdmx/__init__.py 100.00% <100.00%> (ø)
sdmx/compare.py 100.00% <ø> (ø)
sdmx/convert/__init__.py 100.00% <100.00%> (ø)
sdmx/convert/common.py 100.00% <100.00%> (ø)
sdmx/dictlike.py 97.64% <100.00%> (-0.03%) ⬇️
sdmx/format/__init__.py 100.00% <ø> (ø)
sdmx/format/common.py 100.00% <100.00%> (ø)
sdmx/format/csv/common.py 100.00% <100.00%> (ø)
sdmx/format/csv/v1.py 100.00% <100.00%> (ø)
sdmx/format/csv/v2.py 100.00% <100.00%> (ø)
... and 20 more

... and 19 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

khaeru added a commit that referenced this pull request Sep 23, 2025
@khaeru khaeru changed the title Prepare for SDMX-CSV 2.x writer Convert/write SDMX-CSV 2.x Sep 23, 2025
khaeru added a commit that referenced this pull request Sep 23, 2025
khaeru added a commit that referenced this pull request Sep 24, 2025
Ensure BaseMetadataSet appears in the MRO before lower-level classes.
- Add enumerations for common and version-specific format parameters.
- Add FormatOptions.
- Reimplement .pandas.writer as PandasConverter, a DispatchConverter.
- Document conversion options at the class level.
- Adjust tests.
- Add Attributes enumeration.
- Add format, format_options parameters to PandasConverter.
- Separate handling of complex datetime=… parameter to a dedicated
  method.
- Add DATAFLOW column for SDMX-CSV 1.0.
- Don't create an instance of BaseWriter.
- Collapse dataset() into the main method.
- Drop code for creating DATAFLOW column, now handled by
  .convert.pandas.
- Ensure Key.copy() returns same type for subclasses.
- Add ColumnSpec to handle column info for conversion.
- Deprecate datetime=…, rtype=… attributes/arguments; add warnings.
- New methods PandasConverter.{get_components,handle_compat,
  handle_datetime}().
- Use pandas method chaining and .pipe() functions in convert_dataset().
khaeru added a commit that referenced this pull request Sep 24, 2025
khaeru added a commit that referenced this pull request Sep 24, 2025
- Move .writer.pandas docs to new page doc/api/convert.rst.
- Move .format docs to new page doc/api/format.rst.
- Use consistent page and section titles.
- Copyedit descriptive text and cross-link module descriptions.
khaeru added a commit that referenced this pull request Sep 24, 2025
- Move .writer.pandas docs to new page doc/api/convert.rst.
- Move .format docs to new page doc/api/format.rst.
- Use consistent page and section titles.
- Copyedit descriptive text and cross-link module descriptions.
- Move .writer.pandas docs to new page doc/api/convert.rst.
- Move .format docs to new page doc/api/format.rst.
- Use consistent page and section titles.
- Copyedit descriptive text and cross-link module descriptions.
@khaeru khaeru added writer Convert objects to other formats doc Documentation, help, examples, etc. csv SDMX-CSV format labels Sep 24, 2025
@khaeru khaeru merged commit 07683cc into main Sep 24, 2025
20 checks passed
khaeru added a commit that referenced this pull request Sep 24, 2025
@khaeru khaeru mentioned this pull request Sep 24, 2025
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

csv SDMX-CSV format doc Documentation, help, examples, etc. enh Enhancements & new features writer Convert objects to other formats

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants