Skip to content

Commit 07683cc

Browse files
khaerukhaeru
authored
Merge pull request #243 from khaeru/enh/csv-2.0
Convert/write SDMX-CSV 2.x
2 parents 59a90ec + d7a0975 commit 07683cc

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

49 files changed

+2200
-1107
lines changed

‎doc/Makefile‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33

44
# You can set these variables from the command line, and also
55
# from the environment for the first two.
6-
SPHINXOPTS ?=
6+
SPHINXOPTS ?= --jobs=auto
77
SPHINXBUILD ?= sphinx-build
88
SOURCEDIR = .
99
BUILDDIR = _build
@@ -12,7 +12,10 @@ BUILDDIR = _build
1212
help:
1313
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
1414

15-
.PHONY: help Makefile
15+
clean-autosummary:
16+
find . -name "$(BUILDDIR)" -prune -o -iname _autosummary -print0 | xargs -0 rm -rf
17+
18+
.PHONY: clean-autosummary help Makefile
1619

1720
# Catch-all target: route all unknown targets to Sphinx using the new
1821
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).

‎doc/api.rst‎

Lines changed: 17 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -7,10 +7,14 @@ Some parts of the API are described on separate pages:
77
:hidden:
88

99
api/model
10+
api/format
11+
api/convert
1012
api/reader
1113
api/writer
1214

1315
- :mod:`sdmx.model`: :doc:`api/model`.
16+
- :mod:`sdmx.format`: :doc:`api/format`.
17+
- :mod:`sdmx.convert`: :doc:`api/convert`.
1418
- :mod:`sdmx.reader`: :doc:`api/reader`.
1519
- :mod:`sdmx.writer`: :doc:`api/writer`.
1620
- :mod:`sdmx.source` on the page :doc:`sources`.
@@ -57,35 +61,6 @@ Top-level methods and classes
5761
:members:
5862
:show-inheritance:
5963

60-
``format``: SDMX file formats
61-
=============================
62-
63-
.. automodule:: sdmx.format
64-
:members:
65-
:exclude-members: Version
66-
:undoc-members:
67-
:show-inheritance:
68-
69-
This information is used across other modules including :mod:`sdmx.reader`,
70-
:mod:`sdmx.client`, and :mod:`sdmx.writer`.
71-
72-
SDMX-JSON
73-
---------
74-
75-
.. automodule:: sdmx.format.json
76-
:members:
77-
78-
SDMX-ML
79-
-------
80-
81-
.. automodule:: sdmx.format.xml
82-
:members:
83-
:exclude-members: validate_xml
84-
85-
.. automodule:: sdmx.format.xml.common
86-
:members:
87-
:exclude-members: install_schemas, validate_xml
88-
8964
``message``: SDMX messages
9065
==========================
9166

@@ -119,9 +94,9 @@ SDMX-ML
11994

12095
``session``: HTTP sessions and responses
12196
========================================
122-
.. autoclass:: sdmx.session.Session
123-
.. autoclass:: sdmx.session.ResponseIO
12497

98+
.. automodule:: sdmx.session
99+
:members: Session, ResponseIO
125100

126101
``urn``: Uniform Resource Names (URNs) for SDMX objects
127102
=======================================================
@@ -133,12 +108,23 @@ SDMX-ML
133108
Utilities and internals
134109
=======================
135110

111+
.. currentmodule:: sdmx.tools
112+
113+
.. automodule:: sdmx.tools
114+
:members:
115+
:show-inheritance:
116+
136117
.. currentmodule:: sdmx.util
137118

138119
.. automodule:: sdmx.util
139120
:members:
140121
:show-inheritance:
141122

123+
.. currentmodule:: sdmx.types
124+
125+
.. automodule:: sdmx.types
126+
:members:
127+
:show-inheritance:
142128

143129
:class:`.DictLike` collections
144130
------------------------------
@@ -149,7 +135,6 @@ Utilities and internals
149135
:members:
150136
:show-inheritance:
151137

152-
153138
Structure expressions in :class:`.Item` descriptions
154139
----------------------------------------------------
155140

‎doc/api/convert.rst‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
Convert from/to SDMX
2+
********************
3+
4+
:mod:`sdmx.convert` and :mod:`sdmx.convert.common`
5+
provide generic and extensible features for converting:
6+
7+
- from :mod:`sdmx.message` and :mod:`sdmx.model` objects
8+
to arbitrary Python data structures, or
9+
- from arbitrary Python data structures to :mod:`sdmx` objects.
10+
11+
The included :mod:`sdmx.convert.pandas`, :class:`.PandasConverter`, and :func:`.to_pandas`
12+
build on these features to provide conversion to types from the :mod:`pandas` package.
13+
This conversion is **not described by the SDMX standards**;
14+
in other words, it is particular to the :mod:`sdmx` package.
15+
16+
In contrast, submodules of :mod:`sdmx.writer`, :func:`.to_csv`, and :func:`.to_xml`
17+
provide conversion to standard :doc:`/api/format`.
18+
19+
User code can also subclass :class:`~.common.Converter` or :class:`.DispatchConverter`
20+
to convert to/from other Python types or non-standard formats.
21+
22+
.. module:: sdmx.convert
23+
24+
.. automodule:: sdmx.convert.common
25+
:members:
26+
27+
``convert.pandas``: Convert to ``pandas`` objects
28+
=================================================
29+
30+
.. currentmodule:: sdmx.convert.pandas
31+
32+
33+
.. autosummary::
34+
PandasConverter
35+
convert_dataset
36+
convert_datamessage
37+
convert_itemscheme
38+
convert_structuremessage
39+
40+
Other objects are converted as follows:
41+
42+
:class:`.Component`
43+
The :attr:`~.Concept.id` attribute of the :attr:`~.Component.concept_identity` is returned.
44+
45+
:class:`.DataMessage`
46+
The :class:`.DataSet` or data sets within the Message are converted to pandas objects.
47+
Returns:
48+
49+
- :class:`pandas.Series` or :class:`pandas.DataFrame`, if `obj` has only one data set.
50+
- list of (Series or DataFrame), if `obj` has more than one data set.
51+
52+
:class:`.dict`
53+
The values of the mapping are converted individually.
54+
If the resulting values are :class:`str` or Series *with indexes that share the same name*, then they are converted to a Series, possibly with a :class:`pandas.MultiIndex`.
55+
Otherwise, a :class:`.DictLike` is returned.
56+
57+
:class:`.DimensionDescriptor`
58+
The :attr:`~.DimensionDescriptor.components` of the DimensionDescriptor are converted.
59+
60+
:class:`list`
61+
For the following *obj*, returns Series instead of a :class:`list`:
62+
63+
- a list of :class:`Observation <.BaseObservation>`:
64+
the Observations are converted using :func:`.convert_dataset`.
65+
- a list with only 1 :class:`DataSet <.BaseDataSet`
66+
(e.g. the :attr:`~.DataMessage.data` attribute of :class:`.DataMessage`):
67+
the Series for the single element is returned.
68+
- a list of :class:`.SeriesKey`: the key values (but no data) are returned.
69+
70+
:class:`.NameableArtefact`
71+
The :attr:`~.NameableArtefact.name` attribute of `obj` is returned.
72+
73+
.. todo::
74+
Support selection of language for conversion of
75+
:class:`InternationalString <sdmx.model.InternationalString>`.
76+
77+
Code reference
78+
--------------
79+
80+
.. automodule:: sdmx.convert.pandas
81+
:members:
82+
:exclude-members: to_pandas

‎doc/api/format.rst‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
.. _format:
2+
3+
Standard formats
4+
****************
5+
6+
The SDMX Information Model provides terms and concepts for data and metadata,
7+
but does not specify how that (meta)data is stored, represented, or serialized.
8+
Other parts of the SDMX standard describe formats for storing data, metadata, and structures.
9+
10+
:mod:`sdmx.format` captures information about these formats,
11+
including their versions and options/parameters.
12+
This information is used across other modules including :mod:`sdmx.reader`,
13+
:mod:`sdmx.client`, and :mod:`sdmx.writer`.
14+
15+
In general, the :mod:`sdmx` package:
16+
17+
- **reads** most :ref:`sdmx-csv`, :ref:`sdmx-json` 1.0, and :ref:`sdmx-ml` messages;
18+
see details in the individual sections below and the linked :mod:`.reader` submodules.
19+
- **writes** certain :ref:`sdmx-csv` 1.0, and :ref:`sdmx-ml` formats;
20+
see details below and the linked :ref:`.writer` submodules.
21+
- is **tested** using collected specimens of messages in various formats,
22+
stored in the `khaeru/sdmx-test-data <https://github.com/khaeru/sdmx-test-data/>`_ Git repository.
23+
These are used to check that the code functions as intended,
24+
but can also be viewed to understand the data formats.
25+
26+
.. automodule:: sdmx.format
27+
:members:
28+
:exclude-members: Version, f
29+
:undoc-members:
30+
:show-inheritance:
31+
32+
.. autosummary::
33+
34+
MEDIA_TYPES
35+
Flag
36+
MediaType
37+
list_media_types
38+
39+
.. _sdmx-csv:
40+
41+
SDMX-CSV
42+
--------
43+
44+
Reference: https://github.com/sdmx-twg/sdmx-csv;
45+
see in particular the file `sdmx-csv-field-guide.md <https://github.com/sdmx-twg/sdmx-csv/blob/v2.0.0/data-message/docs/sdmx-csv-field-guide.md>`_.
46+
47+
Based on Comma-Separated Value (CSV).
48+
The SDMX-CSV *format* is versioned differently from the overall SDMX *standard*:
49+
50+
- `SDMX-CSV 1.0 <https://github.com/sdmx-twg/sdmx-csv/tree/v1.0>`__ corresponds to SDMX 2.1.
51+
It supports only data and metadata, not structures.
52+
SDMX-CSV 1.0 files are recognizable by the header ``DATAFLOW`` in the first column of the first row.
53+
54+
.. versionadded:: 2.9.0
55+
56+
Support for *writing* SDMX-CSV 1.0.
57+
See :mod:`.writer.csv`.
58+
59+
:mod:`sdmx` does not currently support *reading* SDMX-CSV 1.0.
60+
61+
- `SDMX-CSV 2.0.0 <https://github.com/sdmx-twg/sdmx-csv/tree/v2.0.0>`_ corresponds to SDMX 3.0.0.
62+
The format differs from and is not backwards compatible with SDMX-CSV 1.0.
63+
SDMX-CSV 2.0.0 files are recognizable by the header ``STRUCTURE`` in the first column of the first row.
64+
65+
.. versionadded:: 2.19.0
66+
67+
Initial support for *reading* SDMX-CSV 2.0.0.
68+
See :mod:`.reader.csv`.
69+
70+
.. versionadded:: 2.23.0
71+
72+
Initial support for *writing* SDMX-CSV 2.0.0.
73+
See :mod:`.writer.csv`.
74+
75+
.. automodule:: sdmx.format.csv.common
76+
:members:
77+
78+
.. automodule:: sdmx.format.csv.v1
79+
:members:
80+
81+
.. automodule:: sdmx.format.csv.v2
82+
:members:
83+
84+
.. _sdmx-json:
85+
86+
SDMX-JSON
87+
---------
88+
89+
Reference: https://github.com/sdmx-twg/sdmx-json
90+
91+
Based on JavaScript Object Notation (JSON).
92+
The SDMX-JSON *format* is versioned differently from the overall SDMX *standard*:
93+
94+
- SDMX-JSON 1.0 corresponds to SDMX 2.1.
95+
It supports only data and not structures or metadata.
96+
- SDMX-JSON 2.0.0 corresponds to SDMX 3.0.0.
97+
It adds support for structures.
98+
99+
- See :mod:`.reader.json`.
100+
101+
.. versionadded:: 0.5
102+
103+
Support for reading SDMX-JSON 1.0.
104+
105+
.. automodule:: sdmx.format.json
106+
:members:
107+
108+
.. _sdmx-ml:
109+
110+
SDMX-ML
111+
-------
112+
113+
Reference: https://github.com/sdmx-twg/sdmx-ml
114+
115+
Based on eXtensible Markup Language (XML).
116+
SDMX-ML can represent every class and property in the IM.
117+
118+
- An SDMX-ML document contains exactly one :class:`.Message`.
119+
See :mod:`sdmx.message` for the different classes of Messages and their attributes.
120+
- See :mod:`.reader.xml.v21`, :mod:`.reader.xml.v30`, :mod:`.writer.xml`.
121+
122+
.. versionadded:: 2.11.0
123+
124+
Support for reading SDMX-ML 3.0.0.
125+
.. automodule:: sdmx.format.xml
126+
:members:
127+
:exclude-members: validate_xml
128+
129+
.. automodule:: sdmx.format.xml.common
130+
:members:
131+
:exclude-members: install_schemas, validate_xml
132+
133+
Format API
134+
==========
135+
136+
.. automodule:: sdmx.format.common
137+
:members:

0 commit comments

Comments
 (0)