Merge pull request #227 from khaeru/enh/2025-w08

Miscellaneous improvements from 2025-W08

Author: khaeru

doc/api/model-common-list.rst

@@ -3,8 +3,8 @@
 :obj:`~.common.ActionType`
 :obj:`~.common.Agency`
 :obj:`~.common.AgencyScheme`
+:obj:`~.common.AllDimensions`
 :obj:`~.common.AnnotableArtefact`
-:obj:`~.common.Annotation`
 :obj:`~.common.AttributeDescriptor`
 :obj:`~.common.AttributeRelationship`
 :obj:`~.common.AttributeValue`
@@ -77,6 +77,7 @@
 :obj:`~.common.ToVTLSpaceKey`
 :obj:`~.common.Transformation`
 :obj:`~.common.TransformationScheme`
+:obj:`~.common.UsageStatus`
 :obj:`~.common.UserDefinedOperator`
 :obj:`~.common.UserDefinedOperatorScheme`
 :obj:`~.common.VTLConceptMapping`

doc/api/model-v21-list.rst

@@ -1,6 +1,7 @@
 .. This file is auto-generated by doc/conf.py.
 
 :obj:`~.v21.AfterPeriod`
+:obj:`~.v21.Annotation`
 :obj:`~.v21.BeforePeriod`
 :obj:`~.v21.CodeMap`
 :obj:`~.v21.CodelistMap`

doc/api/model-v30-list.rst

@@ -1,7 +1,9 @@
 .. This file is auto-generated by doc/conf.py.
 
 :obj:`~.v30.AfterPeriod`
+:obj:`~.v30.Annotation`
 :obj:`~.v30.BeforePeriod`
+:obj:`~.v30.CodeSelection`
 :obj:`~.v30.CodedMetadataAttributeValue`
 :obj:`~.v30.CodelistExtension`
 :obj:`~.v30.Constraint`
@@ -12,6 +14,7 @@
 :obj:`~.v30.DataStructureDefinition`
 :obj:`~.v30.Dataflow`
 :obj:`~.v30.DataflowRelationship`
+:obj:`~.v30.ExclusiveCodeSelection`
 :obj:`~.v30.GeoCodelist`
 :obj:`~.v30.GeoFeatureSetCode`
 :obj:`~.v30.GeoGridCode`
@@ -21,6 +24,7 @@
 :obj:`~.v30.Hierarchy`
 :obj:`~.v30.HierarchyAssociation`
 :obj:`~.v30.IdentifiableObjectSelection`
+:obj:`~.v30.InclusiveCodeSelection`
 :obj:`~.v30.Measure`
 :obj:`~.v30.MeasureDescriptor`
 :obj:`~.v30.MeasureRelationship`

doc/example.rst

@@ -24,11 +24,11 @@ These include *structural metadata* that together completely describe the data a
     sm
 
 :py:`sm` is a Python object of class :class:`.StructureMessage`.
-We can explore some of the specific artifacts—for example, two **code lists**—using :meth:`.StructureMessage.get` to retrieve them and :func:`.to_pandas` to convert to :class:`.pandas.Series`:
+We can explore some of the specific artifacts—for example, three **code lists**—using :meth:`.StructureMessage.get` to retrieve them and :func:`.to_pandas` to convert to :class:`.pandas.Series`:
 
 .. ipython:: python
 
-    for cl in "AGE", "SEX":
+    for cl in "AGE(10.3)", "SEX(1.13)", "UNIT(55.0)":
         print(sdmx.to_pandas(sm.get(cl)))
 
 Next, we download a **data set** containing a portion of the data in this data flow, structured by this DSD.

doc/implementation.rst

@@ -4,37 +4,61 @@ Implementation notes
 :mod:`sdmx` aims for a **precise, Pythonic, and useful implementation of the SDMX standards**.
 This means:
 
-- Classes and their attributes have the same names, types, cardinality that appear in the standard.
+- Classes and their attributes have the names, types, cardinality, and directionality given in the standard.
 
-  - Where the standard uses non-Pythonic naming conventions (for instance, "dimensionAtObservation"), :mod:`sdmx` follows the `PEP-8 naming conventions <https://peps.python.org/pep-0008/#naming-conventions>`_ (for instance, "dimension_at_observation" for an attribute).
+  - Where the standard has non-Pythonic names (for instance, "dimensionAtObservation"), :mod:`sdmx` follows the `PEP-8 naming conventions <https://peps.python.org/pep-0008/#naming-conventions>`_ (for instance, "dimension_at_observation" for a class attribute).
   - Where the standard is ambiguous or imprecise itself, implementation (for instance, naming) choices in :mod:`sdmx` are clearly labelled.
 
 - Extensions, additional features, and conveniences in :mod:`sdmx` that do not appear in the standards are clearly labeled.
-- All behaviour visible “in the wild”—that is, from publicly available data sources and files—is supported, so long as it is verifiably standards compliant.
+- All behaviour visible “in the wild”—that is, from publicly available data sources and files—is either:
+
+  - supported, if it is verifiably standards-compliant, or
+  - tolerated if otherwise, so long as this does not complicate the implementation of standards.
+    Non-standard content is flagged using log messages and by other means.
 
 This page gives brief explanations of **how** this implementation is achieved.
 Although this page is organized (see the contents in the sidebar) to correspond to the standards, it (:ref:`again <not-the-standard>`) **does not restate them** or set out to explain all their details.
 For those purposes, see :doc:`resources`; or the :doc:`walkthrough`, which includes some incidental explanations.
 
 .. _sdmx-version-policy:
 
-SDMX standard versions 2.0, 2.1, and 3.0
-========================================
+Standards versions
+==================
 
+SDMX standards documents are available `on the SDMX website <https://sdmx.org/?page_id=5008>`__ and via links to other locations.
 Multiple versions of the SDMX standards have been adopted:
 
 - 2.0 in November 2005.
 - 2.1 in August 2011; published at the International Standards Organization (ISO) in January 2013; and revised multiple times since.
-- 3.0 in October 2021.
-
-The standards are available on the SDMX website: https://sdmx.org/?page_id=5008
-
-- In SDMX 2.1, sections of the standards were numbered from 1 to 7.
-  For instance, the :ref:`im` is Section 2.
-- From SDMX 3.0, some of these section numbers have been removed.
-  For instance, SDMX-ML was described in SDMX 2.1 sections 3A and 3B; in SDMX 3.0, these section numbers are no longer used, replaced with a reference to the SDMX Technical Working Group (TWG) Git repository at https://github.com/sdmx-twg/sdmx-ml .
-- Some of these sections or sub-standards are versioned differently from the overall standard.
-  See in particular :ref:`sdmx-csv`, :ref:`sdmx-json`, and :ref:`sdmx-rest` below.
+- 3.0.0 in October 2021.
+- 3.1 planned for some time in 2025.
+
+Some notes about the organization of the standards:
+
+- In SDMX 2.1, **‘sections’** of the standards were numbered from 1 to 7.
+  For instance, the :ref:`im` is referred to as “Section 2”.
+
+  - This is distinct from numbered sections or headings *within* the particular standards documents.
+    For instance, SDMX 2.1 Section 2 “Information Model/UML Conceptual Design” is a document that contains numbered sections/headings such as “2. Actors and Use Cases” and “2.1. Introduction.”
+    Documentation and code comments for the :mod:`sdmx` package attempts to be unambiguous about such references, and uses the “§” character to refer to these (sub-)headings.
+- From SDMX 3.0.0, some of these section numbers have been removed or disused.
+  For instance, the SDMX-ML file format was described in SDMX 2.1 sections 3A and 3B; in SDMX 3.0.0, these section numbers are no longer used, replaced with a reference to the SDMX Technical Working Group (TWG) Git repository at https://github.com/sdmx-twg/sdmx-ml.
+- Some of the sections or component standards are versioned differently from SDMX as a whole.
+  The following table lists the *apparent* correspondences between versions of the component standards (The SDMX TWG does not publish such a table, so this should not be taken as official):
+
+  ======== ========================== =============== =============== ==========================
+  SDMX     SDMX-REST                  SDMX-CSV        SDMX-JSON       SDMX-ML
+  ======== ========================== =============== =============== ==========================
+  1.0      (not versioned separately) (did not exist) (did not exist) (not versioned separately)
+  2.0      (not versioned separately) (did not exist) (did not exist) (not versioned separately)
+  2.1      1.x; latest 1.5            1.0             1.0             (not versioned separately)
+  3.0.0    2.x; latest 2.2            2.0.0           2.0.0           3.0.0
+  3.1      ?                          ?               ?               ?
+  ======== ========================== =============== =============== ==========================
+
+  See further details under the :ref:`sdmx-csv`, :ref:`sdmx-json`, and :ref:`sdmx-rest` sections, below.
+- The version numbers `do not <https://github.com/sdmx-twg/sdmx-3_1_0/issues/1#issuecomment-2519837607>`_ follow the `semantic versioning <https://semver.org>`_ system.
+  This means that increments to the second (3.0 → 3.1) or first (3.1 → 4.0) version part do not necessarily indicate the presence/absence of 'breaking' or backwards-incompatible changes.
 
 For the current Python package, :mod:`sdmx`:
 
@@ -45,20 +69,20 @@ For the current Python package, :mod:`sdmx`:
     This makes it more difficult and costly to support them.
   - While no SDMX 2.0 implementation is planned, contributions from new developers are possible and welcome.
 
-- **SDMX 2.1 and 3.0** are implemented as described on this page, with exhaustive implementation as the design goal for :mod:`sdmx`.
-- For **SDMX 3.0** specifically, as of v2.14.0 :mod:`sdmx` implements:
+- **SDMX 2.1 and 3.0.0** are implemented as described on this page, with exhaustive implementation as the design goal for :mod:`sdmx`.
+- For **SDMX 3.0.0** specifically, as of v2.14.0 :mod:`sdmx` implements:
 
-  - The SDMX 3.0 information model (:mod:`.model.v30`), to the same extent as SDMX 2.1.
-  - Reading of SDMX-ML 3.0 (:mod:`.reader.xml.v30`).
+  - The SDMX 3.0.0 information model (:mod:`.model.v30`), to the same extent as SDMX 2.1.
+  - Reading of SDMX-ML 3.0.0 (:mod:`.reader.xml.v30`).
   - Construction of URLs and querying SDMX-REST API v2.1.0 data sources (:mod:`.rest.v30`).
 
   This implies the following are not yet supported:
 
-  - Writing SDMX-ML 3.0.
+  - Writing SDMX-ML 3.0.0.
   - Reading and writing SDMX-JSON 2.0 (see :ref:`sdmx-json`).
 
-  Follow the :doc:`whatsnew`; :issue:`87`; and other GitHub issues and pull requests for details.
-  Please `open an issue <https://github.com/khaeru/sdmx/issues>`_ on GitHub to report examples of real-world SDMX 3.0 web services examples and specimens of data that can be added.
+  Follow the :doc:`whatsnew` and GitHub issues and pull requests with the `'sdmx-3' label <https://github.com/khaeru/sdmx/labels/sdmx-3>`__ for details.
+  Please `open an issue <https://github.com/khaeru/sdmx/issues>`_ on GitHub to report examples of real-world SDMX 3.0.0 web services examples and specimens of data that can be added.
 
 .. _im:
 
@@ -68,14 +92,14 @@ Information model (SDMX-IM)
 Reference:
 
 - `SDMX 2.1 Section 2 — Information Model <https://sdmx.org/wp-content/uploads/SDMX_2-1-1_SECTION_2_InformationModel_201108.pdf>`_ (PDF).
-- `SDMX 3.0 Section 2 — Information Model <https://sdmx.org/wp-content/uploads/SDMX_3-0-0_SECTION_2_FINAL-1_0.pdf>`_ (PDF).
+- `SDMX 3.0.0 Section 2 — Information Model <https://sdmx.org/wp-content/uploads/SDMX_3-0-0_SECTION_2_FINAL-1_0.pdf>`_ (PDF).
 
 In general:
 
 - :mod:`sdmx.model.common` implements:
 
-  1. Classes that are fully identical in the SDMX 2.1 and 3.0 information models.
-  2. Base classes like :class:`.BaseDataStructureDefinition` that contain **common attributes and features** shared by SDMX 2.1 and 3.0 classes that differ in some ways.
+  1. Classes that are fully identical in the SDMX 2.1 and 3.0.0 information models.
+  2. Base classes like :class:`.BaseDataStructureDefinition` that contain **common attributes and features** shared by SDMX 2.1 and 3.0.0 classes that differ in some ways.
      These classes should not be instantiated or used directly, except for type checking and hinting.
 
 - :mod:`sdmx.model.v21` and :mod:`sdmx.model.v30` contain:
@@ -254,7 +278,7 @@ The IM provides terms and concepts for data and metadata, but does not specify h
 The SDMX standards include multiple formats for storing data, metadata, and structures.
 In general, :mod:`sdmx`:
 
-- Reads most SDMX-ML 2.1 and 3.0 and SDMX-JSON 1.0 messages.
+- Reads most SDMX-ML 2.1 and 3.0.0 and SDMX-JSON 1.0 messages.
 - Uses collected specimens of messages in various formats, stored in the `khaeru/sdmx-test-data <https://github.com/khaeru/sdmx-test-data/>`_ Git repository.
   These are used by the test suite to check that the code functions as intended, but can also be viewed to understand the data formats.
 
@@ -272,7 +296,7 @@ SDMX-ML can represent every class and property in the IM.
 
 .. versionadded:: 2.11.0
 
-   Support for reading SDMX-ML 3.0.
+   Support for reading SDMX-ML 3.0.0.
 
 .. _sdmx-json:
 
@@ -345,17 +369,17 @@ These generally elaborate the SDMX standards, but in some cases also document so
 The SDMX-REST *web service API* is versioned differently from the overall SDMX *standard*:
 
 - SDMX-REST API v1.5.0 and earlier corresponding to SDMX 2.1 and earlier.
-- SDMX-REST API v2.0.0 and later corresponding to SDMX 3.0 and later.
+- SDMX-REST API v2.0.0 and later corresponding to SDMX 3.0.0 and later.
 
 :mod:`sdmx` aims to support:
 
 - SDMX-REST API versions in the 1.x series from v1.5.0 and later
 - SDMX-REST API versions in the 2.x series from v2.1.0 and later.
-- Data retrieved in SDMX 2.1 and 3.0 :ref:`formats <formats>`.
+- Data retrieved in SDMX 2.1 and 3.0.0 :ref:`formats <formats>`.
   Some existing services offer a parameter to select SDMX 2.1 *or* 2.0 format; :mod:`sdmx` does not support the latter.
   Other services *only* provide SDMX 2.0-formatted data; these cannot be used with :mod:`sdmx` (:ref:`see above <sdmx-version-policy>`).
 
-:class:`.Client` constructs valid URLs (using :class:`~.rest.URL` subclasses :class:`.v21.URL` and :class:`.v30.URL`).
+:class:`.Client` constructs valid URLs using the :class:`~.rest.URL` subclasses :class:`.v21.URL` and :class:`.v30.URL`.
 
 - For example, :meth:`.Client.get` automatically adds the HTTP header ``Accept: application/vnd.sdmx.structurespecificdata+xml;`` when a :py:`structure=...` argument is provided and the data source supports this content type.
 - :class:`.v21.URL` supplies some default parameters in certain cases.

doc/whatsnew.rst

@@ -3,10 +3,70 @@
 What's new?
 ***********
 
-.. _2.21.1:
+.. _2.22.0:
+
+Next release
+============
+
+Migration notes
+---------------
+
+- Modify code that imports :class:`~.v21.Annotation` from :mod:`sdmx.model.common` to import from either :mod:`sdmx.model.v21` or :mod:`sdmx.model.v30`, as appropriate.
+  For example, instead of:
+
+  .. code-block:: python
+
+     from sdmx.model.common import Annotation
+
+     a = Annotation(id="FOO", ...)
+
+  …do:
 
-.. Next release
-.. ============
+  .. code-block:: python
+
+     from sdmx.model.v21 import Annotation
+
+     a = Annotation(id="FOO", ...)
+- Adjust code that accesses :class:`.ReportStructure` via the :attr:`v21.MetadataSet.described_by` attribute:
+
+  1. To access ReportStructure, use the new :attr:`~.v21.MetadataSet.report_structure` attribute.
+  2. To access :class:`.MetadataStructureDefinition`, use :attr:`~v21.MetadataSet.described_by`.
+
+All changes
+-----------
+
+- :meth:`StructureMessage.get` handles full and partial :class:`URNs <URN>` (:pull:`227`).
+- :class:`.v21.Annotation` and :class:`.v30.Annotation` are derived from :class:`.common.BaseAnnotation` (:pull:`227`).
+  This allows to reflect that the latter has an attribute, :attr:`.v30.Annotation.value`, that the former does not.
+  This is a change in the SDMX 3.0.0 Information Model that is not mentioned in the “Summary of major changes and new functionality” or IM document.
+
+  Code like :py:`from sdmx.model.common import Annotation` now emits :class:`DeprecationWarning`, and in the future will raise :class:`ImportError`.
+- :func:`.validate_xml` now supports :xml:`<com:StructuredText>` elements representing, for instance, :class:`.XHTMLAttributeValue` (:pull:`227`).
+  A new function :func:`.construct_schema` modifies the official SDMX-ML schemas to insert an import of the `XML Schema for XHTML 1.0 <https://www.w3.org/TR/xhtml1-schema/>`_, allowing to validate the XHTML content within these elements.
+- Improve :mod:`.model` (:pull:`227`):
+
+  - :class:`.IdentifiableArtefact` is comparable with :class:`str` via its :attr:`~.IdentifiableArtefact.id`.
+    This means that :func:`sorted` can be used with mixed collections of these two types.
+  - :attr:`.Structure.grouping` now returns a list of :class:`.ComponentList`.
+    In :mod:`sdmx` v2.21.1 and earlier, this list would include a :class:`dict` of 0 or more :class:`.GroupDimensionDescriptor`, keyed by the ID of each.
+    Now, each group dimension descriptor is directly an item in the list.
+  - :attr:`.v21.MetadataSet.report_structure` is added and distinguished from :attr:`~.v21.MetadataSet.described_by`.
+    This works around an issue in the SDMX 2.1 IM; see the class docstring for details.
+  - New convenience methods :meth:`.MetadataReport.get`, :meth:`.MetadataReport.get_value`, and :meth:`.ReportedAttribute.get_child`.
+
+- Improve reading and writing of SDMX-ML (:pull:`227`):
+
+  - Read :xml:`<str:AnnotationValue>` in SDMX-ML 3.0.0 (:issue:`226`).
+  - Read :xml:`<str:Hierarchy>` where the optional :xml:`<... leveled="...">` attribute is not present (:issue:`226`).
+  - Read and write XSD-valid :class:`.v21.MetadataSet` and :class:`.v21.HierarchicalCodelist`.
+  - Write :attr:`.Dimension.concept_role`.
+  - Write annotations associated with :class:`DataSet <.BaseDataSet>`, :class:`MetadataSet <.BaseMetadataSet>`, and :class:`.MetadataReport`.
+  - Pending resolution of :issue:`228`, ignore :xml:`<com:Link>` in SDMX-ML 3.0.0 .
+
+- Update and expand :ref:`sdmx-version-policy` in the documentation (:pull:`227`).
+  A table is now included showing the correspondence of versions of component SDMX standards.
+
+.. _2.21.1:
 
 v2.21.1 (2025-01-14)
 ====================