Use more durable IDs for IMF_Beta and IMF_Beta3 (#225)

- Update sources.json
- Add .source.imf_data{,3} modules.
  - Add headers for all requests.
- Update doc/sources.rst for IMF_DATA{,3}.
  - Copyedit IMF_DATA{,3} documentation.
- Update test_sources.py.
  - Add tests for other artifacts.
- Fix handling of <str:Codelist>URN… in SDMX-ML 3.0
- Add #225 to doc/whatsnew

---------

Co-authored-by: Paul Natsuo Kishimoto <[email protected]>

Author: aboddie

.github/workflows/sources.yaml

@@ -38,8 +38,8 @@ jobs:
         - GROW
         - ILO
         - IMF
-        - IMF_beta
-        - IMF_beta3
+        - IMF_DATA
+        - IMF_DATA3
         - INEGI
         - INSEE
         - ISTAT

doc/sources.rst

@@ -312,13 +312,15 @@ API documentation `1 <https://datahelp.imf.org/knowledgebase/articles/1952905-sd
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 SDMX-ML —
-`Website <https://sdmxcentral.imf.org/>`__
+`Website <https://sdmxcentral.imf.org/>`__ —
+`API documentation <https://dsbb.imf.org/content/Pdfs/IMF%20SDMX%20Central%20Web%20Services%20Guide%20Published%2010_17_2019.pdf>`__
 
-- This appears to be an instance of the “Fusion Metadata Registry” software.
+- This source does not contain data and should only be used to query structures.
+- This is an instance of the “Fusion Metadata Registry” software.
   Such instances also expose SDMX 2.1 and 3.0 APIs.
-- No API documentation appears to be available.
 - The :mod:`sdmx` source with ID ``IMF`` corresponds to the SDMX 2.1 (SDMX-REST 1.x) API with base URL https://sdmxcentral.imf.org/ws/public/sdmxapi/rest.
-  The web interface suggests URLs for the SDMX 3.0.0 (SDMX-REST 2.x) API with base URL https://sdmxcentral.imf.org/sdmx/v2.
+
+  The web interface suggests URLs for an SDMX 3.0.0 (SDMX-REST 2.x) API with base URL https://sdmxcentral.imf.org/sdmx/v2.
   This API can be accessed by modifying the :attr:`.Source.url` and :attr:`~.Source.versions` attributes, or by constructing a new Source.
   For example:
 
@@ -331,32 +333,30 @@ SDMX-ML —
      client.source.url = "https://sdmxcentral.imf.org/sdmx/v2"
      client.source.versions = {Version["3.0.0"]}
 
-     # Retrieve an SDMX-ML 3.0.0 structure message
-     message = client.dataflow("01R")
-
-- The source appears to provide a subset of the data available on https://data.imf.org.
-- Supports series-key-only and hence dataset-based key validation and construction.
-
-``IMF_beta``, ``IMF_beta3``: api.imf.org
+``IMF_DATA``, ``IMF_DATA3``: api.imf.org
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 SDMX-ML —
-`Website <https://betadata.imf.org>`__ —
-`API documentation <https://betadata.imf.org/en/Resource-Pages/IMF-API>`__
+Website `(main) <https://data.imf.org>`__, `(beta) <https://betadata.imf.org>`__ —
+API documentation `(main) <https://data.imf.org/en/Resource-Pages/IMF-API>`__, `(beta) <https://betadata.imf.org/en/Resource-Pages/IMF-API>`__.
 
-.. warning:: As of 2025-01-10, this source carries a banner:
+.. note:: As of **2025-01-10**, this source carries a banner:
 
-       We're in Beta!
-       Help us improve by `testing <https://datasupport.imf.org/knowledge?id=kb_article_view&sys_kb_id=372b9c5493019610102cf4647aba1015&category_id=4e49be7c1b6391903dba646fbd4bcb00>`__ and sharing `feedback <https://forms.office.com/pages/responsepage.aspx?id=Q_qFgC4wvUWxcaZkjDtr54N7EnsUWMNKll1Zs-zgwh9UODA5MTFBVlA1MDFaWEpIMFVaSE83TzJYTy4u&route=shorturl>`__.
-       This is a beta version; the data is not final and should not be used for actual work.
+      We're in Beta!
+      Help us improve by `testing <https://datasupport.imf.org/knowledge?id=kb_article_view&sys_kb_id=372b9c5493019610102cf4647aba1015&category_id=4e49be7c1b6391903dba646fbd4bcb00>`__ and sharing `feedback <https://forms.office.com/pages/responsepage.aspx?id=Q_qFgC4wvUWxcaZkjDtr54N7EnsUWMNKll1Zs-zgwh9UODA5MTFBVlA1MDFaWEpIMFVaSE83TzJYTy4u&route=shorturl>`__.
+      This is a beta version; the data is not final and should not be used for actual work.
 
    Users should heed this message.
-   The source IDs used in :mod:`sdmx` may change if and when this source exits beta and enters production, or is designated as the recommended, primary, or sole IMF source.
 
-- The API documentation indicates "Our data are available through SDMX 2.1 and SDMX 3.0 APIs," but the documentation pages mention only the SDMX 2.1 (SDMX-REST 1.x) base URL, https://api.imf.org/external/sdmx/2.1.
-  The base URL used by :mod:`sdmx` for the SDMX 3.0 (SDMX-REST 2.x) API is inferred.
-- :mod:`sdmx` provides access to both versions of the API with IDs ``IMF_beta`` and ``IMF_beta3``.
-  As of 2025-01-10, both return HTTP **403 Forbidden** to every request except the SDMX 2.1 data query illustrated in the API documentation.
+   `This documentation page <https://datasupport.imf.org/knowledge?id=knowledge_category&sys_kb_id=967337049388ea50102cf4647aba1024&category_id=4e49be7c1b6391903dba646fbd4bcb00>`__ states that the “IMF Data portal” associated with the above URL endpoints “will launch at the end of Q1 2025,” thus **on or about 2025-03-31**.
+   At that point, the ‘(main)’ links above will point to the new portal, and the ‘(beta)’ links will no longer be accessible.
+
+- The ``IMF_DATA`` source points to an SDMX-REST 1.x API endpoint that serves SDMX-ML 2.1.
+  This endpoint does not support access of :class:`.v21.MetadataSet`, :class:`.v21.MetadataflowDefinition`, or :class:`.v21.MetadataStructureDefinition`.
+
+- The ``IMF_DATA3`` source points to an SDMX-REST 2.x API endpoint that serves SDMX-ML 3.0.0 and SDMX-JSON 2.0.0.
+  Only the former is supported by :mod:`sdmx`.
+  This endpoint does not support access of :class:`.HierarchicalCodelist`.
 
 .. _INEGI:
 

doc/whatsnew.rst

@@ -63,6 +63,7 @@ All changes
   - Write annotations associated with :class:`DataSet <.BaseDataSet>`, :class:`MetadataSet <.BaseMetadataSet>`, and :class:`.MetadataReport`.
   - Pending resolution of :issue:`228`, ignore :xml:`<com:Link>` in SDMX-ML 3.0.0 .
 
+- Rename :ref:`IMF_beta, IMF_beta3 <IMF>` data sources to :ref:`IMF_DATA, IMF_DATA3 <IMF>` and update documentation on 3 distinct IMF-run web services (thanks :gh-user:`aboddie` for :pull:`225` and :issue:`224`).
 - Update and expand :ref:`sdmx-version-policy` in the documentation (:pull:`227`).
   A table is now included showing the correspondence of versions of component SDMX standards.
 

sdmx/reader/xml/v30.py

@@ -92,14 +92,22 @@ def _link(reader, elem) -> None:
 
 @end("str:Codelist")
 def _cl(reader, elem):
-    try:
-        sdmx.urn.match(elem.text)
-    except ValueError:
-        result = v21._itemscheme(reader, elem)
-        result.extends = reader.pop_all(v30.CodelistExtension)
-        return result
-    else:
-        reader.push(elem, elem.text)
+    # Handle <str:Codelist>urn:…</str:Codelist>; occurs within <str:CodelistExtension>
+    if len(elem) == 0 and elem.text:
+        try:
+            # Validate the contained URN, return to str
+            urn = str(sdmx.urn.URN(elem.text))
+        except Exception:
+            pass  # Malformed, or not a URN
+        else:
+            reader.push(elem, urn)
+            return
+
+    # Use the .v21 reader function
+    result = v21._itemscheme(reader, elem)
+    # Attach CodelistExtensions
+    result.extends = reader.pop_all(v30.CodelistExtension)
+    return result
 
 
 @end("str:CodelistExtension")

sdmx/source/imf_data.py

@@ -0,0 +1,20 @@
+from . import Source as BaseSource
+
+
+class Source(BaseSource):
+    """Hooks for the ``IMF_DATA`` SDMX-REST API."""
+
+    _id = "IMF_DATA"
+
+    def modify_request_args(self, kwargs):
+        """Modify arguments used to build query URL.
+
+        Set default provider agency ID ``all``.
+        """
+        super().modify_request_args(kwargs)
+
+        # Supply 'all' as the default agency_id
+        # NB this is an indirect test for resource_type != 'data'; because of the way
+        #    the hook is called, resource_type is not available directly.
+        if "key" not in kwargs:
+            kwargs.setdefault("agency_id", "all")

sdmx/source/imf_data3.py

@@ -0,0 +1,32 @@
+from requests.structures import CaseInsensitiveDict
+
+from sdmx.format import MediaType
+
+from . import Source as BaseSource
+
+
+class Source(BaseSource):
+    """Hooks for the ``IMF_DATA3`` SDMX-REST API."""
+
+    _id = "IMF_DATA3"
+
+    def modify_request_args(self, kwargs):
+        """Modify arguments used to build query URL.
+
+        1. Set default provider agency ID ``all``.
+        2. Set media-type in header.
+        """
+        super().modify_request_args(kwargs)
+
+        # Supply 'all' as the default agency_id
+        # NB this is an indirect test for resource_type != 'data'; because of the way
+        #    the hook is called, resource_type is not available directly.
+        if "key" not in kwargs:
+            kwargs.setdefault("agency_id", "all")
+
+        # Retrieve SDMX-ML by default
+        # TODO Choose between the data, metadata, or structure media-type according to
+        #      other `kwargs`
+        kwargs.setdefault("headers", CaseInsensitiveDict())
+        mt = [MediaType(x, "xml", "3.0.0") for x in ("data", "metadata", "structure")]
+        kwargs["headers"].setdefault("Accept", ", ".join(map(str, mt)))

sdmx/sources.json

@@ -229,14 +229,38 @@
     }
   },
   {
-    "id": "IMF_beta",
+    "id": "IMF_DATA",
     "url": "https://api.imf.org/external/sdmx/2.1",
-    "name": "International Monetary Fund"
+    "name": "International Monetary Fund",
+    "supports": {
+      "actualconstraint": false,
+      "agencyscheme": false,
+      "allowedconstraint": false,
+      "dataconsumerscheme": false,
+      "dataproviderscheme": false,
+      "organisationscheme": false,
+      "provisionagreement": false,
+      "registration": false,
+      "structure": false,
+      "structureset": false
+    }
   },
   {
-    "id": "IMF_beta3",
+    "id": "IMF_DATA3",
     "url": "https://api.imf.org/external/sdmx/3.0",
-    "name": "International Monetary Fund",
+    "name": "International Monetary Fund (SDMX 3.0)",
+    "supports": {
+      "actualconstraint": false,
+      "agencyscheme": false,
+      "allowedconstraint": false,
+      "dataconsumerscheme": false,
+      "dataproviderscheme": false,
+      "organisationscheme": false,
+      "provisionagreement": false,
+      "registration": false,
+      "structure": false,
+      "structureset": false
+    },
     "versions": [
       "3.0.0"
     ]

sdmx/tests/test_sources.py

@@ -368,73 +368,71 @@ class TestIMF(DataSourceTest):
     source_id = "IMF"
 
 
-# As of 2025-01-10, all endpoints aside from SDMX 2.1 /data/ return 403
-IMF_BETA_XFAIL: dict[str, Union[type[Exception], tuple[type[Exception], str]]] = {
-    k: HTTPError
-    for k in """
-    actualconstraint
-    agencyscheme
-    allowedconstraint
-    categorisation
-    categoryscheme
-    codelist
-    conceptscheme
-    contentconstraint
-    dataconsumerscheme
-    dataflow
-    dataproviderscheme
-    datastructure
-    hierarchicalcodelist
-    metadataflow
-    metadatastructure
-    organisationscheme
-    provisionagreement
-    registration
-    structure
-    structureset
-    """.split()
-}
-
+class TestIMF_DATA(DataSourceTest):
+    source_id = "IMF_DATA"
 
-class TestIMF_beta(DataSourceTest):
-    source_id = "IMF_beta"
-
-    endpoint_args = dict(
+    endpoint_args = {
         # As indicated in the API documentation
-        data=dict(
+        "data": dict(
             resource_id="CPI",
             key="111.CPI.CP01.IX.M",
             params=dict(startPeriod=2018),
-            # Does not appear to affect 403
-            # headers={"User-Agent": "idata-script-client"},
-        )
-    )
+        ),
+        "codelist": dict(resource_id="CL_COUNTRY"),
+        "structure": dict(resource_id="DSD_CPI"),
+        "conceptscheme": dict(resource_id="CS_MASTER_SYSTEM"),
+    }
 
-    xfail = IMF_BETA_XFAIL | dict(
-        metadata=NotImplementedError,
-        registration=ValueError,
-    )
+    xfail = {
+        "actualconstraint": HTTPError,
+        "agencyscheme": HTTPError,
+        "allowedconstraint": HTTPError,
+        "dataconsumerscheme": HTTPError,
+        "dataproviderscheme": HTTPError,
+        "metadata": NotImplementedError,
+        "metadataflow": HTTPError,
+        "metadatastructure": HTTPError,
+        "organisationscheme": HTTPError,
+        "provisionagreement": HTTPError,
+        "registration": HTTPError,
+        "structureset": HTTPError,
+    }
 
 
-class TestIMF_beta3(DataSourceTest):
-    source_id = "IMF_beta3"
+class TestIMF_DATA3(DataSourceTest):
+    source_id = "IMF_DATA3"
 
-    endpoint_args = dict(
-        data=dict(
+    endpoint_args = {
+        "data": dict(
             context="dataflow",
-            agency_id="IMF",
+            agency_id="IMF.STA",
             resource_id="CPI",
             key="111.CPI.CP01.IX.M",
-            # Not yet supported
-            # params={"c[TIME_PERIOD]": "ge:2018"},
+            params={"c[TIME_PERIOD]": "ge:2018"},
         ),
-        metadata=dict(provider_id="IMF"),
-    )
+        "codelist": dict(resource_id="CL_COUNTRY"),
+        "structure": dict(resource_id="DSD_CPI"),
+        "conceptscheme": dict(resource_id="CS_MASTER_SYSTEM"),
+    }
 
-    xfail = IMF_BETA_XFAIL | dict(
-        data=HTTPError,  # 403
-        metadata=HTTPError,  # 403
-    )
+    xfail = {
+        "actualconstraint": HTTPError,
+        "agencyscheme": HTTPError,
+        "allowedconstraint": HTTPError,
+        "codelist": NotImplementedError,
+        "conceptscheme": NotImplementedError,
+        "contentconstraint": HTTPError,
+        "dataconsumerscheme": HTTPError,
+        "dataflow": NotImplementedError,
+        "dataproviderscheme": HTTPError,
+        "datastructure": NotImplementedError,
+        "hierarchicalcodelist": HTTPError,
+        "metadatastructure": NotImplementedError,
+        "organisationscheme": HTTPError,
+        "provisionagreement": HTTPError,
+        "registration": HTTPError,
+        "structureset": HTTPError,
+    }
 
 
 class TestINEGI(DataSourceTest):