Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disallow saving to the filesystem when using APILayoutStrategy #1486

Open
luipir opened this issue Dec 18, 2024 · 8 comments
Open

Disallow saving to the filesystem when using APILayoutStrategy #1486

luipir opened this issue Dec 18, 2024 · 8 comments
Labels
bug Things which are broken
Milestone

Comments

@luipir
Copy link

luipir commented Dec 18, 2024

when writing db but href generated with APILayoutStrategyand error is issued, seems because try to update connection.json but looking it with the same name of created folder that contain collection.json.

Note, collection.json is not (yet) present in the folder

anything else I can add to facilitare replication? a to_dict() of the catalog?

here the stack trace

...hide.../aerofototeca/stac_generator/stac_generator.py:446 in stac_generator                  │
│                                                                                                                               │
│   443 │   )                                                                                                                   │
│   444 │                                                                                                                       │
│   445 │   # save the catalog                                                                                                  │
│ ❱ 446 │   catalog.save(catalog_type=pystac.CatalogType.SELF_CONTAINED)                                                        │
│   447 │                                                                                                                       │
│   448 │   # validate the catalog                                                                                              │
│   449 │   pystac.validation.validate_all(catalog)                                                                             │
│                                                                                                                               │
│ ╭───────────────────────────────────────────────────────── locals ──────────────────────────────────────────────────────────╮ │
│ │        basename = '6023-Lvl02-Color.tif'                                                                                  │ │
│ │            bbox = [302787.0311584755, 4652173.35055104, 303571.45467431337, 4653296.350993636]                            │ │
│ │         catalog = <Catalog id=aerofototeca_roma>                                                                          │ │
│ │      collection = <Collection id=fotoindici>                                                                              │ │
│ │     collections = {                                                                                                       │ │
│ │                   │   '6020': [                                                                                           │ │
│ │                   │   │   '...hide.../temp/6020-Lvl02-Oblique-Backward.ti'+1,           │ │
│ │                   │   │   '...hide.../temp/6020-Lvl02-Color.tif',                       │ │
│ │                   │   │   '...hide.../temp/6020-Lvl02-Oblique-Left.tif',                │ │
│ │                   │   │   '...hide.../temp/6020-Lvl02-Oblique-Right.tif',               │ │
│ │                   │   │   '...hide.../temp/6020-Lvl02-Oblique-Forward.tif'              │ │
│ │                   │   ],                                                                                                  │ │
│ │                   │   '6019': [                                                                                           │ │
│ │                   │   │   '...hide.../temp/6019-Lvl02-Oblique-Right.tif',               │ │
│ │                   │   │   '...hide.../temp/6019-Lvl02-Oblique-Backward.ti'+1,           │ │
│ │                   │   │   '...hide.../temp/6019-Lvl02-Oblique-Left.tif',                │ │
│ │                   │   │   '...hide.../temp/6019-Lvl02-Color.tif',                       │ │
│ │                   │   │   '...hide.../temp/6019-Lvl02-Oblique-Forward.tif'              │ │
│ │                   │   ],                                                                                                  │ │
│ │                   │   '6021': [                                                                                           │ │
│ │                   │   │   '...hide.../temp/6021-Lvl02-Oblique-Forward.tif',             │ │
│ │                   │   │   '...hide.../temp/6021-Lvl02-Color.tif',                       │ │
│ │                   │   │   '...hide.../temp/6021-Lvl02-Oblique-Backward.ti'+1,           │ │
│ │                   │   │   '...hide.../temp/6021-Lvl02-Oblique-Right.tif',               │ │
│ │                   │   │   '...hide.../temp/6021-Lvl02-Oblique-Left.tif'                 │ │
│ │                   │   ],                                                                                                  │ │
│ │                   │   '6023': [                                                                                           │ │
│ │                   │   │   '...hide.../temp/6023-Lvl02-Oblique-Forward.tif',             │ │
│ │                   │   │   '...hide.../temp/6023-Lvl02-Oblique-Left.tif',                │ │
│ │                   │   │   '...hide.../temp/6023-Lvl02-Oblique-Backward.ti'+1,           │ │
│ │                   │   │   '...hide.../temp/6023-Lvl02-Oblique-Right.tif',               │ │
│ │                   │   │   '...hide.../temp/6023-Lvl02-Color.tif'                        │ │
│ │                   │   ],                                                                                                  │ │
│ │                   │   '6018': [                                                                                           │ │
│ │                   │   │   '...hide.../temp/6018-Lvl02-Oblique-Backward.ti'+1,           │ │
│ │                   │   │   '...hide.../temp/6018-Lvl02-Color.tif',                       │ │
│ │                   │   │   '...hide.../temp/6018-Lvl02-Oblique-Forward.tif',             │ │
│ │                   │   │   '...hide.../temp/6018-Lvl02-Oblique-Right.tif',               │ │
│ │                   │   │   '...hide.../temp/6018-Lvl02-Oblique-Left.tif'                 │ │
│ │                   │   ],                                                                                                  │ │
│ │                   │   '6022': [                                                                                           │ │
│ │                   │   │   '...hide.../temp/6022-Lvl02-Oblique-Backward.ti'+1,           │ │
│ │                   │   │   '...hide.../temp/6022-Lvl02-Oblique-Forward.tif',             │ │
│ │                   │   │   '...hide.../temp/6022-Lvl02-Oblique-Right.tif',               │ │
│ │                   │   │   '...hide.../temp/6022-Lvl02-Oblique-Left.tif',                │ │
│ │                   │   │   '...hide.../temp/6022-Lvl02-Color.tif'                        │ │
│ │                   │   ]                                                                                                   │ │
│ │                   }                                                                                                       │ │
│ │   creation_date = datetime.datetime(2022, 10, 15, 9, 17, 40)                                                              │ │
│ │             crs = CRS.from_wkt('PROJCS["WGS 84 / UTM zone 33N",GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS             │ │
│ │                   84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUT… │ │
│ │            dirs = []                                                                                                      │ │
│ │               f = <_io.TextIOWrapper name='...hide.../temp/hash.log' mode='r'           │ │
│ │                   encoding='UTF-8'>                                                                                       │ │
│ │           files = [                                                                                                       │ │
│ │                   │   '6020-Lvl02-Oblique-Backward.tif',                                                                  │ │
│ │                   │   'hash.log',                                                                                         │ │
│ │                   │   '6020-Lvl02-Color.tif',                                                                             │ │
│ │                   │   '6019-Lvl02-Oblique-Right.tif',                                                                     │ │
│ │                   │   '6021-Lvl02-Oblique-Forward.tif',                                                                   │ │
│ │                   │   '6023-Lvl02-Oblique-Forward.tif',                                                                   │ │
│ │                   │   '6018-Lvl02-Oblique-Backward.tif',                                                                  │ │
│ │                   │   '6022-Lvl02-Oblique-Backward.tif',                                                                  │ │
│ │                   │   '6022-Lvl02-Oblique-Forward.tif',                                                                   │ │
│ │                   │   '6018-Lvl02-Color.tif',                                                                             │ │
│ │                   │   ... +21                                                                                             │ │
│ │                   ]                                                                                                       │ │
│ │              fn = '../temp/6023-Lvl02-Color.tif'                                                                          │ │
│ │       footprint = {                                                                                                       │ │
│ │                   │   'type': 'Polygon',                                                                                  │ │
│ │                   │   'coordinates': (                                                                                    │ │
│ │                   │   │   (                                                                                               │ │
│ │                   │   │   │   (302787.0311584755, 4652173.35055104),                                                      │ │
│ │                   │   │   │   (302787.0311584755, 4653296.350993636),                                                     │ │
│ │                   │   │   │   (303571.45467431337, 4653296.350993636),                                                    │ │
│ │                   │   │   │   (303571.45467431337, 4652173.35055104),                                                     │ │
│ │                   │   │   │   (302787.0311584755, 4652173.35055104)                                                       │ │
│ │                   │   │   ),                                                                                              │ │
│ │                   │   )                                                                                                   │ │
│ │                   }                                                                                                       │ │
│ │            hash = '4591a3d890f57eb9ddeb1483d1266c9fae9966b7e6c9455a9ec3cb48b985bf43'                                      │ │
│ │          hashes = {                                                                                                       │ │
│ │                   │   '6019-Lvl02-Oblique-Right.tif': '52c1ca814f2ea2feb357ecb1ed808f655c5984aba6410824ce8aace4898b7218', │ │
│ │                   │   '6020-Lvl02-Oblique-Backward.tif':                                                                  │ │
│ │                   'd89590ba33941ed4665c880afa3f879f6d818a7bda6f349793b74151507351b9',                                     │ │
│ │                   │   '6022-Lvl02-Oblique-Backward.tif':                                                                  │ │
│ │                   '3d02719901ed14f1e43c37e0775a0bc0d427fb9eaa9de176db23fb1bd6bdf9e7',                                     │ │
│ │                   │   '6023-Lvl02-Oblique-Forward.tif':                                                                   │ │
│ │                   'e3f77bbd8a79c0a4299431024afbed238733e21295e20f472cfc07776188bf5e',                                     │ │
│ │                   │   '6021-Lvl02-Oblique-Forward.tif':                                                                   │ │
│ │                   '8801cf489a6712d4f8e8efba0e1b2d0b7e1606b30d976308e65a30ae38344484',                                     │ │
│ │                   │   '6022-Lvl02-Oblique-Forward.tif':                                                                   │ │
│ │                   'faa804078a10b9df710ea46cfb806f2a5aa2ef500d04109be072201a869e4132',                                     │ │
│ │                   │   '6018-Lvl02-Oblique-Backward.tif':                                                                  │ │
│ │                   '54dbdc3ea776d113942e2be3342adb4c4efa756cfff330028d4a4da8be790ba7',                                     │ │
│ │                   │   '6019-Lvl02-Oblique-Backward.tif':                                                                  │ │
│ │                   'b2ab4f1ec320546f396ed5e949846bdac70cf772aa50603a99a155121d4e3ab9',                                     │ │
│ │                   │   '6020-Lvl02-Color.tif': '8937add3c5d795a41d846eb5a75fecc6a2208994b88cdddbfb642e742c9235f4',         │ │
│ │                   │   '6018-Lvl02-Oblique-Forward.tif':                                                                   │ │
│ │                   'd8029a176c2f4a630a70636cce04cf32a86d65698d393efbc5abd92ab99c2e9f',                                     │ │
│ │                   │   ... +20                                                                                             │ │
│ │                   }                                                                                                       │ │
│ │     hashes_path = PosixPath('...hide.../temp/hash.log')                                 │ │
│ │              id = '6022'                                                                                                  │ │
│ │      input_path = PosixPath('...hide.../temp')                                          │ │
│ │            item = <Item id=6022-lvl02-color>                                                                              │ │
│ │ item_collection = <Collection id=6022>                                                                                    │ │
│ │         item_id = '6022-Lvl02-Color'                                                                                      │ │
│ │            line = '../temp/6023-Lvl02-Color.tif: 1119e0579be3eaaa5704faf1d592d792fe72e71105a6e91328'+15                   │ │
│ │             log = PosixPath('...hide.../aerofototeca/stac_generator/stac_generator.log')    │ │
│ │       log_level = 20                                                                                                      │ │
│ │          logger = <Logger stac_generator (INFO)>                                                                          │ │
│ │         matches = <re.Match object; span=(0, 20), match='6022-Lvl02-Color.tif'>                                           │ │
│ │  multihash_hash = '12204591a3d890f57eb9ddeb1483d1266c9fae9966b7e6c9455a9ec3cb48b985bf43'                                  │ │
│ │            root = '...hide.../temp'                                                     │ │
│ │            tags = {                                                                                                       │ │
│ │                   │   'ExifIFD:ExifVersion': '0232',                                                                      │ │
│ │                   │   'ExifIFD:DateTimeOriginal': '2022-10-15T09:17:40',                                                  │ │
│ │                   │   'ExifIFD:OffsetTimeOriginal': '+02:00',                                                             │ │
│ │                   │   'ExifIFD:ComponentsConfiguration': 'Y, Cb, Cr, -',                                                  │ │
│ │                   │   'ExifIFD:FlashpixVersion': '0100',                                                                  │ │
│ │                   │   'ExifIFD:ColorSpace': 'Uncalibrated',                                                               │ │
│ │                   │   'ExifIFD:ImageUniqueID': '6022-Lvl02-Color',                                                        │ │
│ │                   │   'ExifIFD:OwnerName': 'Roma Capitale',                                                               │ │
│ │                   │   'System:FileName': '6022-Lvl02-Color.tif',                                                          │ │
│ │                   │   'System:Directory': '...hide.../temp',                            │ │
│ │                   │   ... +61                                                                                             │ │
│ │                   }                                                                                                       │ │
│ │          target = '...hide.../temp/6022-Lvl02-Color.tif'                                │ │
│ │         targets = [                                                                                                       │ │
│ │                   │   '...hide.../temp/6022-Lvl02-Oblique-Backward.ti'+1,               │ │
│ │                   │   '...hide.../temp/6022-Lvl02-Oblique-Forward.tif',                 │ │
│ │                   │   '...hide.../temp/6022-Lvl02-Oblique-Right.tif',                   │ │
│ │                   │   '...hide.../temp/6022-Lvl02-Oblique-Left.tif',                    │ │
│ │                   │   '...hide.../temp/6022-Lvl02-Color.tif'                            │ │
│ │                   ]                                                                                                       │ │
│ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                                               │
│ ...hide.../aerofototeca/stac_generator/.pixi/envs/default/lib/python3.12/site-packages/pystac/c │
│ atalog.py:975 in save                                                                                                         │
│                                                                                                                               │
│    972 │   │   │   │   │   │   stac_io=stac_io,                                                                               │
│    973 │   │   │   │   │   )                                                                                                  │
│    974 │   │   │   │   else:                                                                                                  │
│ ❱  975 │   │   │   │   │   child.save(stac_io=stac_io)                                                                        │
│    976 │   │                                                                                                                  │
│    977 │   │   for item_link in self.get_item_links():                                                                        │
│    978 │   │   │   if item_link.is_resolved():                                                                                │
│                                                                                                                               │
│ ╭─────────────────────────────────── locals ───────────────────────────────────╮                                              │
│ │            catalog_type = SELF_CONTAINED                                     │                                              │
│ │                   child = <Collection id=fotoindici>                         │                                              │
│ │              child_link = <Link rel=child target=<Collection id=fotoindici>> │                                              │
│ │               dest_href = None                                               │                                              │
│ │ items_include_self_link = False                                              │                                              │
│ │                    root = <Catalog id=aerofototeca_roma>                     │                                              │
│ │                    self = <Catalog id=aerofototeca_roma>                     │                                              │
│ │                 stac_io = None                                               │                                              │
│ ╰──────────────────────────────────────────────────────────────────────────────╯                                              │
│                                                                                                                               │
│ ...hide.../aerofototeca/stac_generator/.pixi/envs/default/lib/python3.12/site-packages/pystac/c │
│ atalog.py:975 in save                                                                                                         │
│                                                                                                                               │
│    972 │   │   │   │   │   │   stac_io=stac_io,                                                                               │
│    973 │   │   │   │   │   )                                                                                                  │
│    974 │   │   │   │   else:                                                                                                  │
│ ❱  975 │   │   │   │   │   child.save(stac_io=stac_io)                                                                        │
│    976 │   │                                                                                                                  │
│    977 │   │   for item_link in self.get_item_links():                                                                        │
│    978 │   │   │   if item_link.is_resolved():                                                                                │
│                                                                                                                               │
│ ╭──────────────────────────────── locals ────────────────────────────────╮                                                    │
│ │            catalog_type = None                                         │                                                    │
│ │                   child = <Collection id=6020>                         │                                                    │
│ │              child_link = <Link rel=child target=<Collection id=6020>> │                                                    │
│ │               dest_href = None                                         │                                                    │
│ │ items_include_self_link = False                                        │                                                    │
│ │                    root = <Catalog id=aerofototeca_roma>               │                                                    │
│ │                    self = <Collection id=fotoindici>                   │                                                    │
│ │                 stac_io = None                                         │                                                    │
│ ╰────────────────────────────────────────────────────────────────────────╯                                                    │
│                                                                                                                               │
│ ...hide.../aerofototeca/stac_generator/.pixi/envs/default/lib/python3.12/site-packages/pystac/c │
│ atalog.py:1011 in save                                                                                                        │
│                                                                                                                               │
│   1008 │   │   │   catalog_dest_href = make_absolute_href(                                                                    │
│   1009 │   │   │   │   rel_href, dest_href, start_is_dir=True                                                                 │
│   1010 │   │   │   )                                                                                                          │
│ ❱ 1011 │   │   self.save_object(                                                                                              │
│   1012 │   │   │   include_self_link=include_self_link,                                                                       │
│   1013 │   │   │   dest_href=catalog_dest_href,                                                                               │
│   1014 │   │   │   stac_io=stac_io,                                                                                           │
│                                                                                                                               │
│ ╭─────────────────────────────────────── locals ────────────────────────────────────────╮                                     │
│ │       catalog_dest_href = None                                                        │                                     │
│ │            catalog_type = None                                                        │                                     │
│ │               dest_href = None                                                        │                                     │
│ │       include_self_link = False                                                       │                                     │
│ │                    item = <Item id=6020-lvl02-oblique-forward>                        │                                     │
│ │               item_link = <Link rel=item target=<Item id=6020-lvl02-oblique-forward>> │                                     │
│ │ items_include_self_link = False                                                       │                                     │
│ │                    root = <Catalog id=aerofototeca_roma>                              │                                     │
│ │                    self = <Collection id=6020>                                        │                                     │
│ │                 stac_io = None                                                        │                                     │
│ ╰───────────────────────────────────────────────────────────────────────────────────────╯                                     │
│                                                                                                                               │
│ ...hide.../aerofototeca/stac_generator/.pixi/envs/default/lib/python3.12/site-packages/pystac/s │
│ tac_object.py:476 in save_object                                                                                              │
│                                                                                                                               │
│   473 │   │   │   │   )                                                                                                       │
│   474 │   │   │   dest_href = self_href                                                                                       │
│   475 │   │                                                                                                                   │
│ ❱ 476 │   │   stac_io.save_json(dest_href, self.to_dict(include_self_link=include_self_link))                                 │
│   477 │                                                                                                                       │
│   478 │   def full_copy(                                                                                                      │
│   479 │   │   self,                                                                                                           │
│                                                                                                                               │
│ ╭───────────────────────────────────────────────── locals ──────────────────────────────────────────────────╮                 │
│ │         dest_href = '...hide.../aerofototeca/stac_generator/stac_catalo'+54 │                 │
│ │ include_self_link = False                                                                                 │                 │
│ │              root = <Catalog id=aerofototeca_roma>                                                        │                 │
│ │      root_stac_io = None                                                                                  │                 │
│ │              self = <Collection id=6020>                                                                  │                 │
│ │         self_href = '...hide.../aerofototeca/stac_generator/stac_catalo'+54 │                 │
│ │           stac_io = <pystac.stac_io.DefaultStacIO object at 0x708e9976edb0>                               │                 │
│ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────╯                 │
│                                                                                                                               │
│ ...hide.../aerofototeca/stac_generator/.pixi/envs/default/lib/python3.12/site-packages/pystac/s │
│ tac_io.py:261 in save_json                                                                                                    │
│                                                                                                                               │
│   258 │   │   │   │   :meth:`StacIO.json_dumps`.                                                                              │
│   259 │   │   """                                                                                                             │
│   260 │   │   txt = self.json_dumps(json_dict, *args, **kwargs)                                                               │
│ ❱ 261 │   │   self.write_text(dest, txt)                                                                                      │
│   262 │                                                                                                                       │
│   263 │   @classmethod                                                                                                        │
│   264 │   def set_default(cls, stac_io_class: Callable[[], StacIO]) -> None:                                                  │
│                                                                                                                               │
│ ╭───────────────────────────────────────────────────────── locals ──────────────────────────────────────────────────────────╮ │
│ │      args = ()                                                                                                            │ │
│ │      dest = '...hide.../aerofototeca/stac_generator/stac_catalo'+54                         │ │
│ │ json_dict = {                                                                                                             │ │
│ │             │   'type': 'Collection',                                                                                     │ │
│ │             │   'id': '6020',                                                                                             │ │
│ │             │   'stac_version': '1.0.0',                                                                                  │ │
│ │             │   'description': 'Collection of fotoindexes n.6020',                                                        │ │
│ │             │   'links': [                                                                                                │ │
│ │             │   │   {'rel': 'root', 'href': '../../..', 'type': 'application/json', 'title': 'Arofototeca of Roma'},      │ │
│ │             │   │   {'rel': 'item', 'href': './6020/items/6020-lvl02-oblique-backward', 'type': 'application/json'},      │ │
│ │             │   │   {'rel': 'item', 'href': './6020/items/6020-lvl02-color', 'type': 'application/json'},                 │ │
│ │             │   │   {'rel': 'item', 'href': './6020/items/6020-lvl02-oblique-left', 'type': 'application/json'},          │ │
│ │             │   │   {'rel': 'item', 'href': './6020/items/6020-lvl02-oblique-right', 'type': 'application/json'},         │ │
│ │             │   │   {'rel': 'item', 'href': './6020/items/6020-lvl02-oblique-forward', 'type': 'application/json'},       │ │
│ │             │   │   {                                                                                                     │ │
│ │             │   │   │   'rel': 'parent',                                                                                  │ │
│ │             │   │   │   'href': './..',                                                                                   │ │
│ │             │   │   │   'type': 'application/json',                                                                       │ │
│ │             │   │   │   'title': 'Drone flight images of Roma'                                                            │ │
│ │             │   │   }                                                                                                     │ │
│ │             │   ],                                                                                                        │ │
│ │             │   'stac_extensions': ['https://stac-extensions.github.io/version/v1.2.0/schema.json'],                      │ │
│ │             │   'version': '1.0.0',                                                                                       │ │
│ │             │   'deprecated': False,                                                                                      │ │
│ │             │   'experimental': False,                                                                                    │ │
│ │             │   'title': 'Drone flight images of Roma with fotoindice n.6020',                                            │ │
│ │             │   ... +4                                                                                                    │ │
│ │             }                                                                                                             │ │
│ │    kwargs = {}                                                                                                            │ │
│ │      self = <pystac.stac_io.DefaultStacIO object at 0x708e9976edb0>                                                       │ │
│ │       txt = '{\n  "type": "Collection",\n  "id": "6020",\n  "stac_version": "1.0.0",\n  "descript'+2073                   │ │
│ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                                               │
│ ...hide.../aerofototeca/stac_generator/.pixi/envs/default/lib/python3.12/site-packages/pystac/s │
│ tac_io.py:316 in write_text                                                                                                   │
│                                                                                                                               │
│   313 │   │   is not already) and delegates to :meth:`DefaultStacIO.write_text_from_href` for                                 │
│   314 │   │   opening and reading the file."""                                                                                │
│   315 │   │   href = str(os.fspath(dest))                                                                                     │
│ ❱ 316 │   │   return self.write_text_to_href(href, txt)                                                                       │
│   317 │                                                                                                                       │
│   318 │   def write_text_to_href(self, href: str, txt: str) -> None:                                                          │
│   319 │   │   """Writes text to file using UTF-8 encoding.                                                                    │
│                                                                                                                               │
│ ╭────────────────────────────────────────────── locals ──────────────────────────────────────────────╮                        │
│ │    _ = ()                                                                                          │                        │
│ │ dest = '...hide.../aerofototeca/stac_generator/stac_catalo'+54       │                        │
│ │ href = '...hide.../aerofototeca/stac_generator/stac_catalo'+54       │                        │
│ │ self = <pystac.stac_io.DefaultStacIO object at 0x708e9976edb0>                                     │                        │
│ │  txt = '{\n  "type": "Collection",\n  "id": "6020",\n  "stac_version": "1.0.0",\n  "descript'+2073 │                        │
│ ╰────────────────────────────────────────────────────────────────────────────────────────────────────╯                        │
│                                                                                                                               │
│ ...hide.../aerofototeca/stac_generator/.pixi/envs/default/lib/python3.12/site-packages/pystac/s │
│ tac_io.py:335 in write_text_to_href                                                                                           │
│                                                                                                                               │
│   332 │   │   dirname = os.path.dirname(href)                                                                                 │
│   333 │   │   if dirname != "" and not os.path.isdir(dirname):                                                                │
│   334 │   │   │   os.makedirs(dirname)                                                                                        │
│ ❱ 335 │   │   with open(href, "w", encoding="utf-8") as f:                                                                    │
│   336 │   │   │   f.write(txt)                                                                                                │
│   337                                                                                                                         │
│   338                                                                                                                         │
│                                                                                                                               │
│ ╭─────────────────────────────────────────────── locals ────────────────────────────────────────────────╮                     │
│ │ dirname = '...hide.../aerofototeca/stac_generator/stac_catalo'+49       │                     │
│ │    href = '...hide.../aerofototeca/stac_generator/stac_catalo'+54       │                     │
│ │    self = <pystac.stac_io.DefaultStacIO object at 0x708e9976edb0>                                     │                     │
│ │     txt = '{\n  "type": "Collection",\n  "id": "6020",\n  "stac_version": "1.0.0",\n  "descript'+2073 │                     │
│ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────╯                     │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
IsADirectoryError: [Errno 21] Is a directory: 
@gadomski
Copy link
Member

Can you provide a minimum-reproducable example? A simple script to create a collection and an item would be best. Thanks!

@luipir
Copy link
Author

luipir commented Dec 18, 2024

I'll try to create a to_dict of the catalog hierarchy and check if starting from there would re-create the issue

@luipir
Copy link
Author

luipir commented Dec 18, 2024

@gadomski this scirpt is similar to that I wrote, depurated of many business detail that are not useful for replicate the issue.

in resume the script accept a path containing COGs, walk it selecting all tif and create the following hierarchy

catalog->highlevel collection->item collection->item

I've to populate with 26k images (the catalog) divided in group of 5 files (item collection in the script simulated having a collection with only a tif file item/asset)
so, in the real case, high level collection would have 500 to item collections in it.

stac_generator_issue.zip

to replicate:

$> pip install typer pystac rio_stac rasterio shapely
$> python stac_generateor_issue.py -i <path where to look for tiffs>

@luipir
Copy link
Author

luipir commented Dec 18, 2024

p.s. I tried to dump all the stac hierarchy dumping json from stacobject.to_dict(), but recreating the hierarchy with stacobject.from_file(...) generate great number of issue, so I did it creating the above script.
Any suggestion how to dump a failing stac catalog and recreate it? This would simplify the creation of test data from my side.

@luipir
Copy link
Author

luipir commented Dec 19, 2024

seems the issue is in inside strategy.get_href(...) that is used during normalize_hrefs. In any case the gnerated href is correct, and it does not work only in the contect to dump the catalog with save because trying to save contect of json in the href that is a dir (should be root/{id}/{id}.json.

@luipir
Copy link
Author

luipir commented Dec 19, 2024

more, if fix creation of json in stac_io contect fail due to:
ValueError: <Catalog id=some_id> does not have a self_href set.
this because for the current strategy APILayoutStrategy() the catalog self_href is not set due to this fix #1116

@gadomski gadomski self-assigned this Dec 20, 2024
@gadomski gadomski added the bug Things which are broken label Dec 20, 2024
@gadomski
Copy link
Member

Thanks for the information! I was able to reproduce the problem, and the key issue is that the ApiLayoutStrategy isn't intended for actually writing to a file system: #1294. If you're writing these out, I would recommend using the BestPracticesLayoutStrategy.

The ApiLayoutStrategy should probably throw an explicit error if you try to use it to write out files, so I'm going to update the title of this issue to reflect that.

FWIW your minimum reproducible example was still pretty verbose, here's the script that I used to reproduce:

import datetime

from pystac import Catalog, CatalogType, Collection, Extent, Item
from pystac.layout import APILayoutStrategy

item = Item("an-item", None, None, datetime.datetime.now(), {})
extent = Extent.from_items([item])
catalog = Catalog("a-catalog", "A description")
collection = Collection("a-collection", "A description", extent)
collection.add_item(item)
catalog.add_child(collection)
catalog.normalize_hrefs(root_href="issues-1486", strategy=APILayoutStrategy())
catalog.save(CatalogType.SELF_CONTAINED)

@gadomski gadomski changed the title IsADirectoryError when catalog.normalize_hrefs with strategy=pystac.layout.APILayoutStrategy() Disallow saving to the filesystem when using APILayoutStrategy Dec 20, 2024
@gadomski gadomski added this to the v1.12 milestone Dec 20, 2024
@gadomski gadomski removed their assignment Dec 20, 2024
@luipir
Copy link
Author

luipir commented Dec 22, 2024

@gadomski thank you for the analysis, I looked in the repo before bu I didn't find this because I looked into issues (open and closed).

And thanks to give me a resumed test, I just realised later that I would be able to simplify it just reading unit test.

cheers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Things which are broken
Projects
None yet
Development

No branches or pull requests

2 participants