Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix metadata topic_hierarchy #800

Merged
merged 5 commits into from
Nov 7, 2024
Merged

fix metadata topic_hierarchy #800

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
0707517
fix metadata topic_hierarchy
tomkralidis Nov 7, 2024
85f0785
test
tomkralidis Nov 7, 2024
a88befa
test
tomkralidis Nov 7, 2024
52939d4
test
tomkralidis Nov 7, 2024
4bd2616
fix
tomkralidis Nov 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions .github/workflows/tests-docker.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,11 +147,11 @@ jobs:
python3 wis2box-ctl.py execute wis2box data ingest -mdi $DISCOVERY_METADATA_ID -p $TEST_DATA
- name: add example hourly ship data (bufr2bufr) WMO
env:
TOPIC_HIERARCHY: origin/a/wis2/int-wmo-test/data/core/weather/surface-based-observations/ship-hourly
DISCOVERY_METADATA: /data/wis2box/metadata/discovery/int-wmo-test-ship-hourly.yml
DISCOVERY_METADATA_ID: urn:wmo:md:int-wmo-test:surface-weather-observations:ship-hourly
STATION_METADATA: /data/wis2box/metadata/station/wmo-test-ship-hourly.csv
TEST_DATA: /data/wis2box/observations/wmo/ship-hourly
TOPIC_HIERARCHY: origin/a/wis2/int-wmo-test/data/core/weather/surface-based-observations/ship
DISCOVERY_METADATA: /data/wis2box/metadata/discovery/int-wmo-test-ship.yml
DISCOVERY_METADATA_ID: urn:wmo:md:int-wmo-test:surface-weather-observations:ship
STATION_METADATA: /data/wis2box/metadata/station/wmo-test-ship.csv
TEST_DATA: /data/wis2box/observations/wmo/ship
run: |
python3 wis2box-ctl.py execute wis2box dataset publish $DISCOVERY_METADATA
python3 wis2box-ctl.py execute wis2box metadata station publish-collection --path $STATION_METADATA --topic-hierarchy $TOPIC_HIERARCHY
Expand Down
6 changes: 3 additions & 3 deletions docs/source/reference/quickstart.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ Publish test datasets:
wis2box dataset publish $WIS2BOX_DATADIR/metadata/discovery/dz-surface-weather-observations.yml
wis2box dataset publish $WIS2BOX_DATADIR/metadata/discovery/ro-synoptic-weather-observations.yml
wis2box dataset publish $WIS2BOX_DATADIR/metadata/discovery/cd-surface-weather-observations.yml
wis2box dataset publish $WIS2BOX_DATADIR/metadata/discovery/int-wmo-test-ship-hourly.yml
wis2box dataset publish $WIS2BOX_DATADIR/metadata/discovery/int-wmo-test-ship.yml
wis2box dataset publish $WIS2BOX_DATADIR/metadata/discovery/int-wmo-test-drifting-buoys.yml
wis2box dataset publish $WIS2BOX_DATADIR/metadata/discovery/int-wmo-test-wind-profile.yml

Expand All @@ -67,7 +67,7 @@ Load initial stations:
wis2box metadata station publish-collection --path /data/wis2box/metadata/station/algeria.csv --topic-hierarchy origin/a/wis2/dz-meteoalgerie/data/core/weather/surface-based-observations/synop
wis2box metadata station publish-collection --path /data/wis2box/metadata/station/romania.csv --topic-hierarchy origin/a/wis2/ro-rnimh-test/data/core/weather/surface-based-observations/synop
wis2box metadata station publish-collection --path /data/wis2box/metadata/station/congo.csv --topic-hierarchy origin/a/wis2/cg-met/data/recommended/weather/surface-based-observations/synop
wis2box metadata station publish-collection --path /data/wis2box/metadata/station/wmo-test-ship-hourly.csv --topic-hierarchy origin/a/wis2/int-wmo-test/data/core/weather/surface-based-observations/ship-hourly
wis2box metadata station publish-collection --path /data/wis2box/metadata/station/wmo-test-ship.csv --topic-hierarchy origin/a/wis2/int-wmo-test/data/core/weather/surface-based-observations/ship
wis2box metadata station publish-collection --path /data/wis2box/metadata/station/wmo-test-drifting-buoys.csv --topic-hierarchy origin/a/wis2/int-wmo-test/data/core/ocean/surface-based-observations/drifting-buoys
wis2box metadata station publish-collection --path /data/wis2box/metadata/station/wmo-test-wind-profile.csv --topic-hierarchy origin/a/wis2/int-wmo-test/data/core/weather/surface-based-observations/wind-profile

Expand All @@ -80,7 +80,7 @@ Ingest data using the data ingest command to push data to the ``wis2box-incoming
wis2box data ingest --metadata-id "urn:wmo:md:dz-meteoalgerie:surface-weather-observations" --path $WIS2BOX_DATADIR/observations/algeria
wis2box data ingest --metadata-id "urn:wmo:md:ro-rnimh-test:synoptic-weather-observations" --path $WIS2BOX_DATADIR/observations/romania
wis2box data ingest --metadata-id "urn:wmo:md:cg-met:surface-weather-observations" --path $WIS2BOX_DATADIR/observations/congo
wis2box data ingest --metadata-id "urn:wmo:md:int-wmo-test:surface-weather-observations:ship-hourly" --path $WIS2BOX_DATADIR/observations/wmo/ship-hourly
wis2box data ingest --metadata-id "urn:wmo:md:int-wmo-test:surface-weather-observations:ship" --path $WIS2BOX_DATADIR/observations/wmo/ship
wis2box data ingest --metadata-id "urn:wmo:md:int-wmo-test:surface-weather-observations:drifting-buoys" --path $WIS2BOX_DATADIR/observations/wmo/drifting-buoys
wis2box data ingest --metadata-id "urn:wmo:md:int-wmo-test:surface-weather-observations:wind-profile" --path $WIS2BOX_DATADIR/observations/wmo/wind-profile

Expand Down
4 changes: 2 additions & 2 deletions ...ta/discovery/int-wmo-test-ship-hourly.yml → .../metadata/discovery/int-wmo-test-ship.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
wis2box:
retention: P30D
topic_hierarchy: int-wmo-test/data/core/weather/surface-based-observations/ship-hourly
topic_hierarchy: int-wmo-test/data/core/weather/surface-based-observations/ship
country: int
centre_id: int-wmo-test
data_mappings:
Expand All @@ -16,7 +16,7 @@ mcf:
version: 1.0

metadata:
identifier: urn:wmo:md:int-wmo-test:surface-weather-observations:ship-hourly
identifier: urn:wmo:md:int-wmo-test:surface-weather-observations:ship
hierarchylevel: dataset

identification:
Expand Down
0 ...metadata/station/wmo-test-ship-hourly.csv → ...s/data/metadata/station/wmo-test-ship.csv
View file
Open in desktop
File renamed without changes.
0 ...700RRA_C_EDZW_20240211170705_50982396.bin → ...700RRA_C_EDZW_20240211170705_50982396.bin
View file
Open in desktop
File renamed without changes.
2 changes: 1 addition & 1 deletion tests/integration/test_workflow.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ def test_wis2downloader():
'origin/a/wis2/cg-met/data/recommended/weather/surface-based-observations/synop': 0, # noqa
'origin/a/wis2/int-wmo-test/data/core/ocean/surface-based-observations/drifting-buoys': 2, # noqa
'origin/a/wis2/int-wmo-test/data/core/weather/surface-based-observations/wind-profile': 1, # noqa
'origin/a/wis2/int-wmo-test/data/core/weather/surface-based-observations/ship-hourly': 5, # noqa
'origin/a/wis2/int-wmo-test/data/core/weather/surface-based-observations/ship': 5, # noqa
'origin/a/wis2/it-meteoam/data/core/weather/surface-based-observations/synop': 31, # noqa
'origin/a/wis2/int-wmo-test/data/core/weather/advisories-warnings': 1 # noqa
}
Expand Down
Loading