Unhandled error in `curate apply-geolocation-rules`

[corneliusroemer]

There's an unhandled error in apply-geolocation-rules:

$ augur curate apply-geolocation-rules \
                --geolocation-rules /Users/cr/code/mpox/ingest/defaults/geolocation-rules.tsv \
                $INPUTFILE

Traceback (most recent call last):
  File "/Users/cr/micromamba/envs/py13/lib/python3.13/site-packages/augur/__init__.py", line 71, in run
    return args.__command__.run(args)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/Users/cr/micromamba/envs/py13/lib/python3.13/site-packages/augur/curate/__init__.py", line 246, in run
    dump_ndjson(validated_output_records)
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cr/micromamba/envs/py13/lib/python3.13/site-packages/augur/io/json.py", line 89, in dump_ndjson
    for item in iterable:
                ^^^^^^^^
  File "/Users/cr/micromamba/envs/py13/lib/python3.13/site-packages/augur/curate/__init__.py", line 154, in validate_records
    for idx, record in enumerate(records):
                       ~~~~~~~~~^^^^^^^^^
  File "/Users/cr/micromamba/envs/py13/lib/python3.13/site-packages/augur/curate/apply_geolocation_rules.py", line 260, in run
    annotated_values = transform_geolocations(
        default_rules,
        [record.get(field, '') for field in location_fields],
        args.case_sensitive)
  File "/Users/cr/micromamba/envs/py13/lib/python3.13/site-packages/augur/curate/apply_geolocation_rules.py", line 169, in transform_geolocations
    annotated_values = get_annotated_geolocation(geolocation_rules, transformed_values, case_sensitive)
  File "/Users/cr/micromamba/envs/py13/lib/python3.13/site-packages/augur/curate/apply_geolocation_rules.py", line 115, in get_annotated_geolocation
    return get_annotated_geolocation(geolocation_rules, raw_geolocation, case_sensitive, rule_traversal)
  File "/Users/cr/micromamba/envs/py13/lib/python3.13/site-packages/augur/curate/apply_geolocation_rules.py", line 148, in get_annotated_geolocation
    return get_annotated_geolocation(geolocation_rules, raw_geolocation, case_sensitive, rule_traversal)
  File "/Users/cr/micromamba/envs/py13/lib/python3.13/site-packages/augur/curate/apply_geolocation_rules.py", line 115, in get_annotated_geolocation
    return get_annotated_geolocation(geolocation_rules, raw_geolocation, case_sensitive, rule_traversal)
  File "/Users/cr/micromamba/envs/py13/lib/python3.13/site-packages/augur/curate/apply_geolocation_rules.py", line 115, in get_annotated_geolocation
    return get_annotated_geolocation(geolocation_rules, raw_geolocation, case_sensitive, rule_traversal)
  File "/Users/cr/micromamba/envs/py13/lib/python3.13/site-packages/augur/curate/apply_geolocation_rules.py", line 148, in get_annotated_geolocation
    return get_annotated_geolocation(geolocation_rules, raw_geolocation, case_sensitive, rule_traversal)
  File "/Users/cr/micromamba/envs/py13/lib/python3.13/site-packages/augur/curate/apply_geolocation_rules.py", line 115, in get_annotated_geolocation
    return get_annotated_geolocation(geolocation_rules, raw_geolocation, case_sensitive, rule_traversal)
  File "/Users/cr/micromamba/envs/py13/lib/python3.13/site-packages/augur/curate/apply_geolocation_rules.py", line 101, in get_annotated_geolocation
    current_rules = current_rules.get(field_value.lower())
                                      ^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'lower'


An error occurred (see above) that has not been properly handled by Augur.
To report this, please open a new issue including the original command and the error above:
    <https://github.com/nextstrain/augur/issues/new/choose>

[corneliusroemer]

Here, you seem to assume that field_value is a string, which isn't validated.

augur/augur/curate/apply_geolocation_rules.py

Lines 94 to 105 in 4a31ada

	if rule_traversal is None:
	rule_traversal = []
	current_rules = geolocation_rules
	# Traverse the geolocation rules based using the rule_traversal values
	for field_value in rule_traversal:
	# Use lowercase for field_value for case-insensitive rule matching
	current_rules = current_rules.get(field_value.lower())
	# If we hit `None`, then we know there are no matching rules, so stop the rule traversal
	if current_rules is None:
	break

Type hints would make reading this code a lot easier IMO.

[corneliusroemer]

I've improved error handling/validation in #1905

This shows why the error happened:

ERROR: Expected the next traversal target to be a string, but got <class 'NoneType'> instead. Raw geolocation: ['', 'Democratic Republic of the Congo', 'Tshopo,Yahuma', None], rule traversal: ['*', 'Democratic Republic of the Congo', '*']. Current rules: {'*': ('Africa', 'Democratic Republic of the Congo', '*', '*')}.

[victorlin]

Thanks for catching this! In case it's not clear to others, this happens when null instead of empty string is used in the data. A reproducible example:

cat >rules.tsv <<~~
r_raw/c_raw//	r_annotated/c_annotated//
~~

echo '{"region": "r_raw", "country": "c_raw", "division": "", "location": null}' \
| augur curate apply-geolocation-rules --geolocation-rules rules.tsv