Skip to content

[AR parser] Add preventive error handling for missing production data #8073

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

taehunlee990803
Copy link

@taehunlee990803 taehunlee990803 commented Apr 25, 2025

Summary

  • Enhanced the AR (Argentina) production parser with advanced error handling.
  • Explicitly check if renewables or non-renewables production data is empty or invalid.
  • Raise ParserException early if data is missing, preventing silent failures.
  • Improve logging: added info logs for record counts and warning logs for partial data issues.

Motivation

Although the AR parser is currently working and Issue #8054 has been closed,
this PR aims to prevent future incidents where missing or invalid production data could silently pass undetected.

By ensuring the parser fails fast and clearly when data is missing,
the ElectricityMap server can detect problems immediately instead of misinterpreting the parser as healthy.

This enhances the robustness and observability of the parser and helps maintain reliable data quality over time.

Changes

  • Update renewables_production_mix() and non_renewables_production_mix() to:
    • Validate HTTP response status
    • Validate JSON structure
    • Check for empty production lists
    • Gracefully handle missing fields in production items
  • Improved logging for easier debugging.
  • Raise ParserException if production data is missing or invalid.

Related Issues

Checklist

  • Successfully ran poetry run test_parser AR production locally and verified correct outputs.
  • Verified that the parser correctly raises ParserException when API returns empty or invalid data.
  • Confirmed that all added logs are meaningful, non-intrusive, and help debugging without noise.
  • Focused changes only on improving parser robustness and preventive error handling.
  • No unrelated changes were introduced.

@github-actions github-actions bot added parser python Pull requests that update Python code labels Apr 25, 2025
@VIKTORVAV99 VIKTORVAV99 requested a review from Copilot April 27, 2025 20:32
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds preventive error handling to the AR production parser to ensure that missing or invalid production data is caught early and handled appropriately.

  • Adds enhanced error handling (timeouts, try/except blocks) for both renewables and non-renewables production data retrieval.
  • Improves logging for debugging by reporting errors and warnings, and raises ParserException when necessary.
Comments suppressed due to low confidence (2)

parsers/CAMMESA.py:118

  • The ParserException's 'parser' field is set to 'AR.py', which may be confusing since this file is named 'CAMMESA.py'. Consider using a consistent identifier that matches the file name to simplify debugging.
raise ParserException(parser="AR.py", message=f"Renewables API error: {str(e)}", zone_key=zone_key)

parsers/CAMMESA.py:152

  • Catching a KeyError during the production record parsing only logs a warning and skips the record, which may lead to incomplete production data. Consider implementing a more robust handling strategy or accumulating and reporting these cases to ensure data integrity.
logger.warning(f"[AR Renewables] Missing expected field {e} in production info: {production_info}")

Copy link
Member

@VIKTORVAV99 VIKTORVAV99 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I fail to see how this solves the problems you describe. If anything it creates more points of failure and complexity as is.

Would love to discuss it more though.

Comment on lines +144 to +149
production=ProductionMix(
biomass=production_info.get("biocombustible", 0.0),
hydro=production_info.get("hidraulica", 0.0),
solar=production_info.get("fotovoltaica", 0.0),
wind=production_info.get("eolica", 0.0),
),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would do the opposite of your intentions here, if we set it to 0 if it's missing there is no way to know if it's actually missing or just 0 when we just look at the data so this we cant do.

api_cammesa_response = session.get(
CAMMESA_DEMANDA_ENDPOINT, params=params, timeout=10
)
api_cammesa_response.raise_for_status()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why raise for status instead of simply checking if the response is okay? with api_cammesa_response.ok and directly raise the parser exception. that way we can avoid the try except statement that adds additional overhead.

)
production_year = production_datetime.year

# use the split from EIA 2022 to split the termico production into gas and oil
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why was this comment removed?

Comment on lines +193 to +194
logger.info(
f"[AR Non-Renewables] Successfully retrieved {len(production_list)} production records."
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will create too many logs, please remove it.

Comment on lines +215 to +219
hydro=production_info.get("hidraulico", 0.0),
nuclear=production_info.get("nuclear", 0.0),
gas=production_info.get("gas", 0.0),
oil=production_info.get("oil", 0.0),
coal=production_info.get("coal", 0.0),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here as above.

Comment on lines +152 to +155
except KeyError as e:
logger.warning(
f"[AR Renewables] Missing expected field {e} in production info: {production_info}"
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this will only work for momento now as the others are accessed used get which means it won't throw any errors.

@VIKTORVAV99 VIKTORVAV99 marked this pull request as draft May 4, 2025 11:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
parser python Pull requests that update Python code
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants