Skip to content

Resource limits now also from services key; Added tests #1173

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 4, 2025

Conversation

ArneTR
Copy link
Member

@ArneTR ArneTR commented May 4, 2025

Greptile Summary

Added comprehensive testing and validation for Docker container resource limits, ensuring consistent configuration between service-level and deploy.resources.limits specifications.

This PR reworks how GMT handles Resource Limits.

Previously this was done through the deploy key in services.
However this key is intended for Docker Swarm.

GMT now also support the cpus and mem_limit keys under services.

When using Docker Compose directly the deploy settings for CPU and Memory have no further implications and will just overwrite any settings given via services top level keys. Implementation is here: https://github.com/docker/compose/blob/5bb46035cf12e5b2fc2f3b09522a392b7b753120/pkg/compose/create.go#L713

  • Added schema validation in lib/schema_checker.py to enforce memory limits as strings and CPU limits as string/float/int
  • Added tests in tests/test_resource_limits.py to verify matching CPU/memory limits between deploy and service-level configs
  • Fixed container name mismatch in test files between service definitions and flow sections
  • Added validation in lib/scenario_runner.py to properly handle and apply resource limits from both configuration locations
  • Added test cases in tests/data/usage_scenarios/ for various resource limit scenarios including invalid configurations

Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

8 file(s) reviewed, 11 comment(s)
Edit PR Review Bot Settings | Greptile

Copy link

github-actions bot commented May 4, 2025

Eco CI Output:

Label 🖥 avg. CPU utilization [%] 🔋 Total Energy [Joules] 🔌 avg. Power [Watts] Duration [Seconds]
Measurement #1 28.6834 3291.37 4.18 786.71
--- --- --- --- ---
Total Run 28.68 3291.37 4.18 786.71
--- --- --- --- ---
Additional overhead from Eco CI N/A 9.32 4.11 2.27

🌳 CO2 Data:
City: Boydton, Lat: 36.6676, Lon: -78.3875
IP: 20.161.77.225
CO₂ from energy is: 1.178310460 g
CO₂ from manufacturing (embodied carbon) is: 0.224459226 g
Carbon Intensity for this location: 358 gCO₂eq/kWh
SCI: 1.402770 gCO₂eq / pipeline run emitted


Total cost of whole PR so far:

@ArneTR ArneTR merged commit 9ef2c87 into main May 4, 2025
@ArneTR ArneTR deleted the resource-limits-extended branch May 4, 2025 07:53
ArneTR added a commit that referenced this pull request May 4, 2025
* main: (27 commits)
  Bump orjson from 3.10.16 to 3.10.18 (#1168)
  Bump pydantic from 2.11.3 to 2.11.4 (#1169)
  Bump psycopg[binary] from 3.2.6 to 3.2.7 (#1171)
  Resource limits now also from services key; Added tests (#1173)
  (fix): Cron job queue check logic was reversed
  Updated ee
  Moving [system] and [machine] to upper case
  Shortened model
  Watchlist must insert usage_scenario_variables
  Comment for MCP
  Job ID now a field to filter by
  Usage Scenario Variables and ScenarioRunner templates (#1172)
  Disabled providers are now removed also from DB entry not only from effective measurement
  (fix): Entries for config options that were null where not correctly showing
  Made temperatur error more helpful
  Typo
  Root DIR of GMT was not accurate
  Renamed Runner to ScenarioRunner and moved to lib/
  Software add now returns job_id on insert; Jobs API now allows filter for job_id [skip ci] (#1170)
  (test-fix): New wording
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant