Skip to content

feat: init data quality dimensions #11953

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 14 commits into
base: main
Choose a base branch
from
Draft

Conversation

benbenben2
Copy link
Collaborator

@benbenben2 benbenben2 commented May 26, 2025

What

TL;DR:
introduced data quality dimensions (lib/ProductOpener/DataQualityDimensions.pm, tests/unit/dataqualitydimensions.t),
based on states and completeness (lib/ProductOpener/Products.pm),
all other changes are consequences of these changes.

discussion

  1. redundancy between states and data quality info-> changes: moved all states into quality info, except pro platform related tests (en:to-be-exported, en:to-be-automatically-exported). "# On the producers platform, keep track of which products have changes to be exported" not sure what to do with that

  2. strange with steps in the actual implementation and half steps divided by 4 -> changes: increment everything we check and divided passed tests by total count. Note that with previous implementation if all tests pass the score is 1.1

  3. typo packaging -> packagings

  4. packaging-code (emb) should be only for EU product and for products from animal origin

  5. ingredients should be extended for all selected languages. Picture for ingredients as well.

  6. en:checked is for accuracy dimension and not for completeness

  7. origins: refers to "Origin of ingredients" not "Origin of the product and/or its ingredients (English)" and not "countries where sold" this is very rarely input. changes: removed this check.

  8. $notempty can be replaced by the completeness score itself, if the score is 0 notempty is also 0. changes: removed this state.

  9. completed_t, why to have this field? 1) poor chance to have all complete 2) what does it bring? 3) there should be another dashboard separately to see the score overtime, this dashboard would replace that field. changes: removed this field.

  10. removed "en:complete" from gen_sugar/gen_sucres as it seems not working, the following 2 products appeared in the game but do not contain complete
    https://world.openfoodfacts.org/api/v2/product/6001052001018
    https://world.openfoodfacts.org/api/v2/product/5038862130530

Screenshot

Screenshot_20250526_150703

Screenshot_20250526_171901

Related issue(s) and discussion

  • Fixes #-none-

@benbenben2 benbenben2 self-assigned this May 26, 2025
@benbenben2 benbenben2 requested a review from a team as a code owner May 26, 2025 13:31
@github-actions github-actions bot added 🧴 Open Beauty Facts Our cosmetic analysis project https://world.openbeautyfacts.org 📚 Documentation Documentation issues improve the project for everyone. 🐾 Open Pet Food Facts Our pet food analysis project https://world.openpetfoodfacts.org 🧽 Data quality https://wiki.openfoodfacts.org/Quality API READ All READ APIs include Product, Search… 🕹️ Gamification 🧪 tests 📸 Open Products Facts Our project to increase the lifespan of objects. https://world.openproductsfacts.org 🍬 How much sugar https://howmuchsugar.in/ - Mini game powered by Open Food Facts data Display Products Tags Web update all products config exports 🧪 unit tests 👮 Moderation labels May 26, 2025
@github-project-automation github-project-automation bot moved this to To discuss and validate in 🍊 Open Food Facts Server issues May 26, 2025
@codecov-commenter
Copy link

codecov-commenter commented May 26, 2025

❌ 26 Tests Failed:

Tests completed Failed Passed Skipped
7264 26 7238 0
View the top 3 failed test(s) by shortest run time
tests_integration_export_t::0017 - export_database - 3770013801303
Stack Traces | 0.0119s run time
not ok 0017 - export_database - 3770013801303
tests_integration_export_t::0015 - export_database - 3661344653573
Stack Traces | 0.0122s run time
not ok 0015 - export_database - 3661344653573
tests_integration_export_t::0026 - export_database - 9002355004345
Stack Traces | 0.0126s run time
not ok 0026 - export_database - 9002355004345

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@benbenben2 benbenben2 marked this pull request as draft May 26, 2025 15:09
@github-actions github-actions bot added the 💥 Merge Conflicts 💥 Merge Conflicts label Jun 6, 2025
@github-actions github-actions bot added 🧬 Taxonomies https://wiki.openfoodfacts.org/Global_taxonomies 🧪 additives labels Jun 7, 2025
@github-actions github-actions bot added status system Products have a number of completion states, eg: https://world.openfoodfacts.org/states GitHub Actions Pull requests that update Github_actions code labels Jun 7, 2025
@benbenben2
Copy link
Collaborator Author

/update_tests_results

@benbenben2
Copy link
Collaborator Author

/update_tests_results

Copy link

sonarqubecloud bot commented Jun 8, 2025

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🧪 additives API READ All READ APIs include Product, Search… config 🧽 Data quality https://wiki.openfoodfacts.org/Quality Display 📚 Documentation Documentation issues improve the project for everyone. exports 🕹️ Gamification GitHub Actions Pull requests that update Github_actions code 🍬 How much sugar https://howmuchsugar.in/ - Mini game powered by Open Food Facts data 💥 Merge Conflicts 💥 Merge Conflicts 👮 Moderation 🧴 Open Beauty Facts Our cosmetic analysis project https://world.openbeautyfacts.org 🐾 Open Pet Food Facts Our pet food analysis project https://world.openpetfoodfacts.org 📸 Open Products Facts Our project to increase the lifespan of objects. https://world.openproductsfacts.org Products status system Products have a number of completion states, eg: https://world.openfoodfacts.org/states Tags 🧬 Taxonomies https://wiki.openfoodfacts.org/Global_taxonomies 🧪 tests 🧪 unit tests update all products Web
Projects
Status: To discuss and validate
Status: In progress
Status: Todo
Status: Todo
Status: In progress
Status: In progress
Status: Backlog
Status: In progress
Status: In progress
Development

Successfully merging this pull request may close these issues.

2 participants