Skip to content

Dev #322

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 325 commits into
base: main
Choose a base branch
from
Open

Dev #322

wants to merge 325 commits into from

Conversation

maxachis
Copy link
Collaborator

No description provided.

maxachis and others added 30 commits March 25, 2025 15:08
…_auto_url_metadata

feat(app): Add Miscellaneous URL Metadata Task
…_additional_information

feat(app): Add additional information to Final Review Process
…_final_review_multiple_agency_ids

Mc 173 final review multiple agency ids
Previously, agency ids not already in the database were rejected. Now these are permitted with a placeholder name
…_agency_ids

feat(app): `/review/approve-source` new agencies added to db
…_validated_urls

fix(database): Fix bug causing validated URLs to show up for some annotations
# Conflicts:
#	collector_db/enums.py
#	core/AsyncCore.py
…_auto_googler_erroring_frequently

Set default for snippet if none exists.
…_filter_by_batch_id

feat(app): Add batch filtering for annotation requests
Additionally, modify `review/approve-source` to no longer accept `relevant` key-value pair.
…_review_reject_source

feat(app): add `review/reject-source` endpoint
Now, URLs marked not relevant by a user should not show up for subsequent annotations such as record type or agency.
…_hide_user_not_relevant

feat(database): Adjust annotation logic for URLs marked not relevant
Previously, the `/annotate/agencies` `POST` method would return a 500 error when an agency whose ID was not yet in the DBI database was submitted. This has been resolved.
…_add_agency_ids_via_annotate_agencies

feat(database): add agency not in database in annotate agencies
BREAKING CHANGE: All `/review/`endpoints now require the `source_collector_final_review` permission
maxachis and others added 26 commits June 16, 2025 12:02
# Conflicts:
#	src/db/queries/implementations/core/get_next_url_for_final_review.py
#	tests/automated/integration/collector_db/test_db_client.py
#	tests/automated/integration/tasks/test_agency_preannotation_task.py
#	tests/manual/pdap_client/test_pdap_client.py
…_add_progress_bar_to_final_review

mc_313_add_progress_bar_to_final_review
Extract logic to AnnotationExistsCTEQueryBuilder
Begin work on GetMetricsURLSAggregatedPendingQueryBuilder
…_metrics_urls_aggregate_pending

mc_312_metrics_urls_aggregate_pending
…_update_progress_bar_for_final_review

mc_313_update_progress_bar_for_final_review
Copy link

gitguardian bot commented Jun 23, 2025

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
16713553 Triggered Generic Password 72ed9c9 local_database/setup_fdw.sh View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

maxachis and others added 2 commits July 5, 2025 14:14
…mocked_url_request_interface

Add logic for storing compressed html when scraping HTML
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants