Add TrueQuery QA Generation by askliar · Pull Request #10559 · pyg-team/pytorch_geometric

askliar · 2025-12-15T18:39:00Z

This PR introduces QA generation procedure, a comprehensive LLM-powered pipeline for generating high-quality question-answer pairs from text documents. The system is designed to create training data for retrieval-augmented generation (RAG) systems and question-answering models.

Note: Original Author of TrueQuery is Vibhor Agrawal of NVIDIA

for more information, see https://pre-commit.ci

…h_geometric

for more information, see https://pre-commit.ci

puririshi98 · 2025-12-16T01:27:01Z

first step, address whatever complaints the CI linter has: https://results.pre-commit.ci/run/github/106024057/1765827433.VRbWJ22KTYipZNbGq422sQ
and update the changelog:
https://github.com/askliar/pytorch_geometric/blob/master/CHANGELOG.md

puririshi98 · 2025-12-16T01:32:51Z

upon quick review no glaring issues. Looking forward to the final restructuring before reviewing deeper and also involving Vibhor for review.
One thing i would ask is that when the final restructuring is done, please attach a log w a succesful run of the full example in the latest pyg container: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pyg/tags

codecov · 2025-12-16T01:33:58Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.91%. Comparing base (c211214) to head (cb696d4).
⚠️ Report is 179 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10559      +/-   ##
==========================================
- Coverage   86.11%   81.91%   -4.20%     
==========================================
  Files         496      511      +15     
  Lines       33655    37778    +4123     
==========================================
+ Hits        28981    30945    +1964     
- Misses       4674     6833    +2159

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

puririshi98

overall lgtm.
2 things:

update changelog.md
please attach a log w a succesful run of the full example in the latest pyg container + pip install vllm: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pyg/tags

puririshi98 · 2026-02-09T19:14:09Z

@askliar plz also add an update to the examples/llm/README.md with enough details for smooth user bringup

…Generator - Implemented automatic dataset download if the input directory does not exist. - Added validation to ensure at least one data source is specified in the configuration. - Enhanced QAGenerator initialization to read additional parameters from the configuration file. - Updated LLMClient to support new parameters for maximum batched tokens. - Introduced new YAML configuration files for different backend setups (nim and vllm). - Minor adjustments in txt2kg_rag.py to ensure proper exit behavior.

…umentation - Fixed FAISS L2→cosine similarity conversion bug in validate_answer_spans_hybrid() that caused 0% validation pass rate (all QA pairs incorrectly rejected) - Fixed AttributeError in LLMClient.cleanup() caused by undefined is_local attribute; now correctly uses the existing backend attribute - Fixed JSON parsing errors by proactively stripping markdown code blocks from LLM responses before parsing, with json_repair as fallback - Added json_repair and langgraph to the rag extras in pyproject.toml - Added comprehensive README for the txt2qa synthetic QA generation pipeline, covering vLLM and NIM backends, quickstart, config reference, troubleshooting, and performance benchmarks - Added success log examples for vLLM and NIM backends in examples/llm/logs/ - Added DEPLOYMENT_SUMMARY.md documenting bugs fixed and test results - Updated vLLM and NIM configuration files Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

for more information, see https://pre-commit.ci

- Add noqa: E402 to late imports required before vLLM initialization - Fix E251 keyword arg formatting by extracting short local variables - Fix E501 long lines across both files by splitting strings and logger calls - Replace Unicode dashes/comparison chars with ASCII equivalents Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

- Keep noqa: E402 for late imports required before vLLM init - Fix broken syntax from pre-commit.ci auto-formatter (E501 split) - Keep our correct long-line fixes; adopt pre-commit.ci's unused-var removal Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

for more information, see https://pre-commit.ci

…rag.py - Replace fabricated vllm_success.log and nim_success.log with actual pipeline run output (real timestamps, model names, progress lines) - Revert examples/llm/txt2kg_rag.py to pyg-team/master (remove stray exit(0)) - Collapse CHANGELOG Unreleased section to a single Added line for txt2qa.py (pyg-team#10559) and add PR links to all four Fixed items - Shorten examples/llm/README.md: update txt2qa table row to one sentence and remove the 260-line TXT2QA guide section - Replace 318-line DEPLOYMENT_SUMMARY.md with a compact ~45-line reference (overview, quick-start commands, critical config flags, output note) Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

puririshi98

overall lgtm. one thing that would make this really solid is add a minimal unit test to test/llm/test_txt2qa.py that generates a single query locally just to make sure this stays working well for posterity. After these changes im happy to merge once Vibhor reviews

puririshi98 · 2026-02-18T20:19:00Z

examples/llm/logs/nim_success.log

+2026-02-18 00:23:18,251 - __main__ - INFO - Final aggregated metrics across all chunks:                             {'average_question_length': 202.625, 'average_answer_length': 413.25, 'complexity_distribution': {5: 13, 4: 3}, 'total_pairs': 16, 'query_type_distribution': {'multi_hop': 8, 'structural': 8}, 'reasoning_type_distribution': {'causal': 9, 'factual': 7}, 'multi_hop_metrics': {'average_hop_count': 3.0, 'hop_count_distribution': {3.0: 6, 2.0: 1, 4.0: 1}}}
+2026-02-18 00:23:18,254 - __main__ - INFO - Wrote 5 QA pairs to /home/scratch.askliar_ent/pytorch_geometric/techqa_output_nim/all_qa_pairs_batch_0.jsonl
+2026-02-18 00:23:18,261 - __main__ - INFO - Done from 0 to 5
+2026-02-18 00:23:18,261 - __main__ - INFO - Process 5 to 10


no need to include the logs in the PR itself, just add them to the top PR description for logging purposes, dont need to fill the examples repo w/ extra file

Yeah, I will remove them once PR is approved. Did not want to add it as a file separately.

…T_SUMMARY.md, and enhance README.md with updated TXT2QA quick start instructions. Introduce tests for txt2qa functionality in test_txt2qa.py.

puririshi98

lgtm just waiting on vibhors stamp but plz move the files to the PR description instead of the actual diff

askliar · 2026-02-20T20:12:20Z

vllm_success.log
nim_success.log

Done @puririshi98

puririshi98 · 2026-02-20T20:19:18Z

@askliar plz also fix the precommit, it wont let us merge otherwise

Andrii Skliar added 2 commits December 15, 2025 19:34

initial truequery_qa_gen commit

6373868

Merge branch 'master' of https://github.com/askliar/pytorch_geometric

c21a068

askliar requested a review from puririshi98 as a code owner December 15, 2025 18:39

askliar changed the title ~~WIP: Add TrueQuery QA Generation~~ Draft: Add TrueQuery QA Generation Dec 15, 2025

pre-commit-ci bot and others added 7 commits December 15, 2025 18:40

[pre-commit.ci] auto fixes from pre-commit.com hooks

a3806bb

for more information, see https://pre-commit.ci

cleanup file structure

742ed00

Merge branches master and master of https://github.com/askliar/pytorc…

b8ada09

…h_geometric

[pre-commit.ci] auto fixes from pre-commit.com hooks

1e2cca6

for more information, see https://pre-commit.ci

Update imports in qa_gen.py to include TypedDict and OpenAI module

54b3c89

Merge branch 'master' of https://github.com/askliar/pytorch_geometric

9e946e8

[pre-commit.ci] auto fixes from pre-commit.com hooks

9919739

for more information, see https://pre-commit.ci

askliar changed the title ~~Draft: Add TrueQuery QA Generation~~ [WIP] Add TrueQuery QA Generation Dec 16, 2025

askliar marked this pull request as draft December 16, 2025 09:27

Andrii and others added 2 commits December 16, 2025 11:35

improve formatting

c4e3570

Merge branch 'master' into master

1067787

puririshi98 requested changes Feb 5, 2026

View reviewed changes

puririshi98 marked this pull request as ready for review February 5, 2026 23:03

puririshi98 added 2 commits February 5, 2026 15:03

Merge branch 'master' into master

0f5b572

Merge branch 'master' into master

04a2385

Andrii and others added 6 commits February 13, 2026 16:02

Merge branch 'master' of https://github.com/askliar/pytorch_geometric

cf35193

Merge branch 'master' into master

90443fe

Merge branch 'master' into master

caa7c0a

Merge branch 'master' of github.com:askliar/pytorch_geometric

580705e

askliar requested a review from rusty1s as a code owner February 17, 2026 19:32

askliar requested a review from akihironitta as a code owner February 17, 2026 19:32

pre-commit-ci bot and others added 5 commits February 17, 2026 19:33

[pre-commit.ci] auto fixes from pre-commit.com hooks

4e4d70d

for more information, see https://pre-commit.ci

[pre-commit.ci] auto fixes from pre-commit.com hooks

b664ae8

for more information, see https://pre-commit.ci

Merge branch 'master' into master

a63304f

askliar marked this pull request as draft February 17, 2026 23:14

root and others added 4 commits February 18, 2026 01:03

Update txt2kg.py

cea73d9

Update CHANGELOG.md

7ee4dd0

Update CHANGELOG.md

4252c7a

askliar changed the title ~~[WIP] Add TrueQuery QA Generation~~ Add TrueQuery QA Generation Feb 18, 2026

askliar marked this pull request as ready for review February 18, 2026 09:06

puririshi98 requested changes Feb 18, 2026

View reviewed changes

Andrii added 2 commits February 20, 2026 18:59

Merge branch 'master' of https://github.com/pyg-team/pytorch_geometric

3d3686c

Add faiss-cpu dependency to pyproject.toml, remove outdated DEPLOYMEN…

92f9966

…T_SUMMARY.md, and enhance README.md with updated TXT2QA quick start instructions. Introduce tests for txt2qa functionality in test_txt2qa.py.

askliar requested a review from wsad1 as a code owner February 20, 2026 18:22

puririshi98 requested changes Feb 20, 2026

View reviewed changes

remove logs

ef89878

Andrii and others added 2 commits February 20, 2026 23:58

format

a88702a

Merge branch 'master' into master

cb696d4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add TrueQuery QA Generation#10559

Add TrueQuery QA Generation#10559
askliar wants to merge 33 commits intopyg-team:masterfrom
askliar:master

askliar commented Dec 15, 2025 •

edited by puririshi98

Loading

Uh oh!

puririshi98 commented Dec 16, 2025

Uh oh!

puririshi98 commented Dec 16, 2025

Uh oh!

codecov bot commented Dec 16, 2025 •

edited

Loading

Uh oh!

puririshi98 left a comment

Uh oh!

puririshi98 commented Feb 9, 2026

Uh oh!

puririshi98 left a comment

Uh oh!

puririshi98 Feb 18, 2026

Uh oh!

askliar Feb 20, 2026

Uh oh!

puririshi98 left a comment •

edited

Loading

Uh oh!

askliar commented Feb 20, 2026

Uh oh!

puririshi98 commented Feb 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

askliar commented Dec 15, 2025 • edited by puririshi98 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

puririshi98 commented Dec 16, 2025

Uh oh!

puririshi98 commented Dec 16, 2025

Uh oh!

codecov bot commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

puririshi98 left a comment

Choose a reason for hiding this comment

Uh oh!

puririshi98 commented Feb 9, 2026

Uh oh!

puririshi98 left a comment

Choose a reason for hiding this comment

Uh oh!

puririshi98 Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

askliar Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

puririshi98 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

askliar commented Feb 20, 2026

Uh oh!

puririshi98 commented Feb 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

askliar commented Dec 15, 2025 •

edited by puririshi98

Loading

codecov bot commented Dec 16, 2025 •

edited

Loading

puririshi98 left a comment •

edited

Loading