Request for the Correct Knowledge base or Abstract Extract Script

[samjia2000]

Hi, I am trying to reproduce the evaluation results and find significant gap as mentioned in a previous issue marked as completed. I guess the gap is mainly caused by the mismatch between the default format of the RAG data provided in the README and the actual RAG data the authors used in the R1-Searcher paper. I wonder have the authors updated the actual knowledge base or the script for extracting abstracts?

Btw. Here are my reproduced evaluation results on HotpotQA.

{'F1': 0.6034644757707149, 'EM': 0.48, 'CEM': 0.544, 'time_use_in_second': 83.04516339302063, 'time_use_in_minite': '1:23'}

[SsmallSong]

hello, I have uploaded the processed kilt dataset to Hugging Face, which can be viewed here: https://huggingface.co/datasets/XXsongLALA/R1-Searcher-kilt . Among them, the contents from 1 to 16 are the split parts, and the complete kilt dataset can be obtained after concatenation. This dataset has not been split into 100 words. You only need to run wiki_corpus_index_build/split_kilt_to_100.py to get the tsv file used for the final evaluation. The results of the three benchmarks, namely bamboogle, 2wiki, and musique, are evaluated based on this dataset.

For hotpotqa, the abstract was downloaded directly without using the extraction script. The download address is https://rocketqa.bj.bcebos.com/corpus/nq.tar.gz. Since the abstract contains less noise, using this dataset for evaluation will yield higher scores.