Could you please offer the code for calculating unit testing metric of function completion in RepoEval dataset?

[wyt2000]

Thank you for your work. I hope to reproduce the experiments related to the RepoEval dataset, but the repository only contains the code for calculating EM and ES. Could you please provide the code for calculating the UT metric to help reproduce the function-level code generation experiments? I would greatly appreciate it.

[xiaowu0162]

Hi @wyt2000 ,

Thank you for your interest. For each repository, we created an environment and installed the repository from source. Then, to test each case, we substitute the model's function completion into the source code and trigger unittest with pytest.

Although we haven't tested on it, it seems that the similar functionality is implemented in CodeRAG-Bench (https://github.com/code-rag-bench/code-rag-bench?tab=readme-ov-file#repoeval-function). You may also give it a try.

Best,
Di