Clarification about different experiment types

[sgdgp]

Thank you for an amazing paper and for publicly releasing the codebase. The paper had great research contributions and insights. Additionally this codebase can be very helpful to the fellow researchers in the community to further advance repository level code completion.

My questions:

• Could you please clarify the different experiment types that I found in several files., like baseline, lrcontext, rcfcl_rg1, rcfcl_oracle, etc. I was wondering which is the type used for reproducing the results in the Table 2 of the paper.
• Also, do I need to make any tweaks in this file (https://github.com/amazon-science/Repoformer/blob/main/repo_eval/run_fim_hf.sh) to reproduce the numbers in Table 2

[xiaowu0162]

Hi @sgdgp ,

Thank you for your interest!

• rcfcl_rg1 is the default RAG setting in the paper, and lrcontext is the default non-RAG setting. The other two settings are popular and relevant settings. baseline is non-RAG with only the left context. rcfcl_oracle is RAG with the oracle retrieved contexts (using the answer code as the query).
• The run_fim_hf.sh can be used to obtain the starcoder results in Table 2. For reproducing repoformer results, the file finetuning/evaluation/run_repoeval.sh should be used.