Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] in evaluation #15

Open
LiuShifeng opened this issue Oct 16, 2019 · 0 comments
Open

[Bug] in evaluation #15

LiuShifeng opened this issue Oct 16, 2019 · 0 comments

Comments

@LiuShifeng
Copy link

I tested the evaluation scripts with the provided codes, config file, and test files.
However, I found the performance is quite different from the human calculation.

I doubt that the parameter name should be labels rather than target_names in semeval2020_0601_eval.py and semeval2020_0602_eval.py.
OR it would be nice that the authors could specify the version of scikit-learn they used.

The script output is shown below and my scikit-learn version is 0.20.3.

              precision    recall  f1-score   support

      HasDef       0.00      0.00      0.00         2
       NoDef       0.60      1.00      0.75         3

   micro avg       0.60      0.60      0.60         5
   macro avg       0.30      0.50      0.37         5
weighted avg       0.36      0.60      0.45         5


              precision    recall  f1-score   support

      B-Term       0.50      0.50      0.50         2
      I-Term       1.00      1.00      1.00         2
B-Definition       0.88      0.78      0.82         9
I-Definition       0.67      0.80      0.73         5

   micro avg       0.78      0.78      0.78        18
   macro avg       0.76      0.77      0.76        18
weighted avg       0.79      0.78      0.78        18


{'Direct-Defines': {'p': 1.0, 'r': 1.0, 'f': 1.0}, 'macro': {'p': 1.0, 'f': 1.0}}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant