Please refer to our EMNLP 2018 paper to learn more about this dataset.
To evaluate your model, please sample a pair of input and standard output file from the validation set, and compare your output with the standard one. The sampling scripts and evaluations scripts used in official evaluation are also accessible through the following work sheet.
@inproceedings{hao2018fewrel, title={FewRel:A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation}, author={Han, Xu and Zhu, Hao and Yu, Pengfei and Wang, Ziyun and Yao, Yuan and Liu, Zhiyuan and Sun, Maosong}, booktitle={EMNLP}, year={2018}, }
# | Model | 5-way-1-shot | 5-way-5-shot | 10-way-1-shot | 10-way-5-shot |
---|---|---|---|---|---|
# |
Human | 92.22 | - | 85.88 | - |
1 |
Matching the Blanks
[paper]
Google Research (Feb 26, 2019) |
93.86 ± 0.08 | 97.06 ± 0.04 | 89.20 ± 0.13 | 94.27 ± 0.07 |
2 |
MLMAN
[paper]
[code]
NEL-SLIP, University of Science and Technology of China (Jan 10, 2019) |
82.98 ± 0.20 | 92.66 ± 0.09 | 73.59 ± 0.26 | 87.29 ± 0.15 |
3 |
Prototypical Network (LM-Transformer)
LAVIS, University of Applied Sciences RheinMain (Jan 18, 2019) |
81.40 ± 0.21 | 92.11 ± 0.10 | 72.51 ± 0.28 | 86.03 ± 0.17 |
4 |
TDCN
Anonymous (Jul 23, 2019) |
81.93 ± 0.21 | 92.08 ± 0.10 | 72.18 ± 0.28 | 85.98 ± 0.17 |
5 |
LM-ProtoNet (FGM)
Anonymous (Feb 15, 2019) |
76.60 ± 0.24 | 89.31 ± 0.13 | 65.31 ± 0.31 | 82.10 ± 0.21 |
6 | Prototypical Network (CNN) | 69.20 ± 0.20 | 84.79 ± 0.16 | 56.44 ± 0.22 | 75.55 ± 0.19 |
7 | SNAIL (CNN) | 67.29 ± 0.26 | 79.40 ± 0.22 | 53.28 ± 0.27 | 68.33 ± 0.259 |
8 | GNN (CNN) | 66.23 ± 0.75 | 81.28 ± 0.62 | 46.27 ± 0.80 | 64.02 ± 0.77 |
9 | Meta Network (CNN) | 64.46 ± 0.54 | 80.57 ± 0.48 | 53.96 ± 0.56 | 69.23 ± 0.52 |
10 | kNN (PCNN) | 60.28 ± 0.43 | 72.41 ± 0.39 | 46.15 ± 0.31 | 59.11 ± 0.30 |
11 | kNN (CNN) | 54.67 ± 0.44 | 68.77 ± 0.41 | 41.24 ± 0.31 | 55.87 ± 0.31 |
12 | Finetune (CNN) | 44.21 ± 0.44 | 68.66 ± 0.41 | 27.30 ± 0.28 | 55.04 ± 0.31 |
13 | Finetune (PCNN) | 45.64 ± 0.62 | 57.86 ± 0.61 | 29.65 ± 0.40 | 37.43 ± 0.42 |