Welcome to FewRel

FewRel is a Few-shot Relation classification dataset, which features 70, 000 natural language sentences expressing 100 relations annotated by crowdworkers.

Please refer to our EMNLP 2018 paper to learn more about this dataset.

Baseline models and a series of toolkits are released in this repo:

It is fairly easy for you to get started with FewRel. First, you can download a copy of the dataset by the following links. The dataset is distributed under the CC BY-SA 4.0 license:

To evaluate your model, please sample a pair of input and standard output file from the validation set, and compare your output with the standard one. The sampling scripts and evaluations scripts used in official evaluation are also accessible through the following work sheet.

Once you have built a model that works to your expectations on the validation set, you can submit it to get official scores on the hidden test set. To preserve the integrity of test results, we do not release the test set to the public. Instead, we require you to submit your model so that we can run it on the test set for you. Instructions to submit your model is here!

@inproceedings{hao2018fewrel,
	title={FewRel:A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation},
	author={Han, Xu and Zhu, Hao and Yu, Pengfei and Wang, Ziyun and Yao, Yuan and Liu, Zhiyuan and Sun, Maosong},
	booktitle={EMNLP},
	year={2018},
}
								

#	Model	5-way-1-shot	5-way-5-shot	10-way-1-shot	10-way-5-shot
#	Human	92.22	-	85.88	-
1	Matching the Blanks [paper] Google Research (Feb 26, 2019)	93.86 ± 0.08	97.06 ± 0.04	89.20 ± 0.13	94.27 ± 0.07
2	MLMAN Anonymous (Jan 10, 2019)	82.98 ± 0.20	92.66 ± 0.09	73.59 ± 0.26	87.29 ± 0.15
3	Prototypical Network (LM-Transformer) LAVIS, University of Applied Sciences RheinMain (Jan 18, 2019)	81.40 ± 0.21	92.11 ± 0.10	72.51 ± 0.28	86.03 ± 0.17
4	LM-ProtoNet (FGM) Anonymous (Feb 15, 2019)	76.60 ± 0.24	89.31 ± 0.13	65.31 ± 0.31	82.10 ± 0.21
5	Prototypical Network (CNN)	69.20 ± 0.20	84.79 ± 0.16	56.44 ± 0.22	75.55 ± 0.19
6	SNAIL (CNN)	67.29 ± 0.26	79.40 ± 0.22	53.28 ± 0.27	68.33 ± 0.259
7	GNN (CNN)	66.23 ± 0.75	81.28 ± 0.62	46.27 ± 0.80	64.02 ± 0.77
8	Meta Network (CNN)	64.46 ± 0.54	80.57 ± 0.48	53.96 ± 0.56	69.23 ± 0.52
9	kNN (PCNN)	60.28 ± 0.43	72.41 ± 0.39	46.15 ± 0.31	59.11 ± 0.30
10	kNN (CNN)	54.67 ± 0.44	68.77 ± 0.41	41.24 ± 0.31	55.87 ± 0.31
11	Finetune (CNN)	44.21 ± 0.44	68.66 ± 0.41	27.30 ± 0.28	55.04 ± 0.31
12	Finetune (PCNN)	45.64 ± 0.62	57.86 ± 0.61	29.65 ± 0.40	37.43 ± 0.42