Submission Tutorial

This tutorial will walk you through the whole submission process. By following it step by step, you will get your results on the hidden test set, and, if you wish, have your results listed on the public leaderboard.

Evaluating on the dev set

Step 0: Create a CodaLab account

  1. Sign up on the website of CodaLab and complete your own information.

  2. If you already have an account, please sign in.

  3. More details can be found in quick start.

Step 1: Create a CodaLab worksheet

  1. Click My Dashboard.

  2. Click New Worksheet to create your own worksheet.

Step 2: Add the dev set and scripts

  1. In the web interface terminal on the top of the page, type in the following command:

    Add Train/val data:

    cl add bundle 0x22dc32baa68f43a1ace0dd3d5921386a .

    Add a script to sample N-way K-shot dataset from a data division:

    cl add bundle 0x2a02b4a8119b4220ad37520d7ee3b5e8 .

    Add the code for evaluating final results:

    cl add bundle 0x1f88373e182e42f5a7a91d3ed0d02124 .

Step 3: Upload your own models and parameters

  1. Click the Upload button in the top-right corner. to upload your own models and parameters. Then, your models and parameters will appear in the contents of the worksheet as several bundles, e.g., we upload our own demo models and parameters as follows,

Step 4: Run evaluation script on the dev set

  1. Write and upload the evaluation script, e.g., we upload our demo evaluation script as follows,

    The code of the script is

    python sample_io.py data/val.json 100 5 5 12345 input > input.json
    python sample_io.py data/val.json 100 5 5 12345 output > output.json
    python test_demo.py input.json > predict.json
    python evaluate.py predict.json output.json

    Note that: More details about data, evaluation scripts and demo examples can be found in our worksheet.

  2. In the web interface terminal on the top of the page, type in the following command:

    Sample instances from the dev set and evaluate the models:

    cl run 'data:0x22dc32' 'sample_io.py:0x2a02b4' 'evaluate.py:0x1f8837' 'checkpoint:0x9f3d8f' 'fewshot_re_kit:0x70219b' '_processed_data:0xb636c7' 'models:0xbe1716' 'test_demo.py:0x8fa335' 'evaluate.sh:0xda5e83' 'bash evaluate.sh'  --request-docker-image codalab/default-cpu

    Note that: for each file, its uuid should be detailed, e.g., 'data:0x22dc32'.

  3. Get the results:

Evaluating on the test set

To preserve the integrity of test results, we do not release the test set to the public. Instead, we require you to upload your model onto CodaLab so that we can run it on the test set for you. Before evaluating your models on the test set, make sure you have completed evaluation of development set as detailed above. Thus we will assume you are already familiar with uploading files and running commands on CodaLab. Note that, our evaluation script is similar to the one used in the public worksheet.

Step 0: Upload your own models and parameters

This step is similar to the step for the dev set.

Step 1: Run your trained model on the dev set

This step is similar to the step for the dev set.

Step 2: Submission

Send a short email to fewrel@googlegroups.com with links to your bundles like:

...
...
checkpoint : 
    cl add bundle 0x9f3d8ff57ebd4449a73a10c9296a15c9 。
test_demo.py :
    cl add bundle 0x8fa33534a6524b24a29254dc86361823 .

...
...

You are also required to email us your evaluation script, commands, and docker information like:

cl run 'data:0x22dc32' 'sample_io.py:0x2a02b4' 'evaluate.py:0x1f8837' \\
'checkpoint:0x9f3d8f' 'fewshot_re_kit:0x70219b' '_processed_data:0xb636c7' \\
 'models:0xbe1716' 'test_demo.py:0x8fa335' \\
'evaluate.sh:0xda5e83' 'bash evaluate.sh'  --request-docker-image codalab/default-cpu

or

cl run 'data:0x22dc32' 'sample_io.py:0x2a02b4' 'evaluate.py:0x1f8837' \\
'checkpoint:0x9f3d8f' 'fewshot_re_kit:0x70219b' '_processed_data:0xb636c7' \\
 'models:0xbe1716' 'test_demo.py:0x8fa335' \\
'evaluate.sh:0xda5e83' 'bash evaluate.sh'  \\
--request-docker-image codalab/default-gpu --request-gpus 1 --request-disk 6g --request-time 2d

If you want your models' result to be listed on the public leaderboard. Please also send us the model name and your affiliation. It would be even better if you could provide a link to your published paper, preprints, website or blogs, describing your submitted model.

Please give up to several days for us to evaluate your model, and give your response.


Should you have any questions, please send an email to fewrel@googlegroups.com.