From 21ab15b39936bda9330cf77005288867ba4ea71e Mon Sep 17 00:00:00 2001 From: Filip Gralinski Date: Fri, 21 May 2021 17:07:00 +0200 Subject: [PATCH] Update README --- README.md | 114 ------------------------------------------------------ 1 file changed, 114 deletions(-) diff --git a/README.md b/README.md index ffd8565..79fb14a 100644 --- a/README.md +++ b/README.md @@ -292,120 +292,6 @@ During the task the following punctuation marks will be evaluated: ellipsis \... blank (no punctuation) -### Submission format - -Results are to be submitted in a JSON file with the format matching the -input data. Files with results will be tested against the gold standard -annotations kept in the file with the matching text ID in the file name. - -#### Example result directory structure - -For a given **poleval_text.test.tar.gz** data set with has the following -structure: - -+---------------------------+ -| **test/** | -| | -| **json/** | -| | -| **wikitalks109264.json** | -| | -| **wikitalks0017548.json** | -| | -| **wikitalks0017518.json** | -| | -| **wikitalks0017499.json** | -| | -| **...** | -| | -| **csv/** | -| | -| **wikitalks109264.csv** | -| | -| **wikitalks0017548.csv** | -| | -| **wikitalks0017518.csv** | -| | -| **wikitalks0017499.csv** | -| | -| **...** | -+---------------------------+ - -This is the directory structure for **poleval_wav.test.tar.gz**: - -+--------------------------------+ -| **poleval_final_dataset_wav/** | -| | -| **test/** | -| | -| **wikitalks109264.wav** | -| | -| **wikitalks0017548.wav** | -| | -| **wikitalks0017518.wav** | -| | -| **wikitalks0017499.wav** | -| | -| **...** | -+--------------------------------+ - -Here is the directory structure for **poleval_fa.test.tar.gz**: - -+--------------------------------+ -| **poleval_final_dataset/** | -| | -| **test/** | -| | -| **wikitalks109264.clntmstmp** | -| | -| **wikitalks0017548.clntmstmp** | -| | -| **wikitalks0017518.clntmstmp** | -| | -| **wikitalks0017499.clntmstmp** | -| | -| **...** | -+--------------------------------+ - -The correct submission format should be: - -+---------------------------+ -| **system_response/** | -| | -| **wikitalks109264.json** | -| | -| **wikitalks0017548.json** | -| | -| **wikitalks0017518.json** | -| | -| **wikitalks0017499.json** | -| | -| **...** | -+---------------------------+ - -#### Schema validation - -Use *jsonschema* to run a sanity check against the file with the -results: -[[https://pypi.org/project/jsonschema/]{.ul}](https://pypi.org/project/jsonschema/) - - ------------------------------------------------ - **\$ jsonschema -i result.json result.schema** - ------------------------------------------------ - -For multiple files: - - ----------------------------------------------------------------------- - **\$ ls result/\*.json \| xargs -I{} jsonschema -i {} result.schema** - ----------------------------------------------------------------------- - -### Evaluation Script - -Evaluation script will be provided on the task's page. - - ------------------------------------------------------------ - **\$ python3 evaluate.py gold_directory system_directory** - ------------------------------------------------------------ ### Metrics