s415366/punctuation_restoration
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Update README

Browse Source
This commit is contained in:
Filip Gralinski 2021-05-21 17:07:00 +02:00
parent 0e2e5a10fd
commit 21ab15b399
1 changed files with 0 additions and 114 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

114
README.md
View File

@ -292,120 +292,6 @@ During the task the following punctuation marks will be evaluated:
ellipsis \...
blank (no punctuation)
### Submission format
Results are to be submitted in a JSON file with the format matching the
input data. Files with results will be tested against the gold standard
annotations kept in the file with the matching text ID in the file name.
#### Example result directory structure
For a given **poleval_text.test.tar.gz** data set with has the following
structure:
+---------------------------+
| **test/** |
| |
| **json/** |
| |
| **wikitalks109264.json** |
| |
| **wikitalks0017548.json** |
| |
| **wikitalks0017518.json** |
| |
| **wikitalks0017499.json** |
| |
| **...** |
| |
| **csv/** |
| |
| **wikitalks109264.csv** |
| |
| **wikitalks0017548.csv** |
| |
| **wikitalks0017518.csv** |
| |
| **wikitalks0017499.csv** |
| |
| **...** |
+---------------------------+
This is the directory structure for **poleval_wav.test.tar.gz**:
+--------------------------------+
| **poleval_final_dataset_wav/** |
| |
| **test/** |
| |
| **wikitalks109264.wav** |
| |
| **wikitalks0017548.wav** |
| |
| **wikitalks0017518.wav** |
| |
| **wikitalks0017499.wav** |
| |
| **...** |
+--------------------------------+
Here is the directory structure for **poleval_fa.test.tar.gz**:
+--------------------------------+
| **poleval_final_dataset/** |
| |
| **test/** |
| |
| **wikitalks109264.clntmstmp** |
| |
| **wikitalks0017548.clntmstmp** |
| |
| **wikitalks0017518.clntmstmp** |
| |
| **wikitalks0017499.clntmstmp** |
| |
| **...** |
+--------------------------------+
The correct submission format should be:
+---------------------------+
| **system_response/** |
| |
| **wikitalks109264.json** |
| |
| **wikitalks0017548.json** |
| |
| **wikitalks0017518.json** |
| |
| **wikitalks0017499.json** |
| |
| **...** |
+---------------------------+
#### Schema validation
Use *jsonschema* to run a sanity check against the file with the
results:
[[https://pypi.org/project/jsonschema/]{.ul}](https://pypi.org/project/jsonschema/)
------------------------------------------------
**\$ jsonschema -i result.json result.schema**
------------------------------------------------
For multiple files:
-----------------------------------------------------------------------
**\$ ls result/\*.json \| xargs -I{} jsonschema -i {} result.schema**
-----------------------------------------------------------------------
### Evaluation Script
Evaluation script will be provided on the task's page.
------------------------------------------------------------
**\$ python3 evaluate.py gold_directory system_directory**
------------------------------------------------------------
### Metrics