--init handles BLEU metric now

This commit is contained in:
Filip Gralinski 2015-08-25 21:26:51 +02:00 committed by Filip Gralinski
parent fc6ef6ed90
commit e6b3d92913
2 changed files with 41 additions and 7 deletions

View File

@ -10,20 +10,22 @@ machine learning algorithms are available here.
## Installing ## Installing
You need [Haskell Stack](https://github.com/commercialhaskell/stack), You need [Haskell Stack](https://github.com/commercialhaskell/stack),
then install with: then install GEval with:
git clone https://github.com/filipg/geval git clone https://github.com/filipg/geval
cd geval cd geval
stack setup stack setup
stack install stack install
By default `geval` library is installed in `$HOME/.local/bin`, so to By default `geval` library is installed in `$HOME/.local/bin`, so in
run `geval` you need to either add `$HOME/.local/bin` to `$PATH` or to order to run `geval` you need to either add `$HOME/.local/bin` to
type: `$PATH` or to type:
PATH="$HOME/.local/bin" geval PATH="$HOME/.local/bin" geval
## Directory structure of a Gonito challenge ## Preparing a Gonito challenge
### Directory structure of a Gonito challenge
A definition of a Gonito challenge should be put in a separate A definition of a Gonito challenge should be put in a separate
directory (preferably as a separate Git repo). Such a directory should directory (preferably as a separate Git repo). Such a directory should
@ -38,8 +40,8 @@ have the following structure:
* `train/` — subdirectory with training data (if training data are * `train/` — subdirectory with training data (if training data are
supplied for a given Gonito challenge at all) supplied for a given Gonito challenge at all)
* `train/train.tsv` — the usual name of training data (this name is * `train/train.tsv` — the usual name of training data (this name is
not required and could be more than file), the first column is the not required and could be more than one file), the first column is the
target (predicted value), the other columns represent features, no target (predicted) value, the other columns represent features, no
header is assumed header is assumed
* `dev-0/` — subdirectory with a development set (a sample test set, * `dev-0/` — subdirectory with a development set (a sample test set,
which won't be used for the final evaluation) which won't be used for the final evaluation)
@ -56,3 +58,16 @@ have the following structure:
organizers of a Gonito challenge, see notes on the structure of organizers of a Gonito challenge, see notes on the structure of
commits below commits below
* `test-B`, `test-C`, ... — other alternative test sets (if supplied) * `test-B`, `test-C`, ... — other alternative test sets (if supplied)
### Initiating a Gonito challenge with geval
You can use `geval` to initiate a Gonito challenge:
geval --init --expected-directory my-challenge
(This will generate a sample toy challenge with guessing the mass of a planet).
A metric (other than the default root-mean-square error) can be given
to generate another type of a toy challenge:
geval --init --expected-directory my-mt-challenge --metric BLEU

View File

@ -68,6 +68,11 @@ configContents metric testName = [i|
|] |]
trainContents :: Metric -> String trainContents :: Metric -> String
trainContents BLEU = [hereLit|alussa loi jumala taivaan ja maan he mea hanga na te atua i te timatanga te rangi me te whenua
ja maa oli autio ja tyhjä , ja pimeys oli syvyyden päällä a kahore he ahua o te whenua , i takoto kau ; he pouri ano a runga i te mata o te hohonu
ja jumalan henki liikkui vetten päällä na ka whakapaho te wairua o te atua i runga i te kare o nga wai
|]
trainContents _ = [hereLit|0.06 0.39 0 0.206 trainContents _ = [hereLit|0.06 0.39 0 0.206
1.00 1.00 1 0.017 1.00 1.00 1 0.017
317.8 5.20 67 0.048 317.8 5.20 67 0.048
@ -75,21 +80,35 @@ trainContents _ = [hereLit|0.06 0.39 0 0.206
|] |]
devInContents :: Metric -> String devInContents :: Metric -> String
devInContents BLEU = [hereLit|ja jumala sanoi : " tulkoon valkeus " , ja valkeus tuli
ja jumala näki , että valkeus oli hyvä ; ja jumala erotti valkeuden pimeydestä
|]
devInContents _ = [hereLit|0.72 0 0.007 devInContents _ = [hereLit|0.72 0 0.007
9.54 62 0.054 9.54 62 0.054
|] |]
devExpectedContents :: Metric -> String devExpectedContents :: Metric -> String
devExpectedContents BLEU = [hereLit|a ka ki te atua , kia marama : na ka marama
a ka kite te atua i te marama , he pai : a ka wehea e te atua te marama i te pouri
|]
devExpectedContents _ = [hereLit|0.82 devExpectedContents _ = [hereLit|0.82
95.2 95.2
|] |]
testInContents :: Metric -> String testInContents :: Metric -> String
testInContents BLEU = [hereLit|ja jumala kutsui valkeuden päiväksi , ja pimeyden hän kutsui yöksi
ja tuli ehtoo , ja tuli aamu , ensimmäinen päivä
|]
testInContents _ = [hereLit|1.52 2 0.093 testInContents _ = [hereLit|1.52 2 0.093
30.06 14 0.009 30.06 14 0.009
|] |]
testExpectedContents :: Metric -> String testExpectedContents :: Metric -> String
testExpectedContents BLEU = [hereLit|na ka huaina e te atua te marama ko te awatea , a ko te pouri i huaina e ia ko te po
a ko te ahiahi , ko te ata , he ra kotahi
|]
testExpectedContents _ = [hereLit|0.11 testExpectedContents _ = [hereLit|0.11
17.2 17.2
|] |]