filipg/geval
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
472 Commits 1 Branch 0 Tags 681 KiB
61eb437909
Commit Graph

320 Commits

Author SHA1 Message Date
Filip Gralinski
dab2646798 Start working on --list-metrics options 2019-08-21 23:44:18 +02:00
Filip Gralinski
4d069e8102 Handle preprocessing operations for metrics 2019-08-12 17:50:48 +02:00
Filip Gralinski
4452095538 More checks in validation 2019-08-10 16:31:54 +02:00
Filip Gralinski
9b79b8761d Check whether the maximum values is obtained during the validation 2019-08-10 15:55:51 +02:00
Filip Gralinski
fcb5c454c4 Add forgotten file 2019-08-10 13:00:10 +02:00
Filip Gralinski
2236899c3d Refactor (introduce GEval.Metric) 2019-08-10 12:30:17 +02:00
Karol Kaczmarek
19d231e140 Add simple validation of a challenge (--validate option) 2019-08-05 09:28:12 +02:00
Filip Graliński
b4ad774623 Trying to get references 2019-05-23 16:16:05 +02:00
Filip Graliński
780b7016c5 Refactor feature extraction 2019-05-23 10:03:26 +02:00
Filip Gralinski
e5220d71d8 Clip loess and switch to gaussian in Calibration 2019-03-19 20:41:06 +01:00
Filip Gralinski
de40851b5a Fix corner cases in calibration 2019-03-19 19:44:19 +01:00
Filip Gralinski
feacd5844c Minor fix in plotting the graph 2019-03-19 15:11:04 +01:00
Filip Gralinski
9beb68867a Fix grave error in calculating calibration 2019-03-19 15:10:23 +01:00
Filip Gralinski
c578efe28d Change lambda parameter in Loess 2019-03-19 08:45:58 +01:00
Filip Gralinski
d98003fc0e Introduce lambda constant in Loess 2019-03-19 08:35:41 +01:00
Filip Gralinski
adadf14888 Fix simple mistake when plotting a Loess graph 2019-03-19 07:46:10 +01:00
Filip Graliński
eb10a4c3b4 Add plotting graphs for selected metrics 2019-03-19 07:31:17 +01:00
Filip Gralinski
816c83f183 Handle Probabilistic-Soft-F1 when creating a challenge 2019-03-12 22:39:32 +01:00
Filip Graliński
ae27029f61 Implement Probabilistic-Soft-F1 2019-03-12 22:35:19 +01:00
Filip Graliński
8393bec3ae Implement auxiliary calibration function 2019-03-12 08:58:21 +01:00
Filip Graliński
19642db43f Add auxilliary functions 2019-02-22 11:22:12 +01:00
Filip Gralinski
fcb16d43f1 Export extensionsHandled 2019-02-14 22:29:44 +01:00
Filip Gralinski
1a9fe36a9e Handle JSONl (for MultiLabel-F) 2019-02-14 19:01:53 +01:00
Filip Graliński
2ea53f92c7 Refactor gevalCore 2019-02-14 16:48:55 +01:00
Filip Graliński
709eeec4ef Merge branch 'master' into jsonl 2019-02-14 16:35:41 +01:00
Filip Graliński
4b85c4c1bb Merge branch 'master' of ssh://gonito.net/geval 2019-02-14 16:26:19 +01:00
Filip Graliński
b2e3293a12 Refactor line-by-line mode 2019-02-14 16:25:28 +01:00
Filip Gralinski
26e9735d31 Handle jsonl files 2019-02-14 10:54:25 +01:00
Filip Gralinski
872724722a Fix SMAPE on zero values 2019-02-12 08:36:52 +01:00
Filip Graliński
af21031172 Do not preprocess outputs for some metrics 2019-02-01 13:10:45 +01:00
Filip Graliński
ad5d614f48 Filter out NaN values so that sorting is not poisoned 2019-02-01 12:32:56 +01:00
Filip Graliński
f7bd1b2ccd Add missing file, when generating a challenge 2019-02-01 12:04:52 +01:00
Filip Gralinski
abcce9bf68 Add numerical features 2019-02-01 10:58:29 +01:00
Filip Gralinski
d5a8908599 Refactor Features into Factors 2019-01-26 19:26:45 +01:00
Filip Gralinski
1c3908b273 Refactor CartesianFeature type 2019-01-26 18:00:36 +01:00
Filip Gralinski
ea5de5c719 Introduce existential features 2019-01-26 17:18:41 +01:00
Filip Gralinski
dbf5c961af Start numerical factors 2019-01-23 13:00:37 +01:00
Filip Gralinski
1aee476434 Add --filtre option 2019-01-14 23:23:50 +01:00
Filip Gralinski
4003715726 Fix issue with sorting 2019-01-13 12:09:15 +01:00
Filip Graliński
b0c75cac3a Change features into "factors" 
(Just the terminology was changed)
 2019-01-11 16:08:56 +01:00
Filip Graliński
de901d4c64 Add min-cartesian-feature (as optional value) 2019-01-11 10:16:39 +01:00
Filip Gralinski
dbe1613052 Filtre out unwanted Cartesian features 2019-01-11 08:47:11 +01:00
Filip Gralinski
39bc3964b3 Speed up cartesian features 2019-01-10 22:53:43 +01:00
Filip Graliński
23aad86e72 Add cartesian features to black-box debugging 
But it's very slow now, needs to be sped up
 2019-01-10 14:01:29 +01:00
Filip Graliński
99e3a10791 Add bigram features in black-box debugging 2019-01-10 10:41:55 +01:00
Filip Graliński
13f9629cbc Minor refactor 2019-01-10 10:00:51 +01:00
Filip Graliński
212457077f Consider word shapes in black-box debugging 2019-01-10 09:58:04 +01:00
Filip Gralinski
e0cfb9c4b0 Add --min-frequency for black box debugging 2019-01-10 08:15:34 +01:00
Filip Graliński
1832a23b75 Refactor features 2019-01-09 17:45:06 +01:00
Filip Gralinski
5d19fc7585 Add character-by-character tokenization. 2018-12-17 07:54:12 +01:00
Filip Gralinski
d671989a09 Merge branch 'fix-haddock' of git.applica.pl:piotr.halama/geval 2018-12-14 07:28:27 +01:00
Piotr Halama
95fd1787ac Reformat type expressions to fix Haddock parse errors 2018-12-13 17:14:21 +01:00
Filip Gralinski
310ece7f2d Get rid of f**king Windows end-of-line characters 
I hate it so much. Die, die, stupid \r
 2018-12-11 13:25:19 +01:00
Filip Graliński
6f1344e73e Git-annex remote can be specified with --submit option 2018-12-07 09:21:02 +01:00
Filip Gralinski
57ec495bc6 Fix mistakes in the implementation of 13a and v14 tokenizers. 2018-11-17 16:56:16 +01:00
Filip Gralinski
9322307813 Handle more than one possibility in TokenAccuracy 2018-10-24 08:02:34 +02:00
Filip Gralinski
2e816c4e38 Add TokenAccuracy metric 2018-10-23 17:01:33 +02:00
Filip Gralinski
f814fc2c79 Merge branch 'warnings' of https://gitlab.com/fintara/geval 2018-10-23 08:46:49 +02:00
Filip Graliński
8735610745 Implement soft f-score 2018-10-17 22:41:46 +02:00
Filip Gralinski
5dc6e13191 Add Pearson and Spearman correlation measures 2018-09-27 21:52:02 +02:00
Filip Gralinski
b3800bc1d9 Add Macro-F-measure to help 2018-09-27 18:23:26 +02:00
Filip Graliński
782c556f8c Add Macro-F1 metric 2018-09-27 18:21:56 +02:00
Filip Gralinski
3a852ed081 Speed up GLEU (cntd.) 2018-09-26 22:27:59 +02:00
Filip Gralinski
eb395d9be0 Add WER metric 2018-09-25 08:13:57 +02:00
Filip Gralinski
4f09a1802f Speed up GLEU 2018-09-25 07:10:17 +02:00
Filip Gralinski
b419aa7b08 Handle dot decimal separator in parameters 2018-09-12 20:52:56 +02:00
Filip Graliński
83b6b39fca Fix error message 2018-09-12 13:48:31 +02:00
Filip Graliński
b46caaa702 Fix README.md generated for LikelihoodHashed 2018-09-12 12:44:36 +02:00
Filip Gralinski
5cff29cf06 Add GLEU 2018-09-11 08:03:07 +02:00
Filip Gralinski
c6d48c57f6 improve documentation on geval --submit 2018-09-01 16:39:00 +02:00
Filip Gralinski
e2c3102cc4 check whether the remote tracking branch exists 2018-09-01 14:39:34 +02:00
Filip Graliński
eaa791cf2f improvement for "submit" special command 2018-08-28 18:58:51 +02:00
Piotr Halama
bd7c789bae Implement --submit command 2018-08-27 17:57:07 +02:00
Filip Gralinski
421d2e9797 add minimalistic tokenizer 2018-08-17 18:13:27 +02:00
Filip Gralinski
c79c4b356e fix some warnings 2018-08-17 17:52:41 +02:00
Filip Gralinski
8b7a18b4c7 v14 tokenizer added 2018-08-17 17:45:01 +02:00
Filip Gralinski
5e5a58210e use tokenization when looking for worst features 2018-08-17 17:27:25 +02:00
Filip Gralinski
0871b57bbc add --just-tokenize option 2018-08-17 16:57:47 +02:00
Filip Gralinski
83550688ce first tokenizer 2018-08-13 10:09:55 +02:00
Filip Gralinski
d3da3a0ca5 WIP 2018-08-13 07:39:06 +02:00
Filip Gralinski
8388ab4d27 towards tokenization 2018-08-11 22:59:43 +02:00
Filip Graliński
de52a12b03 export some functions from OptionsParser 2018-08-10 16:09:41 +02:00
Filip Graliński
5098225bc1 improvements in challenge creation 2018-08-10 13:05:42 +02:00
Filip Graliński
e10f92cf9c create challenge with MultiLabelLikelihood/LogLoss 2018-08-09 16:35:31 +02:00
Filip Graliński
efcceae26a implement MultiLabel-LogLoss and MultiLabel-Likelihood 2018-08-09 16:00:19 +02:00
Filip Graliński
bd2bfde287 MultiLabel-F1 works on labels given with probs now 2018-08-09 14:08:54 +02:00
Filip Graliński
82bdf70031 add missing metric to help 2018-08-09 12:47:52 +02:00
Filip Graliński
da2114e6d2 reverse sides when diffing 2018-08-07 16:21:37 +02:00
Filip Graliński
e55b8539f1 option -r can be used with -m 2018-08-07 15:55:04 +02:00
Filip Gralinski
c385710719 showing most worsening features 2018-08-06 22:22:33 +02:00
Filip Gralinski
3f3d1fd287 refactor worst features 2018-08-06 21:34:38 +02:00
Filip Graliński
7503644bbe sort in --worst-features 2018-08-06 12:09:31 +02:00
Filip Graliński
bc1de4c3e6 worst features show average score now 2018-08-06 11:59:04 +02:00
Filip Graliński
51abed6fa4 count the number of lines correctly 2018-08-03 11:16:28 +02:00
Filip Gralinski
8dac79fab2 clean up listing worst features 2018-08-02 22:09:25 +02:00
Filip Graliński
020b93ccf8 p-value for features counted 2018-08-02 12:50:13 +02:00
Filip Graliński
f8418894fb Merge branch 'worst-features' of ssh://gonito.net/geval into worst-features 2018-08-02 08:31:08 +02:00
Filip Graliński
cd30d88998 fix some warnings 2018-08-02 08:29:52 +02:00
Filip Gralinski
2b1cf80601 implement ranking conduit 2018-08-01 22:39:34 +02:00
Filip Graliński
4b3a4fa665 implement MultiLabel-F metric 2018-07-26 13:01:10 +02:00
Filip Gralinski
c0fd359590 refactor for Gonito 2018-07-14 09:48:45 +02:00
Filip Graliński
0c6032d166 print params 2018-07-10 16:22:28 +02:00
Filip Graliński
9f5882719b param can take an empty value 2018-07-10 12:10:02 +02:00
Filip Graliński
ab635f2594 add helper function for parsing params in file paths 2018-07-10 11:18:52 +02:00
Filip Graliński
0708b746a9 fix handling compressed files 2018-06-29 16:59:00 +02:00
Filip Graliński
010f0f46ab export function needed by Gonito 2018-06-28 17:00:18 +02:00
Filip Graliński
1278081a48 results are sorted in the natural manner when multiple outputs are evaluated 2018-06-28 16:32:46 +02:00
Filip Graliński
338ddb7fbf fully handle multiple outputs 2018-06-28 16:22:22 +02:00
Filip Graliński
ba26cdb9e0 multiple outs are recognised but not handled 2018-06-28 15:36:47 +02:00
Filip Graliński
656a194f42 start refactoring to enable evaluating multiple outputs 2018-06-28 14:49:44 +02:00
Tsvetan Ovedenski
f6ad2f0a85
Remove warnings in Core 2018-06-20 11:48:03 +02:00
Filip Gralinski
0a2e1fcc32 docs on PrecisionAndRecall 2018-06-13 15:36:23 +02:00
Filip Graliński
e0e06196f0 Merge branch 'handle-version-option' into 'master' 
Added version flag handling, added changelog

Closes #7

See merge request filipg/geval!1
 2018-06-13 10:46:34 +00:00
Filip Gralinski
012578f32a implement mean absolute error 2018-06-13 12:30:11 +02:00
Tomasz Weissbek
964957b1db Added version flag handling, added changelog 2018-06-13 12:19:06 +02:00
Filip Gralinski
1073407760 improve documentation 2018-06-12 21:52:18 +02:00
Filip Graliński
86d50b92b7 multiple metrics can be specified 2018-06-08 12:38:45 +02:00
Filip Gralinski
ffb24509d7 handle http(s):// 2018-06-02 23:27:49 +02:00
Filip Gralinski
57ee8a1296 switch to smart sources 2018-06-02 20:24:34 +02:00
Filip Gralinski
18ed47322e Merge branch 'master' into smart-conduit 2018-06-02 16:31:36 +02:00
Filip Gralinski
f9dfbc1466 accuracy can work on probablity distributions now 2018-06-02 12:24:14 +02:00
Filip Gralinski
d370e375a0 add --alt-metric option 2018-06-02 11:29:54 +02:00
Filip Gralinski
4768931221 add BIO-F1-Labels metric 2018-05-29 22:04:19 +02:00
Filip Gralinski
65e8d2562e underscores can be used in the BIO format 2018-05-29 20:59:00 +02:00
Filip Gralinski
3f7384f467 add --sort and --reverse-sort options 2018-05-28 10:04:27 +02:00
Filip Gralinski
ab1056301e add sorting for --line-by-line internally 2018-05-28 09:45:08 +02:00
Filip Gralinski
f68223409e add test for the line-by-line mode 2018-05-26 21:10:22 +02:00
Filip Gralinski
cb655cd2ae refactor LineByLine 2018-05-26 14:40:26 +02:00
Filip Gralinski
c71c7a019d remove warning in LineByLine.hs 2018-05-26 13:09:06 +02:00
Filip Graliński
881a77e239 better diagnostic messages for BIO 2018-05-25 14:44:19 +02:00
Filip Gralinski
3e201d11ef update for Stack LTS 11.9 2018-05-19 13:49:53 +02:00
Filip Graliński
192d531969 add likelihood as evaluation metrics 2018-05-17 15:21:03 +02:00
Filip Gralinski
438f013914 automatic decompression 2018-05-17 08:26:57 +02:00
Filip Gralinski
01b93dd243 improve help for geval --init 2018-05-16 21:00:45 +02:00
Filip Gralinski
b01f9439b7 log probs 2018-05-16 20:59:40 +02:00
Filip Gralinski
82e794ae3c implement BIO-F1 2018-05-16 10:51:50 +02:00
Filip Gralinski
9fc4beaba1 improve sample challenge for LogLossHashed 2018-05-15 08:14:52 +02:00
Filip Gralinski
06fd093349 probs can be given for LogLossHashed 2018-05-15 08:07:47 +02:00
Filip Gralinski
bdcd26cddc WIP 2018-05-14 10:37:58 +02:00
Filip Gralinski
cea084c789 accuracy can work on probs now 2018-04-07 21:13:37 +02:00
Filip Gralinski
ff8ec8880e add LogLoss 2018-04-07 08:29:58 +02:00
Filip Gralinski
9d4aab5f2c diff 2018-02-20 21:28:14 +01:00
Filip Gralinski
88f69156e7 refactor code 2018-02-20 21:28:14 +01:00
Filip Gralinski
5ae8036efc add short options, improve help 2018-02-20 21:28:14 +01:00
Filip Gralinski
b51944b930 --init considers --precision now 2018-02-20 21:28:14 +01:00
Filip Gralinski
6cfefed0c1 precision is part of specification now 2018-02-20 21:28:14 +01:00
Filip Gralinski
f32564a42a write wrong line number correctly in line-by-line mode 2018-02-20 21:28:14 +01:00
Filip Gralinski
5c00ab6d26 show line number when something wrong 2018-02-20 21:28:14 +01:00
Filip Gralinski
b323e6148c refactor parse errors (use Either instead of throwing an error) 2018-02-20 21:28:14 +01:00
Filip Gralinski
c70d49c418 add line-by-line mode 2018-02-20 21:28:13 +01:00