5 Commits

Author	SHA1	Message	Date
rjawor	0a8d2fdd39	tokenize by whitespace option	2015-12-27 20:54:40 +01:00
rjawor	68fecaddf8	adding all tokenized examples	2015-08-19 20:49:26 +02:00
rjawor	5a57406875	finished original word positions	2015-06-27 12:40:24 +02:00
rjawor	9b1735516c	working sentence tokenizer	2015-06-25 20:49:22 +02:00
rjawor	8432dd321f	tokenizer in progress	2015-06-25 10:12:51 +02:00