s478874/Modelowanie-Corpus
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
2 Commits 1 Branch 0 Tags 69 KiB
Shell 53.7%
Python 46.3%
120587d98d
Go to file
Kuba 120587d98d Small script correction
2023-03-16 00:07:03 +01:00
.gitignore Create corpus script 2023-03-15 23:59:24 +01:00
readme.md Create corpus script 2023-03-15 23:59:24 +01:00
wiki.sh Small script correction 2023-03-16 00:07:03 +01:00

readme.md

Text corbus

based on Wikipedia EN Latest Articles -- enwiki-latest-pages-articles.xml.bz2 created with wiki.sh script

Code

  • include only article text in between marks
  • exclude lines without alphabet letters
  • clear excessive white spaces
  • remove special characters
  • remove unintentional empty lines

Stats

No lines

wc -l

Size

du -sh enwiki-latest-corpus.txt.bz2

Head of file

bzcat enwiki-latest-corpus.txt.bz2 | head -n 5

Random lines from file

bzcat enwiki-latest-corpus.txt.bz2 | shuf -n 5