73 lines
2.2 KiB
Markdown
73 lines
2.2 KiB
Markdown
# Corpus
|
||
News-Commentary v16 przefiltrowany korpus -- News-Commentary-v16.xz
|
||
|
||
# Statystyki
|
||
|
||
## Po filtrowaniu:
|
||
### Ilość linijek
|
||
|
||
`wc -l`
|
||
|
||
632985
|
||
|
||
### Rozmiar
|
||
|
||
83.0 MiB
|
||
|
||
## Filtrowanie
|
||
|
||
# Użyto biblioteki opusfilter
|
||
|
||
`opusfilter filter_config.yaml`
|
||
|
||
1. Usuwanie duplikatów (2.40% duplicate lines)
|
||
2. Użyto następujących filtrów:
|
||
```
|
||
filters:
|
||
- LengthFilter:
|
||
unit: word
|
||
min_length: 1
|
||
max_length: 300
|
||
|
||
- LengthRatioFilter:
|
||
unit: word
|
||
threshold: 3
|
||
|
||
- LongWordFilter:
|
||
threshold: 40
|
||
|
||
```
|
||
|
||
# Użyto skryptu filter.py, który:
|
||
1. Usuwa linijki w których nie ma ani jednej litery Unicode
|
||
2. Usuwa linijki składające się z jednego słowa, który jest linkiem lub jest alfanumeryczny
|
||
|
||
|
||
## Przed przefiltrowaniem
|
||
|
||
### Ilość linijek
|
||
|
||
`wc -l`
|
||
|
||
648886
|
||
|
||
### Rozmiar
|
||
|
||
84.8 MiB
|
||
|
||
### 10 przykładowych losowych zdań z korpusu:
|
||
|
||
`lm shuf corpora.eng | head`
|
||
|
||
```
|
||
1 The crash is followed by a flight to safety, which is followed by a steep fall in the velocity of money as investors hoard cash.
|
||
2 In this sense, the pandemic represents a unique opportunity to advance European integration like never before.
|
||
3 As depositors flee from a weak bank, they can destroy the bank’s liquidity.
|
||
4 But progress is nonetheless being made.
|
||
5 Critics of the growth model argue that it is imperative to redistribute income and wealth as soon as possible.
|
||
6 All told, countries that have pursued greater economic openness have enjoyed improved nutritional, health, and educational outcomes, as well as higher productivity and incomes.
|
||
7 The periods around World War I and World War II are routinely overlooked in discussions that focus on deregulation of capital markets since the 1980s.
|
||
8 The Greek people deserve some real choices in the near future.
|
||
9 LONDON – The outbreak of the Zika virus, like Ebola before it, has highlighted the risk that infectious diseases can pose to the health of entire countries – and the importance of vaccines to the fight against fast-moving epidemics.
|
||
10 Controls may even require curbing individual freedoms, like accessing hospitals or getting on airplanes.
|
||
``` |