Corpus

News-Commentary v16 przefiltrowany korpus -- News-Commentary-v16.xz

Statystyki

Po filtrowaniu:

Ilość linijek

wc -l

632985

Rozmiar

83.0 MiB

Filtrowanie

Użyto biblioteki opusfilter

opusfilter filter_config.yaml

1. Usuwanie duplikatów (2.40% duplicate lines)
2. Użyto następujących filtrów:

      filters:
        - LengthFilter:
            unit: word
            min_length: 1
            max_length: 300

        - LengthRatioFilter:
            unit: word
            threshold: 3

        - LongWordFilter:
            threshold: 40
        

Użyto skryptu filter.py, który:

1. Usuwa linijki w których nie ma ani jednej litery Unicode
2. Usuwa linijki składające się z jednego słowa, który jest linkiem lub jest alfanumeryczny

Przed przefiltrowaniem

Ilość linijek

wc -l

648886

Rozmiar

84.8 MiB

10 przykładowych losowych zdań z korpusu:

lm shuf corpora.eng | head

     1  The crash is followed by a flight to safety, which is followed by a steep fall in the velocity of money as investors hoard cash.
     2  In this sense, the pandemic represents a unique opportunity to advance European integration like never before.
     3  As depositors flee from a weak bank, they can destroy the bank’s liquidity.
     4  But progress is nonetheless being made.
     5  Critics of the growth model argue that it is imperative to redistribute income and wealth as soon as possible.
     6  All told, countries that have pursued greater economic openness have enjoyed improved nutritional, health, and educational outcomes, as well as higher productivity and incomes.
     7  The periods around World War I and World War II are routinely overlooked in discussions that focus on deregulation of capital markets since the 1980s.
     8  The Greek people deserve some real choices in the near future.
     9  LONDON – The outbreak of the Zika virus, like Ebola before it, has highlighted the risk that infectious diseases can pose to the health of entire countries – and the importance of vaccines to the fight against fast-moving epidemics.
    10  Controls may even require curbing individual freedoms, like accessing hospitals or getting on airplanes.