linear model
This commit is contained in:
s444417
commit
ca70a52e9f
142
README.md
Normal file
142
README.md
Normal file
@ -0,0 +1,142 @@
|
|||||||
|
|
||||||
|
RetroC2 temporal classification challenge
|
||||||
|
=========================================
|
||||||
|
|
||||||
|
Guess the publication year of a Polish text.
|
||||||
|
|
||||||
|
This is the second (larger and improved) edition of the challenge, see
|
||||||
|
[http://gonito.net/challenge/retroc]() for the first edition.
|
||||||
|
|
||||||
|
Example
|
||||||
|
-------
|
||||||
|
|
||||||
|
For instance, you are expected to guess the publication year of this 500-word
|
||||||
|
text:
|
||||||
|
|
||||||
|
> Gazet, a tam o osobie zamformuie się. Uwiadomienie. Stosownie do
|
||||||
|
> dodatku gazety W. Xiestwa Poznańskiego Nru 74. Ig15. niźey podpisany
|
||||||
|
> odwoluiąc się, w którey wszelkie pełnomocnictwa komukolwiek priez
|
||||||
|
> niego dane, od daty teyźe gazety za nieważne mieć chce, dziś więc
|
||||||
|
> potwierdza to, kassuiac i umarzaiąc pełnomocnictw» Ur. Podgurskiemu
|
||||||
|
> przez siebie uczyni o n e, w P o z n a n i u dnia 3. Mala 1816 r.
|
||||||
|
> Psirohońshi. Odmienienie mieszkania. Donoszę Szanowney Publiczności,
|
||||||
|
> iż mieszkanie moie z Dominikjńskiey ulicy przeniosłem na Szeroką
|
||||||
|
> ulicę do JP. Fi asa pod Nr 114 na pierwszym piętrze, i handel zboża
|
||||||
|
> nadal prowadzić będę. Poznań dnia 6. Maia 1816. Meyer Marcuse. III
|
||||||
|
> 111---»-- Do przedania. Kamienica w rynku podNrcra 62, o trzech
|
||||||
|
> piętrach, wraz z zabudowaniami, w bardzo dobrym znayduiąca się
|
||||||
|
> stanie, do szynku i przyimowania gości urządzona, iest zwolney ręki
|
||||||
|
> do przedania. Dokładnieyszą wiadomość powziąść można u właściciela.
|
||||||
|
> Do przedania. Dom za Świętym Marcinem pod N rem 42. z browarnią,
|
||||||
|
> staynią, studnią i wielkićm podwórzem, niemniey kilkanaście szachtów
|
||||||
|
> kamieni, iest na dniu 24m Czerwca r. b. z wolney .ręki do
|
||||||
|
> sprzedania. Każdy ochotę mający kupna, o kondycyach sprzedaży
|
||||||
|
> dowiedzieć się mole tu w Voznaniu w rynku pod N rem 57. u S tanisław
|
||||||
|
> a PoweIskiego. Do przedania. Na mocy w Prześwietnym Sądzie Pokoiti
|
||||||
|
> Powiatu tuteyszego pomiędzy Szl. Henrykiem Eichbaum, właścicielem
|
||||||
|
> młyna papierni w MuchodzU 5 A 7 II nie, Powiatu Międzyrzeckiego, a
|
||||||
|
> Szl. Wilhelmem Ferdynandem Naukę, Kredy torem pryncypalnym z młyna
|
||||||
|
> wodnego w Muchodzinie, na dniu 29m miesiąca Marca roku bieżąesgo
|
||||||
|
> itawartey i w ley n.ierze do podpisanego uczynionego wniosku,
|
||||||
|
> zesunie młyn papiernia, wraz do tego należącemi gruntami, w wsi
|
||||||
|
> Muchodzinie w Powiecie Międzyrzeckim leżąca, według urzedowey na
|
||||||
|
> dniu I I Kwietnia roku bieżącego zdziałaney taxy, na summe. 2246
|
||||||
|
> Tal. 12 dgr, oszacowana, w drodze lieytacyi public-zney więcey
|
||||||
|
> daiącemu za gotowa Ziraz zjpłatę, i wypełnieniem kondycyi kupna,
|
||||||
|
> sprzedana; do którey to sprzedaży termin pierwszy do publikacyi
|
||||||
|
> kondycyi kupna 1 przedsunowczego przysądzenia, na żądanie
|
||||||
|
> Iineressentow, na dzieli 12. miesiąca Czerwca roku bieżącego
|
||||||
|
> w.kascelląryi Urzędnika podpisanego o godzinie iotey przed południem
|
||||||
|
> wyznaczonym zostaie.- Wzywa się więc ninśeyszem Publiczność kupna
|
||||||
|
> tego ochotę maiącą, oraz wszelcy Kredyiorowie e x q u o c u n q u e
|
||||||
|
> jur e d o młyna tego papierni twierdzić prawa sobie mogący, aby w
|
||||||
|
> terminie wzwyż wyrażonym osobiście lub przez prawnie umocowanych
|
||||||
|
> Pełnomocników stawilisię; pierwsi swe licyta, drudzy zaś swe realne
|
||||||
|
> pretensye do protokółu podali, a nay więcey licytuiącemu nie«
|
||||||
|
> ruchomości powyż wymienioney zprzyległościami przygotowawcze
|
||||||
|
> przysądzenie nastąpi. Kredytprowie zaś "Z swerni pretensyami do
|
||||||
|
> nieruchomości tey za prekludowanych, a to sub prejudicio perpetui
|
||||||
|
> silentii uważani zostaną. Zbiór obiaśnień i kondycyi kupna przeyrzeć
|
||||||
|
> każdy interessuiący może u podpisanego. Międzyrzecz dnia 20.
|
||||||
|
> Kwietnia i816\\ Ur ząd P isars t wa Aktowego Powiatu
|
||||||
|
> Międzyrzeckiego. M. GądkowskL Do przedania. Podaie się do publiczney
|
||||||
|
> wiadomości, iż podpisany Komornik Sądowy Powiatu Krobskiego,
|
||||||
|
> zatradowane inwentarze, to iest: konie, woły, krowy, owce i t. d. i
|
||||||
|
> porządki gospodarskie, Wmu Kamieńskiemu, Possessocpwi dóbr
|
||||||
|
> Sobiałkowskich, za kaucyą na zabezpieczenie inwentarzy gruntowych,
|
||||||
|
> do massy konkursowey JOO.XiazatSujftoivsftjcji należących, w wsi
|
||||||
|
> Sobiałkowie pod Rawiczem
|
||||||
|
|
||||||
|
(Yes, there might be a lot of OCR noise there!)
|
||||||
|
|
||||||
|
The perfect answer for this text is 1816.37021856342 (year with a
|
||||||
|
fraction representing a specific day, May, 15th, 1816 for this
|
||||||
|
example). You could as well return non-integer numbers, for instance
|
||||||
|
if you are sure that the text was published in 1977, but you have no
|
||||||
|
idea on which day, the optimal decision is to return 1977.5.
|
||||||
|
|
||||||
|
The metric is root mean squared error.
|
||||||
|
|
||||||
|
Directory structure
|
||||||
|
-------------------
|
||||||
|
|
||||||
|
* `README.md` — this file
|
||||||
|
* `config.txt` — GEval configuration file
|
||||||
|
* `train/` — directory with training data
|
||||||
|
* `train/train.tsv.xz` — train set (compressed with xz, not gzip!)
|
||||||
|
* `train/meta.tsv.xz` — metadata (do **not** use in training)
|
||||||
|
* `dev-0/` — directory with dev (test) data from the same sources as the train set
|
||||||
|
* `dev-0/in.tsv` — input text for the dev set
|
||||||
|
* `dev-0/expected.tsv` — expected data for the dev set (publication years)
|
||||||
|
* `dev-0/meta.tsv.xz` — metadata (do **not** use while testing)
|
||||||
|
* `dev-1/` — directory with dev (test) data from different source than the train set
|
||||||
|
* `dev-1/in.tsv` — input text for the dev set
|
||||||
|
* `dev-1/expected.tsv` — expected data for the dev set (publication years)
|
||||||
|
* `dev-1/meta.tsv.xz` — metadata (do **not** use while testing)
|
||||||
|
* `test-A` — directory with test data
|
||||||
|
* `test-A/in.tsv` — input text for the test set
|
||||||
|
* `test-A/expected.tsv` — expected data for the test set (hidden)
|
||||||
|
* `test-A/meta.tsv.xz` — hidden metadata
|
||||||
|
|
||||||
|
Structure of data sets
|
||||||
|
----------------------
|
||||||
|
|
||||||
|
Dev and tests test sets are balanced for years (or at least it was
|
||||||
|
attempted to balance them for years — for some years there was not enough material).
|
||||||
|
|
||||||
|
The `dev-0` dataset was created using the same sources as the train set, but
|
||||||
|
`dev-1` and `test-A` were generated using sources different from
|
||||||
|
`dev-0` (and different to each other), so `dev-0` is likely to be
|
||||||
|
easier than `dev-1`.
|
||||||
|
|
||||||
|
Metadata files are given for reference, do not use them for training.
|
||||||
|
|
||||||
|
Format of the train set
|
||||||
|
-----------------------
|
||||||
|
|
||||||
|
The format of the train set is different from test sets. There is more
|
||||||
|
information there and you are free to exploit it.
|
||||||
|
|
||||||
|
TAB-separated columns:
|
||||||
|
|
||||||
|
* beginning of the period in which a text is known to be published, given as a year
|
||||||
|
with a possible fraction (note that various time granularities are given in this
|
||||||
|
data set — daily, monthly, yearly, etc.),
|
||||||
|
* end of the period in which a text is known to be published,
|
||||||
|
* title normalised,
|
||||||
|
* symbol of the source (usually a Polish digital library).
|
||||||
|
* ~500-word-long text snippet.
|
||||||
|
|
||||||
|
Format of the test sets
|
||||||
|
-----------------------
|
||||||
|
|
||||||
|
The input file is just a list of ~500-word-long text snippets, each
|
||||||
|
given in a separate line.
|
||||||
|
|
||||||
|
The `expected.tsv` file is a list of publication years (with fractions).
|
||||||
|
|
||||||
|
Format of the output files
|
||||||
|
--------------------------
|
||||||
|
|
||||||
|
For each input line, publication year should be given (it is the same
|
||||||
|
as `expected.tsv` files). The name of the output files is `out.tsv`.
|
||||||
1
config.txt
Normal file
1
config.txt
Normal file
@ -0,0 +1 @@
|
|||||||
|
--metric RMSE --precision 4
|
||||||
20000
dev-0/expected.tsv
Normal file
20000
dev-0/expected.tsv
Normal file
File diff suppressed because it is too large
Load Diff
20000
dev-0/in.tsv
Normal file
20000
dev-0/in.tsv
Normal file
File diff suppressed because one or more lines are too long
BIN
dev-0/meta.tsv.xz
Normal file
BIN
dev-0/meta.tsv.xz
Normal file
Binary file not shown.
19998
dev-0/out.tsv
Normal file
19998
dev-0/out.tsv
Normal file
File diff suppressed because it is too large
Load Diff
11563
dev-1/expected.tsv
Normal file
11563
dev-1/expected.tsv
Normal file
File diff suppressed because it is too large
Load Diff
11563
dev-1/in.tsv
Normal file
11563
dev-1/in.tsv
Normal file
File diff suppressed because one or more lines are too long
BIN
dev-1/meta.tsv.xz
Normal file
BIN
dev-1/meta.tsv.xz
Normal file
Binary file not shown.
11562
dev-1/out.tsv
Normal file
11562
dev-1/out.tsv
Normal file
File diff suppressed because it is too large
Load Diff
570
run.ipynb
Normal file
570
run.ipynb
Normal file
@ -0,0 +1,570 @@
|
|||||||
|
{
|
||||||
|
"cells": [
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 86,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"import lzma\n",
|
||||||
|
"import sys\n",
|
||||||
|
"from io import StringIO\n",
|
||||||
|
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
|
||||||
|
"import pandas as pd\n",
|
||||||
|
"import numpy\n",
|
||||||
|
"\n",
|
||||||
|
"pathX = \"./train/train.tsv.xz\"\n",
|
||||||
|
"# pathX = \"./train/in.tsv\"\n",
|
||||||
|
"# pathY = \"./train/meta.tsv.xz\"\n",
|
||||||
|
"nrows = 100000"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 87,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"# data = lzma.open(pathX, mode='rt', encoding='utf-8').read()\n",
|
||||||
|
"# stringIO = StringIO(data)\n",
|
||||||
|
"# df = pd.read_csv(stringIO, sep=\"\\t\", header=None)\n",
|
||||||
|
"df = pd.read_csv(pathX, sep='\\t', nrows=nrows, header=None)\n",
|
||||||
|
"# df = df.drop(df.columns, axis=1)\n",
|
||||||
|
"# topics = pd.read_csv(pathY, sep='\\t', nrows=nrows, header=None)"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 88,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"name": "stdout",
|
||||||
|
"output_type": "stream",
|
||||||
|
"text": [
|
||||||
|
"100000\n"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"print(len(df.index))\n",
|
||||||
|
"\n",
|
||||||
|
"# print(len(topics.index))\n"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 66,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"def mergeTexts(a, b, c):\n",
|
||||||
|
" return str(a) + \" \" + str(b) + \" \" + str(c)"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 67,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"def getMean(a, b):\n",
|
||||||
|
" return ((a + b)/2)"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 89,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"df[\"year\"] = df.apply(lambda x: getMean(x[0], x[1]), axis = 1)\n",
|
||||||
|
"df[\"text\"] = df.apply(lambda x: x[4], axis = 1)"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 90,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"data": {
|
||||||
|
"text/html": [
|
||||||
|
"<div>\n",
|
||||||
|
"<style scoped>\n",
|
||||||
|
" .dataframe tbody tr th:only-of-type {\n",
|
||||||
|
" vertical-align: middle;\n",
|
||||||
|
" }\n",
|
||||||
|
"\n",
|
||||||
|
" .dataframe tbody tr th {\n",
|
||||||
|
" vertical-align: top;\n",
|
||||||
|
" }\n",
|
||||||
|
"\n",
|
||||||
|
" .dataframe thead th {\n",
|
||||||
|
" text-align: right;\n",
|
||||||
|
" }\n",
|
||||||
|
"</style>\n",
|
||||||
|
"<table border=\"1\" class=\"dataframe\">\n",
|
||||||
|
" <thead>\n",
|
||||||
|
" <tr style=\"text-align: right;\">\n",
|
||||||
|
" <th></th>\n",
|
||||||
|
" <th>year</th>\n",
|
||||||
|
" <th>text</th>\n",
|
||||||
|
" </tr>\n",
|
||||||
|
" </thead>\n",
|
||||||
|
" <tbody>\n",
|
||||||
|
" <tr>\n",
|
||||||
|
" <th>63552</th>\n",
|
||||||
|
" <td>2013.212329</td>\n",
|
||||||
|
" <td>dnia 10 października 2012r., znak (...)/, Zakł...</td>\n",
|
||||||
|
" </tr>\n",
|
||||||
|
" <tr>\n",
|
||||||
|
" <th>89500</th>\n",
|
||||||
|
" <td>2013.656164</td>\n",
|
||||||
|
" <td>postępowania, skarżąca wniosła, jak na wstępie...</td>\n",
|
||||||
|
" </tr>\n",
|
||||||
|
" <tr>\n",
|
||||||
|
" <th>94039</th>\n",
|
||||||
|
" <td>1925.015068</td>\n",
|
||||||
|
" <td>dzieją się ciągle awantury, nieustają podejrze...</td>\n",
|
||||||
|
" </tr>\n",
|
||||||
|
" <tr>\n",
|
||||||
|
" <th>62566</th>\n",
|
||||||
|
" <td>2012.348361</td>\n",
|
||||||
|
" <td>Samodzielnego Publicznego Zespołu Opieki Zdrow...</td>\n",
|
||||||
|
" </tr>\n",
|
||||||
|
" <tr>\n",
|
||||||
|
" <th>87553</th>\n",
|
||||||
|
" <td>1975.494521</td>\n",
|
||||||
|
" <td>doprowadzających przeładowywane produkty od st...</td>\n",
|
||||||
|
" </tr>\n",
|
||||||
|
" </tbody>\n",
|
||||||
|
"</table>\n",
|
||||||
|
"</div>"
|
||||||
|
],
|
||||||
|
"text/plain": [
|
||||||
|
" year text\n",
|
||||||
|
"63552 2013.212329 dnia 10 października 2012r., znak (...)/, Zakł...\n",
|
||||||
|
"89500 2013.656164 postępowania, skarżąca wniosła, jak na wstępie...\n",
|
||||||
|
"94039 1925.015068 dzieją się ciągle awantury, nieustają podejrze...\n",
|
||||||
|
"62566 2012.348361 Samodzielnego Publicznego Zespołu Opieki Zdrow...\n",
|
||||||
|
"87553 1975.494521 doprowadzających przeładowywane produkty od st..."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"execution_count": 90,
|
||||||
|
"metadata": {},
|
||||||
|
"output_type": "execute_result"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"df = df.drop(columns = [0,1,2,3,4], axis=1)\n",
|
||||||
|
"df.sample(5)"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 91,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"data": {
|
||||||
|
"text/html": [
|
||||||
|
"<div>\n",
|
||||||
|
"<style scoped>\n",
|
||||||
|
" .dataframe tbody tr th:only-of-type {\n",
|
||||||
|
" vertical-align: middle;\n",
|
||||||
|
" }\n",
|
||||||
|
"\n",
|
||||||
|
" .dataframe tbody tr th {\n",
|
||||||
|
" vertical-align: top;\n",
|
||||||
|
" }\n",
|
||||||
|
"\n",
|
||||||
|
" .dataframe thead th {\n",
|
||||||
|
" text-align: right;\n",
|
||||||
|
" }\n",
|
||||||
|
"</style>\n",
|
||||||
|
"<table border=\"1\" class=\"dataframe\">\n",
|
||||||
|
" <thead>\n",
|
||||||
|
" <tr style=\"text-align: right;\">\n",
|
||||||
|
" <th></th>\n",
|
||||||
|
" <th>text</th>\n",
|
||||||
|
" </tr>\n",
|
||||||
|
" </thead>\n",
|
||||||
|
" <tbody>\n",
|
||||||
|
" <tr>\n",
|
||||||
|
" <th>85466</th>\n",
|
||||||
|
" <td>pokoJenie dOInu Burbonów, zaprzestalo we Frano...</td>\n",
|
||||||
|
" </tr>\n",
|
||||||
|
" <tr>\n",
|
||||||
|
" <th>416</th>\n",
|
||||||
|
" <td>non. Jakiekolwiek próby odbudowy takich instyt...</td>\n",
|
||||||
|
" </tr>\n",
|
||||||
|
" <tr>\n",
|
||||||
|
" <th>36354</th>\n",
|
||||||
|
" <td>000 ludzi, a który był iście tryumf.ilnym. trw...</td>\n",
|
||||||
|
" </tr>\n",
|
||||||
|
" <tr>\n",
|
||||||
|
" <th>95566</th>\n",
|
||||||
|
" <td>do robienia lodów. Ogrodzenia do kląbów w wiel...</td>\n",
|
||||||
|
" </tr>\n",
|
||||||
|
" <tr>\n",
|
||||||
|
" <th>58632</th>\n",
|
||||||
|
" <td>ftcitnciu>a4ii. Dzień już dal nogę i mrugali g...</td>\n",
|
||||||
|
" </tr>\n",
|
||||||
|
" </tbody>\n",
|
||||||
|
"</table>\n",
|
||||||
|
"</div>"
|
||||||
|
],
|
||||||
|
"text/plain": [
|
||||||
|
" text\n",
|
||||||
|
"85466 pokoJenie dOInu Burbonów, zaprzestalo we Frano...\n",
|
||||||
|
"416 non. Jakiekolwiek próby odbudowy takich instyt...\n",
|
||||||
|
"36354 000 ludzi, a który był iście tryumf.ilnym. trw...\n",
|
||||||
|
"95566 do robienia lodów. Ogrodzenia do kląbów w wiel...\n",
|
||||||
|
"58632 ftcitnciu>a4ii. Dzień już dal nogę i mrugali g..."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"execution_count": 91,
|
||||||
|
"metadata": {},
|
||||||
|
"output_type": "execute_result"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"topics = df.pop('year')\n",
|
||||||
|
"df.sample(5)"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 92,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"data": {
|
||||||
|
"text/plain": [
|
||||||
|
"22843 1991.500000\n",
|
||||||
|
"12830 1937.157534\n",
|
||||||
|
"63119 1919.500000\n",
|
||||||
|
"77638 2010.130137\n",
|
||||||
|
"5577 1934.768493\n",
|
||||||
|
"Name: year, dtype: float64"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"execution_count": 92,
|
||||||
|
"metadata": {},
|
||||||
|
"output_type": "execute_result"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"topics.sample(5)"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 93,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"data": {
|
||||||
|
"text/plain": [
|
||||||
|
"array(['00', '000', '0000', ..., 'תורהט', 'תותיירב', 'תשדוקמ'],\n",
|
||||||
|
" dtype=object)"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"execution_count": 93,
|
||||||
|
"metadata": {},
|
||||||
|
"output_type": "execute_result"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"vectorizer = TfidfVectorizer(lowercase=True, stop_words=['polish'])\n",
|
||||||
|
"X = vectorizer.fit_transform(df.to_numpy().ravel())\n",
|
||||||
|
"vectorizer.get_feature_names_out()\n"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 73,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"# vectorizer.transform(\"Ala ma kotka\".lower().split())"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 74,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"# df = df.reset_index()"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 94,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"tfidfVector = vectorizer.transform(df[\"text\"])\n",
|
||||||
|
"\n",
|
||||||
|
" "
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 76,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"# from sklearn.model_selection import train_test_split\n",
|
||||||
|
"# from sklearn.naive_bayes import GaussianNB\n",
|
||||||
|
"# \n",
|
||||||
|
"# gnb = GaussianNB()\n",
|
||||||
|
"# gnb.fit(tfidfVector.todense(), topics)"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 95,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"from sklearn.linear_model import LinearRegression\n",
|
||||||
|
"\n",
|
||||||
|
"reg = LinearRegression().fit(tfidfVector, topics)\n"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 106,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"testXPath = \"./dev-0/in.tsv\"\n",
|
||||||
|
"testYPath = \"./dev-0/expected.tsv\"\n",
|
||||||
|
"\n",
|
||||||
|
"testX = pd.read_csv(testXPath, sep='\\t', nrows=19998, header=None)\n",
|
||||||
|
"\n",
|
||||||
|
"testY = pd.read_csv(testYPath, sep='\\t', nrows=19998, header=None)\n"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 107,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"data": {
|
||||||
|
"text/html": [
|
||||||
|
"<div>\n",
|
||||||
|
"<style scoped>\n",
|
||||||
|
" .dataframe tbody tr th:only-of-type {\n",
|
||||||
|
" vertical-align: middle;\n",
|
||||||
|
" }\n",
|
||||||
|
"\n",
|
||||||
|
" .dataframe tbody tr th {\n",
|
||||||
|
" vertical-align: top;\n",
|
||||||
|
" }\n",
|
||||||
|
"\n",
|
||||||
|
" .dataframe thead th {\n",
|
||||||
|
" text-align: right;\n",
|
||||||
|
" }\n",
|
||||||
|
"</style>\n",
|
||||||
|
"<table border=\"1\" class=\"dataframe\">\n",
|
||||||
|
" <thead>\n",
|
||||||
|
" <tr style=\"text-align: right;\">\n",
|
||||||
|
" <th></th>\n",
|
||||||
|
" <th>0</th>\n",
|
||||||
|
" </tr>\n",
|
||||||
|
" </thead>\n",
|
||||||
|
" <tbody>\n",
|
||||||
|
" <tr>\n",
|
||||||
|
" <th>9194</th>\n",
|
||||||
|
" <td>że w moich oczach umizgasz się do niej. Rzeczy...</td>\n",
|
||||||
|
" </tr>\n",
|
||||||
|
" <tr>\n",
|
||||||
|
" <th>3959</th>\n",
|
||||||
|
" <td>reką dopoty, dopóki nic wyiaanię !Prawiü Bye m...</td>\n",
|
||||||
|
" </tr>\n",
|
||||||
|
" <tr>\n",
|
||||||
|
" <th>19210</th>\n",
|
||||||
|
" <td>końcach klapy wentylacyjnej i zapobiegają odch...</td>\n",
|
||||||
|
" </tr>\n",
|
||||||
|
" <tr>\n",
|
||||||
|
" <th>6080</th>\n",
|
||||||
|
" <td>lat cię- owe g\\\\\\\\ allowne walki w dziennikars...</td>\n",
|
||||||
|
" </tr>\n",
|
||||||
|
" <tr>\n",
|
||||||
|
" <th>18845</th>\n",
|
||||||
|
" <td>elektr,yczne] ny, poszanowania ładu i po- roc1...</td>\n",
|
||||||
|
" </tr>\n",
|
||||||
|
" </tbody>\n",
|
||||||
|
"</table>\n",
|
||||||
|
"</div>"
|
||||||
|
],
|
||||||
|
"text/plain": [
|
||||||
|
" 0\n",
|
||||||
|
"9194 że w moich oczach umizgasz się do niej. Rzeczy...\n",
|
||||||
|
"3959 reką dopoty, dopóki nic wyiaanię !Prawiü Bye m...\n",
|
||||||
|
"19210 końcach klapy wentylacyjnej i zapobiegają odch...\n",
|
||||||
|
"6080 lat cię- owe g\\\\\\\\ allowne walki w dziennikars...\n",
|
||||||
|
"18845 elektr,yczne] ny, poszanowania ładu i po- roc1..."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"execution_count": 107,
|
||||||
|
"metadata": {},
|
||||||
|
"output_type": "execute_result"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"testX.sample(5)"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 108,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"data": {
|
||||||
|
"text/html": [
|
||||||
|
"<div>\n",
|
||||||
|
"<style scoped>\n",
|
||||||
|
" .dataframe tbody tr th:only-of-type {\n",
|
||||||
|
" vertical-align: middle;\n",
|
||||||
|
" }\n",
|
||||||
|
"\n",
|
||||||
|
" .dataframe tbody tr th {\n",
|
||||||
|
" vertical-align: top;\n",
|
||||||
|
" }\n",
|
||||||
|
"\n",
|
||||||
|
" .dataframe thead th {\n",
|
||||||
|
" text-align: right;\n",
|
||||||
|
" }\n",
|
||||||
|
"</style>\n",
|
||||||
|
"<table border=\"1\" class=\"dataframe\">\n",
|
||||||
|
" <thead>\n",
|
||||||
|
" <tr style=\"text-align: right;\">\n",
|
||||||
|
" <th></th>\n",
|
||||||
|
" <th>0</th>\n",
|
||||||
|
" </tr>\n",
|
||||||
|
" </thead>\n",
|
||||||
|
" <tbody>\n",
|
||||||
|
" <tr>\n",
|
||||||
|
" <th>3849</th>\n",
|
||||||
|
" <td>1956.476776</td>\n",
|
||||||
|
" </tr>\n",
|
||||||
|
" </tbody>\n",
|
||||||
|
"</table>\n",
|
||||||
|
"</div>"
|
||||||
|
],
|
||||||
|
"text/plain": [
|
||||||
|
" 0\n",
|
||||||
|
"3849 1956.476776"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"execution_count": 108,
|
||||||
|
"metadata": {},
|
||||||
|
"output_type": "execute_result"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"testY.sample()\n"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 109,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"testXtfidfVector = vectorizer.transform(testX[0])\n"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 110,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"data": {
|
||||||
|
"text/plain": [
|
||||||
|
"-0.3101240322770993"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"execution_count": 110,
|
||||||
|
"metadata": {},
|
||||||
|
"output_type": "execute_result"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"reg.score(testXtfidfVector, testY[0])\n"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 115,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"testXPath = \"./dev-1/in.tsv\"\n",
|
||||||
|
"testYPath = \"./dev-1/out.tsv\"\n",
|
||||||
|
"\n",
|
||||||
|
"testX = pd.read_csv(testXPath, sep='\\t', nrows=nrows, header=None)\n",
|
||||||
|
"\n",
|
||||||
|
"# testY = pd.read_csv(testYPath, sep='\\t', nrows=nrows, header=None)\n",
|
||||||
|
"testXtfidfVector = vectorizer.transform(testX[0])\n"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 116,
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"name": "stdout",
|
||||||
|
"output_type": "stream",
|
||||||
|
"text": [
|
||||||
|
"[1967.93839413 1941.34584207 1967.15515902 ... 1935.84850316 1940.56249116\n",
|
||||||
|
" 1958.40938993]\n"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"pred = reg.predict(testXtfidfVector)\n",
|
||||||
|
"print(pred)\n",
|
||||||
|
"\n",
|
||||||
|
"import csv\n",
|
||||||
|
"with open(testYPath, 'w', newline='') as f_output:\n",
|
||||||
|
" tsv_output = csv.writer(f_output, delimiter='\\n')\n",
|
||||||
|
" tsv_output.writerow(pred)"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"metadata": {
|
||||||
|
"interpreter": {
|
||||||
|
"hash": "369f2c481f4da34e4445cda3fffd2e751bd1c4d706f27375911949ba6bb62e1c"
|
||||||
|
},
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python 3.10.4 64-bit",
|
||||||
|
"language": "python",
|
||||||
|
"name": "python3"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"codemirror_mode": {
|
||||||
|
"name": "ipython",
|
||||||
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.10.4"
|
||||||
|
},
|
||||||
|
"orig_nbformat": 4
|
||||||
|
},
|
||||||
|
"nbformat": 4,
|
||||||
|
"nbformat_minor": 2
|
||||||
|
}
|
||||||
124
run.py
Normal file
124
run.py
Normal file
@ -0,0 +1,124 @@
|
|||||||
|
# %%
|
||||||
|
import lzma
|
||||||
|
import sys
|
||||||
|
from io import StringIO
|
||||||
|
from sklearn.feature_extraction.text import TfidfVectorizer
|
||||||
|
import pandas as pd
|
||||||
|
import numpy
|
||||||
|
|
||||||
|
pathX = "./train/train.tsv.xz"
|
||||||
|
# pathX = "./train/in.tsv"
|
||||||
|
# pathY = "./train/meta.tsv.xz"
|
||||||
|
nrows = 100000
|
||||||
|
|
||||||
|
# %%
|
||||||
|
# data = lzma.open(pathX, mode='rt', encoding='utf-8').read()
|
||||||
|
# stringIO = StringIO(data)
|
||||||
|
# df = pd.read_csv(stringIO, sep="\t", header=None)
|
||||||
|
df = pd.read_csv(pathX, sep='\t', nrows=nrows, header=None)
|
||||||
|
# df = df.drop(df.columns, axis=1)
|
||||||
|
# topics = pd.read_csv(pathY, sep='\t', nrows=nrows, header=None)
|
||||||
|
|
||||||
|
# %%
|
||||||
|
print(len(df.index))
|
||||||
|
|
||||||
|
# print(len(topics.index))
|
||||||
|
|
||||||
|
|
||||||
|
# %%
|
||||||
|
def mergeTexts(a, b, c):
|
||||||
|
return str(a) + " " + str(b) + " " + str(c)
|
||||||
|
|
||||||
|
# %%
|
||||||
|
def getMean(a, b):
|
||||||
|
return ((a + b)/2)
|
||||||
|
|
||||||
|
# %%
|
||||||
|
df["year"] = df.apply(lambda x: getMean(x[0], x[1]), axis = 1)
|
||||||
|
df["text"] = df.apply(lambda x: x[4], axis = 1)
|
||||||
|
|
||||||
|
# %%
|
||||||
|
df = df.drop(columns = [0,1,2,3,4], axis=1)
|
||||||
|
df.sample(5)
|
||||||
|
|
||||||
|
# %%
|
||||||
|
topics = df.pop('year')
|
||||||
|
df.sample(5)
|
||||||
|
|
||||||
|
# %%
|
||||||
|
topics.sample(5)
|
||||||
|
|
||||||
|
# %%
|
||||||
|
vectorizer = TfidfVectorizer(lowercase=True, stop_words=['polish'])
|
||||||
|
X = vectorizer.fit_transform(df.to_numpy().ravel())
|
||||||
|
vectorizer.get_feature_names_out()
|
||||||
|
|
||||||
|
|
||||||
|
# %%
|
||||||
|
# vectorizer.transform("Ala ma kotka".lower().split())
|
||||||
|
|
||||||
|
# %%
|
||||||
|
# df = df.reset_index()
|
||||||
|
|
||||||
|
# %%
|
||||||
|
tfidfVector = vectorizer.transform(df["text"])
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
# %%
|
||||||
|
# from sklearn.model_selection import train_test_split
|
||||||
|
# from sklearn.naive_bayes import GaussianNB
|
||||||
|
#
|
||||||
|
# gnb = GaussianNB()
|
||||||
|
# gnb.fit(tfidfVector.todense(), topics)
|
||||||
|
|
||||||
|
# %%
|
||||||
|
from sklearn.linear_model import LinearRegression
|
||||||
|
|
||||||
|
reg = LinearRegression().fit(tfidfVector, topics)
|
||||||
|
|
||||||
|
|
||||||
|
# %%
|
||||||
|
testXPath = "./dev-0/in.tsv"
|
||||||
|
testYPath = "./dev-0/expected.tsv"
|
||||||
|
|
||||||
|
testX = pd.read_csv(testXPath, sep='\t', nrows=19998, header=None)
|
||||||
|
|
||||||
|
testY = pd.read_csv(testYPath, sep='\t', nrows=19998, header=None)
|
||||||
|
|
||||||
|
|
||||||
|
# %%
|
||||||
|
testX.sample(5)
|
||||||
|
|
||||||
|
# %%
|
||||||
|
testY.sample()
|
||||||
|
|
||||||
|
|
||||||
|
# %%
|
||||||
|
testXtfidfVector = vectorizer.transform(testX[0])
|
||||||
|
|
||||||
|
|
||||||
|
# %%
|
||||||
|
reg.score(testXtfidfVector, testY[0])
|
||||||
|
|
||||||
|
|
||||||
|
# %%
|
||||||
|
testXPath = "./dev-1/in.tsv"
|
||||||
|
testYPath = "./dev-1/out.tsv"
|
||||||
|
|
||||||
|
testX = pd.read_csv(testXPath, sep='\t', nrows=nrows, header=None)
|
||||||
|
|
||||||
|
# testY = pd.read_csv(testYPath, sep='\t', nrows=nrows, header=None)
|
||||||
|
testXtfidfVector = vectorizer.transform(testX[0])
|
||||||
|
|
||||||
|
|
||||||
|
# %%
|
||||||
|
pred = reg.predict(testXtfidfVector)
|
||||||
|
print(pred)
|
||||||
|
|
||||||
|
import csv
|
||||||
|
with open(testYPath, 'w', newline='') as f_output:
|
||||||
|
tsv_output = csv.writer(f_output, delimiter='\n')
|
||||||
|
tsv_output.writerow(pred)
|
||||||
|
|
||||||
|
|
||||||
14220
test-A/in.tsv
Normal file
14220
test-A/in.tsv
Normal file
File diff suppressed because one or more lines are too long
14219
test-A/out.tsv
Normal file
14219
test-A/out.tsv
Normal file
File diff suppressed because it is too large
Load Diff
BIN
train/meta.tsv.xz
Normal file
BIN
train/meta.tsv.xz
Normal file
Binary file not shown.
BIN
train/train.tsv.xz
Normal file
BIN
train/train.tsv.xz
Normal file
Binary file not shown.
Loading…
Reference in New Issue
Block a user