326 lines
9.2 KiB
Plaintext
326 lines
9.2 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "35c19016",
|
||
"metadata": {},
|
||
"source": [
|
||
"\n",
|
||
"<div class=\"alert alert-block alert-info\">\n",
|
||
"<h1> Ekstrakcja informacji </h1>\n",
|
||
"<h2> 8. <i>Regresja liniowa</i> [wykład]</h2> \n",
|
||
"<h3> Filip Graliński (2021)</h3>\n",
|
||
"</div>\n",
|
||
"\n",
|
||
""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "cathedral-newark",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Regresja liniowa\n",
|
||
"\n",
|
||
"Regresja liniowa jest prosta...\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"... dosłownie — dopasuj prostą $y = ax + b$ do punktów\n",
|
||
"\n",
|
||
"Należy odgadnąć $a$ i $b$ tak, aby zminimalizować błąd\n",
|
||
"kwadratowy, tj. wartość:\n",
|
||
"\n",
|
||
"$$\\sum_{i=1}^n (y_i - (ax_i + b))^2$$\n",
|
||
"\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "heard-clinton",
|
||
"metadata": {},
|
||
"source": [
|
||
"Regresje liniowa (jednej zmiennej) jest łatwa do rozwiązania — wystarczy podstawić do wzoru!\n",
|
||
"\n",
|
||
"$$\\hat{b} = \\frac{ \\sum_{i=1}^{n}{x_i y_i} - \\frac{1}{n} \\sum_{i=1}^n x_i\n",
|
||
" \\sum_{j=1}^n y_j}{ \\sum_{i=1}^n {x_i^2} - \\frac{1}{n} (\\sum_{i=1}^n\n",
|
||
" x_i)^2 }$$\n",
|
||
"\n",
|
||
"$$\\hat{a} = \\bar{y} - \\hat{b}\\,\\bar{x}$$\n",
|
||
"\n",
|
||
"\n",
|
||
"Na przykład dla mieszkań: $b =$ -30809.203 zł, $a =$ 5733.693 zł/m$^2$.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "preceding-impression",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Regresja wielu zmiennych\n",
|
||
"\n",
|
||
"W praktyce mamy do czynienia z **wielowymiarową** regresją\n",
|
||
"liniową.\n",
|
||
"\n",
|
||
"Cena mieszkań może być prognozowana na podstawie:\n",
|
||
"\n",
|
||
"* powierzchni w m$^2$ ($x_1 = 32.3$) $w_1 = 7000$\n",
|
||
"\n",
|
||
"* liczby pokoi ($x_2 = 3$) $w_2 = -30000$\n",
|
||
" \n",
|
||
"* nr piętra ($x_3 = 4$) \n",
|
||
"\n",
|
||
"* wieku ($x_4 = 13$) $w_3 = -1000$\n",
|
||
"\n",
|
||
"* odległości od Dworca Centralnego w Warszawie ($x_5 = 371.3$)\n",
|
||
"\n",
|
||
"* wielkość miasta\n",
|
||
"\n",
|
||
"* gęstość zaludnienia\n",
|
||
"\n",
|
||
"* cech zerojedynkowych:\n",
|
||
"\n",
|
||
" * czy wielka płyta? ($x_6 = 0$)\n",
|
||
"\n",
|
||
" * czy jest jacuzzi? ($x_7 = 1$) $w_7 = 5000$\n",
|
||
"\n",
|
||
" * czy jest grzyb? ($x_8 = 0$) $w_8 = -40000$\n",
|
||
" \n",
|
||
" * czy to Kielce? ($x_9 = 1$)\n",
|
||
" \n",
|
||
" * czy to Kraków ($x_{10} = 0$)\n",
|
||
" \n",
|
||
" * czy to Katowice ($x_{11} = 0$)\n",
|
||
" \n",
|
||
" * czy obok budynku jest parking \n",
|
||
" \n",
|
||
" * czy w budynku jest parking\n",
|
||
"\n",
|
||
"* zakodowany opis \n",
|
||
"\n",
|
||
" * $(x_{12}, x_{|V|+12})$ - wektor tf-idf \n",
|
||
"\n",
|
||
"... więc uogólniamy na wiele ($k$) wymiarów:\n",
|
||
"\n",
|
||
"$$ y = w_0 + w_1x_1 + \\ldots + w_kx_k = w_0 + \\sum_{j=1}^{k} w_jx_j = w_0 + \\vec{w}\\vec{x}$$\n",
|
||
"\n",
|
||
"gdzie:\n",
|
||
"\n",
|
||
"* $x_1,\\dots,x_k$ -- zmienne, na podstawie których zgadujemy\n",
|
||
"\n",
|
||
"* $w_0, w_1,\\dots,w_k$ -- wagi modelu (do wymyślenia na\n",
|
||
" podstawie przykładów)\n",
|
||
"\n",
|
||
"* $y$ -- odgadywana wartość\n",
|
||
"\n",
|
||
"Też istnieje wzór ładny wzór na wyliczenie wektora wag!\n",
|
||
"\n",
|
||
"$$\\mathbf{w} = (\\mathbf{X}^{\\rm T}\\mathbf{X})^{-1} \\mathbf{X}^{\\rm T}\\mathbf{y}$$\n",
|
||
"\n",
|
||
"... niestety odwracanie macierzy nie jest tanie :("
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "confused-increase",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Kilka spostrzeżeń\n",
|
||
"\n",
|
||
"Regresja liniowa to najprostszy możliwy model:\n",
|
||
"\n",
|
||
"* im czegoś więcej na wejściu, tym proporcjonalnie (troszkę) więcej/mniej na wyjściu\n",
|
||
"\n",
|
||
"* nic prostszego nie da się wymyślić (funkcja stała??)\n",
|
||
"\n",
|
||
"* niestety model liniowy czasami kompletnie nie ma sensu (np. wzrost człowieka w\n",
|
||
" stosunku do wieku)\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "freelance-controversy",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Uczenie\n",
|
||
"\n",
|
||
"A jak nauczyć się wag z przykładów?\n",
|
||
"\n",
|
||
"* wzór (z odwracaniem macierzy) — problematyczny\n",
|
||
"\n",
|
||
"### Metoda gradientu prostego\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"(Źródło: https://pl.wikipedia.org/wiki/Morskie_Oko#/media/Plik:Morskie_Oko_ze_szlaku_przez_%C5%9Awist%C3%B3wk%C4%99.jpg, licencja CC BY 3.0)\n",
|
||
"\n",
|
||
"Schodź wzdłuż lokalnego spadku funkcji błędu.\n",
|
||
"\n",
|
||
"Tak więc w praktyce zamiast podstawiać do wzoru lepiej się uczyć iteracyjnie -\n",
|
||
" metodą **gradientu prostego** (ang. _gradient descent_).\n",
|
||
"\n",
|
||
"1. Zacznij od byle jakich wag $w_i$ (np. wylosuj)\n",
|
||
"2. Weź losowy przykład uczący $x_1,\\dots,x_n$, $y$.\n",
|
||
"3. Oblicz wyjście $\\hat{y}$ na podstawie $x_1,\\dots,x_n$.\n",
|
||
"4. Oblicz funkcję błędu między $y$ a $\\hat{y}$.\n",
|
||
"5. Zmodyfikuj lekko wagi $(w_i)$ w kierunku spadku funkcji błędu.\n",
|
||
"6. Jeśli błąd jest duży, idź do 2.\n",
|
||
"\n",
|
||
"Modyfikacja wag:\n",
|
||
"\n",
|
||
"$$w_i := w_i - x_i (\\hat{y} - y) \\eta$$\n",
|
||
"\n",
|
||
"gdzie $\\eta$ to **współczynnik uczenia** _learning rate_.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "divine-medium",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Ewaluacja regresji\n",
|
||
"\n",
|
||
"To miary błędu (im mniej, tym lepiej!)}\n",
|
||
"\n",
|
||
"### Błąd bezwzględny (Mean Absolute Error, MAE)\n",
|
||
"\n",
|
||
"$$\\frac{1}{n}\\sum_{i=1}^n |\\hat{y}_i - y_i| $$\n",
|
||
"\n",
|
||
"### Mean Squared Error (MSE)\n",
|
||
"\n",
|
||
"$$\\frac{1}{n}\\sum_{i=1}^n (\\hat{y}_i - y_i)^2$$\n",
|
||
"\n",
|
||
"### Root Mean Squared Error (RMSE)\n",
|
||
"\n",
|
||
"$$\\sqrt{\\frac{1}{n}\\sum_{i=1}^n (\\hat{y}_i - y_i)^2}$$\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "supreme-tennessee",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Regresja liniowa dla tekstu\n",
|
||
"\n",
|
||
"Czym jest wektor $\\vec{x} = (x_1,\\dots,x_n)$? Wiemy, np. reprezentacja tf-idf (być z trikiem z haszowaniem, Word2vec etc.).\n",
|
||
"\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "seasonal-syndication",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Przykład \n",
|
||
"\n",
|
||
"Wyzwanie RetroC2 - odgadywanie roku dla krótkiego tekstu (1814-2013), <https://gonito.net/challenge/retroc2>.\n",
|
||
" \n",
|
||
"Lista słów (obcięta do 7 znaków) z największą/najmniejszymi wagami. \n",
|
||
"\n",
|
||
"```\n",
|
||
"wzbudze -0.08071490\n",
|
||
"paczka -0.08000180\n",
|
||
"szarpi -0.05906200\n",
|
||
"spadoch -0.05784140\n",
|
||
"rzymsko -0.05466660\n",
|
||
"sosnowy -0.05162170\n",
|
||
"dębowyc -0.04778910\n",
|
||
"nawinię -0.04649400\n",
|
||
"odmówie -0.04522140\n",
|
||
"zacisko -0.04480620\n",
|
||
"funkcją -0.04479500\n",
|
||
"werben -0.04423350\n",
|
||
"nieumyś -0.04415200\n",
|
||
"wodomie -0.04351570\n",
|
||
"szczote -0.04313390\n",
|
||
"exekucy -0.04297940\n",
|
||
"listew -0.04214090\n",
|
||
"daley -0.04145400\n",
|
||
"metro -0.04080110\n",
|
||
"wyjąwsz -0.04078060\n",
|
||
"salda -0.04042050\n",
|
||
"tkach -0.04020180\n",
|
||
"cetnar -0.03999050\n",
|
||
"zgóry -0.03855980\n",
|
||
"belek -0.03833100\n",
|
||
"formier -0.03805890\n",
|
||
"wekslu -0.03796510\n",
|
||
"odmową -0.03753760\n",
|
||
"\n",
|
||
"odwadni 0.04662140\n",
|
||
"dozując 0.04672770\n",
|
||
"wyników 0.04744650\n",
|
||
"sprawst 0.04746330\n",
|
||
"jakub 0.04750710\n",
|
||
"ścieran 0.04791070\n",
|
||
"wrodzon 0.04799800\n",
|
||
"koryguj 0.04843560\n",
|
||
"odnotow 0.04854360\n",
|
||
"tłumiąc 0.04917320\n",
|
||
"leasing 0.04963200\n",
|
||
"ecznej 0.04994810\n",
|
||
"2013r 0.05009500\n",
|
||
"kompens 0.05049060\n",
|
||
"comarch 0.05058620\n",
|
||
"pojazde 0.05078540\n",
|
||
"badanyc 0.05340480\n",
|
||
"kontakc 0.05377990\n",
|
||
"sygnali 0.05601120\n",
|
||
"piasta 0.05658670\n",
|
||
"2000r 0.05716820\n",
|
||
"stropni 0.06123470\n",
|
||
"oszone 0.06124600\n",
|
||
"zamonto 0.06424310\n",
|
||
"……….. 0.06498500\n",
|
||
"kumulat 0.06596770\n",
|
||
"faktura 0.07313080\n",
|
||
"wielost 0.09677770\n",
|
||
"wielomi 0.12307300\n",
|
||
"```\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "encouraging-martial",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": []
|
||
}
|
||
],
|
||
"metadata": {
|
||
"author": "Filip Graliński",
|
||
"email": "filipg@amu.edu.pl",
|
||
"kernelspec": {
|
||
"display_name": "Python 3 (ipykernel)",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"lang": "pl",
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.9.6"
|
||
},
|
||
"subtitle": "8.Regresja liniowa[wykład]",
|
||
"title": "Ekstrakcja informacji",
|
||
"year": "2021"
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
}
|