2693 lines
307 KiB
Plaintext
2693 lines
307 KiB
Plaintext
|
{
|
|||
|
|
"cells": [
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "slide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### Uczenie maszynowe\n",
|
|||
|
|
"# 3. Ewaluacja, regularyzacja, optymalizacja"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "slide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"## 3.1. Metodologia testowania"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"W uczeniu maszynowym bardzo ważna jest ewaluacja budowanego modelu. Dlatego dobrze jest podzielić posiadane dane na odrębne zbiory – osobny zbiór danych do uczenia i osobny do testowania. W niektórych przypadkach potrzeba będzie dodatkowo wyodrębnić tzw. zbiór walidacyjny."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### Zbiór uczący a zbiór testowy"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "fragment"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"* Na zbiorze uczącym (treningowym) uczymy algorytmy, a na zbiorze testowym sprawdzamy ich poprawność.\n",
|
|||
|
|
"* Zbiór uczący powinien być kilkukrotnie większy od testowego (np. 4:1, 9:1 itp.).\n",
|
|||
|
|
"* Zbiór testowy często jest nieznany.\n",
|
|||
|
|
"* Należy unikać mieszania danych testowych i treningowych – nie wolno „zanieczyszczać” danych treningowych danymi testowymi!"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"Czasami potrzebujemy dobrać parametry modelu, np. $\\alpha$ – który zbiór wykorzystać do tego celu?"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### Zbiór walidacyjny"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "fragment"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"Do doboru parametrów najlepiej użyć jeszcze innego zbioru – jest to tzw. **zbiór walidacyjny**"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "fragment"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
" * Zbiór walidacyjny powinien mieć wielkość zbliżoną do wielkości zbioru testowego, czyli np. dane można podzielić na te trzy zbiory w proporcjach 3:1:1, 8:1:1 itp."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "slide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### Walidacja krzyżowa"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "fragment"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"Którą część danych wydzielić jako zbiór walidacyjny tak, żeby było „najlepiej”?"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "fragment"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
" * Niech każda partia danych pełni tę rolę naprzemiennie!"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"<img width=\"100%\" src=\"https://chrisjmccormick.files.wordpress.com/2013/07/10_fold_cv.png\"/>\n",
|
|||
|
|
"Żródło: https://chrisjmccormick.wordpress.com/2013/07/31/k-fold-cross-validation-with-matlab-code/"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### Walidacja krzyżowa\n",
|
|||
|
|
"\n",
|
|||
|
|
"* Podziel dane $D = \\left\\{ (x^{(1)}, y^{(1)}), \\ldots, (x^{(m)}, y^{(m)})\\right\\} $ na $N$ rozłącznych zbiorów $T_1,\\ldots,T_N$\n",
|
|||
|
|
"* Dla $i=1,\\ldots,N$, wykonaj:\n",
|
|||
|
|
" * Użyj $T_i$ do walidacji i zbiór $S_i$ do trenowania, gdzie $S_i = D \\smallsetminus T_i$. \n",
|
|||
|
|
" * Zapisz model $\\theta_i$.\n",
|
|||
|
|
"* Akumuluj wyniki dla modeli $\\theta_i$ dla zbiorów $T_i$.\n",
|
|||
|
|
"* Ustalaj parametry uczenia na akumulowanych wynikach."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### Walidacja krzyżowa – wskazówki\n",
|
|||
|
|
"\n",
|
|||
|
|
"* Zazwyczaj ustala się $N$ w przedziale od $4$ do $10$, tzw. $N$-krotna walidacja krzyżowa (*$N$-fold cross validation*). \n",
|
|||
|
|
"* Zbiór $D$ warto zrandomizować przed podziałem.\n",
|
|||
|
|
"* W jaki sposób akumulować wyniki dla wszystkich zbiórow $T_i$?\n",
|
|||
|
|
"* Po ustaleniu parametrów dla każdego $T_i$, trenujemy model na całych danych treningowych z ustalonymi parametrami.\n",
|
|||
|
|
"* Testujemy na zbiorze testowym (jeśli nim dysponujemy)."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### _Leave-one-out_\n",
|
|||
|
|
"\n",
|
|||
|
|
"Jest to szczególny przypadek walidacji krzyżowej, w której $N = m$."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "fragment"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"* Jaki jest rozmiar pojedynczego zbioru $T_i$?\n",
|
|||
|
|
"* Jakie są zalety i wady tej metody?\n",
|
|||
|
|
"* Kiedy może być przydatna?"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### Zbiór walidujący a algorytmy optymalizacji\n",
|
|||
|
|
"\n",
|
|||
|
|
"* Gdy błąd rośnie na zbiorze uczącym, mamy źle dobrany parametr $\\alpha$. Należy go wtedy zmniejszyć.\n",
|
|||
|
|
"* Gdy błąd zmniejsza się na zbiorze trenującym, ale rośnie na zbiorze walidującym, mamy do czynienia ze zjawiskiem **nadmiernego dopasowania** (*overfitting*).\n",
|
|||
|
|
"* Należy wtedy przerwać optymalizację. Automatyzacja tego procesu to _early stopping_."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "slide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"## 3.2. Miary jakości"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"Aby przeprowadzić ewaluację modelu, musimy wybrać **miarę** (**metrykę**), jakiej będziemy używać.\n",
|
|||
|
|
"\n",
|
|||
|
|
"Jakiej miary użyc najlepiej?\n",
|
|||
|
|
" * To zależy od rodzaju zadania.\n",
|
|||
|
|
" * Innych metryk używa się do regresji, a innych do klasyfikacji"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "slide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### Metryki dla zadań regresji\n",
|
|||
|
|
"\n",
|
|||
|
|
"Dla zadań regresji możemy zastosować np.:\n",
|
|||
|
|
" * błąd średniokwadratowy (*root-mean-square error*, RMSE):\n",
|
|||
|
|
" $$ \\mathrm{RMSE} \\, = \\, \\sqrt{ \\frac{1}{m} \\sum_{i=1}^{m} \\left( \\hat{y}^{(i)} - y^{(i)} \\right)^2 } $$\n",
|
|||
|
|
" * średni błąd bezwzględny (*mean absolute error*, MAE):\n",
|
|||
|
|
" $$ \\mathrm{MAE} \\, = \\, \\frac{1}{m} \\sum_{i=1}^{m} \\left| \\hat{y}^{(i)} - y^{(i)} \\right| $$"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"W powyższych wzorach $y^{(i)}$ oznacza **oczekiwaną** wartości zmiennej $y$ w $i$-tym przykładzie, a $\\hat{y}^{(i)}$ oznacza wartość zmiennej $y$ w $i$-tym przykładzie wyliczoną (**przewidzianą**) przez nasz model."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "slide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### Metryki dla zadań klasyfikacji\n",
|
|||
|
|
"\n",
|
|||
|
|
"Aby przedstawić kilka najpopularniejszych metryk stosowanych dla zadań klasyfikacyjnych, posłużmy się następującym przykładem:"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 1,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [],
|
|||
|
|
"source": [
|
|||
|
|
"# Przydatne importy\n",
|
|||
|
|
"\n",
|
|||
|
|
"import ipywidgets as widgets\n",
|
|||
|
|
"import matplotlib.pyplot as plt\n",
|
|||
|
|
"import numpy as np\n",
|
|||
|
|
"import pandas\n",
|
|||
|
|
"import random\n",
|
|||
|
|
"import seaborn\n",
|
|||
|
|
"\n",
|
|||
|
|
"%matplotlib inline"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 2,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [],
|
|||
|
|
"source": [
|
|||
|
|
"def powerme(x1,x2,n):\n",
|
|||
|
|
" \"\"\"Funkcja, która generuje n potęg dla zmiennych x1 i x2 oraz ich iloczynów\"\"\"\n",
|
|||
|
|
" X = []\n",
|
|||
|
|
" for m in range(n+1):\n",
|
|||
|
|
" for i in range(m+1):\n",
|
|||
|
|
" X.append(np.multiply(np.power(x1,i),np.power(x2,(m-i))))\n",
|
|||
|
|
" return np.hstack(X)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 3,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [],
|
|||
|
|
"source": [
|
|||
|
|
"def plot_data_for_classification(X, Y, xlabel=None, ylabel=None, Y_predicted=[], highlight=None):\n",
|
|||
|
|
" \"\"\"Wykres danych dla zadania klasyfikacji\"\"\"\n",
|
|||
|
|
" fig = plt.figure(figsize=(16*.6, 9*.6))\n",
|
|||
|
|
" ax = fig.add_subplot(111)\n",
|
|||
|
|
" fig.subplots_adjust(left=0.1, right=0.9, bottom=0.1, top=0.9)\n",
|
|||
|
|
" X = X.tolist()\n",
|
|||
|
|
" Y = Y.tolist()\n",
|
|||
|
|
" X1n = [x[1] for x, y in zip(X, Y) if y[0] == 0]\n",
|
|||
|
|
" X1p = [x[1] for x, y in zip(X, Y) if y[0] == 1]\n",
|
|||
|
|
" X2n = [x[2] for x, y in zip(X, Y) if y[0] == 0]\n",
|
|||
|
|
" X2p = [x[2] for x, y in zip(X, Y) if y[0] == 1]\n",
|
|||
|
|
" \n",
|
|||
|
|
" if len(Y_predicted) > 0:\n",
|
|||
|
|
" Y_predicted = Y_predicted.tolist()\n",
|
|||
|
|
" X1tn = [x[1] for x, y, yp in zip(X, Y, Y_predicted) if y[0] == 0 and yp[0] == 0]\n",
|
|||
|
|
" X1fn = [x[1] for x, y, yp in zip(X, Y, Y_predicted) if y[0] == 1 and yp[0] == 0]\n",
|
|||
|
|
" X1tp = [x[1] for x, y, yp in zip(X, Y, Y_predicted) if y[0] == 1 and yp[0] == 1]\n",
|
|||
|
|
" X1fp = [x[1] for x, y, yp in zip(X, Y, Y_predicted) if y[0] == 0 and yp[0] == 1]\n",
|
|||
|
|
" X2tn = [x[2] for x, y, yp in zip(X, Y, Y_predicted) if y[0] == 0 and yp[0] == 0]\n",
|
|||
|
|
" X2fn = [x[2] for x, y, yp in zip(X, Y, Y_predicted) if y[0] == 1 and yp[0] == 0]\n",
|
|||
|
|
" X2tp = [x[2] for x, y, yp in zip(X, Y, Y_predicted) if y[0] == 1 and yp[0] == 1]\n",
|
|||
|
|
" X2fp = [x[2] for x, y, yp in zip(X, Y, Y_predicted) if y[0] == 0 and yp[0] == 1]\n",
|
|||
|
|
" \n",
|
|||
|
|
" if highlight == 'tn':\n",
|
|||
|
|
" ax.scatter(X1tn, X2tn, c='r', marker='x', s=100, label='Dane')\n",
|
|||
|
|
" ax.scatter(X1fn, X2fn, c='k', marker='o', s=50, label='Dane')\n",
|
|||
|
|
" ax.scatter(X1tp, X2tp, c='k', marker='o', s=50, label='Dane')\n",
|
|||
|
|
" ax.scatter(X1fp, X2fp, c='k', marker='x', s=50, label='Dane')\n",
|
|||
|
|
" elif highlight == 'fn':\n",
|
|||
|
|
" ax.scatter(X1tn, X2tn, c='k', marker='x', s=50, label='Dane')\n",
|
|||
|
|
" ax.scatter(X1fn, X2fn, c='g', marker='o', s=100, label='Dane')\n",
|
|||
|
|
" ax.scatter(X1tp, X2tp, c='k', marker='o', s=50, label='Dane')\n",
|
|||
|
|
" ax.scatter(X1fp, X2fp, c='k', marker='x', s=50, label='Dane')\n",
|
|||
|
|
" elif highlight == 'tp':\n",
|
|||
|
|
" ax.scatter(X1tn, X2tn, c='k', marker='x', s=50, label='Dane')\n",
|
|||
|
|
" ax.scatter(X1fn, X2fn, c='k', marker='o', s=50, label='Dane')\n",
|
|||
|
|
" ax.scatter(X1tp, X2tp, c='g', marker='o', s=100, label='Dane')\n",
|
|||
|
|
" ax.scatter(X1fp, X2fp, c='k', marker='x', s=50, label='Dane')\n",
|
|||
|
|
" elif highlight == 'fp':\n",
|
|||
|
|
" ax.scatter(X1tn, X2tn, c='k', marker='x', s=50, label='Dane')\n",
|
|||
|
|
" ax.scatter(X1fn, X2fn, c='k', marker='o', s=50, label='Dane')\n",
|
|||
|
|
" ax.scatter(X1tp, X2tp, c='k', marker='o', s=50, label='Dane')\n",
|
|||
|
|
" ax.scatter(X1fp, X2fp, c='r', marker='x', s=100, label='Dane')\n",
|
|||
|
|
" else:\n",
|
|||
|
|
" ax.scatter(X1tn, X2tn, c='r', marker='x', s=50, label='Dane')\n",
|
|||
|
|
" ax.scatter(X1fn, X2fn, c='g', marker='o', s=50, label='Dane')\n",
|
|||
|
|
" ax.scatter(X1tp, X2tp, c='g', marker='o', s=50, label='Dane')\n",
|
|||
|
|
" ax.scatter(X1fp, X2fp, c='r', marker='x', s=50, label='Dane')\n",
|
|||
|
|
"\n",
|
|||
|
|
" else:\n",
|
|||
|
|
" ax.scatter(X1n, X2n, c='r', marker='x', s=50, label='Dane')\n",
|
|||
|
|
" ax.scatter(X1p, X2p, c='g', marker='o', s=50, label='Dane')\n",
|
|||
|
|
" \n",
|
|||
|
|
" if xlabel:\n",
|
|||
|
|
" ax.set_xlabel(xlabel)\n",
|
|||
|
|
" if ylabel:\n",
|
|||
|
|
" ax.set_ylabel(ylabel)\n",
|
|||
|
|
" \n",
|
|||
|
|
" ax.margins(.05, .05)\n",
|
|||
|
|
" return fig"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 4,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [],
|
|||
|
|
"source": [
|
|||
|
|
"# Wczytanie danych\n",
|
|||
|
|
"import pandas\n",
|
|||
|
|
"import numpy as np\n",
|
|||
|
|
"\n",
|
|||
|
|
"alldata = pandas.read_csv('data-metrics.tsv', sep='\\t')\n",
|
|||
|
|
"data = np.matrix(alldata)\n",
|
|||
|
|
"\n",
|
|||
|
|
"m, n_plus_1 = data.shape\n",
|
|||
|
|
"n = n_plus_1 - 1\n",
|
|||
|
|
"\n",
|
|||
|
|
"X2 = powerme(data[:, 1], data[:, 2], n)\n",
|
|||
|
|
"Y2 = np.matrix(data[:, 0]).reshape(m, 1)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 5,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAm8AAAFmCAYAAAA70X3dAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3df3Dc913n8ddbieS5rndo7bjFdWKSYk2H2NCS6kKhmiqFJpeIay2LFiVnIDeXOZOjmXHtwMR33EGHH3PAQHzuEcqlpkPL+JrNTSXFUBU3DZRieoUomSS160ulhpC4yiWqXdq1OCQn+74/vt+v/dVqV/pK2t3v97v7fMzsaPfz/X7Xn/16V/vS9/PL3F0AAADIh660KwAAAIDkCG8AAAA5QngDAADIEcIbAABAjhDeAAAAcoTwBgAAkCNXpl2BNFx11VV+7bXXpl0NAACARZ544olvufuW5fbpyPB27bXXanJyMu1qAAAALGJm/7DSPjSbAgAA5AjhDQAAIEdSD29m9gkze8XMTtXZbmb2UTObNrNnzOyG2LZbzezZcNuh1tUaAAAgHamHN0l/LOnWZbbfJqk3vO2T9DFJMrMrJD0Qbr9e0h1mdn1TawoAAJCy1MObu39J0vlldtkt6VMe+Iqk15vZVkk3Spp29+fcfUHSQ+G+AAAAbSv18JbANkkvxh6fDcvqlQMAALStPIQ3q1Hmy5TXfhKzfWY2aWaTs7OzDascAABAK+UhvJ2VdE3s8dWSZpYpr8ndH3T3Pnfv27Jl2bnvAAAAMisP4e24pJ8LR52+U9J33P0lSY9L6jWz68ysR9Lt4b4AqrlLY2PBzyTlAIDMSj28mdmnJf1vSW81s7NmdpeZ3W1md4e7TEh6TtK0pI9L+gVJcvdXJd0j6YSkM5IedvfTLX8BQB6Mj0vDw9KBA5eDmnvweHg42A4AyIXUl8dy9ztW2O6SPlRn24SCcAdgOUND0v790pEjwePDh4PgduRIUD40lG79AACJpR7eALSAWRDYpCCwRSFu//6g3GqN/wEAZJF5B/Z16evrcxamR0dyl7pivSUqFYIbAGSImT3h7n3L7ZN6nzcALRL1cYuL94EDAOQC4Q3oBFFwi/q4VSqX+8AR4AAgV+jzBnSC8fHLwS3q4xbvAzcwIO3Zk24dAQCJEN6ATjA0JI2OBj+jPm5RgBsYYLQpAOQI4Q3oBGa1r6zVKwcAZBZ93gAAAHKE8AYAAJAjhDcAAIAcIbwBAADkCOENAAAgRwhvAAAAOUJ4AwAAyBHCGwAAQI4wSS8AADWU58sqnS5p6tyUejf3amTniIobimlXCyC8AQBQ7eQLJzV4bFAVr2ju4pwK3QUdPHFQE3sn1L+9P+3qocPRbAoAQEx5vqzBY4MqL5Q1d3FOkjR3cU7lhaD8wsKFlGuITkd4AwAgpnS6pIpXam6reEWlU6UW1whYjPAGAEDM1LmpS1fcqs1dnNP0+ekW1whYjPAGAEBM76YdKtiGmtsKtkE7Nn1/i2sELEZ4AwAgZuTvC+r65/ma27r+eV4jz72uxTUCFiO8AQAQU/ypOzQx/wEV56WCB5MyFPxKFeelifkPaONP3ZFyDdHpmCoEAIA4M/X/7sOaOfAhlf78Y5reJO04/6pG3v0ftPHwA5JZ2jVEhzN3T7sOLdfX1+eTk5NpVwMAkGXuUlesgapSIbih6czsCXfvW24fmk0BAKjmLh04sLjswIGgHEgZ4Q0AgLgouB05Iu3fH1xx278/eEyAQwbQ5w0AgLjx8cvB7fDhoKn08OFg25Ej0sCAtGdPunVER8tEeDOzWyUdkXSFpKPu/ltV239J0t7w4ZWSfkDSFnc/b2bPSypLek3Sqyu1EwMAsKyhIWl0NPgZ9XGLAtzAQFAOpCj1AQtmdoWkr0u6WdJZSY9LusPdv1Zn//dJOuDuPx4+fl5Sn7t/K+m/yYAFAACQRXkZsHCjpGl3f87dFyQ9JGn3MvvfIenTLakZAABAxmQhvG2T9GLs8dmwbAkze52kWyV9Jlbskj5vZk+Y2b56/4iZ7TOzSTObnJ2dbUC1AXQ8d2lsbGkH9nrlANAAWQhvtSbNqfcb732S/sbdz8fK3uXuN0i6TdKHzOzdtQ509wfdvc/d+7Zs2bK+GgOAFHRsHx5ePAIxGqk4PBxsB4AGy0J4OyvpmtjjqyXN1Nn3dlU1mbr7TPjzFUljCpphAaD5hoaWTiERn2KCju0AmiALo00fl9RrZtdJ+qaCgPZvqncys++RNCDpZ2JlBUld7l4O798i6ddaUmsAqJ5C4siR4H58igkAaLDUr7y5+6uS7pF0QtIZSQ+7+2kzu9vM7o7tukfS5919Llb2JkknzexpSX8n6bPu/uetqjsALApwEYIbgCbKwpU3ufuEpImqsj+sevzHkv64quw5SW9rcvUAoL56yygR4AA0SepX3gAgt1hGCUAKMnHlDQByiWWUAKSA8AYAa8UySgBSQHgDgLUyq31lrV45ADQAfd4AAAByhPAGAACQI4Q3AACAHCG8AQAA5AjhDQAAIEcIbwAAADlCeAMAAMgRwhsAAECOEN4AAAByhPAGAACQI4Q3AACAHCG8AQAA5AjhDQAAIEcIbwAAADlCeAMAAMgRwhsAAECOEN4AAAByhPAGAACQI4Q3AACAHCG8AQAA5AjhDQAAIEcIbwAAADlCeAMAAMiRTIQ3M7vVzJ41s2kzO1Rj+01m9h0zeyq8/UrSYwEAANrJlWlXwMyukPSApJslnZX0uJkdd/evVe361+7+r9d4LAAAQFvIwpW3GyVNu/tz7r4g6SFJu1twLAAAQO5kIbxtk/Ri7PHZsKzaj5rZ02b2OTPbucpjZWb7zGzSzCZnZ2cbUW8AAICWS73ZVJLVKPOqx09K+j53v2Bmg5LGJfUmPDYodH9Q0oOS1NfXV3MfIOvK82WVTpc0dW5KvZt7NbJzRMUNxbSrBQBooSyEt7OSrok9vlrSTHwHd/9u7P6Emf2BmV2V5FigXZx84aQGjw2q4hXNXZxTobuggycOamLvhPq396ddPQBAi2Sh2fRxSb1mdp2Z9Ui6XdLx+A5m9r1mZuH9GxXU+1ySY4F2UJ4va/DYoMoLZc1dnJMkzV2cU3khKL+wcCHlGgIAWiX18Obur0q6R9IJSWckPezup83sbjO7O9ztA5JOmdnTkj4q6XYP1Dy29a8CaK7S6ZIqXqm5reIVlU6VWlwjAEBastBsKnefkDRRVfaHsfu/L+n3kx4LtJupc1OXrrhVm7s4p+nz0y2uEQAgLalfeQOwst7NvSp0F2puK3QXtGPTjhbXCACQFsIbkAMjO0fUZbU/rl3WpZFdIy2uEQAgLYQ3IAeKG4qa2DuhYk/x0hW4QndBxZ6gfGPPxpRrGHKXxsaCn0nKAQCrlok+bwBW1r+9XzP3zqh0qqTp89PasWmHRnaNZCe4SdL4uDQ8LO3fLx0+LJkFge3AAenIEWl0VNqzJ+1aAkCuEd6AHNnYs1F33XBX2tWob2goCG5HjgSPDx++HNz27w+2AwDWhfAGoHHMgsAmBYEtCnHxK3EAgHWhz1u7ou8R0hIPcBGCGwA0DOGtXUV9jw4cuBzUor5Hw8PBdqAZovdZXPx9CABYF8Jbu4r3PYq+OOl7hGarfp9VKkvfhwCAdaHPW7ui7xHSMD5+ObhF77P4+3BggNGmALBO5h34l3BfX59PTk6mXY3WcJe6YhdYKxWCG5rHPQhwQ0OL32f1ygEAi5jZE+7et9w+NJu2M/oeodXMgitr1QGtXjkAYNUIb+2KvkcAALQl+ry1K/oeAQDQlghv7WpoKFiKKN7HKApwAwOMNgUAIKdoNm1X9D3CajGxMwDkAuENQICJnTtWeb6so08e1X2P3qejTx5Veb6cdpUALINmUwAqz5dV2v4tTd33DvU+ekQjBxZUPPwAEzt3gJMvnNTgsUFVvKK5i3MqdBd08MRBTeydUP/2/rSr11xMbYOcYp43oMMt+fL2K9W18Komjkn9L4iJndtYeb6sbfdvU3lh6ZW2Yk9RM/fOaGPPxhRq1iJjY8F
|
|||
|
|
"text/plain": [
|
|||
|
|
"<Figure size 691.2x388.8 with 1 Axes>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"metadata": {
|
|||
|
|
"needs_background": "light"
|
|||
|
|
},
|
|||
|
|
"output_type": "display_data"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"fig = plot_data_for_classification(X2, Y2, xlabel=r'$x_1$', ylabel=r'$x_2$')"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 6,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [],
|
|||
|
|
"source": [
|
|||
|
|
"def safeSigmoid(x, eps=0):\n",
|
|||
|
|
" \"\"\"Funkcja sigmoidalna zmodyfikowana w taki sposób, \n",
|
|||
|
|
" żeby wartości zawsze były odległe od asymptot o co najmniej eps\n",
|
|||
|
|
" \"\"\"\n",
|
|||
|
|
" y = 1.0/(1.0 + np.exp(-x))\n",
|
|||
|
|
" if eps > 0:\n",
|
|||
|
|
" y[y < eps] = eps\n",
|
|||
|
|
" y[y > 1 - eps] = 1 - eps\n",
|
|||
|
|
" return y\n",
|
|||
|
|
"\n",
|
|||
|
|
"def h(theta, X, eps=0.0):\n",
|
|||
|
|
" \"\"\"Funkcja hipotezy (regresja logistyczna)\"\"\"\n",
|
|||
|
|
" return safeSigmoid(X*theta, eps)\n",
|
|||
|
|
"\n",
|
|||
|
|
"def J(h,theta,X,y, lamb=0):\n",
|
|||
|
|
" \"\"\"Funkcja kosztu dla regresji logistycznej\"\"\"\n",
|
|||
|
|
" m = len(y)\n",
|
|||
|
|
" f = h(theta, X, eps=10**-7)\n",
|
|||
|
|
" j = -np.sum(np.multiply(y, np.log(f)) + \n",
|
|||
|
|
" np.multiply(1 - y, np.log(1 - f)), axis=0)/m\n",
|
|||
|
|
" if lamb > 0:\n",
|
|||
|
|
" j += lamb/(2*m) * np.sum(np.power(theta[1:],2))\n",
|
|||
|
|
" return j\n",
|
|||
|
|
"\n",
|
|||
|
|
"def dJ(h,theta,X,y,lamb=0):\n",
|
|||
|
|
" \"\"\"Gradient funkcji kosztu\"\"\"\n",
|
|||
|
|
" g = 1.0/y.shape[0]*(X.T*(h(theta,X)-y))\n",
|
|||
|
|
" if lamb > 0:\n",
|
|||
|
|
" g[1:] += lamb/float(y.shape[0]) * theta[1:] \n",
|
|||
|
|
" return g\n",
|
|||
|
|
"\n",
|
|||
|
|
"def classifyBi(theta, X):\n",
|
|||
|
|
" \"\"\"Funkcja predykcji - klasyfikacja dwuklasowa\"\"\"\n",
|
|||
|
|
" prob = h(theta, X)\n",
|
|||
|
|
" return prob"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 7,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [],
|
|||
|
|
"source": [
|
|||
|
|
"def GD(h, fJ, fdJ, theta, X, y, alpha=0.01, eps=10**-3, maxSteps=10000):\n",
|
|||
|
|
" \"\"\"Metoda gradientu prostego dla regresji logistycznej\"\"\"\n",
|
|||
|
|
" errorCurr = fJ(h, theta, X, y)\n",
|
|||
|
|
" errors = [[errorCurr, theta]]\n",
|
|||
|
|
" while True:\n",
|
|||
|
|
" # oblicz nowe theta\n",
|
|||
|
|
" theta = theta - alpha * fdJ(h, theta, X, y)\n",
|
|||
|
|
" # raportuj poziom błędu\n",
|
|||
|
|
" errorCurr, errorPrev = fJ(h, theta, X, y), errorCurr\n",
|
|||
|
|
" # kryteria stopu\n",
|
|||
|
|
" if abs(errorPrev - errorCurr) <= eps:\n",
|
|||
|
|
" break\n",
|
|||
|
|
" if len(errors) > maxSteps:\n",
|
|||
|
|
" break\n",
|
|||
|
|
" errors.append([errorCurr, theta]) \n",
|
|||
|
|
" return theta, errors"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 8,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"theta = [[ 1.37136167]\n",
|
|||
|
|
" [ 0.90128948]\n",
|
|||
|
|
" [ 0.54708112]\n",
|
|||
|
|
" [-5.9929264 ]\n",
|
|||
|
|
" [ 2.64435168]\n",
|
|||
|
|
" [-4.27978238]]\n"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"# Uruchomienie metody gradientu prostego dla regresji logistycznej\n",
|
|||
|
|
"theta_start = np.matrix(np.zeros(X2.shape[1])).reshape(X2.shape[1],1)\n",
|
|||
|
|
"theta, errors = GD(h, J, dJ, theta_start, X2, Y2, \n",
|
|||
|
|
" alpha=0.1, eps=10**-7, maxSteps=10000)\n",
|
|||
|
|
"print('theta = {}'.format(theta))"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 9,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [],
|
|||
|
|
"source": [
|
|||
|
|
"def plot_decision_boundary(fig, theta, X):\n",
|
|||
|
|
" \"\"\"Wykres granicy klas\"\"\"\n",
|
|||
|
|
" ax = fig.axes[0]\n",
|
|||
|
|
" xx, yy = np.meshgrid(np.arange(-1.0, 1.0, 0.02),\n",
|
|||
|
|
" np.arange(-1.0, 1.0, 0.02))\n",
|
|||
|
|
" l = len(xx.ravel())\n",
|
|||
|
|
" C = powerme(xx.reshape(l, 1), yy.reshape(l, 1), n)\n",
|
|||
|
|
" z = classifyBi(theta, C).reshape(int(np.sqrt(l)), int(np.sqrt(l)))\n",
|
|||
|
|
"\n",
|
|||
|
|
" plt.contour(xx, yy, z, levels=[0.5], lw=3);"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 10,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [],
|
|||
|
|
"source": [
|
|||
|
|
"Y_expected = Y2.astype(int)\n",
|
|||
|
|
"Y_predicted = (classifyBi(theta, X2) > 0.5).astype(int)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 11,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [],
|
|||
|
|
"source": [
|
|||
|
|
"# Przygotowanie interaktywnego wykresu\n",
|
|||
|
|
"\n",
|
|||
|
|
"dropdown_highlight = widgets.Dropdown(options=['all', 'tp', 'fp', 'tn', 'fn'], value='all', description='highlight')\n",
|
|||
|
|
"\n",
|
|||
|
|
"def interactive_classification(highlight):\n",
|
|||
|
|
" fig = plot_data_for_classification(X2, Y2, xlabel=r'$x_1$', ylabel=r'$x_2$',\n",
|
|||
|
|
" Y_predicted=Y_predicted, highlight=highlight)\n",
|
|||
|
|
" plot_decision_boundary(fig, theta, X2)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 12,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"application/vnd.jupyter.widget-view+json": {
|
|||
|
|
"model_id": "b208b75eb3484bef8d52e8aec9b89448",
|
|||
|
|
"version_major": 2,
|
|||
|
|
"version_minor": 0
|
|||
|
|
},
|
|||
|
|
"text/plain": [
|
|||
|
|
"interactive(children=(Dropdown(description='highlight', options=('all', 'tp', 'fp', 'tn', 'fn'), value='all'),…"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "display_data"
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"text/plain": [
|
|||
|
|
"<function __main__.interactive_classification(highlight)>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"execution_count": 12,
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "execute_result"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"widgets.interact(interactive_classification, highlight=dropdown_highlight)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"Zadanie klasyfikacyjne z powyższego przykładu polega na przypisaniu punktów do jednej z dwóch kategorii:\n",
|
|||
|
|
" 0. <font color=\"red\">czerwone krzyżyki</font>\n",
|
|||
|
|
" 1. <font color=\"green\">zielone kółka</font>\n",
|
|||
|
|
"\n",
|
|||
|
|
"W tym celu zastosowano regresję logistyczną."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"W rezultacie otrzymano model, który dzieli płaszczyznę na dwa obszary:\n",
|
|||
|
|
" 0. <font color=\"red\">na zewnątrz granatowej krzywej</font>\n",
|
|||
|
|
" 1. <font color=\"green\">wewnątrz granatowej krzywej</font>\n",
|
|||
|
|
" \n",
|
|||
|
|
"Model przewiduje klasę <font color=\"red\">0 („czerwoną”)</font> dla punktów znajdujący się w obszarze na zewnątrz krzywej, natomiast klasę <font color=\"green\">1 („zieloną”)</font> dla punktów znajdujących sie w obszarze wewnąrz krzywej."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"Wszysktie obserwacje możemy podzielić zatem na cztery grupy:\n",
|
|||
|
|
" * **true positives (TP)** – prawidłowo sklasyfikowane pozytywne przykłady (<font color=\"green\">zielone kółka</font> w <font color=\"green\">wewnętrznym obszarze</font>)\n",
|
|||
|
|
" * **true negatives (TN)** – prawidłowo sklasyfikowane negatywne przykłady (<font color=\"red\">czerwone krzyżyki</font> w <font color=\"red\">zewnętrznym obszarze</font>)\n",
|
|||
|
|
" * **false positives (FP)** – negatywne przykłady sklasyfikowane jako pozytywne (<font color=\"red\">czerwone krzyżyki</font> w <font color=\"green\">wewnętrznym obszarze</font>)\n",
|
|||
|
|
" * **false negatives (FN)** – pozytywne przykłady sklasyfikowane jako negatywne (<font color=\"green\">zielone kółka</font> w <font color=\"red\">zewnętrznym obszarze</font>)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"Innymi słowy:\n",
|
|||
|
|
"\n",
|
|||
|
|
"<img width=\"50%\" src=\"https://blog.aimultiple.com/wp-content/uploads/2019/07/positive-negative-true-false-matrix.png\">"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 13,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "skip"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"TP = 5\n",
|
|||
|
|
"TN = 35\n",
|
|||
|
|
"FP = 3\n",
|
|||
|
|
"FN = 6\n"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"# Obliczmy TP, TN, FP i FN\n",
|
|||
|
|
"\n",
|
|||
|
|
"tp = 0\n",
|
|||
|
|
"tn = 0\n",
|
|||
|
|
"fp = 0\n",
|
|||
|
|
"fn = 0\n",
|
|||
|
|
"\n",
|
|||
|
|
"for i in range(len(Y_expected)):\n",
|
|||
|
|
" if Y_expected[i] == 1 and Y_predicted[i] == 1:\n",
|
|||
|
|
" tp += 1\n",
|
|||
|
|
" elif Y_expected[i] == 0 and Y_predicted[i] == 0:\n",
|
|||
|
|
" tn += 1\n",
|
|||
|
|
" elif Y_expected[i] == 0 and Y_predicted[i] == 1:\n",
|
|||
|
|
" fp += 1\n",
|
|||
|
|
" elif Y_expected[i] == 1 and Y_predicted[i] == 0:\n",
|
|||
|
|
" fn += 1\n",
|
|||
|
|
" \n",
|
|||
|
|
"print('TP =', tp)\n",
|
|||
|
|
"print('TN =', tn)\n",
|
|||
|
|
"print('FP =', fp)\n",
|
|||
|
|
"print('FN =', fn)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "skip"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"Możemy teraz zdefiniować następujące metryki:"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "slide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"#### Dokładność (*accuracy*)\n",
|
|||
|
|
"$$ \\mbox{accuracy} = \\frac{\\mbox{przypadki poprawnie sklasyfikowane}}{\\mbox{wszystkie przypadki}} = \\frac{TP + TN}{TP + TN + FP + FN} $$"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"Dokładność otrzymujemy przez podzielenie liczby przypadków poprawnie sklasyfikowanych przez liczbę wszystkich przypadków:"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 14,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Accuracy: 0.8163265306122449\n"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"accuracy = (tp + tn) / (tp + tn + fp + fn)\n",
|
|||
|
|
"print('Accuracy:', accuracy)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"**Uwaga:** Nie zawsze dokładność będzie dobrą miarą, zwłaszcza gdy klasy są bardzo asymetryczne!\n",
|
|||
|
|
"\n",
|
|||
|
|
"*Przykład:* Wyobraźmy sobie test na koronawirusa, który **zawsze** zwraca wynik negatywny. Jaką przydatność będzie miał taki test w praktyce? Żadną. A jaka będzie jego *dokładność*? Policzmy:\n",
|
|||
|
|
"$$ \\mbox{accuracy} \\, = \\, \\frac{\\mbox{szacowana liczba osób zdrowych na świecie}}{\\mbox{populacja Ziemi}} \\, \\approx \\, \\frac{7\\,700\\,000\\,000 - 600\\,000}{7\\,700\\,000\\,000} \\, \\approx \\, 0.99992 $$\n",
|
|||
|
|
"(zaokrąglone dane z 27 marca 2020)\n",
|
|||
|
|
"\n",
|
|||
|
|
"Powyższy wynik jest tak wysoki, ponieważ zdecydowana większość osób na świecie nie jest zakażona, więc biorąc losowego Ziemianina możemy w ciemno strzelać, że nie ma koronawirusa.\n",
|
|||
|
|
"\n",
|
|||
|
|
"W tym przypadku duża różnica w liczności obu zbiorów (zakażeni/niezakażeni) powoduje, że *accuracy* nie jest dobrą metryką.\n",
|
|||
|
|
"\n",
|
|||
|
|
"Dlatego dysponujemy również innymi metrykami:"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "slide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"#### Precyzja (*precision*)\n",
|
|||
|
|
"$$ \\mbox{precision} = \\frac{TP}{TP + FP} $$"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 15,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Precision: 0.625\n"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"precision = tp / (tp + fp)\n",
|
|||
|
|
"print('Precision:', precision)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"Precyzja określa, jaka część przykładów sklasyfikowanych jako pozytywne to faktycznie przykłady pozytywne."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "slide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"#### Pokrycie (czułość, *recall*)\n",
|
|||
|
|
"$$ \\mbox{recall} = \\frac{TP}{TP + FN} $$"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 16,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"Recall: 0.45454545454545453\n"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"recall = tp / (tp + fn)\n",
|
|||
|
|
"print('Recall:', recall)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"Pokrycie mówi nam, jaka część przykładów pozytywnych została poprawnie sklasyfikowana."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "slide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"#### *$F$-measure* (*$F$-score*)\n",
|
|||
|
|
"$$ F = \\frac{2 \\cdot \\mbox{precision} \\cdot \\mbox{recall}}{\\mbox{precision} + \\mbox{recall}} $$"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 17,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stdout",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"F-score: 0.5263157894736842\n"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"fscore = (2 * precision * recall) / (precision + recall)\n",
|
|||
|
|
"print('F-score:', fscore)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"$F$-_measure_ jest kompromisem między precyzją a pokryciem (a ściślej: jest średnią harmoniczną precyzji i pokrycia)."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"$F$-_measure_ jest szczególnym przypadkiem ogólniejszej miary:"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"*$F_\\beta$-measure*:\n",
|
|||
|
|
"$$ F_\\beta = \\frac{(1 + \\beta) \\cdot \\mbox{precision} \\cdot \\mbox{recall}}{\\beta^2 \\cdot \\mbox{precision} + \\mbox{recall}} $$"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "fragment"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"Dla $\\beta = 1$ otrzymujemy:\n",
|
|||
|
|
"$$ F_1 \\, = \\, \\frac{(1 + 1) \\cdot \\mbox{precision} \\cdot \\mbox{recall}}{1^2 \\cdot \\mbox{precision} + \\mbox{recall}} \\, = \\, \\frac{2 \\cdot \\mbox{precision} \\cdot \\mbox{recall}}{\\mbox{precision} + \\mbox{recall}} \\, = \\, F $$"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "slide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"## 3.3. Obserwacje odstające"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"**Obserwacje odstające** (*outliers*) – to wszelkie obserwacje posiadające nietypową wartość.\n",
|
|||
|
|
"\n",
|
|||
|
|
"Mogą być na przykład rezultatem błędnego pomiaru albo pomyłki przy wprowadzaniu danych do bazy, ale nie tylko.\n",
|
|||
|
|
"\n",
|
|||
|
|
"Obserwacje odstające mogą niekiedy znacząco wpłynąć na parametry modelu, dlatego ważne jest, żeby takie obserwacje odrzucić zanim przystąpi się do tworzenia modelu."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"W poniższym przykładzie można zobaczyć wpływ obserwacji odstających na wynik modelowania na przykładzie danych dotyczących cen mieszkań zebranych z ogłoszeń na portalu Gratka.pl: tutaj przykładem obserwacji odstającej może być ogłoszenie, w którym podano cenę w tys. zł zamiast ceny w zł."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 18,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [],
|
|||
|
|
"source": [
|
|||
|
|
"# Przydatne funkcje\n",
|
|||
|
|
"\n",
|
|||
|
|
"def h_linear(Theta, x):\n",
|
|||
|
|
" \"\"\"Funkcja regresji liniowej\"\"\"\n",
|
|||
|
|
" return x * Theta\n",
|
|||
|
|
"\n",
|
|||
|
|
"def linear_regression(theta):\n",
|
|||
|
|
" \"\"\"Ta funkcja zwraca funkcję regresji liniowej dla danego wektora parametrów theta\"\"\"\n",
|
|||
|
|
" return lambda x: h_linear(theta, x)\n",
|
|||
|
|
"\n",
|
|||
|
|
"def cost(theta, X, y):\n",
|
|||
|
|
" \"\"\"Wersja macierzowa funkcji kosztu\"\"\"\n",
|
|||
|
|
" m = len(y)\n",
|
|||
|
|
" J = 1.0 / (2.0 * m) * ((X * theta - y).T * (X * theta - y))\n",
|
|||
|
|
" return J.item()\n",
|
|||
|
|
"\n",
|
|||
|
|
"def gradient(theta, X, y):\n",
|
|||
|
|
" \"\"\"Wersja macierzowa gradientu funkcji kosztu\"\"\"\n",
|
|||
|
|
" return 1.0 / len(y) * (X.T * (X * theta - y)) \n",
|
|||
|
|
"\n",
|
|||
|
|
"def gradient_descent(fJ, fdJ, theta, X, y, alpha=0.1, eps=10**-5):\n",
|
|||
|
|
" \"\"\"Algorytm gradientu prostego (wersja macierzowa)\"\"\"\n",
|
|||
|
|
" current_cost = fJ(theta, X, y)\n",
|
|||
|
|
" logs = [[current_cost, theta]]\n",
|
|||
|
|
" while True:\n",
|
|||
|
|
" theta = theta - alpha * fdJ(theta, X, y)\n",
|
|||
|
|
" current_cost, prev_cost = fJ(theta, X, y), current_cost\n",
|
|||
|
|
" if abs(prev_cost - current_cost) > 10**15:\n",
|
|||
|
|
" print('Algorithm does not converge!')\n",
|
|||
|
|
" break\n",
|
|||
|
|
" if abs(prev_cost - current_cost) <= eps:\n",
|
|||
|
|
" break\n",
|
|||
|
|
" logs.append([current_cost, theta]) \n",
|
|||
|
|
" return theta, logs\n",
|
|||
|
|
"\n",
|
|||
|
|
"def plot_data(X, y, xlabel, ylabel):\n",
|
|||
|
|
" \"\"\"Wykres danych (wersja macierzowa)\"\"\"\n",
|
|||
|
|
" fig = plt.figure(figsize=(16*.6, 9*.6))\n",
|
|||
|
|
" ax = fig.add_subplot(111)\n",
|
|||
|
|
" fig.subplots_adjust(left=0.1, right=0.9, bottom=0.1, top=0.9)\n",
|
|||
|
|
" ax.scatter([X[:, 1]], [y], c='r', s=50, label='Dane')\n",
|
|||
|
|
" \n",
|
|||
|
|
" ax.set_xlabel(xlabel)\n",
|
|||
|
|
" ax.set_ylabel(ylabel)\n",
|
|||
|
|
" ax.margins(.05, .05)\n",
|
|||
|
|
" plt.ylim(y.min() - 1, y.max() + 1)\n",
|
|||
|
|
" plt.xlim(np.min(X[:, 1]) - 1, np.max(X[:, 1]) + 1)\n",
|
|||
|
|
" return fig\n",
|
|||
|
|
"\n",
|
|||
|
|
"def plot_regression(fig, fun, theta, X):\n",
|
|||
|
|
" \"\"\"Wykres krzywej regresji (wersja macierzowa)\"\"\"\n",
|
|||
|
|
" ax = fig.axes[0]\n",
|
|||
|
|
" x0 = np.min(X[:, 1]) - 1.0\n",
|
|||
|
|
" x1 = np.max(X[:, 1]) + 1.0\n",
|
|||
|
|
" L = [x0, x1]\n",
|
|||
|
|
" LX = np.matrix([1, x0, 1, x1]).reshape(2, 2)\n",
|
|||
|
|
" ax.plot(L, fun(theta, LX), linewidth='2',\n",
|
|||
|
|
" label=(r'$y={theta0:.2}{op}{theta1:.2}x$'.format(\n",
|
|||
|
|
" theta0=float(theta[0][0]),\n",
|
|||
|
|
" theta1=(float(theta[1][0]) if theta[1][0] >= 0 else float(-theta[1][0])),\n",
|
|||
|
|
" op='+' if theta[1][0] >= 0 else '-')))"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 19,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [],
|
|||
|
|
"source": [
|
|||
|
|
"# Wczytanie danych (mieszkania) przy pomocy biblioteki pandas\n",
|
|||
|
|
"\n",
|
|||
|
|
"alldata = pandas.read_csv('data_flats_with_outliers.tsv', sep='\\t',\n",
|
|||
|
|
" names=['price', 'isNew', 'rooms', 'floor', 'location', 'sqrMetres'])\n",
|
|||
|
|
"data = np.matrix(alldata[['price', 'sqrMetres']])\n",
|
|||
|
|
"\n",
|
|||
|
|
"m, n_plus_1 = data.shape\n",
|
|||
|
|
"n = n_plus_1 - 1\n",
|
|||
|
|
"Xn = data[:, 0:n]\n",
|
|||
|
|
"\n",
|
|||
|
|
"Xo = np.matrix(np.concatenate((np.ones((m, 1)), Xn), axis=1)).reshape(m, n + 1)\n",
|
|||
|
|
"yo = np.matrix(data[:, -1]).reshape(m, 1)\n",
|
|||
|
|
"\n",
|
|||
|
|
"Xo /= np.amax(Xo, axis=0)\n",
|
|||
|
|
"yo /= np.amax(yo, axis=0)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 20,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAmwAAAFoCAYAAADq7KeuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAcJUlEQVR4nO3dfbBkZ10n8O9vJkOik9EISQjkBbQyC6IrAa9JkNRuYMWFKZboLu7E3ZKIWQIKFi/KGrVKFP9YastXJJCKvGZXcXwDUsUARgqNKTaGSUyAvOBMAco4kYSXDTcDiZnMs390z2a46Zn0vbdv93N7Pp+qW919znNO/+65p3u+85xznlOttQAA0K8Nsy4AAICjE9gAADonsAEAdE5gAwDonMAGANA5gQ0AoHMzC2xVdWZVfayq7qiq26rq1SPaVFW9uar2VNUnq+qZs6gVAGCWjpvhex9I8nOttZurakuSm6rq2tba7Ye1eUGSrcOf85K8bfgIAHDMmFkPW2vtrtbazcPni0nuSHL6kmYXJbm6DdyQ5KSqesKUSwUAmKkuzmGrqicneUaSv10y6/QkXzjs9d48MtQBAMy1WR4STZJU1YlJ/izJa1prX1s6e8QiI++lVVWXJbksSTZv3vz9T33qUydaJwDAat10001faq2dstzlZhrYqmpTBmHtD1prfz6iyd4kZx72+owk+0atq7V2VZKrkmRhYaHt2rVrwtUCAKxOVf3DSpab5VWileQdSe5orf3WEZpdk+Qlw6tFz09yb2vtrqkVCQDQgVn2sD07yU8k+VRV3TKc9ktJzkqS1tqVSXYm2ZZkT5KvJ3npDOoEAJipmQW21tr1GX2O2uFtWpJXTqciAIA+dXGVKAAARyawAQB0TmADAOicwAYA0DmBDQCgcwIbAEDnBDYAgM4JbAAAnRPYAAA6J7ABAHROYAMA6JzABgDQOYENAKBzAhsAQOcENgCAzglsAACdE9gAADonsAEAdE5gAwDonMAGANA5gQ0AoHMCGwBA5wQ2AIDOCWwAAJ0T2AAAOiewAQB0TmADAOicwAYA0DmBDQCgcwIbAEDnBDYAgM4JbAAAnRPYAAA6J7ABAHROYAMA6JzABgDQOYENAKBzAhsAQOcENgCAzglsAACdE9gAADonsAEAdE5gAwDo3EwDW1W9s6rurqpPH2H+hVV1b1XdMvz5lWnXCAAwa8fN+P3fneQtSa4+Spu/aa29cDrlAAD0Z6Y9bK2165J8ZZY1AAD0bj2cw/asqrq1qj5UVd8z62IAAKZt1odEH83NSZ7UWruvqrYleX+SraMaVtVlSS5LkrPOOmt6FQIArLGue9haa19rrd03fL4zyaaqOvkIba9qrS201hZOOeWUqdYJALCWug5sVXVaVdXw+bkZ1Pvl2VYFADBdMz0kWlXvTXJhkpOram+SNyTZlCSttSuTvDjJT1fVgSTfSHJxa63NqFwAgJmYaWBrrf34o8x/SwbDfgAAHLO6PiQKAIDABgDQPYENAKBzAhsAQOcENgCAzglsAACdE9gAADonsAEAdE5gAwDonMAGANA5gQ0AoHMCGwBA5wQ2AIDOCWwAAJ0T2AAAOiewAQB0TmADAOicwAYA0DmBDQCgcwIbAEDnBDYAgM4JbAAAnRPYAAA6J7ABAHROYAMA6JzABgDQOYENAKBzAhsAQOcENgCAzglsAACdE9gAADonsAEAdE5gAwDonMAGANA5gQ0AoHMCGwBA5wQ2AIDOCWwAAJ0T2AAAOiewAQB0TmADAOicwAYA0DmBDQCgczMNbFX1zqq6u6o+fYT5VVVvrqo9VfXJqnrmtGuEri0uJm9/e/ILvzB4XFycdUUArIHjZvz+707yliRXH2H+C5JsHf6cl+Rtw0fg+uuTbduSgweT/fuTzZuT170u2bkzueCCWVcHwATNtIettXZdkq8cpclFSa5uAzckOamqnjCd6qBji4uDsLa4OAhryeDx0PT77pttfQBMVO/nsJ2e5AuHvd47nAbHth07Bj1roxw8OJgPwNzoPbDViGltZMOqy6pqV1Xtuueee9a4LJix3bsf7llbav/+ZM+e6dYDwJrqPbDtTXLmYa/PSLJvVMPW2lWttYXW2sIpp5wyleJgZrZuHZyzNsrmzcnZZ0+3HgDWVO+B7ZokLxleLXp+kntba3fNuiiYue3bkw1H+Phu2DCYD8DcmOlVolX13iQXJjm5qvYmeUOSTUnSWrsyyc4k25LsSfL1JC+dTaXQmS1bBleDLr1KdMOGwfQTT5x1hQBM0EwDW2vtxx9lfkvyyimVA+vLBRck+/YNLjDYs2dwGHT7dmENYA7Nehw2YDVOPDG59NJZVwHAGuv9HDYAgGOewAYA0DmBDQCgcwIbAEDnBDYAgM4JbAAAnRPYAAA6J7ABAHROYAMA6JzABgDQOYENAKBzAhsAQOcENgCAzglsAACdE9gAADonsAEAdE5gAwDonMAGANA5gQ0AoHMCGwBA546bdQHAKi0uJjt2JLt3J1u3Jtu3J1u2zLoqACZIYIP17Prrk23bkoMHk/37k82bk9e9Ltm5M7nggllXB8CEOCQK69Xi4iCsLS4OwloyeDw0/b77ZlsfABMjsMF6tWPHoGdtlIMHB/MBmAsCG6xXu3c/3LO21P79yZ49060HgDUjsMF6tXXr4Jy1UTZvTs4+e7r1ALBmBDZYr7ZvTzYc4SO8YcNgPgBzQWCD9WrLlsHVoFu2PNzTtnnzw9NPPHG29QEwMYb1gPXsgguSffsGFxjs2TM4DLp9u7AGMGcENljvTjwxufTSh18vLiZvf7uBdAHmiMAG88RAugBzyTlsMC8MpAswtwQ2mBcG0gWYWwIbzAsD6QLMLYEN5oWBdAHmlsAG88JAugBza+yrRKvqO5JsTXLCoWmttevWoihgBQ4NmLv0KtENGwykC7DOjRXYquq/JXl1kjOS3JLk/CT/J8lz1640YNkMpAswl8btYXt1kh9IckNr7TlV9dQkv7Z2ZQErtnQgXQDWvXHPYbu/tXZ/klTV8a21O5M8Ze3KAgDgkHF72PZW1UlJ3p/k2qr6apJ9a1cWsGKLi4NDom5NBTA3qrW2vAWq/m2Sb0/y4dbav6xJVau0sLDQdu3aNesyYPpG3Zrq0EUHbk0FMHNVdVNrbWG5y409rEdVbayqJyb5XAYXHpy23Dcbsc7nV9VnqmpPVV0+Yv6FVXVvVd0y/PmV1b4nzC23pgKYW+NeJfqzSd6Q5ItJDt37piX5vpW+cVVtTHJFkucl2ZvkE1V1TWvt9iVN/6a19sKVvg/MtcMPf/7zPycPPji63YMPDtq5GAFgXVrOVaJPaa19eYLvfW6SPa21zyZJVf1RkouSLA1swChLD38ed1xy4MDotvffn9zuowWwXo17SPQLSe6d8HufPlzvIXuH05Z6VlXdWlUfqqrvOdLKquqyqtpVVbvuueeeCZcKnRl1+PNIYe2QL0/y/1sATNO4PWyfTfJXVfXBJA8cmtha+61VvHeNmLb0CoibkzyptXZfVW3L4CrVraNW1lq7KslVyeCig1XUBf3bsWPQs7Ycj3vc2tQCwJobt4ftH5Ncm+QxSbYc9rMae5OcedjrM7JkqJDW2tdaa/cNn+9MsqmqTl7l+8L6t3v3wz1r4zjhhORpT1u7egBYU2P1sLXWfi1Jqmpza20Z/0oc1SeSbK2q70zyT0kuTvJfDm9QVacl+WJrrVXVuRkETMd1YOvWZOPG5KGHxmu/aZObvwOsY2P1sFXVs6rq9iR3DF8/vareupo3bq0dSPKqJB8ZrvePW2u3VdUrquoVw2YvTvLpqro1yZuTXNyWO3AczKNt28YLa5s3P3xTePcTBVi3xj2H7XeS/Psk1yRJa+3Wqvo3q33z4WHOnUumXXnY87ckectq3wfmzu/+7qO3qUouuyx54xuFNYB1buyBc1trX1gyacxjMcBELS4mv/M7j96uteS3fzv5+MfXviYA1tTYw3pU1Q8maVX1mKr6+QwPjwJTtmPH+OeuJcmLXuQuBwDr3LiB7RVJXpnBOGl7k5yT5Gf
|
|||
|
|
"text/plain": [
|
|||
|
|
"<Figure size 691.2x388.8 with 1 Axes>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"metadata": {
|
|||
|
|
"needs_background": "light"
|
|||
|
|
},
|
|||
|
|
"output_type": "display_data"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"fig = plot_data(Xo, yo, xlabel=u'metraż', ylabel=u'cena')\n",
|
|||
|
|
"theta_start = np.matrix([0.0, 0.0]).reshape(2, 1)\n",
|
|||
|
|
"theta, logs = gradient_descent(cost, gradient, theta_start, Xo, yo, alpha=0.01)\n",
|
|||
|
|
"plot_regression(fig, h_linear, theta, Xo)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"Na powyższym przykładzie obserwacja odstająca jawi sie jako pojedynczy punkt po prawej stronie wykresu. Widzimy, że otrzymana krzywa regresji zamiast odwzorowywać ogólny trend, próbuje „dopasować się” do tej pojedynczej obserwacji.\n",
|
|||
|
|
"\n",
|
|||
|
|
"Dlatego taką obserwację należy usunąć ze zbioru danych (zobacz ponizej)."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 21,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [],
|
|||
|
|
"source": [
|
|||
|
|
"# Odrzućmy obserwacje odstające\n",
|
|||
|
|
"alldata_no_outliers = [\n",
|
|||
|
|
" (index, item) for index, item in alldata.iterrows() \n",
|
|||
|
|
" if item.price > 100 and item.sqrMetres > 10]\n",
|
|||
|
|
"\n",
|
|||
|
|
"alldata_no_outliers = alldata.loc[(alldata['price'] > 100) & (alldata['sqrMetres'] > 100)]"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 22,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [],
|
|||
|
|
"source": [
|
|||
|
|
"data = np.matrix(alldata_no_outliers[['price', 'sqrMetres']])\n",
|
|||
|
|
"\n",
|
|||
|
|
"m, n_plus_1 = data.shape\n",
|
|||
|
|
"n = n_plus_1 - 1\n",
|
|||
|
|
"Xn = data[:, 0:n]\n",
|
|||
|
|
"\n",
|
|||
|
|
"Xo = np.matrix(np.concatenate((np.ones((m, 1)), Xn), axis=1)).reshape(m, n + 1)\n",
|
|||
|
|
"yo = np.matrix(data[:, -1]).reshape(m, 1)\n",
|
|||
|
|
"\n",
|
|||
|
|
"Xo /= np.amax(Xo, axis=0)\n",
|
|||
|
|
"yo /= np.amax(yo, axis=0)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 23,
|
|||
|
|
"metadata": {
|
|||
|
|
"scrolled": true,
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAmwAAAFoCAYAAADq7KeuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de5BcZ3nn8e8zmtFtZmSbsoxsWeayaO2YXWxYxZjgWi4JBKvYOAvEgs3GCsuW17GcNbdUTLIFhCSLUxtMAsgYr+PY3k3AW4GAKwgIIUk5TkJAJjbgu+IkWEjYigHP6K6Rnv3j9Eg9Mz0zPZfu82rm+6nq6tPnnO5+Z1ot/fSe533fyEwkSZJUrp66GyBJkqSpGdgkSZIKZ2CTJEkqnIFNkiSpcAY2SZKkwhnYJEmSCldbYIuIdRHxFxHxUEQ8EBHXtjgnIuIjEbEjIr4ZES+po62SJEl16q3xvUeAd2XmNyJiELg3Ir6cmQ82nXMpsL5xeynw8ca9JEnSolFbD1tm7s7MbzS2h4GHgLXjTrsMuCMrXwVOjYgzu9xUSZKkWhVRwxYRzwVeDPzduENrgSeaHu9kYqiTJEla0Oq8JApARAwAnwbenplD4w+3eErLtbQi4krgSoD+/v5/d955581rOyVJkubq3nvv/ZfMXD3T59Ua2CKijyqs/UFmfqbFKTuBdU2PzwZ2tXqtzLwZuBlgw4YNuX379nlurSRJ0txExD/P5nl1jhIN4PeAhzLzhklOuwu4ojFa9GLgmczc3bVGSpIkFaDOHraXAz8HfCsi7mvs+xXgHIDMvAnYBmwEdgD7gbfW0E5JkqRa1RbYMvMeWteoNZ+TwJbutEiSJKlMRYwSlSRJ0uQMbJIkSYUzsEmSJBXOwCZJklQ4A5skSVLhDGySJEmFM7BJkiQVzsAmSZJUOAObJElS4QxskiRJhTOwSZIkFc7AJkmSVDgDmyRJUuEMbJIkSYUzsEmSJBXOwCZJklQ4A5skSVLhDGySJEmFM7BJkiQVzsAmSZJUOAObJElS4QxskiRJhTOwSZIkFc7AJkmSVDgDmyRJUuEMbJIkSYUzsEmSJBXOwCZJklQ4A5skSVLhDGySJEmFM7BJkiQVzsAmSZJUOAObJElS4QxskiRJhTOwSZIkFc7AJkmSVDgDmyRJUuEMbJIkSYUzsEmSJBXOwCZJklQ4A5skSVLhDGySJEmFM7BJkiQVzsAmSZJUOAObJElS4QxskiRJhTOwSZIkFc7AJkmSVLhaA1tE3BoRT0XEtyc5/sqIeCYi7mvc3tvtNkqSJNWtt+b3vw34GHDHFOf8VWa+vjvNkSRJKk+tPWyZeTfw/TrbIEmSVLqToYbtZRFxf0R8ISJeONlJEXFlRGyPiO179uzpZvskSZI6qvTA9g3gOZl5AfBR4LOTnZiZN2fmhszcsHr16q41UJIkqdOKDmyZOZSZexvb24C+iDi95mZJkiR1VdGBLSLWREQ0ti+iau/T9bZKkiSpu2odJRoRnwReCZweETuB9wF9AJl5E/Am4BciYgQ4ALw5M7Om5kqSJNWi1sCWmW+Z5vjHqKb9kCRJWrSKviQqSZIkA5skSVLxDGySJEmFM7BJkiQVzsAmSZJUOAObJElS4QxskiRJhTOwSZIkFc7AJkmSVDgDmyRJUuEMbJIkSYUzsEmSJBXOwCZJklQ4A5skSVLhDGySJEmFM7BJkiQVzsAmSZJUOAObJElS4QxskiRJhTOwSZIkFc7AJkmSVDgDmyRJUuEMbJIkSYUzsEmSJBXOwCZJklQ4A5skSVLhDGySJEmFM7BJkiQVzsAmSZJUOAObJElS4QxskiRJhTOwSZIkFc7AJkmSVDgDmyRJUuEMbJIkSYUzsEmSJBXOwCZJklQ4A5skSVLhDGySJEmFM7BJkiQVzsAmSZJUOAObJElS4QxskiRJhTOwSZIkFc7AJkmSVDgDmyRJUuEMbJIkSYWrNbBFxK0R8VREfHuS4xERH4mIHRHxzYh4SbfbKKlmw8Nwyy3wy79c3Q8P190iSeq63prf/zbgY8Adkxy/FFjfuL0U+HjjXtJicM89sHEjHDsG+/ZBfz+8852wbRtcckndrZOkrqm1hy0z7wa+P8UplwF3ZOWrwKkRcWZ3WiepVsPDVVgbHq7CGlT3o/v37q23fZLURaXXsK0Fnmh6vLOxT9JCd+edVc9aK8eOVcclaZEoPbBFi33Z8sSIKyNie0Rs37NnT4ebJanjHnvsRM/aePv2wY4d3W2PJNWo9MC2E1jX9PhsYFerEzPz5szckJkbVq9e3ZXGSeqg9eurmrVW+vvhBS/obnskqUalB7a7gCsao0UvBp7JzN11N0pSF2zaBD2T/BXV01Mdl6RFotZRohHxSeCVwOkRsRN4H9AHkJk3AduAjcAOYD/w1npaKqnrBger0aDjR4n29FT7BwbqbqEkdU2tgS0z3zLN8QS2dKk5kkpzySWwa1c1wGDHjuoy6KZNhjVJi07d87BJ0tQGBuBtb6u7FZJUq9Jr2CRJkhY9A5skSVLhDGySJEmFM7BJkiQVzsAmSZJUOAObJElS4QxskiRJhTOwSZIkFc7AJkmSVDgDmyRJUuEMbJIkSYUzsEmSJBXOwCZJklQ4A5skSVLhDGySJEmFM7BJkiQVzsAmSZJUOAObJElS4QxskiRJhTOwSZIkFc7AJkmSVDgDmyRJUuEMbJIkSYXrrbsBknRSGB6GO++Exx6D9eth0yYYHKy7VZIWCQObJE3nnntg40Y4dgz27YP+fnjnO2HbNrjkkrpbJ2kR8JKoJE1leLgKa8PDVViD6n50/9699bZP0qJgYJOkqdx5Z9Wz1sqxY9VxSeqwti+JRsRpwHpg+ei+zLy7E42SpGI89tiJnrXx9u2DHTu62x5Ji1JbgS0i/itwLXA2cB9wMfC3wKs71zRJKsD69VXNWqvQ1t8PL3hB99skadFp95LotcCPAv+cma8CXgzs6VirJKkUmzZBzyR/Vfb0VMclqcPaDWwHM/MgQEQsy8yHgXM71yxJKsTgYDUadHCw6lGD6n50/8BAve2TtCi0W8O2MyJOBT4LfDkifgDs6lyzJKkgl1wCu3ZVAwx27Kgug27aZFiT1DWRmTN7QsQrgFOAL2bm4Y60ao42bNiQ27dvr7sZkiRJY0TEvZm5YabPm8ko0SXAs4F/bOxaA3xnpm8oSZKkmWl3lOgvAu8DngRGJyRK4EUdapckqRQuyyXVrt0etmuBczPz6U42RpJUGJflkorQ7ijRJ4BnOtkQSVJhXJZLKka7PWyPA38ZEZ8HDo3uzMwbOtIqSVL92lmW621v626bpEWq3cD2ncZtaeMmSVroXJZLKkZbgS0zfw0gIvozc5JvryRpQXFZLqkYbdWwRcTLIuJB4KHG4wsi4saOtkySVC+X5ZKK0e6gg98BfhJ4GiAz7wf+facaJUkqgMtyScVoe+LczHwiIpp3HZ3/5kiSiuKyXFIR2g1sT0TEjwEZEUuB/07j8qgkaYEbGHA0qFSzdi+JXgVsAdYCO4ELgas71ShJkiSd0G4P24eAazLzBwARcVpj33/pVMMkSQVxeSqpVu0GtheNhjWAzPxBRLy4Q22SJJXE5amk2rV7SbSn0asGQEQ8ixkMWJhMRLwuIh6JiB0RcV2L46+MiGci4r7G7b1zfU9J0gy4PJVUhJlcEv2biPgjIIHLgd+cyxtHxBJgK/Aaqrq4r0fEXZn54LhT/yozXz+X95IkzZLLU0lFaHelgzsiYjvwaiCAN7QIVjN1EbAjMx8HiIhPAZcBc31dSYuZtVbzy+WppCLMZB62B5nfMLUWeKLp8U7gpS3Oe1lE3A/sAt6dmQ/MYxskLSSdrLXatQve8x54+GE47zz44AfhrLPmp90lW7du6uNnn92ddkiLXLs1bJ0QLfbluMffAJ6TmRcAHwU+O+mLRVwZEdsjYvuePXvmsZmSTgqdrLW68UZYuxbuuAO+9rXqfu3aar8kdUGdgW0n0Pxft7OpetGOy8yhzNzb2N4G9EXE6a1eLDNvzswNmblh9erVnWq
|
|||
|
|
"text/plain": [
|
|||
|
|
"<Figure size 691.2x388.8 with 1 Axes>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"metadata": {
|
|||
|
|
"needs_background": "light"
|
|||
|
|
},
|
|||
|
|
"output_type": "display_data"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"fig = plot_data(Xo, yo, xlabel=u'metraż', ylabel=u'cena')\n",
|
|||
|
|
"theta_start = np.matrix([0.0, 0.0]).reshape(2, 1)\n",
|
|||
|
|
"theta, logs = gradient_descent(cost, gradient, theta_start, Xo, yo, alpha=0.01)\n",
|
|||
|
|
"plot_regression(fig, h_linear, theta, Xo)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"Na powyższym wykresie widać, że po odrzuceniu obserwacji odstających otrzymujemy dużo bardziej „wiarygodną” krzywą regresji."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "slide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"## 3.4. Problem nadmiernego dopasowania"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### Obciążenie a wariancja"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 24,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [],
|
|||
|
|
"source": [
|
|||
|
|
"# Dane do prostego przykładu\n",
|
|||
|
|
"\n",
|
|||
|
|
"data = np.matrix([\n",
|
|||
|
|
" [0.0, 0.0],\n",
|
|||
|
|
" [0.5, 1.8],\n",
|
|||
|
|
" [1.0, 4.8],\n",
|
|||
|
|
" [1.6, 7.2],\n",
|
|||
|
|
" [2.6, 8.8],\n",
|
|||
|
|
" [3.0, 9.0],\n",
|
|||
|
|
" ])\n",
|
|||
|
|
"\n",
|
|||
|
|
"m, n_plus_1 = data.shape\n",
|
|||
|
|
"n = n_plus_1 - 1\n",
|
|||
|
|
"Xn1 = data[:, 0:n]\n",
|
|||
|
|
"Xn1 /= np.amax(Xn1, axis=0)\n",
|
|||
|
|
"Xn2 = np.power(Xn1, 2) \n",
|
|||
|
|
"Xn2 /= np.amax(Xn2, axis=0)\n",
|
|||
|
|
"Xn3 = np.power(Xn1, 3) \n",
|
|||
|
|
"Xn3 /= np.amax(Xn3, axis=0)\n",
|
|||
|
|
"Xn4 = np.power(Xn1, 4) \n",
|
|||
|
|
"Xn4 /= np.amax(Xn4, axis=0)\n",
|
|||
|
|
"Xn5 = np.power(Xn1, 5) \n",
|
|||
|
|
"Xn5 /= np.amax(Xn5, axis=0)\n",
|
|||
|
|
"\n",
|
|||
|
|
"X1 = np.matrix(np.concatenate((np.ones((m, 1)), Xn1), axis=1)).reshape(m, n + 1)\n",
|
|||
|
|
"X2 = np.matrix(np.concatenate((np.ones((m, 1)), Xn1, Xn2), axis=1)).reshape(m, 2 * n + 1)\n",
|
|||
|
|
"X5 = np.matrix(np.concatenate((np.ones((m, 1)), Xn1, Xn2, Xn3, Xn4, Xn5), axis=1)).reshape(m, 5 * n + 1)\n",
|
|||
|
|
"y = np.matrix(data[:, -1]).reshape(m, 1)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 25,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAmEAAAFoCAYAAAAfEiweAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAT+0lEQVR4nO3df4zteV3f8dd79kKQmWlcwgXWhRbqnYCWP8TeEpRJQ0Xa9bZxW6OZNVFXc5NNm1Kx17RS20jSNC1pGlPbWJvNQtEUYQhi3dhblaJEb7Rk765bYbmSmVCF27u6lzbB2Wkb3M6nf5y5vdfLvXtnl5nve+6cxyPZnJnzPXPOO9987/Dk+2tqjBEAAKa10D0AAMA8EmEAAA1EGABAAxEGANBAhAEANBBhAAANDizCquq9VfVUVX3qmudeUlUfraqN3cc7D+rzAQAOs4PcE/a+JPdc99w7k3xsjLGS5GO73wMAzJ06yJu1VtWrk/ziGOP1u99/JslbxhhPVtVdST4+xnjtgQ0AAHBITX1O2MvHGE8mye7jyyb+fACAQ+FY9wA3U1UPJHkgSRYXF//86173uuaJAAD+pEcfffQLY4zjz+dnp46wP6yqu645HPnUzV44xngwyYNJcvLkyXH+/PmpZgQA2JOq+v3n+7NTH458OMn9u1/fn+QXJv58AIBD4SBvUfGBJL+V5LVVdbGqTid5d5K3VdVGkrftfg8AMHcO7HDkGOO7b7LorQf1mQAAtwt3zAcAaCDCAAAaiDAAgAYiDACggQgDAGggwgAAGogwAIAGIgwAoIEIAwBoIMIAABqIMACABiIMAKCBCAMAaCDCAAAaiDAAgAYiDACggQgDAGggwgAAGogwAIAGx7oHAOCI2tpK1teTjY1kZSVZW0uWl7ungkNDhAGw/86dS06dSnZ2ku3tZHExOXMmOXs2WV3tng4OBYcjAdhfW1uzANvamgVYMnu88vzTT/fOB4eECANgf62vz/aA3cjOzmw5IMIA2GcbG1f3gF1vezvZ3Jx2HjikRBgA+2tlZXYO2I0sLiYnTkw7DxxSIgxgHm1tJQ89lPzIj8wet7b2773X1pKFm/zPy8LCbDng6kiAuXPQVy4uL8/e6/rPWFiYPb+09JV/BhwBIgxgnlx75eIVV87fOnUquXRpfyJpdXX2Xuvrs3PATpyY7QETYPD/iTCAebKXKxdPn96fz1pa2r/3giPIOWEA88SVi3BoiDCAeeLKRTg0RBjAPHHlIhwaIgxgnly5cnF5+eoescXFq887cR4m48R8gHnjykU4FEQYwDxy5SK0czgSAKCBCAMAaCDCAAAaiDAAgAYiDACggQgDAGggwgAAGogwAIAGIgwAoIEIAwBoIMIAABqIMACABiIMAKBBS4RV1d+tqieq6lNV9YGqelHHHAAAXSaPsKq6O8kPJjk5xnh9kjuS3Df1HAAAnboORx5L8lVVdSzJi5NcapoDAKDF5BE2xvjvSf5Fks8leTLJF8cYv3L966rqgao6X1XnL1++PPWYAAAHquNw5J1J7k3ymiRfk2Sxqr7n+teNMR4cY5wcY5w8fvz41GMCAByojsOR35rkv40xLo8x/jjJR5J8c8McAABtOiLsc0neVFUvrqpK8tYkFxrmAABo03FO2CeSfDjJY0k+uTvDg1PPAQDQ6VjHh44x3pXkXR2fDQBwGLhjPgBAAxEGANBAhAEANBBhAAANRBgAQAMRBgDQQIQBADQQYQAADUQYAEADEQYA0ECEAQA0EGEAAA1EGABAg2PdAwC029pK1teTjY1kZSVZW0uWl7unAo44EQbMt3PnklOnkp2dZHs7WVxMzpxJzp5NVle7pwOOMIcjgfm1tTULsK2tWYAls8crzz/9dO98wJEmwoD5tb4+2wN2Izs7s+UAB0SEAfNrY+PqHrDrbW8nm5vTzgPMFREGzK+Vldk5YDeyuJicODHtPMBcEWHA/FpbSxZu8mtwYWG2HOCAiDBgfi0vz66CXF6+ukdscfHq80tLvfMBR5pbVADzbXU1uXRpdhL+5ubsEOTamgADDpwIA1haSk6f7p4CmDMORwIANBBhAAANRBgAQAMRBgDQQIQBADQQYQAADUQYAEADEQYA0ECEAQA0EGEAAA1EGABAAxEGANBAhAEANBBhAAANRBgAQAMRBgDQQIQBADQQYQAADUQYAEADEQYA0ECEAQA0EGEAAA1EGABAAxEGANCgJcKq6qur6sNV9btVdaGqvqljDgCALseaPvcnkvzSGOM7q+qFSV7cNAcAQIvJI6yq/lSSv5jk+5NkjPGlJF+aeg4AgE4dhyP/bJLLSf5dVf12VT1UVYsNcwAAtOmIsGNJvjHJT40x3pBkO8k7r39RVT1QVeer6vzly5ennhEA4EB1RNjFJBfHGJ/Y/f7DmUXZnzDGeHCMcXKMcfL48eOTDggAcNAmj7Axxh8k+XxVvXb3qbcm+fTUcwAAdOq6OvLvJHn/7pWRn03yA01zAAC0aImwMcbjSU52fDYAwGHgjvkAAA1EGABAAxEGANBAhAEANBBhAAANRBgAQAMRBgDQQIQBADQQYQAADUQYAEADEQYA0ECEAQA0EGEAAA1EGABAAxEGANBAhAEANBBhAAANRBgAQAMRBgDQQIQBADQQYQAADUQYAEADEQYA0ECEAQA0EGEAAA2OdQ8AzJmtrWR9PdnYSFZWkrW1ZHm5eyqAyYkwYDrnziWnTiU7O8n2drK4mJw5k5w9m6yudk8HMCmHI4FpbG3NAmxraxZgyezxyvNPP907H8DERBgwjfX12R6wG9nZmS0HmCMiDJjGxsbVPWDX295ONjennQegmQgDprGyMjsH7EYWF5MTJ6adB6CZCAOmsbaWLNzkV87Cwmw5wBwRYcA0lpdnV0EuL1/dI7a4ePX5paXe+QAm5hYVwHRWV5NLl2Yn4W9uzg5Brq0JMGAuiTBgWktLyenT3VMAtHM4EgCgwS0jrKreXlV3TjEMAMC82MuesFckeaSqPlRV91RVHfRQAABH3S0jbIzxj5KsJHlPku9PslFV/7SqvvaAZwMAOLL2dE7YGGMk+YPd/55JcmeSD1fVPz/A2QAAjqxbXh1ZVT+Y5P4kX0jyUJK/N8b446paSLKR5O8f7IgAAEfPXm5R8dIk3zHG+P1rnxxj7FTVXzuYsQAAjrZbRtgY48eeZdmF/R0HAGA+uE8YAEADEQYA0ECEAQA0EGEAAA1EGABAg7YIq6o7quq3q+oXu2YAAOjSuSfsHUnc4gIAmEstEVZVr0zyVzO7Az8AwNzp2hP2LzP7c0c7N3tBVT1QVeer6vzly5enmwwAYAKTR9junzp6aozx6LO9bozx4Bjj5Bjj5PHjxyeaDgBgGh17wt6c5Nur6veSfDDJt1TVv2+YAwCgzeQRNsb4B2OMV44xXp3kviS/Osb4nqnnAADo5D5hAAANjnV++Bjj40k+3jkDAEAHe8IAABqIMACABiIMAKCBCAMAaCDCAAAaiDAAgAYiDACggQgDAGggwgAAGogwAIAGIgwAoIEIAwBoIMIAABqIMACABiIMAKCBCAMAaCDCAAAaiDAAgAYiDACggQgDAGggwgAAGogwAIAGIgwAoIEIAwBoIMIAABqIMACABiIMAKCBCAMAaCDCAAAaiDAAgAYiDACggQgDAGggwgAAGogwAIAGIgwAoIEIAwBoIMIAABqIMACABiIMAKCBCAMAaCDCAAAaiDAAgAYiDACggQgDAGggwgAAGogwAIAGIgwAoMHkEVZVr6qqX6uqC1X1RFW9Y+oZAAC6HWv4zGeS/PAY47GqWk7yaFV9dIzx6YZZAABaTL4nbIzx5Bjjsd2vt5JcSHL31HMAAHRqPSesql6d5A1JPtE5BwDA1NoirKqWkvxckh8aY/zRDZY/UFXnq+r85cuXpx8QAOAAtURYVb0gswB7/xjjIzd6zRjjwTHGyTHGyePHj087IADAAZv8xPyqqiTvSXJhjPHjU38+kGRrK1lfTzY2kpWVZG0tWV7ungpgrnRcHfnmJN+b5JNV9fjucz86xjjbMAv
|
|||
|
|
"text/plain": [
|
|||
|
|
"<Figure size 691.2x388.8 with 1 Axes>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"metadata": {
|
|||
|
|
"needs_background": "light"
|
|||
|
|
},
|
|||
|
|
"output_type": "display_data"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"fig = plot_data(X1, y, xlabel='x', ylabel='y')"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 29,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [],
|
|||
|
|
"source": [
|
|||
|
|
"# Funkcja regresji wielomianowej\n",
|
|||
|
|
"\n",
|
|||
|
|
"def h_poly(Theta, x):\n",
|
|||
|
|
" \"\"\"Funkcja wielomianowa\"\"\"\n",
|
|||
|
|
" return sum(theta * np.power(x, i) for i, theta in enumerate(Theta.tolist()))\n",
|
|||
|
|
"\n",
|
|||
|
|
"def polynomial_regression(theta):\n",
|
|||
|
|
" \"\"\"Funkcja regresji wielomianowej\"\"\"\n",
|
|||
|
|
" return lambda x: h_poly(theta, x)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 30,
|
|||
|
|
"metadata": {},
|
|||
|
|
"outputs": [],
|
|||
|
|
"source": [
|
|||
|
|
"def plot_fun(fig, fun, X):\n",
|
|||
|
|
" \"\"\"Wykres funkcji `fun`\"\"\"\n",
|
|||
|
|
" ax = fig.axes[0]\n",
|
|||
|
|
" x0 = np.min(X[:, 1]) - 1.0\n",
|
|||
|
|
" x1 = np.max(X[:, 1]) + 1.0\n",
|
|||
|
|
" Arg = np.arange(x0, x1, 0.1)\n",
|
|||
|
|
" Val = fun(Arg)\n",
|
|||
|
|
" return ax.plot(Arg, Val, linewidth='2')"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 31,
|
|||
|
|
"metadata": {
|
|||
|
|
"scrolled": true,
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"text/plain": [
|
|||
|
|
"[<matplotlib.lines.Line2D at 0x19d045ece20>]"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"execution_count": 31,
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "execute_result"
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAmEAAAFoCAYAAAAfEiweAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXiU9b3+8fubHSZhD/sOkR0SS13Ro3VHLXXBhJ722Nbfse2pBY0b7latlbpC29MeThfb05YEXFHRui9oXdCEsJOwh7AkLGGyL/P9/TGxUJpAEmbmO8v7dV1eSeaZzHNfz/VkvHmWzxhrrQAAABBaca4DAAAAxCJKGAAAgAOUMAAAAAcoYQAAAA5QwgAAAByghAEAADgQtBJmjPm9MWavMWb1EY/1Msa8YYwpbvnaM1jrBwAACGfBPBL2tKSLj3psrqS3rLUZkt5q+RkAACDmmGAOazXGDJf0srV2YsvPGySdY63dZYwZIOlda+2YoAUAAAAIU6G+JqyftXaXJLV87Rvi9QMAAISFBNcB2mKMuV7S9ZLk8Xi+MnbsWMeJAAAdUVZZq31VDUpKiFNG31TFGeM6EhBwn3/+eYW1Nr0zvxvqErbHGDPgiNORe9t6orV2oaSFkjR16lS7YsWKUGUEAJygdzbs1Xf/8JmGxBk9+8MzNGVID9eRgKAwxmzr7O+G+nTkUknXtnx/raQXQ7x+AECQlXvrdeuSlZKkmy8cQwED2hDMERWLJP1d0hhjTKkx5jpJj0i6wBhTLOmClp8BAFHCWqtbn1mpiqoGnT6yt75/9kjXkYCwFbTTkdbaWW0sOi9Y6wQAuPX0R1v17oZy9eiaqCeypygujuvAgLYwMR8AEBDrdh3Sz15dL0l65MrJGtC9i+NEQHijhAEATlhdY7NmLypQQ5NPs04Zqosn9ncdCQh7lDAAwAn76SvrVLy3SiPTPbrnsnGu4wARgRIGADghb67do//7eJsS440W5GSpa1LYjqAEwgolDADQaXsP1em2Z4skSbddNFYTB3V3nAiIHJQwAECn+HxWNy9Zqf3VDToro4+umzbCdSQgolDCAACd8rvlW/RBcYV6eZL0+EzGUQAdRQkDAHTY6p2V+vnf/OMo5l01WX27pThOBEQeShgAoENqGpo0O69Ajc1W3z5tmC4Y3891JCAiUcIAAB3y4MvrtLm8Whl9U3XXpYyjADqLEgYAaLfXVu/Wok+3KykhTgtmZSklMd51JCBiUcIAAO2yq7JWc5/zj6OYe/FYjRvQzXEiILJRwgAAx9Xss8rNX6mDNY06Z0y6vnvmcNeRgIhHCQMAHNfC9zfr75v3qU9qkh69eoqMYRwFcKIoYQCAY1q546Aef32DJOnRq6coPS3ZcSIgOlDCAABtqq5v0py8AjX5rL5zxnCdO7av60hA1KCEAQDadP/SNdq6r0Zj+6dp7iVjXccBogolDADQqpeLyrTk81IlM44CCApKGADgX5QeqNEdz62SJN196Tid1C/NcSIg+iS4DgAACC9fjqPw1jXp/HF99a3ThnXuhbxeKT9fKi6WMjKk7GwpjTIHfIkSBgD4J//9Tok+3bpf6WnJmnfV5M6No1i+XJo+XfL5pOpqyeORcnOlZcukadMCHxqIQJyOBAD8w+fbDuipt4olSY/PnKLeqZ0YR+H1+guY1+svYJL/65ePV1UFMDEQuShhAABJkreuUTfmF6jZZ/WfZ43Q2Seld+6F8vP9R8Ba4/P5lwOghAEA/O59cY127K/V+AHddMtFYzr/QsXFh4+AHa26Wiop6fxrA1GEEgYA0AsFO/V8wU6lJPrHUSQnnMA4iowM/zVgrfF4pNGjO//aQBThwnwAiEVH3Lm4Y/hY3b1rgCTp3ssmaHTf1BN77exs/0X4rYmL8y8HQAkDgJhzxJ2LTTW1mvPtR1U1oK8u6p+gWacMOfHXT0vz3wV59N2RcXH+x1NPsOQBUYISBgCx5Mg7FyUtmPZNfTFgjPp7K/TI7+6Q+X8bA1OSpk2Tysr8R9tKSvynILOzKWDAEShhABBLjrhz8bNB4/XL07NlrE9PvPyEetYe8i+/7rrArCs1NXCvBUQhShgAxJKWOxcrkz268fJb5IuL1w8+XqIzthf5l3PnIhAylDAAiCUZGbIej+4670fa2b2vJu/aqNwP/uJfxp2LQEgxogIAYkl2tp4dd45eHne2ujbUav5LjynJ1+Rfxp2LQEhxJAwAYsjW+jjdd+EPpWbp/g+e1ogDZdy5CDhCCQOAGNHY7NOc/EJVN0uXjkvXzNFXSOdP4s5FwBFKGADEiKfe3KiVOw5qYPcUPTwzS6brKa4jATGNa8IAIAZ8vHmf/vvdTYoz0pPZmereNdF1JCDmUcIAIModrGnQTfmFslb60bmjderI3q4jARAlDACimrVWdz6/Srsq65Q5pIdmn5fhOhKAFpQwAIhii1fs0LJVu5WanKAFOVlKjOdtHwgX/DUCQJTaVF6l+5eulSQ9MGOChvbu6jgRgCNRwgAgCjU0+XRjXqFqG5s1I3Ogrsga5DoSgKNQwgAgCj3+xgat2lmpwT276MFvTJQxxnUkAEehhAFAlPmwpEL/895mxRlpfk6muqUwjgIIR5QwAIgi+6sblLu4UJI0+7wMfWVYL8eJALSFEgYAUcJaq9ufLdKeQ/WaOqynbjh3tOtIAI6BEgYAUeKvn27XG2v3KC05QU9mZyqBcRRAWOMvFACiQMlerx582T+O4qdXTtKQXoyjAMIdJQwAIlx9U7N+vKhQdY0+XXnyIH19ykDXkQC0g5MSZoy5yRizxhiz2hizyBiT4iIHAESDn7+2Qet2HdLQXl31wIyJruMAaKeQlzBjzCBJsyVNtdZOlBQvKSfUOQAgGry3sVy/W75F8XFG83MylZqc4DoSgHZydToyQVIXY0yCpK6SyhzlAICIVVFVr5sXr5Qk5V5wkrKG9nScCEBHhLyEWWt3SnpM0nZJuyRVWmtfP/p5xpjrjTErjDErysvLQx0TAMKatVa3PVOkiqp6nTqil37wb6NcRwLQQS5OR/aUNEPSCEkDJXmMMd86+nnW2oXW2qnW2qnp6emhjgkAYe1Pf9+mt9fvVbcU/ziK+Dg+lgiINC5OR54vaYu1ttxa2yjpOUlnOMgBABFpw26vfrpsnSTpkasma2CPLo4TAegMFyVsu6TTjDFdjf8TZc+TtM5BDgCIOHWNzZq9qEANTT5lTx2i6ZMGuI4EoJNcXBP2iaRnJH0haVVLhoWhzgEAkeiRV9drwx6vRvTx6N7Lx7uOA+AEOLmX2Vp7n6T7XKwbACLV2+v36OmPtiox3mhBTpY8jKMAIhoT8wEgAuz11unWJUWSpJsvHKNJg7s7TgTgRFHCACDM+XxWtywp0r7qBp0xqreuP2uk60gAAoASBgBh7g8fbdX7G8vVo2uinrgmU3GMowCiAiUMAMLYmrJKzXt1vSRp3lWT1b87H7ULRAtKGACEqdqGZs3JK1RDs0/fPHWoLprQ33UkAAFECQOAMPXQK2tVsrdKo9I9uudSxlEA0YYSBgBh6PU1u/WXT7YrKT5OC2ZlqUtSvOtIAAKMEgYAYWbPoTrd/qx/HMVtF4/RhIGMowCiESUMAMKIz2eVu7hQB2oadVZGH33vzBGuIwEIEkoYAISR//1gsz4s2afeniQ9fs0UxlEAUYwSBgBhYlVppR57fYMk6edXT1bfNMZRANGMEgYAYaCmoUlz8grU2Gx17enDdN64fq4jAQgyShgAhIEHXlqrzRXVGtMvTXdMH+c6DoAQSHAdAACc83ql/HypuFjKyJCys6W0tJCt/tVVu5T32Q4lJcRp/qxMpSQyjgKIBZQwALFt+XJp+nTJ55OqqyWPR8rNlZYtk6ZNC/rqyw7Wau5zqyRJd14yVmP7dwv6OgGEB05HAohdXq+/gHm9/gIm+b9++XhVVVBX3+yzuim/UJW1jTp3TLquPWN4UNcHILxQwgDErvx8/xGw1vh8/uVB9Jv3NumTLfvVJzVZj86cImMYRwHEEkoYgNhVXHz4CNjRqqulkpKgrbp
|
|||
|
|
"text/plain": [
|
|||
|
|
"<Figure size 691.2x388.8 with 1 Axes>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"metadata": {
|
|||
|
|
"needs_background": "light"
|
|||
|
|
},
|
|||
|
|
"output_type": "display_data"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"fig = plot_data(X1, y, xlabel='x', ylabel='y')\n",
|
|||
|
|
"theta_start = np.matrix([0, 0]).reshape(2, 1)\n",
|
|||
|
|
"theta, _ = gradient_descent(cost, gradient, theta_start, X1, y, eps=0.00001)\n",
|
|||
|
|
"plot_fun(fig, polynomial_regression(theta), X1)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"Ten model ma duże **obciążenie** (**błąd systematyczny**, *bias*) – zachodzi **niedostateczne dopasowanie** (*underfitting*)."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 32,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"text/plain": [
|
|||
|
|
"[<matplotlib.lines.Line2D at 0x19d0462d880>]"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"execution_count": 32,
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "execute_result"
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAmEAAAFoCAYAAAAfEiweAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3yV5cH/8e91ssmAACFA2CTsKWGDIm4caBVxVlstWldt+1StHc/z9Pm1tba1tVatVq0LFQsOFMRRQUWGhr1JGBkEQoCQBZnn+v1xAiKCBEjOdcbn/XrxSnLOCfl6v26O31zXfV+XsdYKAAAA/uVxHQAAACAcUcIAAAAcoIQBAAA4QAkDAABwgBIGAADgACUMAADAgWYrYcaY54wxu40xa494rLUx5kNjTHbDx+Tm+vkAAACBrDlHwp6XdOFRjz0g6T/W2gxJ/2n4GgAAIOyY5lys1RjTTdK71toBDV9vkjTBWrvTGNNB0gJrbe9mCwAAABCg/H1NWKq1dqckNXxs5+efDwAAEBAiXQc4HmPMNEnTJCk+Pn5Ynz59HCcCAAD4umXLlu2x1qacyvf6u4QVGWM6HDEduft4L7TWPi3paUnKzMy0WVlZ/soIAADQKMaY3FP9Xn9PR86WdFPD5zdJetvPPx8AACAgNOcSFa9KWiyptzGmwBhzi6SHJJ1njMmWdF7D1wAAAGGn2aYjrbXXHuepc5rrZwIAAAQLVswHAABwgBIGAADgACUMAADAAUoYAACAA5QwAAAAByhhAAAADlDCAAAAHKCEAQAAOEAJAwAAcIASBgAA4AAlDAAAwAFKGAAAgAOUMAAAAAcoYQAAAA5QwgAAAByghAEAADhACQMAAHCAEgYAAOBApOsAAIDwYa1Vndeqps7r+1Pv+1h91Ne+z+u/9lyrFtE6o0srtUmIcf2fATQJShgAoEmVHqzV5zl79MnaQn2xvkDlNV5VR0SqxhOpmnqvrD29v79H23id0TVZmV2TNaxrsnqmJMjjMU0THvAjShgA4LRYa7WusEyfbC7WJ5uKtSyvRPXeQ00rUjKSvJK8XklShMcoOsKj6MiGPxEexUR+/etjfb6j5KBWFezX1j2V2rqnUjOXFUiSWsZFaVhDIRvWNVmDO7VSXHSEk2MBnAxKGADgpJUeqNVnOcVasKlYn2wuVnF59eHnIow0YscGTcheqjO3r1C7in2Krq/1/YmLVeSOAikh4ZR+bm29V+sLy7Qst0TLckuUlbtPRWXV+njjbn28cbckKdJj1L9jkoZ1ba3Mbr5ilpoU2yT/3UBTMvZ0x4X9IDMz02ZlZbmOAQBhy+v1jXYt2LRbCzYXa0VeibxH/O+jfVKsJvRO0Vm9UjR20Vwl/fRHUmXlN/+i+Hjp0UelW25pklzWWu3Yf/CrUra9RBt3lX0tmyR1So47PH05vHtr9U5NlDFMYeL0GWOWWWszT+V7GQkDABxTSWWNPs32jXR9urlYeypqDj8X6TEa2b21zuqdogm9U75eal7efOwCJvkez8lpsozGGHVKbqFOyS00eUiaJKmiuk4r8/YrK3efluWWaEXefhWUHFRByUG9tbJQknRGl1a6e2KGJvROoYzBGUoYAOCwmjqvZmTl643lBVqVv/9rI0odW8bqrN7tNKF3isamt1VCzHH+F5KR4RvxOt5IWHp684RvkBATqXEZbTUuo60kqd5rtbmoXFm5JVqeW6L5m3Zred5+fe/5LzUgLUl3nZ2h8/ulcnE//I7pSAAIR+Xl0owZUna2lJEh79VXa862Cv3pg03K3XtAkhQVYTSie2tN6OUrXuntEho3alReLqWl+T4eLTFRKiw85WvCmkJldZ1eWZqnpz7dqj0VvmvZeqcm6s6J6bp4YAdFUMZwEk5nOpISBgDhZuFCadIk392KlZVa1GuEHhp7vVa36ylJ6pkSrx+d20vn9Gmn+OONdp3kz1B8vOTxSHPnSuPGNeF/zKmrqq3XjC/z9Y9PtmhnaZUk3/IXd5ydrslDOioqgvXMcWKUMABA4xwxSrU+pbv+MOEmfdLD9/+PdpUl+vGUkZoytqcim6KAVFT4RttycnxTkFOnOh0BO56aOq9mLS/QEwtylL/voCTfhfw/nNBTVw3rpJhIlrvA8VHCAACN88wzyv/Vb/XIsO/orf4TZI1HidWVun3JTH1vw3/U4s8PN9mdi8Gmtt6r2SsL9fiCHG0t9l3P1j4pVred1UPXDO/C2mM4Ju6OBACcUElljR7PrteL1/9FNZFRiqqv1Y3LZ+uuxa+r9cEy34ua8M7FYBMV4dGVwzrp8qFpmrtmp/7+cY42FZXrf99Zr8fn5+gH43vo+lFdj39DAnCSOJMAIMQdrKnXc59v0z8WbFG56SRFSpevm6+ffvayOpcWffVCP9y5GAwiPEaXDu6oiwd20EcbivTYxzlas6NUv39vo578ZIu+P7a7bhrTTS3jolxHRZBjOhIAQlRdvVczlxXoLx9tVlGZ7y7A8T2Sdf9Dt2vAtjXf/IYAuHMxEFlr9cnmYj32cY6W5ZZIkhJjInXTmG66ZVx3JcdHO04Il7gmDABwmLVWH64v0sPvb1LO7gpJ0oC0JD1wYV/f2llBcOdiILLWavHWvfr7xzlatGWvJKltQrT+MnWIxmekOE4HVyhhAABJUtb2fXrovY3Kahix6dw6Tv91fm9dOqjj1xcjDZI7FwPVstx9+sO8Tfpi2z4ZI90xoad+fG6vprmrFEGFEgYAYa64vFq/fGuN3l/nu8ardXy07p6YrutHdlV0JMWgOdR7rR6fn6O/frRZXisN75asv107VB1axrmOBj+ihAFAGFuVv1+3v7xMO0urFBcVoVvHd9e0M3soMZYLx/1hyda9+tFrK1RUVq3kFlH689WDNbFPqutY8JPTKWH8egQAQez1rHxNeWqxdpZW6YwurfTxf52ln57fmwLmR6N6tNHce8ZrQu8UlRyo1fefz9Jv56xXTZ3XdTQEOEoYAAShmjqvfv32Wt03c7Vq6ry6fmQXvTZtNFNhjrRJiNFzNw3Xzy/qowiP0T8/26YpTy1W/r4DrqMhgFHCACDI7C6v0vXPLNGLi3MVHeHRQ98ZqN9eMZBrvxzzeIxuO6unXr9ttNJaxWlV/n5N+ttnem/NTtfREKD4FwsAQWRFXokufWyhvtxeotSkGM24bZSuGdHFdSwcYVjXZM25Z5zO75eq8qo6/XD6cv367bWqqq13HQ0BhhIGAEFixpd5mvrUEhWVVWt4t2S9c/c4De2S7DoWjqFVi2g9deMw/c+l/RQd4dGLi3P1nScWaWtxhetoCCCUMAAIcDV1Xv3izTW6f9Ya1dR79d3RXTX91lFqlxjrOhq+hTFGN4/trlk/HKOubVpo/c4yXfrYQr29cofraAgQlDAACGC7y6p07T+XaPrSPEVHevTwVYP0m8kDuP4riAzs1FLv3j1OlwzqoMqaev3otZW6f+ZqHaxhejLc8a8YAALUstx9uuSxhVqWW6IOLWP179tG6+rMzq5j4RQkxkbpsWuH6ndXDFRMpEczsvI1+fGF2lxU7joaHKKEAUAAemVpnq55eol2l1drRPfWeufucRrcuZXrWDgNxhhdN7KL3rpzrHqmxGtzUYUu+/tCvf5lvoJh4XQ0PUoYAASQ6rp6/fyN1XrwzTWqrbe6eUw3Tb91pNomxLiOhibSt0OSZt81Tlee0UlVtV7dN2u1fjxjJXdPhqFI1wEAAD5FZVW6/eVlWpG3X9GRHv3uioG6algn17HQDOJjIvXnqwdrdM82+tVba/XWykLtqajRMzdlKjYqwnU8+AkjYQAQALK2+67/WpG3Xx1bxmrW7WMoYGHgqmGd9NadY9U2IUYLc/bo1heyGBELI05KmDHmx8aYdcaYtcaYV40x3GcNICxZa/XSklxd8/QSFZdXa1QP3/VfAzu1dB0NftK7faJe/cFIilgY8nsJM8akSbpHUqa1doCkCEnX+DsHALhmrdVv3l2vX721VnVeq1vGddfLt4xUG67/CjsZqRSxcORqOjJSUpwxJlJSC0mFjnIAgDM
|
|||
|
|
"text/plain": [
|
|||
|
|
"<Figure size 691.2x388.8 with 1 Axes>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"metadata": {
|
|||
|
|
"needs_background": "light"
|
|||
|
|
},
|
|||
|
|
"output_type": "display_data"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"fig = plot_data(X2, y, xlabel='x', ylabel='y')\n",
|
|||
|
|
"theta_start = np.matrix([0, 0, 0]).reshape(3, 1)\n",
|
|||
|
|
"theta, _ = gradient_descent(cost, gradient, theta_start, X2, y, eps=0.000001)\n",
|
|||
|
|
"plot_fun(fig, polynomial_regression(theta), X1)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "fragment"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"Ten model jest odpowiednio dopasowany."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 33,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"text/plain": [
|
|||
|
|
"[<matplotlib.lines.Line2D at 0x19d05047b20>]"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"execution_count": 33,
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "execute_result"
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAmEAAAFoCAYAAAAfEiweAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxU1eH///fJvhMgCzsBEkABFUFEBeqCG221tSK471pbq1Zr66efbp/+PvXbRWu1WvtRUNGqgFatVeuKG4LsyCokQIAQyAYkkz0zc35/TLBUWRLIzJnl9Xw8fCSZmcy8vY/L5c2595xrrLUCAABAaMW5DgAAABCLKGEAAAAOUMIAAAAcoIQBAAA4QAkDAABwgBIGAADgQNBKmDHmCWNMpTFmzX6P9TDGvGOMKW7/2j1Ynw8AABDOgjkS9pSk87702D2S3rPWFkl6r/1nAACAmGOCuVirMaZA0mvW2pHtP2+QdLq1dqcxprekD6y1w4IWAAAAIEyF+pqwfGvtTklq/5oX4s8HAAAICwmuAxyMMeYmSTdJUnp6+pjhw4c7TgQACFdev9WGXR75rdWgnHRlJIftX2+IMsuWLau21uYeye+Gei+tMMb03u90ZOXBXmitfUzSY5I0duxYu3Tp0lBlBABEmP9+ebX2Ltqms4bnaeY1J7mOgxhijNl6pL8b6tORr0q6uv37qyX9I8SfDwCIMiWV9Zq9ZLvijHTP+Zw1QeQI5hIVz0taKGmYMabMGHO9pN9KOtsYUyzp7PafAQA4Yr/91+fy+a2mjxugovxM13GADgva6Uhr7aUHeeqsYH0mACC2fLq5Ru+ur1BaUrzumFzkOg7QKayYDwCISH6/1b1vrJck3TxpiPIyUxwnAjqHEgYAiEj/XFWuVWW1ystM1o2TBrmOA3QaJQwAEHGa23z6/ZsbJEl3nTNUaUksSYHIQwkDAEScpxeWasfeJg3Lz9TFY/q7jgMcEUoYACCi7G1s1cPzSiRJ90wZrvg44zgRcGQoYQCAiPLneSWqa/ZqQmGOTh96RAuVA2GBEgYAiBhbaxr09MJSGSP915ThMoZRMEQuShgAIGL8/q0NavNZfXt0X43o0811HOCoUMIAABFh+bY9en3VTiUnxOlH5wxzHQc4apQwAEDYs9bq3tcDC7NeP2GQ+mSnOk4EHD1KGAAg7L21tkJLt+5Rj/Qkfff0Ia7jAF2CEgYACGsLSqr1q1fXSpLumFykrJREx4mArsESwwCAsLS3sVW/eX29XlhWJkk6oX+2Lh03wHEqoOtQwgAAYcVaq3+u2qlf/3OtqutblRQfpx+cWaibvzZEifGcwEH0oIQBAMLGjr1N+vkrazTv80pJ0rhBPfT/LhqlIbkZjpMBXY8SBgBwzue3enphqf7w1gY1tvqUmZKgn045RtPG9lcctyVClKKEAQCc+nxXne75+2qt3L5XknT+yF76nwtGKC8rxXEyILgoYQAAJ5rbfHp4Xon++uEmef1W+VnJ+v8uHKlzRvRyHQ0ICUoYACDkPt1co5++tFqbqxskSVeMH6Afnzec5ScQUyhhAIDg8HikOXOk4mKpqEiaNk21CSn67b/W6/nF2yVJhXkZ+u1FozS2oIfjsEDoUcIAAF1v/nxpyhTJ75caGmTT0/Wvh2frlxfeqaoWq8R4o++fUahbTh+i5IR412kBJyhhAICu5fEECpjHI0naldFTPz/7u3pn6ClSi9WYfln67dQTVJSf6Tgo4BYlDADQtebMkd9vVZ6Vq3cLT9Z9k65SfXKaMloa9ZOFz+nyG76huPyJrlMCzlHCAABHrMXr05bqBm2qbNCmqnqVVNZrU3GGNt/0lJqS/r3ExNkbF+rX7/5VvT010hnDHSYGwgclDABwWLWNbSqp8mhTZYNKquq1qbJeJVX12r67UX77pRebTClJyq3frcKa7bpq+es6b+MCGUlKT5cKCx38HwDhhxIGALHoADMXbUaGKj0t2rDLo5L2krWpsl6bqupVXd96wLeJM9KgnHQNyU3XkLwMDcnNUGFGnIZMHKtu1bsO8Atx0rRpQf6fAyIDJQwAYoz9+GNVX3yZNnbvq40Zedq4tlXFHz+mjQOGq67twL+TmhivIXnpgZKVm6EheRkqzMvQwJ5pB57d+PIL/zE7UunpgQL2xhtSBveBBCRKGABEtd0NrdpY4fn3f+W1Kt6wQ3uu/etXX9wmdUtJ0LBeWSrMD5StwrxA4eqdldK5ezhOmCCVlwdG20pKAqcgp02jgAH7oYQBQBSw1mr1jlqtKqtVcYVHGyvqVVzpOfBpxNRMZbY0qKh6m4ZVbVVR9TYNrd6qoY3Vyr33f2RuuL5rQmVkSNd30XsBUYgSBgARzFqrj4qr9dB7xVq2dc9Xnk9PildhfqaG5mVoWK9MFb02V0MfvU+9PDU64LjWppKgZwYQQAkDgAhkrdX7Gyr14Hsl+mz7XklSdlqizhyep6H5mRqWn6mi/Az16Zb6n6cRN+RI/uYDvykzF4GQooQBQASx1uqddRV6aF6x1uyokyT1TE/SjZMG64rxA5WRfJjD+rRp0p13Hvg5Zi4CIUUJA4AI4Pdbvbl2l/48r0TrdwbKV05Gsr77tcG67OQBSkvq4OE8MzMwQ5GZi4BzlDAACGM+v9Xrq3fq4XnF2lhRL0nKz0rWd782RJeOG6CUxCO4+TUzF4GwQAkDgDDk9fn1z1Xl+vO8Em2uapAk9emWolvOKNTUMf2OrHztj5mLgHOUMAAII20+v15ZsUOPvF+i0ppGSVK/7qn6/hmF+s6J/ZSUEOc4IYCuQgkDgDDQ6vXrpeVleuSDEm3f3SRJGtgzTd8/o1DfHt1XifGULyDaUMIAwKEWr08vLC3Tox9s0o69gfI1OCddt55ZqAuO76MEyhcQtShhAOBIXXObrntyiZa2L7JamJehH5xZqG8c10fxnblFEICIRAkDAAdqG9t01ZOL9dn2verdLUU/+/qxOn9kr87dnxFARKOEAUCI7Wlo1RUzF2lteZ36dU/V8zeOV/8eaa5jAQgxShgAhFB1fYuumLFIn+/yaGDPND1/43j1yU51HQuAA5QwAAiRyrpmXT5jkYor6zU4N13P3zhe+VkprmMBcIQSBgAhsKu2WZc9/qk2VzdoaH6Gnr1hvHIzk13HAuAQJQwAgqxsT6Mue3yRtu1u1DG9s/S368epZwYFDIh1lDAACKJtNY269PFPtWNvk0b17aZnrh+n7LQk17EAhAFKGAAEyZbqBl32+KfaWdus0QOy9dS149QtNdF1LABhghIGAEFQUunRZY8vUqWnRScVdNcT15ykzBQKGIB/c3I/DGPMD40xa40xa4wxzxtjmB4EIGps2OXR9Mc+VaWnRacM7qmnrh1HAQPwFSEvYcaYvpJukzTWWjtSUryk6aHOAQDBsLa8VtMfW6jq+lZNLMrRE9ecpPRkTjoA+CpXR4YESanGmDZJaZLKHeUAgC6zqmyvrpy5WLVNbTpzeJ7+cvmJSkmMdx0LQJgK+UiYtXaHpPskbZO0U1KttfbtL7/OGHOTMWapMWZpVVVVqGMCQKcs27pHlz++SLVNbTrn2Hz99YoxFDAAh+TidGR3SRdKGiSpj6R0Y8wVX36dtfYxa+1Ya+3Y3NzcUMcEgA5bvGW3rpq5SJ4Wr74+qrceufxEJSU4ueQWQARxcZSYLGmLtbbKWtsm6SVJpzrIAQBHbUFJta5+YrEaWn361gl99OD0E5QYTwEDcHgujhTbJI03xqQZY4yksyStd5ADAI7KRxurdO1TS9TU5tPFY/rp/ktOUAIFDEAHubgmbJGkFyUtl7S6PcNjoc4BAEdj3ucVumHWUrV4/bp03AD9/jvHKT7OuI4FIII4mR1prf2lpF+6+GwAOFrryuv03WeWq9Xn19WnDNSvLhihwMA+AHQc4+YA0AmtXr9+9MJnavX5NW1sfwoYgCNGCQOATnjk/RKt21mnAT3S9ItvHksBA3DEKGEA0EFrdtTqkfdLJEm/v/g4VsIHcFQoYQDQAftOQ3r9VtecWqDxg3u6jgQgwlHCAKAD/jy
|
|||
|
|
"text/plain": [
|
|||
|
|
"<Figure size 691.2x388.8 with 1 Axes>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"metadata": {
|
|||
|
|
"needs_background": "light"
|
|||
|
|
},
|
|||
|
|
"output_type": "display_data"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"fig = plot_data(X5, y, xlabel='x', ylabel='y')\n",
|
|||
|
|
"theta_start = np.matrix([0, 0, 0, 0, 0, 0]).reshape(6, 1)\n",
|
|||
|
|
"theta, _ = gradient_descent(cost, gradient, theta_start, X5, y, alpha=0.5, eps=10**-7)\n",
|
|||
|
|
"plot_fun(fig, polynomial_regression(theta), X1)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "fragment"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"Ten model ma dużą **wariancję** (*variance*) – zachodzi **nadmierne dopasowanie** (*overfitting*)."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"(Zwróć uwagę na dziwny kształt krzywej w lewej części wykresu – to m.in. efekt nadmiernego dopasowania)."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"Nadmierne dopasowanie występuje, gdy model ma zbyt dużo stopni swobody w stosunku do ilości danych wejściowych.\n",
|
|||
|
|
"\n",
|
|||
|
|
"Jest to zjawisko niepożądane.\n",
|
|||
|
|
"\n",
|
|||
|
|
"Możemy obrazowo powiedzieć, że nadmierne dopasowanie występuje, gdy model zaczyna modelować szum/zakłócenia w danych zamiast ich „głównego nurtu”. "
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"Zobacz też: https://pl.wikipedia.org/wiki/Nadmierne_dopasowanie"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"<img style=\"margin:auto\" width=\"90%\" src=\"fit.png\"/>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### Obciążenie (błąd systematyczny, *bias*)\n",
|
|||
|
|
"\n",
|
|||
|
|
"* Wynika z błędnych założeń co do algorytmu uczącego się.\n",
|
|||
|
|
"* Duże obciążenie powoduje niedostateczne dopasowanie."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### Wariancja (*variance*)\n",
|
|||
|
|
"\n",
|
|||
|
|
"* Wynika z nadwrażliwości na niewielkie fluktuacje w zbiorze uczącym.\n",
|
|||
|
|
"* Wysoka wariancja może spowodować nadmierne dopasowanie (modelując szum zamiast sygnału)."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"<img style=\"margin:auto\" width=\"60%\" src=\"bias2.png\"/>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"<img style=\"margin:auto\" width=\"60%\" src=\"curves.jpg\"/>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "slide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"## 3.5. Regularyzacja"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 56,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [],
|
|||
|
|
"source": [
|
|||
|
|
"def SGD(h, fJ, fdJ, theta, X, Y, \n",
|
|||
|
|
" alpha=0.001, maxEpochs=1.0, batchSize=100, \n",
|
|||
|
|
" adaGrad=False, logError=False, validate=0.0, valStep=100, lamb=0, trainsetsize=1.0):\n",
|
|||
|
|
" \"\"\"Stochastic Gradient Descent - stochastyczna wersja metody gradientu prostego\n",
|
|||
|
|
" (więcej na ten temat na wykładzie 11)\n",
|
|||
|
|
" \"\"\"\n",
|
|||
|
|
" errorsX, errorsY = [], []\n",
|
|||
|
|
" errorsVX, errorsVY = [], []\n",
|
|||
|
|
" \n",
|
|||
|
|
" XT, YT = X, Y\n",
|
|||
|
|
" \n",
|
|||
|
|
" m_end=int(trainsetsize*len(X))\n",
|
|||
|
|
" \n",
|
|||
|
|
" if validate > 0:\n",
|
|||
|
|
" mv = int(X.shape[0] * validate)\n",
|
|||
|
|
" XV, YV = X[:mv], Y[:mv] \n",
|
|||
|
|
" XT, YT = X[mv:m_end], Y[mv:m_end] \n",
|
|||
|
|
" m, n = XT.shape\n",
|
|||
|
|
"\n",
|
|||
|
|
" start, end = 0, batchSize\n",
|
|||
|
|
" maxSteps = (m * float(maxEpochs)) / batchSize\n",
|
|||
|
|
" \n",
|
|||
|
|
" if adaGrad:\n",
|
|||
|
|
" hgrad = np.matrix(np.zeros(n)).reshape(n,1)\n",
|
|||
|
|
" \n",
|
|||
|
|
" for i in range(int(maxSteps)):\n",
|
|||
|
|
" XBatch, YBatch = XT[start:end,:], YT[start:end,:]\n",
|
|||
|
|
"\n",
|
|||
|
|
" grad = fdJ(h, theta, XBatch, YBatch, lamb=lamb)\n",
|
|||
|
|
" if adaGrad:\n",
|
|||
|
|
" hgrad += np.multiply(grad, grad)\n",
|
|||
|
|
" Gt = 1.0 / (10**-7 + np.sqrt(hgrad))\n",
|
|||
|
|
" theta = theta - np.multiply(alpha * Gt, grad)\n",
|
|||
|
|
" else:\n",
|
|||
|
|
" theta = theta - alpha * grad\n",
|
|||
|
|
" \n",
|
|||
|
|
" if logError:\n",
|
|||
|
|
" errorsX.append(float(i*batchSize)/m)\n",
|
|||
|
|
" errorsY.append(fJ(h, theta, XBatch, YBatch).item())\n",
|
|||
|
|
" if validate > 0 and i % valStep == 0:\n",
|
|||
|
|
" errorsVX.append(float(i*batchSize)/m)\n",
|
|||
|
|
" errorsVY.append(fJ(h, theta, XV, YV).item())\n",
|
|||
|
|
" \n",
|
|||
|
|
" if start + batchSize < m:\n",
|
|||
|
|
" start += batchSize\n",
|
|||
|
|
" else:\n",
|
|||
|
|
" start = 0\n",
|
|||
|
|
" end = min(start + batchSize, m)\n",
|
|||
|
|
" return theta, (errorsX, errorsY, errorsVX, errorsVY)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 57,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [],
|
|||
|
|
"source": [
|
|||
|
|
"# Przygotowanie danych do przykładu regularyzacji\n",
|
|||
|
|
"\n",
|
|||
|
|
"n = 6\n",
|
|||
|
|
"\n",
|
|||
|
|
"data = np.matrix(np.loadtxt(\"ex2data2.txt\", delimiter=\",\"))\n",
|
|||
|
|
"np.random.shuffle(data)\n",
|
|||
|
|
"\n",
|
|||
|
|
"X = powerme(data[:,0], data[:,1], n)\n",
|
|||
|
|
"Y = data[:,2]"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 58,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [],
|
|||
|
|
"source": [
|
|||
|
|
"def draw_regularization_example(X, Y, lamb=0, alpha=1, adaGrad=True, maxEpochs=2500, validate=0.25):\n",
|
|||
|
|
" \"\"\"Rusuje przykład regularyzacji\"\"\"\n",
|
|||
|
|
" plt.figure(figsize=(16,8))\n",
|
|||
|
|
" plt.subplot(121)\n",
|
|||
|
|
" plt.scatter(X[:, 2].tolist(), X[:, 1].tolist(),\n",
|
|||
|
|
" c=Y.tolist(),\n",
|
|||
|
|
" s=100, cmap=plt.cm.get_cmap('prism'));\n",
|
|||
|
|
"\n",
|
|||
|
|
" theta = np.matrix(np.zeros(X.shape[1])).reshape(X.shape[1],1)\n",
|
|||
|
|
" thetaBest, err = SGD(h, J, dJ, theta, X, Y, alpha=alpha, adaGrad=adaGrad, maxEpochs=maxEpochs, batchSize=100, \n",
|
|||
|
|
" logError=True, validate=validate, valStep=1, lamb=lamb)\n",
|
|||
|
|
"\n",
|
|||
|
|
" xx, yy = np.meshgrid(np.arange(-1.5, 1.5, 0.02),\n",
|
|||
|
|
" np.arange(-1.5, 1.5, 0.02))\n",
|
|||
|
|
" l = len(xx.ravel())\n",
|
|||
|
|
" C = powerme(xx.reshape(l, 1),yy.reshape(l, 1), n)\n",
|
|||
|
|
" z = classifyBi(thetaBest, C).reshape(int(np.sqrt(l)), int(np.sqrt(l)))\n",
|
|||
|
|
"\n",
|
|||
|
|
" plt.contour(xx, yy, z, levels=[0.5], lw=3);\n",
|
|||
|
|
" plt.ylim(-1,1.2);\n",
|
|||
|
|
" plt.xlim(-1,1.2);\n",
|
|||
|
|
" plt.legend();\n",
|
|||
|
|
" plt.subplot(122)\n",
|
|||
|
|
" plt.plot(err[0],err[1], lw=3, label=\"Training error\")\n",
|
|||
|
|
" if validate > 0:\n",
|
|||
|
|
" plt.plot(err[2],err[3], lw=3, label=\"Validation error\");\n",
|
|||
|
|
" plt.legend()\n",
|
|||
|
|
" plt.ylim(0.2,0.8);"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 59,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"name": "stderr",
|
|||
|
|
"output_type": "stream",
|
|||
|
|
"text": [
|
|||
|
|
"<ipython-input-6-09634685a32b>:5: RuntimeWarning: overflow encountered in exp\n",
|
|||
|
|
" y = 1.0/(1.0 + np.exp(-x))\n",
|
|||
|
|
"<ipython-input-58-f0220c89a5e3>:19: UserWarning: The following kwargs were not used by contour: 'lw'\n",
|
|||
|
|
" plt.contour(xx, yy, z, levels=[0.5], lw=3);\n",
|
|||
|
|
"No handles with labels found to put in legend.\n"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA7AAAAHWCAYAAABOj2WsAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3SURRfA4d+b3iD0GkroAqEmICC9S5EqTaWI9CIISFMQpUixAIKAFEUE2ycg0hGkKoQivdfQi0B6ne+PCZCQsptkk03Ifc7Zk+zuvPPe3ZTduzNzx1BKIYQQQgghhBBCpHc21g5ACCGEEEIIIYQwhySwQgghhBBCCCEyBElghRBCCCGEEEJkCJLACiGEEEIIIYTIECSBFUIIIYQQQgiRIUgCK4QQQgghhBAiQ5AEVgghhMhkDMNoZhjGGcMwzhuGMTqe+90Nw/jdMIx/DcM4YRhGT2vEKYQQQjzPkH1ghRBCiMzDMAxb4CzQGPADDgBdlFInY7QZC7grpd43DCM3cAbIp5QKs0bMQgghxBMyAiuEEEJkLtWA80qpi9EJ6SrgtefaKCCLYRgG4AY8ACLSNkwhhBAiLklghRBCiMylIHAtxnW/6Ntimgu8BNwAjgFDlVJRaROeEEIIkTA7aweQHLly5VJFixa1dhgihfwfBHD7yl1sbW0oUCIfji6OFu0/4GEgNy7cJnuerOQulMuifQshXiwHDx68p5TKbe040ogRz23PrydqChwBGgDFgS2GYexSSj2O1ZFh9AH6ALi6ulYtU6ZMyiK7dQyiogd683mBTYZ8myKEEMICEnptzpCvDEWLFsXX19faYQgLOH/4Eh++9in+VwMYtrAv9bu8gp6xljJ3rt2jX+WR1K+Smy/3TsbB0d4C0QohXlSGYVyxdgxpyA8oFOO6B3qkNaaewDSlC2WcNwzjElAG2B+zkVJqIbAQwNvbW6X4tXlaEQh5qL8f9Se45EhZf0IIITKshF6bZQqxsKoSlT2Zu38qxSoWYeobsxnXcio3L95OUZ+REZFMe2M2EWERjFs1TJJXIYSI7QBQ0jAMT8MwHIDOwNrn2lwFGgIYhpEXKA1cTNMohRBCiHhIAiusLke+7Hz21yT6f9aD47tO0bv8MNYt2EJyK2Qvn/Qzx3adYuj8PniUzG/haIUQImNTSkUAg4BNwCngJ6XUCcMw+hmG0S+62cdATcMwjgHbgPeVUvesE7EQQgjxTIacQixePLZ2trR7twV1Or7MZ+98zZf9F7Lr1310n9SZsi+XMquP8LBwfpqxlh8m/48mPerRsFvtVI5aCCEyJqXUemD9c7d9HeP7G0CTtI5LCCGEMEUSWJGu5CqYk49/H82aORv5YcqvDK05jmqvVubNDztS2qdEgutjT+47w+d9F3D5+DXqdKzBoDlvp3HkQoj0Ijw8HD8/P0JCQuLc5+TkhIeHB/b2srQgXbJADQQhROaT2P99kf4l9bVZEliR7tja6tHY5r0bsGbuRn6asYbBL48le153KtQtS4U65chTOBcPbj3kwc3/uHrajx2r9pKrYA4mrXmfGq28rf0QhBBW5OfnR5YsWShatGisD72UUty/fx8/Pz88PT2tGKEQQghLSuj/vkj/kvPaLAmsSLec3ZzpPLotrQY0ZefP+/j3rxP8u/0Ef/20L1a7LDnceG1QM3p+0gWXLM5WilYIkV6EhITE+ybGMAxy5szJ3bt3rRSZEEKI1JDQ/32R/iXntVkSWJHuuWZ1ofnbDWn+dkOUUty8eJtH9/zJmT8b2fJmkyrDQog4EnoTI29uhBDixST/3zOupP7spAqxyFAMw6BA8Xy8VL0keQrnluRVCCFeVMmsRC+EEGnt/v37VKpUiUqVKpEvXz4KFiz49HpYWFiix/r6+jJkyBCT56hZs6alws3wZARWCCGEEOmEjKAIITKenDlzcuTIEQAmTpyIm5sbI0aMeHp/REQEdnbxp13e3t54e5uu37J3717LBGuGyMhIbG1tE7yekMQepyXJCKwQQogXTkL7SCd3f2khhBAiKXr06MHw4cOpX78+77//Pvv376dmzZpUrlyZmjVrcubMGQB27NhBy5YtAZ389urVi3r16lGsWDFmz579tD83N7en7evVq0eHDh0oU6YM3bp1e/ratn79esqUKcMrr7zCkCFDnvYbU2RkJCNHjsTHx4cKFSqwYMGCp/3Wr1+frl274uXlFed6SEgIPXv2xMvLi8qVK7N9+3YAli1bRseOHWnVqhVNmqTN7msyAiuEJZ08CT/+CPfuQaFC0LUrFC5s7aiEyFScnJy4f/8+OXPmjLcKsZOTkxWjE0IIkZqKjv4j1fq+PK1FktqfPXuWrVu3Ymtry+PHj9m5cyd2dnZs3bqVsWPH8uuvv8Y55vTp02zfvh1/f39Kly5N//7942wvc/jwYU6cOEGBAgWoVasWe/bswdvbm759+7Jz5048PT3p0qVLvDEtXrwYd3d3Dhw4QGhoKLVq1XqaeO7fv5/jx4/j6enJjh07Yl2fNWsWAMeOHeP06dM0adKEs2fPArBv3z6OHj1Kjhw5kvT8JJcksEJYwr170K4d+PpCeDhERICDA0ycCK1awXffgbNUSBYiLXh4eODn5xdvRcMne80JIYQQqa1jx45Pp94+evSI7t27c+7cOQzDIDw8PN5jWrRogaOjI46OjuTJk4fbt2/Hed2qVq3a09sqVarE5cuXcXNzo1ixYk+3ounSpQsLFy6M0//mzZs5evQov/zyy9O4zp07h4ODA9WqVYu1lU3M67t372bw4MEAlClThiJFijxNYBs3bpxmyStIAitEygUEQI0acOWKTl6feLJof906aN4ctm0DM9YPCCFSxt7eXvZ5fSHIdG8hRMbm6ur69PsPPviA+vXr89tvv3H58mXq1asX7zGOjo5Pv7e1tSUiIsKsNuYukVFKMWfOHJo2bRrr9h07dsSK9/n4E+v/+eNSmySwQqTU/Plw/Xrs5DWmkBA4eFAnsq+9lraxCSFERiLbYAghUiip03zTyqNHjyhYsCCg141aWpkyZbh48SKXL1+maNGi/Pjjj/G2a9q0KfPnz6dBgwbY29tz9uzZp3Elpk6dOqxYsYIGDRpw9uxZrl69SunSpTl06JClH4pJUsRJiJRQCj77DIKDE28XEADTp6dNTOmJUnpd8J49cOGCtaMRQgghhLCKUaNGMWbMGGrVqkVkZKTF+3d2dmbevHk0a9aMV155hbx58+Lu7h6nXe/evSlbtixVqlShfPny9O3bN95R3ucNGDCAyMhIvLy86NSpE8uWLYs1EpyWjIxYkdHb21v5+vpaOwwhIDAQsmXTa15NyZoVHj1K/ZjSA6Vg6VKYNEmvD7az01OqixeHyZOhdWtrRyhELIZhHFRKmd7HQCTIIq/N04tB0H39/cgL4Jor5YEJIV54p06d4qWXXrJ2GFYXEBCAm5sbSikGDhxIyZIlGTZsmLXDMkt8P8OEXptlBFaIlLCx0cmauW0zA6VgwAAYMkSvCw4M1Il7cDAcPw5duuhRayGESEwG/IBdCCGsadGiRVSqVIly5crx6NEj+vbta+2QUoWsgRUiJZydwdMTzp9PvJ1hQM2aaROTta1bB8uX68Q1PkFBMH48NGoEFSqkbWxCiHRO1sAKIURyDRs2LMOMuKZEJhkSEiIVjRoFLi6Jt3FxgZEj0yYea5s6NeHk9YmwMPj887SJRwghhBBCvDAkgRUipXr0gKpVE97n1cUF2reHunXTNCyrCA+Hf/4x3S4yEtauTf14hBBCCCHEC0USWCFSyt4eNm+GN98EJydwc9Nfs2TR348YoQsaZYbtIUJDzV/r+2SfXCGEEEIIIcwka2CFsAQnJ1iwQG+Vs349PHwIefNC8+YJj8y+iFxd9Yjz48em2xYqlPrxCCEyMCniJIQQIi4ZgRXCktzddZXd/v2hXbvMlbyCHmXu0wccHBJv5+oKw4enTUxCiIwjM8xUEUK8cOrVq8emTZti3fbFF18wYMCARI95svXYq6++ysOHD+O0mThxIjNnzkz03KtXr+b
|
|||
|
|
"text/plain": [
|
|||
|
|
"<Figure size 1152x576 with 2 Axes>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"metadata": {
|
|||
|
|
"needs_background": "light"
|
|||
|
|
},
|
|||
|
|
"output_type": "display_data"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"draw_regularization_example(X, Y)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "slide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### Regularyzacja\n",
|
|||
|
|
"\n",
|
|||
|
|
"Regularyzacja jest metodą zapobiegania zjawisku nadmiernego dopasowania (*overfitting*) poprzez odpowiednie zmodyfikowanie funkcji kosztu.\n",
|
|||
|
|
"\n",
|
|||
|
|
"Do funkcji kosztu dodawane jest specjalne wyrażenie (**wyrazenie regularyzacyjne** – zaznaczone na czerwono w poniższych wzorach), będące „karą” za ekstremalne wartości parametrów $\\theta$.\n",
|
|||
|
|
"\n",
|
|||
|
|
"W ten sposób preferowane są wektory $\\theta$ z mniejszymi wartosciami parametrów – mają automatycznie niższy koszt.\n",
|
|||
|
|
"\n",
|
|||
|
|
"Jak silną regularyzację chcemy zastosować? Możemy o tym zadecydować, dobierajac odpowiednio **parametr regularyzacji** $\\lambda$."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### Regularyzacja dla regresji liniowej – funkcja kosztu\n",
|
|||
|
|
"\n",
|
|||
|
|
"$$\n",
|
|||
|
|
"J(\\theta) \\, = \\, \\dfrac{1}{2m} \\left( \\displaystyle\\sum_{i=1}^{m} h_\\theta(x^{(i)}) - y^{(i)} \\color{red}{ + \\lambda \\displaystyle\\sum_{j=1}^{n} \\theta^2_j } \\right)\n",
|
|||
|
|
"$$"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "fragment"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"* $\\lambda$ – parametr regularyzacji\n",
|
|||
|
|
"* jeżeli $\\lambda$ jest zbyt mały, skutkuje to nadmiernym dopasowaniem\n",
|
|||
|
|
"* jeżeli $\\lambda$ jest zbyt duży, skutkuje to niedostatecznym dopasowaniem"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### Regularyzacja dla regresji liniowej – gradient\n",
|
|||
|
|
"\n",
|
|||
|
|
"$$\\small\n",
|
|||
|
|
"\\begin{array}{llll}\n",
|
|||
|
|
"\\dfrac{\\partial J(\\theta)}{\\partial \\theta_0} &=& \\dfrac{1}{m}\\displaystyle\\sum_{i=1}^m \\left( h_{\\theta}(x^{(i)})-y^{(i)} \\right) x^{(i)}_0 & \\textrm{dla $j = 0$ }\\\\\n",
|
|||
|
|
"\\dfrac{\\partial J(\\theta)}{\\partial \\theta_j} &=& \\dfrac{1}{m}\\displaystyle\\sum_{i=1}^m \\left( h_{\\theta}(x^{(i)})-y^{(i)} \\right) x^{(i)}_j \\color{red}{+ \\dfrac{\\lambda}{m}\\theta_j} & \\textrm{dla $j = 1, 2, \\ldots, n $} \\\\\n",
|
|||
|
|
"\\end{array} \n",
|
|||
|
|
"$$"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### Regularyzacja dla regresji logistycznej – funkcja kosztu\n",
|
|||
|
|
"\n",
|
|||
|
|
"$$\n",
|
|||
|
|
"\\begin{array}{rtl}\n",
|
|||
|
|
"J(\\theta) & = & -\\dfrac{1}{m} \\left( \\displaystyle\\sum_{i=1}^{m} y^{(i)} \\log h_\\theta(x^{(i)}) + \\left( 1-y^{(i)} \\right) \\log \\left( 1-h_\\theta(x^{(i)}) \\right) \\right) \\\\\n",
|
|||
|
|
"& & \\color{red}{ + \\dfrac{\\lambda}{2m} \\displaystyle\\sum_{j=1}^{n} \\theta^2_j } \\\\\n",
|
|||
|
|
"\\end{array}\n",
|
|||
|
|
"$$"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### Regularyzacja dla regresji logistycznej – gradient\n",
|
|||
|
|
"\n",
|
|||
|
|
"$$\\small\n",
|
|||
|
|
"\\begin{array}{llll}\n",
|
|||
|
|
"\\dfrac{\\partial J(\\theta)}{\\partial \\theta_0} &=& \\dfrac{1}{m}\\displaystyle\\sum_{i=1}^m \\left( h_{\\theta}(x^{(i)})-y^{(i)} \\right) x^{(i)}_0 & \\textrm{dla $j = 0$ }\\\\\n",
|
|||
|
|
"\\dfrac{\\partial J(\\theta)}{\\partial \\theta_j} &=& \\dfrac{1}{m}\\displaystyle\\sum_{i=1}^m \\left( h_{\\theta}(x^{(i)})-y^{(i)} \\right) x^{(i)}_j \\color{red}{+ \\dfrac{\\lambda}{m}\\theta_j} & \\textrm{dla $j = 1, 2, \\ldots, n $} \\\\\n",
|
|||
|
|
"\\end{array} \n",
|
|||
|
|
"$$"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "slide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### Implementacja metody regularyzacji"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 60,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [],
|
|||
|
|
"source": [
|
|||
|
|
"def J_(h,theta,X,y,lamb=0):\n",
|
|||
|
|
" \"\"\"Funkcja kosztu z regularyzacją\"\"\"\n",
|
|||
|
|
" m = float(len(y))\n",
|
|||
|
|
" f = h(theta, X, eps=10**-7)\n",
|
|||
|
|
" j = 1.0/m \\\n",
|
|||
|
|
" * -np.sum(np.multiply(y, np.log(f)) + \n",
|
|||
|
|
" np.multiply(1 - y, np.log(1 - f)), axis=0) \\\n",
|
|||
|
|
" + lamb/(2*m) * np.sum(np.power(theta[1:] ,2))\n",
|
|||
|
|
" return j\n",
|
|||
|
|
"\n",
|
|||
|
|
"def dJ_(h,theta,X,y,lamb=0):\n",
|
|||
|
|
" \"\"\"Gradient funkcji kosztu z regularyzacją\"\"\"\n",
|
|||
|
|
" m = float(y.shape[0])\n",
|
|||
|
|
" g = 1.0/y.shape[0]*(X.T*(h(theta,X)-y))\n",
|
|||
|
|
" g[1:] += lamb/m * theta[1:]\n",
|
|||
|
|
" return g"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 65,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [],
|
|||
|
|
"source": [
|
|||
|
|
"slider_lambda = widgets.FloatSlider(min=0.0, max=0.5, step=0.005, value=0.01, description=r'$\\lambda$', width=300)\n",
|
|||
|
|
"\n",
|
|||
|
|
"def slide_regularization_example_2(lamb):\n",
|
|||
|
|
" draw_regularization_example(X, Y, lamb=lamb)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 66,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"application/vnd.jupyter.widget-view+json": {
|
|||
|
|
"model_id": "9a01a44941e544cb9df51b38c035da62",
|
|||
|
|
"version_major": 2,
|
|||
|
|
"version_minor": 0
|
|||
|
|
},
|
|||
|
|
"text/plain": [
|
|||
|
|
"interactive(children=(FloatSlider(value=0.01, description='$\\\\lambda$', max=0.5, step=0.005), Button(descripti…"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "display_data"
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"text/plain": [
|
|||
|
|
"<function __main__.slide_regularization_example_2(lamb)>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"execution_count": 66,
|
|||
|
|
"metadata": {},
|
|||
|
|
"output_type": "execute_result"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"widgets.interact_manual(slide_regularization_example_2, lamb=slider_lambda)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 67,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [],
|
|||
|
|
"source": [
|
|||
|
|
"def cost_lambda_fun(lamb):\n",
|
|||
|
|
" \"\"\"Koszt w zależności od parametru regularyzacji lambda\"\"\"\n",
|
|||
|
|
" theta = np.matrix(np.zeros(X.shape[1])).reshape(X.shape[1],1)\n",
|
|||
|
|
" thetaBest, err = SGD(h, J, dJ, theta, X, Y, alpha=1, adaGrad=True, maxEpochs=2500, batchSize=100, \n",
|
|||
|
|
" logError=True, validate=0.25, valStep=1, lamb=lamb)\n",
|
|||
|
|
" return err[1][-1], err[3][-1]\n",
|
|||
|
|
"\n",
|
|||
|
|
"def plot_cost_lambda():\n",
|
|||
|
|
" \"\"\"Wykres kosztu w zależności od parametru regularyzacji lambda\"\"\"\n",
|
|||
|
|
" plt.figure(figsize=(16,8))\n",
|
|||
|
|
" ax = plt.subplot(111)\n",
|
|||
|
|
" Lambda = np.arange(0.0, 1.0, 0.01)\n",
|
|||
|
|
" Costs = [cost_lambda_fun(lamb) for lamb in Lambda]\n",
|
|||
|
|
" CostTrain = [cost[0] for cost in Costs]\n",
|
|||
|
|
" CostCV = [cost[1] for cost in Costs]\n",
|
|||
|
|
" plt.plot(Lambda, CostTrain, lw=3, label='training error')\n",
|
|||
|
|
" plt.plot(Lambda, CostCV, lw=3, label='validation error')\n",
|
|||
|
|
" ax.set_xlabel(r'$\\lambda$')\n",
|
|||
|
|
" ax.set_ylabel(u'cost')\n",
|
|||
|
|
" plt.legend()\n",
|
|||
|
|
" plt.ylim(0.2,0.8)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 68,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA7AAAAHmCAYAAABK9WIBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzde3Rc5WHv/d+jkTQajUb3i23ZxgYMNjY2GGM7gRLMLSTnhFxIiMmttCfQ0CYk7XsotF1NKKc9TdM0K6EtZUFKkpVD4VBybQ4hlDSEkJDEdgAHzMXG+CLLtu7SaKTRzGie94+95ybLsizNntFI389as2bv0Z6ZR5A4/uZ59t7GWisAAAAAAOa6smIPAAAAAACA6SBgAQAAAAAlgYAFAAAAAJQEAhYAAAAAUBIIWAAAAABASSBgAQAAAAAlwdOANcZca4x5zRizzxhz5yQ/rzPG/Icx5kVjzMvGmN/zcjwAAAAAgNJlvLoPrDHGJ+l1SVdL6pC0Q9KN1to9Wcf8uaQ6a+0dxpgWSa9JWmStjXkyKAAAAABAyfJyBnazpH3W2v1ukD4i6d0TjrGSQsYYI6lGUp+khIdjAgAAAACUKC8Dtl3S4az9Dve1bP8kaY2kTkm/lfRpa23SwzEBAAAAAEpUuYefbSZ5beJ65bdLekHSFZLOkvSfxpifWWuHcj7ImFsk3SJJwWDwotWrV3swXAAAAABAse3atavHWtsy2c+8DNgOScuy9pfKmWnN9nuSPm+dE3H3GWPelLRa0q+zD7LW3i/pfknatGmT3blzp2eDBgAAAAAUjzHm4Ml+5uUS4h2SVhljVhpjKiVtl/T9CcccknSlJBlj2iSdK2m/h2MCAAAAAJQoz2ZgrbUJY8wnJf1Ikk/Sg9bal40xn3B/fp+k/yXp68aY38pZcnyHtbbHqzEBAAAAAEqXl0uIZa19XNLjE167L2u7U9I1Xo4BAAAAADA/eBqwAAAAAOCVeDyujo4ORaPRYg8FM1BVVaWlS5eqoqJi2u8hYAEAAACUpI6ODoVCIa1YsULGTHYTFMxV1lr19vaqo6NDK1eunPb7vLyIEwAAAAB4JhqNqqmpiXgtQcYYNTU1nfbsOQELAAAAoGQRr6VrJv/uCFgAAAAAmIGBgQHde++9M3rvO9/5Tg0MDEx5zGc/+1k99dRTM/r8+YqABQAAAIAZmCpgx8fHp3zv448/rvr6+imPufvuu3XVVVfNeHyna+KYT/U7nO5x+UDAAgAAAMAM3HnnnXrjjTd0wQUX6Pbbb9fTTz+tbdu26UMf+pDOP/98SdJ73vMeXXTRRVq7dq3uv//+9HtXrFihnp4eHThwQGvWrNHNN9+stWvX6pprrtHo6Kgk6aabbtJjjz2WPv5zn/ucNm7cqPPPP1+vvvqqJKm7u1tXX321Nm7cqD/4gz/QGWecoZ6enhPG+uSTT+otb3mLNm7cqA984AMaHh5Of+7dd9+tSy+9VP/+7/9+wv7DDz+s888/X+vWrdMdd9yR/ryamhp99rOf1ZYtW/Tcc8958w94ElyFGAAAAEDJW3Hn//Pssw98/r9N+vrnP/95vfTSS3rhhRckSU8//bR+/etf66WXXkpfWffBBx9UY2OjRkdHdfHFF+v6669XU1NTzufs3btXDz/8sB544AHdcMMN+ta3vqWPfOQjJ3xfc3OzfvOb3+jee+/VF7/4RX31q1/VX/3VX+mKK67Qn/3Zn+mJJ57IieSUnp4e/fVf/7WeeuopBYNB/d3f/Z2+9KUv6bOf/awk53Y2zz77rCQnylP7nZ2d2rp1q3bt2qWGhgZdc801+u53v6v3vOc9ikQiWrdune6+++6Z/4OdAWZgAQAAACBPNm/enHNbmHvuuUcbNmzQ1q1bdfjwYe3du/eE96xcuVIXXHCBJOmiiy7SgQMHJv3s973vfScc8+yzz2r79u2SpGuvvVYNDQ0nvO+Xv/yl9uzZo0suuUQXXHCBvvGNb+jgwYPpn3/wgx/MOT61v2PHDl1++eVqaWlReXm5PvzhD+uZZ56RJPl8Pl1//fXT+UeSV8zAAgAAAECeBIPB9PbTTz+tp556Ss8995yqq6t1+eWXT3rbGL/fn972+XzpJcQnO87n8ymRSEhy7qd6KtZaXX311Xr44YdPOebs/ak+u6qqSj6f75TfnW8ELAAAAICSd7Jlvl4KhUIKh8Mn/fng4KAaGhpUXV2tV199Vb/85S/zPoZLL71Ujz76qO644w49+eST6u/vP+GYrVu36o/+6I+0b98+nX322RoZGVFHR4fOOeecKT97y5Yt+vSnP62enh41NDTo4Ycf1qc+9am8/w6ngyXEAAAAADADTU1NuuSSS7Ru3TrdfvvtJ/z82muvVSKR0Pr16/WXf/mX2rp1a97H8LnPfU5PPvmkNm7cqB/+8IdavHixQqFQzjEtLS36+te/rhtvvFHr16/X1q1b0xeBmsrixYv1t3/7t9q2bZs2bNigjRs36t3vfnfef4fTYaYz5TyXbNq0ye7cubPYwwAAAABQZK+88orWrFlT7GEU1djYmHw+n8rLy/Xcc8/p1ltvTV9UqhRM9u/QGLPLWrtpsuNZQgwAAAAAJerQoUO64YYblEwmVVlZqQceeKDYQ/IUAQsAAAAAJWrVqlV6/vnniz2MguEcWAAAAABASSBgAQAAAAAlgYAFAAAAAJQEAhYAAAAAUBIIWAAAAAAokJqaGklSZ2en3v/+9096zOWXX65T3Tr0y1/+skZGRtL773znOzUwMJC/gc5RBCwAAAAAFNiSJUv02GOPzfj9EwP28ccfV319fT6GdkqJRGLK/em+byYIWAAAAACYgTvuuEP33ntvev+uu+7SP/zDP2h4eFhXXnmlNm7cqPPPP1/f+973TnjvgQMHtG7dOknS6Oiotm/frvXr1+uDH/ygRkdH08fdeuut2rRpk9auXavPfe5zkqR77rlHnZ2d2rZtm7Zt2yZJWrFihXp6eiRJX/rSl7Ru3TqtW7dOX/7yl9Pft2bNGt18881au3atrrnmmpzvSenu7tb111+viy++WBdffLF+/vOfp3+3W265Rddcc40+9rGPnbB/8OBBXXnllVq/fr2uvPJKHTp0SJJ000036U/+5E+0bds23XHHHbP+Z859YAEAAACUvrvqPPzswUlf3r59uz7zmc/oD//wDyVJjz76qJ544glVVVXpO9/5jmpra9XT06OtW7fquuuukzFm0s/5l3/5F1VXV2v37t3avXu3Nm7cmP7Z3/zN36ixsVHj4+O68sortXv3bt1222360pe+pJ/85Cdqbm7O+axdu3bpa1/7mn71q1/JWqstW7bobW97mxoaGrR37149/PDDeuCBB3TDDTfoW9/6lj7ykY/kvP/Tn/60/viP/1iXXnqpDh06pLe//e165ZVX0p/97LPPKhAI6K677srZf9e73qWPfexj+t3f/V09+OCDuu222/Td735XkvT666/rqaeeks/nm9k//ywELAAAAADMwIUXXqiuri51dnaqu7tbDQ0NWr58ueLxuP78z/9czzzzjMrKynTkyBEdP35cixYtmvRznnnmGd12222SpPXr12v9+vXpnz366KO6//77lUgkdPToUe3Zsyfn5xM9++yzeu9736tgMChJet/73qef/exnuu6667Ry5UpdcMEFkqSLLrpIBw4cOOH9Tz31lPbs2ZPeHxoaUjgcliRdd911CgQC6Z9l7z/33HP69re/LUn66Ec/qj/90z9NH/eBD3wgL/EqEbAAAAAAMGPvf//79dhjj+nYsWPavn27JOmhhx5Sd3e3du3apYqKCq1YsULRaHTKz5lsdvbNN9/UF7/4Re3YsUMNDQ266aabTvk51tqT/szv96e3fT7fpEuIk8mknnvuuZxQTUlF8cn2s2X/PlMdd7oIWAAAAACl7yTLfL22fft23Xzzzerp6dFPf/pTSdLg4KBaW1tVUVGhn/zkJzp48OCUn3HZZZfpoYce0rZt2/TSSy9p9+7dkpzZz2AwqLq6Oh0/flw//OEPdfnll0uSQqGQwuHwCUuIL7v
|
|||
|
|
"text/plain": [
|
|||
|
|
"<Figure size 1152x576 with 1 Axes>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"metadata": {
|
|||
|
|
"needs_background": "light"
|
|||
|
|
},
|
|||
|
|
"output_type": "display_data"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"plot_cost_lambda()"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "slide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"## 3.6. Krzywa uczenia się"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"* Krzywa uczenia pozwala sprawdzić, czy uczenie przebiega poprawnie.\n",
|
|||
|
|
"* Krzywa uczenia to wykres zależności między wielkością zbioru treningowego a wartością funkcji kosztu.\n",
|
|||
|
|
"* Wraz ze wzrostem wielkości zbioru treningowego wartość funkcji kosztu na zbiorze treningowym rośnie.\n",
|
|||
|
|
"* Wraz ze wzrostem wielkości zbioru treningowego wartość funkcji kosztu na zbiorze walidacyjnym maleje."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 69,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [],
|
|||
|
|
"source": [
|
|||
|
|
"def cost_trainsetsize_fun(m):\n",
|
|||
|
|
" \"\"\"Koszt w zależności od wielkości zbioru uczącego\"\"\"\n",
|
|||
|
|
" theta = np.matrix(np.zeros(X.shape[1])).reshape(X.shape[1],1)\n",
|
|||
|
|
" thetaBest, err = SGD(h, J, dJ, theta, X, Y, alpha=1, adaGrad=True, maxEpochs=2500, batchSize=100, \n",
|
|||
|
|
" logError=True, validate=0.25, valStep=1, lamb=0.01, trainsetsize=m)\n",
|
|||
|
|
" return err[1][-1], err[3][-1]\n",
|
|||
|
|
"\n",
|
|||
|
|
"def plot_learning_curve():\n",
|
|||
|
|
" \"\"\"Wykres krzywej uczenia się\"\"\"\n",
|
|||
|
|
" plt.figure(figsize=(16,8))\n",
|
|||
|
|
" ax = plt.subplot(111)\n",
|
|||
|
|
" M = np.arange(0.3, 1.0, 0.05)\n",
|
|||
|
|
" Costs = [cost_trainsetsize_fun(m) for m in M]\n",
|
|||
|
|
" CostTrain = [cost[0] for cost in Costs]\n",
|
|||
|
|
" CostCV = [cost[1] for cost in Costs]\n",
|
|||
|
|
" plt.plot(M, CostTrain, lw=3, label='training error')\n",
|
|||
|
|
" plt.plot(M, CostCV, lw=3, label='validation error')\n",
|
|||
|
|
" ax.set_xlabel(u'trainset size')\n",
|
|||
|
|
" ax.set_ylabel(u'cost')\n",
|
|||
|
|
" plt.legend()"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### Krzywa uczenia a obciążenie i wariancja\n",
|
|||
|
|
"\n",
|
|||
|
|
"Wykreślenie krzywej uczenia pomaga diagnozować nadmierne i niedostateczne dopasowanie:\n",
|
|||
|
|
"\n",
|
|||
|
|
"<img width=\"100%\" src=\"learning-curves.png\"/>\n",
|
|||
|
|
"\n",
|
|||
|
|
"Źródło: http://www.ritchieng.com/machinelearning-learning-curve"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 70,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [
|
|||
|
|
{
|
|||
|
|
"data": {
|
|||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA7AAAAHgCAYAAACcrIEcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3yW9b3/8deVASHMMFSGyBAkgixxtFjFvQCtE6tt1To6bXvOsbY9v6PW03Pa/tp6Wn+2x6pVOyzWrbjFusUKOBCZIsOAIHsGyLh+f3wDdxYhxNy5ciev5+NxP3J97+t73/kkKOSd74riOEaSJEmSpOYuK+kCJEmSJEmqDwOsJEmSJCkjGGAlSZIkSRnBACtJkiRJyggGWEmSJElSRjDASpIkSZIyQk7SBeyr7t27x/369Uu6DEmSJElSGsycOXNNHMc9aruXcQG2X79+zJgxI+kyJEmSJElpEEXR0j3dcwqxJEmSJCkjGGAlSZIkSRnBACtJkiRJyggZtwZWkiRJkgBKSkooKipi+/btSZeiBsjLy6NPnz7k5ubW+zUGWEmSJEkZqaioiI4dO9KvXz+iKEq6HO2DOI5Zu3YtRUVF9O/fv96vcwqxJEmSpIy0fft2unXrZnjNQFEU0a1bt30ePTfASpIkScpYhtfM1ZA/OwOsJEmSJDXAhg0b+P3vf9+g155xxhls2LChzj7XX389U6dObdD7t1QGWEmSJElqgLoCbFlZWZ2vfeqpp+jSpUudfW666SZOOumkBte3r6rXvLevYV/7NQYDrCRJkiQ1wA9/+EMWLVrEyJEjufbaa3nppZc4/vjj+dKXvsRhhx0GwNlnn83hhx/O0KFDuf3223e/tl+/fqxZs4YlS5ZQWFjIlVdeydChQznllFMoLi4G4NJLL+XBBx/c3f+GG25g9OjRHHbYYcybNw+A1atXc/LJJzN69GiuvvpqDjroINasWVOj1ueee47Pfe5zjB49mvPPP58tW7bsft+bbrqJY445hgceeKBGe/LkyRx22GEMGzaM6667bvf7dejQgeuvv56jjjqKadOmpecbXAt3IZYkSZKU8fr98Mm0vfeSn59Z6/M///nPmT17Nu+++y4AL730Em+99RazZ8/evbPuXXfdRdeuXSkuLuaII47g3HPPpVu3blXeZ+HChUyePJk77riDCy64gIceeohLLrmkxufr3r07b7/9Nr///e/51a9+xZ133slPfvITTjjhBH70ox/xzDPPVAnJu6xZs4af/vSnTJ06lfbt2/OLX/yCm2++meuvvx4Ix9m89tprQAjlu9orVqzg6KOPZubMmRQUFHDKKafw6KOPcvbZZ7N161aGDRvGTTfd1PBvbAM4AitJkiRJjeTII4+scizMLbfcwogRIzj66KP5+OOPWbhwYY3X9O/fn5EjRwJw+OGHs2TJklrf+5xzzqnR57XXXmPSpEkAnHbaaRQUFNR43ZtvvsmcOXMYO3YsI0eO5E9/+hNLly7dff/CCy+s0n9Xe/r06YwbN44ePXqQk5PDxRdfzCuvvAJAdnY25557bn2+JY3KEVhJkiRJaiTt27ffff3SSy8xdepUpk2bRn5+PuPGjav12Ji2bdvuvs7Ozt49hXhP/bKzsyktLQXCeap7E8cxJ598MpMnT95rzZXbdb13Xl4e2dnZe/3cjc0AK0mSJCnj7Wmabzp17NiRzZs37/H+xo0bKSgoID8/n3nz5vHmm282eg3HHHMM999/P9dddx3PPfcc69evr9Hn6KOP5lvf+hYffvghBx98MNu2baOoqIjBgwfX+d5HHXUU3/3ud1mzZg0FBQVMnjyZ73znO43+NewLpxBLkiRJUgN069aNsWPHMmzYMK699toa90877TRKS0sZPnw4//Ef/8HRRx/d6DXccMMNPPfcc4wePZqnn36anj170rFjxyp9evTowT333MNFF13E8OHDOfroo3dvAlWXnj178rOf/Yzjjz+eESNGMHr0aM4666xG/xr2RVSfIefmZMyYMfGMGTOSLkOSJElSwubOnUthYWHSZSRqx44dZGdnk5OTw7Rp0/jGN76xe1OpTFDbn2EURTPjOB5TW3+nEEtSfW34GGbeA8vehDGXwWHnJV2RJElq5ZYtW8YFF1xAeXk5bdq04Y477ki6pLQywEpSXcrLYfFL8NadsOBpiMvD88tnwKCTIa9zouVJkqTWbdCgQbzzzjtJl9FkDLCSVJvi9fDuZJh+J6xbVPN+6XZY8BwMP7/pa5MkSWqlDLCSVNkn74XQOusBKK1lC/tOfWBTUbie+7gBVpIkqQkZYCWpdAd88GgIrkVv1bzfthOM/BKM+RoQw++ODM9/OBV2boM2+U1ariRJUmtlgJXUem1YBjPuhrf/DNvW1Ly//zA44goYfgG0qXTAd/fBsGYBlGyDRf+AwvFNV7MkSVIr5jmwklqX8vIwcjr5IvjtCHjt5qrhNSsXDjsfLn8Wvv5a2G24cngFKJyYup77eNPULUmSWoQOHToAsGLFCs47r/YTDcaNG8fejg79zW9+w7Zt23a3zzjjDDZs2NB4hTZTjsBKah2K18M798KMP8K6j2re79QHxlwKo78KHfar+70KJ8CrvwrX85+B0p2Q06bRS5YkSS1Xr169ePDBBxv8+t/85jdccskl5OeHpUxPPfVUY5W2V6WlpeTk5OyxXd/XNYQBVlLLtuLdsLb1/Qdr35RpwPFhmvDg0yC7nn8l9hwBnfvCxmWwYyMseQUOPqlx65YkSc3eddddx0EHHcQ3v/lNAG688UY6duzI1VdfzVlnncX69espKSnhpz/9KWeddVaV1y5ZsoTx48cze/ZsiouLueyyy5gzZw6FhYUUF6d+ZvnGN77B9OnTKS4u5rzzzuMnP/kJt9xyCytWrOD444+ne/fuvPjii/Tr148ZM2bQvXt3br75Zu666y4ArrjiCr73ve+xZMkSTj/9dI455hjeeOMNevfuzWOPPUa7du2q1LV69Wq+/vWvs2zZMiAE5bFjx3LjjTeyYsUKlixZQvfu3Rk8eHCV9s9+9jMuv/xyVq9eTY8ePbj77rvp27cvl156KV27duWdd95h9OjR/PrXv/5M33MDrKSWp2Q7zNm1KdP0mvfbdoZRF8OYy6H7oH1//ygKo7Bv/i60504xwEqSlLQb03g2+40ba3160qRJfO9739sdYO+//36eeeYZ8vLyeOSRR+jUqRNr1qzh6KOPZuLEiURRVOv7/O///i/5+fnMmjWLWbNmMXr06N33/uu//ouuXbtSVlbGiSeeyKxZs7jmmmu4+eabefHFF+nevXuV95o5cyZ33303//znP4njmKOOOorjjjuOgoICFi5cyOTJk7njjju44IILeOihh7jkkkuqvP673/0u3//+9znmmGNYtmwZp556KnPnzt393q+99hrt2rXjxhtvrNKeMGECX/nKV/jqV7/KXXfdxTXXXMOjjz4KwIIFC5g6dSrZ2dkN+/5XYoCV1HKsXwoz7oJ3/gLb1ta8f8BhcMSVcNh5Nde17qvKAXbek3DmzZD12f9SliRJmWPUqFF8+umnrFixgtWrV1NQUEDfvn0pKSnhxz/+Ma+88gpZWVksX76cVatWccABB9T6Pq+88grXXHMNAMOHD2f48OG7791///3cfvvtlJaW8sknnzBnzpwq96t77bXX+OIXv0j79uFnnXPOOYdXX32ViRMn0r9/f0aOHAnA4YcfzpIlS2q8furUqcyZM2d3e9OmTWzevBmAiRMnVhmxrdyeNm0aDz/8MABf/vKX+cEPfrC73/nnn98o4RXSGGCjKLoLGA98GsfxsDr6HQG8CVwYx3HDJ4FLap3Ky8NOwNPvgAXPAnHV+9lt4NCz4cgroc8RYfS0MRx4JLTfD7Z+CltXw8f/hIM+3zjvLUmSMsZ5553Hgw8+yMqVK5k0aRIA9957L6tXr2bmzJnk5ubSr18/tm/fXuf71DY6u3jxYn71q18xffp0CgoKuPTSS/f6PnEc7/Fe27Ztd19nZ2dXmaq8S3l5OdOmTasxtRjYHYr31K6
|
|||
|
|
"text/plain": [
|
|||
|
|
"<Figure size 1152x576 with 1 Axes>"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
"metadata": {
|
|||
|
|
"needs_background": "light"
|
|||
|
|
},
|
|||
|
|
"output_type": "display_data"
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"source": [
|
|||
|
|
"plot_learning_curve()"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "slide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"## 3.7. Warianty metody gradientu prostego\n",
|
|||
|
|
"\n",
|
|||
|
|
"* Batch gradient descent\n",
|
|||
|
|
"* Stochastic gradient descent\n",
|
|||
|
|
"* Mini-batch gradient descent"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### _Batch gradient descent_\n",
|
|||
|
|
"\n",
|
|||
|
|
"* Klasyczna wersja metody gradientu prostego\n",
|
|||
|
|
"* Obliczamy gradient funkcji kosztu względem całego zbioru treningowego:\n",
|
|||
|
|
" $$ \\theta := \\theta - \\alpha \\cdot \\nabla_\\theta J(\\theta) $$\n",
|
|||
|
|
"* Dlatego może działać bardzo powoli\n",
|
|||
|
|
"* Nie można dodawać nowych przykładów na bieżąco w trakcie trenowania modelu (*online learning*)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### *Stochastic gradient descent* (SGD)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "fragment"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"#### Algorytm\n",
|
|||
|
|
"\n",
|
|||
|
|
"Powtórz określoną liczbę razy (liczba epok):\n",
|
|||
|
|
" 1. Randomizuj dane treningowe\n",
|
|||
|
|
" 1. Powtórz dla każdego przykładu $i = 1, 2, \\ldots, m$:\n",
|
|||
|
|
" $$ \\theta := \\theta - \\alpha \\cdot \\nabla_\\theta \\, J \\! \\left( \\theta, x^{(i)}, y^{(i)} \\right) $$"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "notes"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"**Randomizacja danych** to losowe potasowanie przykładów uczących (wraz z odpowiedziami)."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"#### SGD - zalety\n",
|
|||
|
|
"\n",
|
|||
|
|
"* Dużo szybszy niż _batch gradient descent_\n",
|
|||
|
|
"* Można dodawać nowe przykłady na bieżąco w trakcie trenowania (*online learning*)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"#### SGD\n",
|
|||
|
|
"\n",
|
|||
|
|
"* Częsta aktualizacja parametrów z dużą wariancją:\n",
|
|||
|
|
"\n",
|
|||
|
|
"<img src=\"http://ruder.io/content/images/2016/09/sgd_fluctuation.png\" style=\"margin: auto;\" width=\"50%\" />\n",
|
|||
|
|
"\n",
|
|||
|
|
"* Z jednej strony dzięki temu nie utyka w złych minimach lokalnych, ale z drugiej strony może „wyskoczyć” z dobrego minimum"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### _Mini-batch gradient descent_"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "fragment"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"#### Algorytm\n",
|
|||
|
|
"\n",
|
|||
|
|
"1. Ustal rozmiar \"paczki/wsadu\" (*batch*) $b \\leq m$.\n",
|
|||
|
|
"2. Powtórz określoną liczbę razy (liczba epok):\n",
|
|||
|
|
" 1. Powtórz dla każdego batcha (czyli dla $i = 1, 1 + b, 1 + 2 b, \\ldots$):\n",
|
|||
|
|
" $$ \\theta := \\theta - \\alpha \\cdot \\nabla_\\theta \\, J \\left( \\theta, x^{(i : i+b)}, y^{(i : i+b)} \\right) $$"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"#### _Mini-batch gradient descent_\n",
|
|||
|
|
"\n",
|
|||
|
|
"* Kompromis między _batch gradient descent_ i SGD\n",
|
|||
|
|
"* Stabilniejsza zbieżność dzięki redukcji wariancji aktualizacji parametrów\n",
|
|||
|
|
"* Szybszy niż klasyczny _batch gradient descent_\n",
|
|||
|
|
"* Typowa wielkość batcha: między kilka a kilkaset przykładów\n",
|
|||
|
|
" * Im większy batch, tym bliżej do BGD; im mniejszy batch, tym bliżej do SGD\n",
|
|||
|
|
" * BGD i SGD można traktować jako odmiany MBGD dla $b = m$ i $b = 1$"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "code",
|
|||
|
|
"execution_count": 71,
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "skip"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"outputs": [],
|
|||
|
|
"source": [
|
|||
|
|
"# Mini-batch gradient descent - przykładowa implementacja\n",
|
|||
|
|
"\n",
|
|||
|
|
"def MiniBatchSGD(h, fJ, fdJ, theta, X, y, \n",
|
|||
|
|
" alpha=0.001, maxEpochs=1.0, batchSize=100, \n",
|
|||
|
|
" logError=True):\n",
|
|||
|
|
" errorsX, errorsY = [], []\n",
|
|||
|
|
" \n",
|
|||
|
|
" m, n = X.shape\n",
|
|||
|
|
" start, end = 0, batchSize\n",
|
|||
|
|
" \n",
|
|||
|
|
" maxSteps = (m * float(maxEpochs)) / batchSize\n",
|
|||
|
|
" for i in range(int(maxSteps)):\n",
|
|||
|
|
" XBatch, yBatch = X[start:end,:], y[start:end,:]\n",
|
|||
|
|
"\n",
|
|||
|
|
" theta = theta - alpha * fdJ(h, theta, XBatch, yBatch)\n",
|
|||
|
|
" \n",
|
|||
|
|
" if logError:\n",
|
|||
|
|
" errorsX.append(float(i*batchSize)/m)\n",
|
|||
|
|
" errorsY.append(fJ(h, theta, XBatch, yBatch).item())\n",
|
|||
|
|
" \n",
|
|||
|
|
" if start + batchSize < m:\n",
|
|||
|
|
" start += batchSize\n",
|
|||
|
|
" else:\n",
|
|||
|
|
" start = 0\n",
|
|||
|
|
" end = min(start + batchSize, m)\n",
|
|||
|
|
" \n",
|
|||
|
|
" return theta, (errorsX, errorsY)"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"Porównanie uśrednionych krzywych uczenia na przykładzie klasyfikacji dwuklasowej zbioru [MNIST](https://en.wikipedia.org/wiki/MNIST_database):\n",
|
|||
|
|
"\n",
|
|||
|
|
"<img src=\"sgd-comparison.png\" width=\"70%\" />"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### Wady klasycznej metody gradientu prostego, czyli dlaczego potrzebujemy optymalizacji\n",
|
|||
|
|
"\n",
|
|||
|
|
"* Trudno dobrać właściwą szybkość uczenia (*learning rate*)\n",
|
|||
|
|
"* Jedna ustalona wartość stałej uczenia się dla wszystkich parametrów\n",
|
|||
|
|
"* Funkcja kosztu dla sieci neuronowych nie jest wypukła, więc uczenie może utknąć w złym minimum lokalnym lub punkcie siodłowym"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "slide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"## 3.8. Algorytmy optymalizacji metody gradientu\n",
|
|||
|
|
"\n",
|
|||
|
|
"* Momentum\n",
|
|||
|
|
"* Nesterov Accelerated Gradient\n",
|
|||
|
|
"* Adagrad\n",
|
|||
|
|
"* Adadelta\n",
|
|||
|
|
"* RMSprop\n",
|
|||
|
|
"* Adam\n",
|
|||
|
|
"* Nadam\n",
|
|||
|
|
"* AMSGrad"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### Momentum\n",
|
|||
|
|
"\n",
|
|||
|
|
"* SGD źle radzi sobie w „wąwozach” funkcji kosztu\n",
|
|||
|
|
"* Momentum rozwiązuje ten problem przez dodanie współczynnika $\\gamma$, który można trakować jako „pęd” spadającej piłki:\n",
|
|||
|
|
" $$ v_t := \\gamma \\, v_{t-1} + \\alpha \\, \\nabla_\\theta J(\\theta) $$\n",
|
|||
|
|
" $$ \\theta := \\theta - v_t $$"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### Przyspiesony gradient Nesterova (*Nesterov Accelerated Gradient*, NAG)\n",
|
|||
|
|
"\n",
|
|||
|
|
"* Momentum czasami powoduje niekontrolowane rozpędzanie się piłki, przez co staje się „mniej sterowna”\n",
|
|||
|
|
"* Nesterov do piłki posiadającej pęd dodaje „hamulec”, który spowalnia piłkę przed wzniesieniem:\n",
|
|||
|
|
" $$ v_t := \\gamma \\, v_{t-1} + \\alpha \\, \\nabla_\\theta J(\\theta - \\gamma \\, v_{t-1}) $$\n",
|
|||
|
|
" $$ \\theta := \\theta - v_t $$"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### Adagrad\n",
|
|||
|
|
"\n",
|
|||
|
|
"* “<b>Ada</b>ptive <b>grad</b>ient”\n",
|
|||
|
|
"* Adagrad dostosowuje współczynnik uczenia (*learning rate*) do parametrów: zmniejsza go dla cech występujących częściej, a zwiększa dla występujących rzadziej:\n",
|
|||
|
|
"* Świetny do trenowania na rzadkich (*sparse*) zbiorach danych\n",
|
|||
|
|
"* Wada: współczynnik uczenia może czasami gwałtownie maleć\n",
|
|||
|
|
"* Wyniki badań pokazują, że często **starannie** dobrane $\\alpha$ daje lepsze wyniki na zbiorze testowym"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### Adadelta i RMSprop\n",
|
|||
|
|
"* Warianty algorytmu Adagrad, które radzą sobie z problemem gwałtownych zmian współczynnika uczenia"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### Adam\n",
|
|||
|
|
"\n",
|
|||
|
|
"* “<b>Ada</b>ptive <b>m</b>oment estimation”\n",
|
|||
|
|
"* Łączy zalety algorytmów RMSprop i Momentum\n",
|
|||
|
|
"* Można go porównać do piłki mającej ciężar i opór\n",
|
|||
|
|
"* Obecnie jeden z najpopularniejszych algorytmów optymalizacji"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### Nadam\n",
|
|||
|
|
"* “<b>N</b>esterov-accelerated <b>ada</b>ptive <b>m</b>oment estimation”\n",
|
|||
|
|
"* Łączy zalety algorytmów Adam i Nesterov Accelerated Gradient"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### AMSGrad\n",
|
|||
|
|
"* Wariant algorytmu Adam lepiej dostosowany do zadań takich jak rozpoznawanie obiektów czy tłumaczenie maszynowe"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"<img src=\"contours_evaluation_optimizers.gif\" style=\"margin: auto;\" width=\"80%\" />"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"<img src=\"saddle_point_evaluation_optimizers.gif\" style=\"margin: auto;\" width=\"80%\" />"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "slide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"## 3.9. Metody zbiorcze"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
" * **Metody zbiorcze** (*ensemble methods*) używają połączonych sił wielu modeli uczenia maszynowego w celu uzyskania lepszej skuteczności niż mogłaby być osiągnięta przez każdy z tych modeli z osobna."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "fragment"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
" * Na metodę zbiorczą składa się:\n",
|
|||
|
|
" * dobór modeli\n",
|
|||
|
|
" * sposób agregacji wyników"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "fragment"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
" * Warto zastosować randomizację, czyli przetasować zbiór uczący przed trenowaniem każdego modelu."
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "slide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### Uśrednianie prawdopodobieństw"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "fragment"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"#### Przykład\n",
|
|||
|
|
"\n",
|
|||
|
|
"Mamy 3 modele, które dla klas $c=1, 2, 3, 4, 5$ zwróciły prawdopodobieństwa:\n",
|
|||
|
|
"\n",
|
|||
|
|
"* $M_1$: [0.10, 0.40, **0.50**, 0.00, 0.00]\n",
|
|||
|
|
"* $M_2$: [0.10, **0.60**, 0.20, 0.00, 0.10]\n",
|
|||
|
|
"* $M_3$: [0.10, 0.30, **0.40**, 0.00, 0.20]\n",
|
|||
|
|
"\n",
|
|||
|
|
"Która klasa zostanie wybrana według średnich prawdopodobieństw dla każdej klasy?"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"Średnie prawdopodobieństwo: [0.10, **0.43**, 0.36, 0.00, 0.10]\n",
|
|||
|
|
"\n",
|
|||
|
|
"Została wybrana klasa $c = 2$"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "slide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### Głosowanie klas"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "fragment"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"#### Przykład\n",
|
|||
|
|
"\n",
|
|||
|
|
"Mamy 3 modele, które dla klas $c=1, 2, 3, 4, 5$ zwróciły prawdopodobieństwa:\n",
|
|||
|
|
"\n",
|
|||
|
|
"* $M_1$: [0.10, 0.40, **0.50**, 0.00, 0.00]\n",
|
|||
|
|
"* $M_2$: [0.10, **0.60**, 0.20, 0.00, 0.10]\n",
|
|||
|
|
"* $M_3$: [0.10, 0.30, **0.40**, 0.00, 0.20]\n",
|
|||
|
|
"\n",
|
|||
|
|
"Która klasa zostanie wybrana według głosowania?"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "subslide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"Liczba głosów: [0, 1, **2**, 0, 0]\n",
|
|||
|
|
"\n",
|
|||
|
|
"Została wybrana klasa $c = 3$"
|
|||
|
|
]
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"cell_type": "markdown",
|
|||
|
|
"metadata": {
|
|||
|
|
"slideshow": {
|
|||
|
|
"slide_type": "slide"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"source": [
|
|||
|
|
"### Inne metody zbiorcze\n",
|
|||
|
|
"\n",
|
|||
|
|
" * Bagging\n",
|
|||
|
|
" * Boostng\n",
|
|||
|
|
" * Stacking\n",
|
|||
|
|
" \n",
|
|||
|
|
"https://towardsdatascience.com/ensemble-methods-bagging-boosting-and-stacking-c9214a10a205"
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"metadata": {
|
|||
|
|
"celltoolbar": "Slideshow",
|
|||
|
|
"kernelspec": {
|
|||
|
|
"display_name": "Python 3",
|
|||
|
|
"language": "python",
|
|||
|
|
"name": "python3"
|
|||
|
|
},
|
|||
|
|
"language_info": {
|
|||
|
|
"codemirror_mode": {
|
|||
|
|
"name": "ipython",
|
|||
|
|
"version": 3
|
|||
|
|
},
|
|||
|
|
"file_extension": ".py",
|
|||
|
|
"mimetype": "text/x-python",
|
|||
|
|
"name": "python",
|
|||
|
|
"nbconvert_exporter": "python",
|
|||
|
|
"pygments_lexer": "ipython3",
|
|||
|
|
"version": "3.8.3"
|
|||
|
|
},
|
|||
|
|
"livereveal": {
|
|||
|
|
"start_slideshow_at": "selected",
|
|||
|
|
"theme": "white"
|
|||
|
|
}
|
|||
|
|
},
|
|||
|
|
"nbformat": 4,
|
|||
|
|
"nbformat_minor": 4
|
|||
|
|
}
|