2023-01-17 12:26:02 +01:00
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Uczenie maszynowe\n",
2023-04-14 10:47:15 +02:00
"# 10. Sieci neuronowe –
2023-01-17 12:26:02 +01:00
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"slideshow": {
"slide_type": "notes"
}
},
"outputs": [],
"source": [
"# Przydatne importy\n",
"\n",
"import matplotlib\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"\n",
"%matplotlib inline"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
2023-04-14 10:47:15 +02:00
"## 10.1. Perceptron"
2023-01-17 12:26:02 +01:00
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"https://www.youtube.com/watch?v=cNxadbrN_aI"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Pierwszy perceptron liniowy\n",
"\n",
"* Frank Rosenblatt, 1957\n",
"* aparat fotograficzny podłączony do 400 fotokomórek (rozdzielczość obrazu: 20 x 20)\n",
"* wagi –
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Uczenie perceptronu\n",
"\n",
"Cykl uczenia perceptronu Rosenblatta:\n",
"\n",
"1. Sfotografuj planszę z kolejnym obiektem.\n",
"1. Zaobserwuj, która lampka zapaliła się na wyjściu.\n",
"1. Sprawdź, czy to jest właściwa lampka.\n",
"1. Wyślij sygnał „nagrody” lub „kary”."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Funkcja aktywacji perceptronu\n",
"\n",
"Funkcja bipolarna:\n",
"\n",
"$$ g(z) = \\left\\{ \n",
"\\begin{array}{rl}\n",
"1 & \\textrm{gdy $z > \\theta_0$} \\\\\n",
"-1 & \\textrm{wpp.}\n",
"\\end{array}\n",
"\\right. $$\n",
"\n",
2023-05-11 10:37:43 +02:00
"gdzie $z = \\theta_1x_1 + \\ldots + \\theta_nx_n$,<br/>\n",
2023-01-17 12:26:02 +01:00
"$\\theta_0$ to próg aktywacji,<br/>\n",
"$x_0 = 1$. "
]
},
{
"cell_type": "code",
2023-05-11 10:37:43 +02:00
"execution_count": 2,
2023-01-17 12:26:02 +01:00
"metadata": {
"slideshow": {
"slide_type": "notes"
}
},
"outputs": [],
"source": [
"def bipolar_plot():\n",
" matplotlib.rcParams.update({'font.size': 16})\n",
"\n",
" plt.figure(figsize=(8,5))\n",
" x = [-1,-.23,1] \n",
" y = [-1, -1, 1]\n",
" plt.ylim(-1.2,1.2)\n",
" plt.xlim(-1.2,1.2)\n",
" plt.plot([-2,2],[1,1], color='black', ls=\"dashed\")\n",
" plt.plot([-2,2],[-1,-1], color='black', ls=\"dashed\")\n",
" plt.step(x, y, lw=3)\n",
" ax = plt.gca()\n",
" ax.spines['right'].set_color('none')\n",
" ax.spines['top'].set_color('none')\n",
" ax.xaxis.set_ticks_position('bottom')\n",
" ax.spines['bottom'].set_position(('data',0))\n",
" ax.yaxis.set_ticks_position('left')\n",
" ax.spines['left'].set_position(('data',0))\n",
"\n",
" plt.annotate(r'$\\theta_0$',\n",
" xy=(-.23,0), xycoords='data',\n",
" xytext=(-50, +50), textcoords='offset points', fontsize=26,\n",
" arrowprops=dict(arrowstyle=\"->\"))\n",
"\n",
" plt.show()"
]
},
{
"cell_type": "code",
2023-05-11 10:37:43 +02:00
"execution_count": 3,
2023-01-17 12:26:02 +01:00
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
2023-05-11 10:37:43 +02:00
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAoAAAAGVCAYAAABuPkCWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA23UlEQVR4nO3deXRU9f3/8ddkJQsJYCAm7CirLEoBQbYIVBMsqIBo+RW0hxZEpUKgCoJWgQa0qI2CIupXrYWKoqFSAipFkYBlF6xiWIQmLAIBzQoJJPf3B800YzJhhkwyyXyej3NyzuTe9/3kPZk7k1fuarMsyxIAAACM4eftBgAAAFCzCIAAAACGIQACAAAYhgAIAABgGAIgAACAYQiAAAAAhiEAAgAAGIYACAAAYBgCIAAjWZalnJwccS18ACYiAAIwUm5uriIjI5Wbm+vtVgCgxhEAAQAADEMABAAAMAwBEAAAwDAEQAAAAMMQAAEAAAxDAAQAADAMARAAAMAwBEAAAADDEAABAAAMQwAEAAAwDAEQAADAMARAAAAAwxAAAQAADEMABAAAMAwBEAAAwDAEQAAAAMMQAAEAAAxDAAQAADAMARAAAMAwBEAAAADDEAABAAAMQwAEAAAwDAEQAADAMARAAB6Vnp6uF198Uffdd5+6dOmigIAA2Ww2zZs3r0rjrl+/XkOHDlVUVJRCQkLUoUMHzZo1S3l5eR7qHADMEeDtBgD4lpdfflnJyckeHfP5559XYmKibDab+vfvr+joaG3atElJSUl6//33lZaWpqioKI/+TADwZWwBBOBRnTt31vTp07Vs2TLt27dPY8eOrdJ4u3fv1rRp0+Tv7681a9Zo48aNevfdd3Xo0CENHjxY6enpuv/++z3UPQCYgS2AADzqN7/5jcP3fn5V+z9z/vz5sixLv/71r5WQkGCfHhoaqtdff11t2rTR+++/r2+//VYdOnSo0s8CAFOwBRBArVVUVKQ1a9ZIksaMGVNufsuWLdW3b19JUkpKSo32BgB1GVsAAdhdLC5R9rkLHh2zyBYkv5AInbMCdCav0K1lv/32W523AuQXEqE2HbtWuHyXHn20eccebdvzjVvj5/639mxeoS74udcXAN/TMDRIfn42b7dRY2yWZVmuFObn5zud5+fnp5CQkCuqLSgokLMWbDabQkNDr6j23LlzKikpcdpHWFjYFdWeP39excXFHqkNDQ2VzXZpZSssLNTFixc9UhsSEmLf7VZUVKQLF5z/QXentl69evL393e79sKFCyoqKnJaGxwcrICAALdrL168qMJC53+4g4KCFBgY6HZtcXGxzp8/77Q2MDBQQUFBbteWlJTo3LlzHqkNCAhQcHCwJMmyLBUUFFS5dvXe7/XHdQeVW+h83fIlJYUFyvzzaDWf8q78gkMvvwAAn5Y2ra+aNW5g/76u5oiyy1fG5V3ATZo0UXh4eIVfZY/LkaRWrVo5rR0wYIBDbadOnZzW9uzZ06G2Z8+eTms7derkUDtgwACnta1atXKoTUhIcFrbpEkTh9qRI0c6rQ0PD3eoHTt2bKW1Zf8QT5w4sdLarKwse21iYmKltRkZGfbaWbNmVVq7b98+e21SUlKltbt27bLXJicnV1q7adMme+3SpUsrrf3oo4/stcuWLau0tuxuvpSUlEprly1bZq/96KOPKq1dunSpvXbTpk2V1pY9w3XXrl2V1iYlJdlr9+3bV2ntrFmz7LUZGRmV1iYmJtprs7KyKq2dOHGivbagoKDiuvoRemTFDmPCHwD8VL9+/Ry+r6s5wlUcAwhAfvXC5VfP9Q8OAEDdxi5gN2vZBcwuYF/cBXw2v0j9n9vidAxfxC5gAGWZtgvY5QAIwHedySvUz+atd5i2PnGAGoYGVXnsyZMna8WKFZoxc6YSp051a9l9+/Zp4MCBkqTvvvuuwt0bTzzxhJYsWaJhw4bp9ddfd3ns3JwctW7aRIePnVL9iAi3+gLge0w7CYSzgAFUqGFokK4KD67yOEFWkUrO5SjEdtHt8Xp166R6tosqKCjQd/v26uabby5X89WOL1RyLkc3Xn+dW+MHllyqbRQerAgPPE8AqEs4BhBArRUUFKTbbrtNkrR8+fJy8//zn/9oy5ZLu67vvPPOGu0NAOoyAiAAr1u0aJE6dOigcePGlZs3Y8YM2Ww2vfHGG1q3bp19ekFBgcaPH6/i4mKNHDmSu4AAgBvYBQzAo3bt2qUHHnjA/v2hQ4ckSa+88or+8Y9/2KenpKQoJiZG0qVL2aSnp+vqq68uN1737t317LPPKjExUUOHDtXAgQPVpEkTbdq0SSdOnFD79u21ZMmSan5WAOBbCIAAPConJ0dbt24tN/3o0aM6evSo/fvKzsr+qalTp6pLly569tlntW3bNuXn56tFixaaOXOmZs6cqfr163ukdwAwBWcBA6jwLOCds4d45CSQ2ionJ0eRkZHKzs5WBGcBAzAMxwACAAAYhgAIAABgGAIgAACAYQiAAAAAhiEAAgAAGIYACAAAYBgCIAAAgGEIgAAAAIYhAAIAABiGAAgAAGAYAiAAAIBhCIAAAACGIQACAAAYhgAIAABgGAIgAACAYQiAAAAAhiEAAgAAGIYACAAAYBgCIAAAgGEIgAAAAIYhAAIAABiGAAgAAGAYAiAAAIBhCIAAAACGIQACAAAYhgAIAABgGAIgAACAYQiAAAAAhiEAAgAAGIYACAAAYBgCIAAAgGEIgAAAAIYhAAIAABiGAAgAAGAYAiAAAIBhCIAAAACGIQACAAAYhgAIAABgGAIgAACAYQiAAAAAhiEAAgAAGIYACAAAYBgCIAAAgGEIgAAAAIYhAAIAABiGAAgAAGAYAiAAAIBhCIAAAACGIQACAAAYhgAIAABgGAIgAACAYQiAAAAAhiEAAgAAGIYACAAAYBgCIAAAgGEIgMBlFBUV6f3339eoUaPUrl07hYaGqkmTJrrxxhv1pz/9SWfPnvV2iwAAuCXA2w0AtdmmTZv029/+Vunp6Q7Tz507p9OnT2vbtm164YUXtGbNGnXt2tVLXQIA4B62AAJOvPTSSxo4cKA9/AUEBKhLly7q16+foqKi7HVHjx7VkCFD2BIIAKgzCIBABRYvXqwHH3xQlmVJku6//35lZGRo79692rRpk44dO6Y5c+bY60+fPq0ZM2Z4q10AANxCAAR+4osvvtCUKVPs3y9ZskQvv/yyYmJi7NOCgoL0+OOP695777VPW7ZsmQoKCmqyVQAArggBECgjLy9PY8aM0cWLFyVJU6ZM0cSJE53WP/roo/bHBQUF+uijj6q9RwAAqooACJTx5z//WUeOHJEkNW3aVPPmzau0vmPHjmrYsKH9+6+//ro62wMAwCMIgMB//fDDD1q4cKH9+9///vcKCwu77HLR0dH2x6XhEQCA2owACPzXG2+8oezsbElSWFiYxo8f79JygYGB9sf5+fnV0ltd9N577ykuLk4NGzZUWFiYunXrpmeeeUYXLlxwa5w333xTNput0q9169ZV07MAAN/EdQCB//rLX/5ifzxq1CiFh4e7tFzZQFOvXj2P91UXTZkyRcnJyQoICNCgQYMUHh6uDRs26NFHH9Xq1av18ccfKyQkxK0xr7nmGvXr16/CeU2bNvVE2wBgDAIgIOnf//639uzZY//+rrvucnnZH3/80f44NDTUk23VSatWrVJycrLCw8O1ceNGde/eXZKUlZWlQYMGKS0tTY8//rjD7nZX9OvXT2+++WY1dAwA5mEXMCA5nL0bGhqqwYMHu7zsmTNn7I9btmzp0b7qoqSkJEnSjBkz7OFPkqKiovTSSy9JkhYtWmTf3Q4AqHkEQEDSxo0b7Y8LCgoUEhJy2ePOSr/K7gJu3769N9qvNY4dO6bt27dLksaMGVNufr9+/dS8eXMVFhYqNTW1ptsDAPwXu4BhPMuytGnTJo+MZXoA3L17tySpUaNGat26dYU1PXr0UGZmpnbv3q1f/vKXLo998OBBzZ49W6dOnVJ4eLg6d+6s4cOHO9yWDwDgGgIgjJeZmelwHF+fPn0UERHh0rJff/21jh49KulS6DE9AB4
2023-01-17 12:26:02 +01:00
"text/plain": [
2023-05-11 10:37:43 +02:00
"<Figure size 800x500 with 1 Axes>"
2023-01-17 12:26:02 +01:00
]
},
2023-05-11 10:37:43 +02:00
"metadata": {},
2023-01-17 12:26:02 +01:00
"output_type": "display_data"
}
],
"source": [
"bipolar_plot()"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Schemat perceptronu\n",
"\n",
"<img src=\"perceptron.png\" alt=\"Rys. 11.1. Schemat perceptronu\" style=\"height: 80%\"/>"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Zasada działania perceptronu\n",
"\n",
"1. Ustal wartości początkowe $\\theta$ (wektor 0 lub liczby losowe blisko 0).\n",
"1. Dla każdego przykładu $(x^{(i)}, y^{(i)})$, dla $i=1,\\ldots,m$\n",
" * Oblicz wartość wyjścia $o^{(i)} = g(\\theta^{T}x^{(i)}) = g(\\sum_{j=0}^{n} \\theta_jx_j^{(i)})$\n",
" * Wykonaj aktualizację
" $$ \\theta := \\theta + \\Delta \\theta $$\n",
" $$ \\Delta \\theta = \\alpha(y^{(i)}-o^{(i)})x^{(i)} $$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"$$\\theta_j := \\theta_j + \\Delta \\theta_j $$\n",
"\n",
"Jeżeli przykład został sklasyfikowany **poprawnie**:\n",
"\n",
"* $y^{(i)}=1$ oraz $o^{(i)}=1$ : $$\\Delta\\theta_j = \\alpha(1 - 1)x_j^{(i)} = 0$$\n",
"* $y^{(i)}=-1$ oraz $o^{(i)}=-1$ : $$\\Delta\\theta_j = \\alpha(-1 - -1)x_j^{(i)} = 0$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Czyli: jeżeli trafiłeś, to nic nie zmieniaj."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"$$\\theta_j := \\theta_j + \\Delta \\theta_j $$\n",
"\n",
"Jeżeli przykład został sklasyfikowany **niepoprawnie**:\n",
"\n",
"* $y^{(i)}=1$ oraz $o^{(i)}=-1$ : $$\\Delta\\theta_j = \\alpha(1 - -1)x_j^{(i)} = 2 \\alpha x_j^{(i)}$$\n",
"* $y^{(i)}=-1$ oraz $o^{(i)}=1$ : $$\\Delta\\theta_j = \\alpha(-1 - 1)x_j^{(i)} = -2 \\alpha x_j^{(i)}$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Czyli: przesuń wagi w odpowiednią stronę."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Zalety perceptronu\n",
"\n",
"* intuicyjny i prosty\n",
"* łatwy w implementacji\n",
"* jeżeli dane można liniowo oddzielić, algorytm jest zbieżny w skończonym czasie"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Wady perceptronu\n",
"\n",
"* jeżeli danych nie można oddzielić liniowo, algorytm nie jest zbieżny"
]
},
{
"cell_type": "code",
2023-05-11 10:37:43 +02:00
"execution_count": 4,
2023-01-17 12:26:02 +01:00
"metadata": {
"slideshow": {
"slide_type": "notes"
}
},
"outputs": [],
"source": [
"def plot_perceptron():\n",
" plt.figure(figsize=(12,3))\n",
"\n",
" plt.subplot(131)\n",
" plt.ylim(-0.2,1.2)\n",
" plt.xlim(-0.2,1.2)\n",
"\n",
" plt.title('AND')\n",
" plt.plot([1,0,0], [0,1,0], 'ro', markersize=10)\n",
" plt.plot([1], [1], 'go', markersize=10)\n",
"\n",
" ax = plt.gca()\n",
" ax.spines['right'].set_color('none')\n",
" ax.spines['top'].set_color('none')\n",
" ax.xaxis.set_ticks_position('none')\n",
" ax.spines['bottom'].set_position(('data',0))\n",
" ax.yaxis.set_ticks_position('none')\n",
" ax.spines['left'].set_position(('data',0))\n",
"\n",
" plt.xticks(np.arange(0, 2, 1.0))\n",
" plt.yticks(np.arange(0, 2, 1.0))\n",
"\n",
"\n",
" plt.subplot(132)\n",
" plt.ylim(-0.2,1.2)\n",
" plt.xlim(-0.2,1.2)\n",
"\n",
" plt.plot([1,0,1], [0,1,1], 'go', markersize=10)\n",
" plt.plot([0], [0], 'ro', markersize=10)\n",
"\n",
" ax = plt.gca()\n",
" ax.spines['right'].set_color('none')\n",
" ax.spines['top'].set_color('none')\n",
" ax.xaxis.set_ticks_position('none')\n",
" ax.spines['bottom'].set_position(('data',0))\n",
" ax.yaxis.set_ticks_position('none')\n",
" ax.spines['left'].set_position(('data',0))\n",
"\n",
" plt.title('OR')\n",
" plt.xticks(np.arange(0, 2, 1.0))\n",
" plt.yticks(np.arange(0, 2, 1.0))\n",
"\n",
"\n",
" plt.subplot(133)\n",
" plt.ylim(-0.2,1.2)\n",
" plt.xlim(-0.2,1.2)\n",
"\n",
" plt.title('XOR')\n",
" plt.plot([1,0], [0,1], 'go', markersize=10)\n",
" plt.plot([0,1], [0,1], 'ro', markersize=10)\n",
"\n",
" ax = plt.gca()\n",
" ax.spines['right'].set_color('none')\n",
" ax.spines['top'].set_color('none')\n",
" ax.xaxis.set_ticks_position('none')\n",
" ax.spines['bottom'].set_position(('data',0))\n",
" ax.yaxis.set_ticks_position('none')\n",
" ax.spines['left'].set_position(('data',0))\n",
"\n",
" plt.xticks(np.arange(0, 2, 1.0))\n",
" plt.yticks(np.arange(0, 2, 1.0))\n",
"\n",
" plt.show()"
]
},
{
"cell_type": "code",
2023-05-11 10:37:43 +02:00
"execution_count": 5,
2023-01-17 12:26:02 +01:00
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"data": {
2023-05-11 10:37:43 +02:00
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA7YAAAEXCAYAAACZJdj6AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAm8UlEQVR4nO3deXRU9f3/8dfNnpCQEGgVDesIrVUOiyxWQEArqLS2Gpe6piqtiIDy07rUpUfleKhaPUlEVKBKFI5CAat1ibQGIa1lacBavwoYwiKiNgkhA2FJyOf3x2RuEpjsQ5LPzPNxTo5h5nPv/Qzcz9v7mnvv5zrGGCMAAAAAACwV0dEdAAAAAACgLQi2AAAAAACrEWwBAAAAAFYj2AIAAAAArEawBQAAAABYjWALAAAAALAawRYAAAAAYDWCLQAAAADAagRbAAAAAIDVCLZhaNasWXIcR47j6Mc//nGT7Xfs2OG2dxxH119/fZPL+Ns+9NBDJ7y3evXqeuvz/8TExOh73/ueBgwYoMsuu0yPPfaYNm3a1KrPCAAny5YtW/TQQw9p5MiROvXUUxUTE6NTTjlFI0aM0IMPPqgvvvii0eVfeeWVgDUwIiJCycnJGjRokKZOnaqCgoJ2+kQAcKL//e9/6tGjhxzH0bBhw1RVVdVo+08//VQxMTFyHEdXXnnlCe9XVVVp8eLFuuaaa+TxeJSYmKjExER5PB5dddVVevXVV1VZWdlkv/r27RuwhsbFxem0007TpEmT9Pzzz6uioqLVnx2WMggrlZWV5vvf/76R5P5s2bKl0WWKiorqtY+IiDD//e9/G13G3/bBBx884b28vLx662vqZ9SoUWbdunVt+twA0FZHjhwx06dPN1FRUY3WrMjISHPHHXeYI0eOBFzPyy+/3KzaFxERYR577LF2/pQAUOvVV191a9ITTzzRYLuqqiozfPhwI8mkpqaab775pt77a9asMQMHDmyy7g0cONCsWbOm0T716dOnWTV0wIABZvv27UH5e4AdOGMbZt577z1999139V7Lyclp0Tqqq6v1yCOPBKU/L7zwgrxer7xer8rLy7V7925t2LBBc+fO1bhx4yRJ69at0+jRo/Xiiy8GZZsA0FIHDx7UxRdfrOeee05VVVU666yz9OKLL+qLL75QSUmJtmzZovnz52vQoEE6duyY5s6dq4svvrjJMwbvvvuuWwP37dungoICPfroo4qLi3Nr7YoVK9rpUwJAfTfccIMuvfRSSdJjjz2mrVu3Bmz3zDPPaOPGje7vp5xyivveypUrddFFF2nr1q2KiorS7bffrtWrV+vrr7/W3r17tWbNGt1xxx2KiorS1q1bddFFF+nNN99ssm9jxoxx66fX69WuXbv07rvvauTIkZKkbdu26fLLL1d1dXUb/xZgjY5O1mhf6enpRpLp0aOHOe+884wk07t3b1NdXd3gMnXP2Ho8HiPJOI5jCgoKGlzG376pM7Yvv/xyo/3Nzc01PXr0cM+CvP/++83+rAAQLDfffLNbt2655RZz9OjRgO2OHj1qbr31VrftrbfeekKbumds8/LyAq7nz3/+s9vm3HPPDeZHAYAW2bVrl0lKSjKSzJgxY044Zty6dauJj483ksykSZPqvff555+bhIQE90zu+vXrG9zOhg0bTGpqqpFkunTpYr744ouA7fxnbMeNGxfw/aqqKjN48GC3hnLsGD44YxtGSktL9fbbb0uSrr32Wt16662SpF27dikvL69Z67jjjjvUvXt3GWP08MMPn7S++k2cOFFvvfWWoqKidOzYMd1555188wagXa1evVovv/yyJOnCCy/UggULFB0dHbBtdHS05s+frwsuuECStHDhQn300Uct3mZ6eroGDBggSdqwYYOOHj3ayt4DQNv06tVLc+bMkSTl5+fr+eefd98zxujXv/61Dh06pMTERL300kv1lp06daoqKirkOI6WL1+uESNGNLid4cOHa/ny5XIcRwcPHtTUqVNb1d/IyEjde++97p//8Y9/tGo9sA/BNoy8/vrr7sHRjTfeqKuuukrx8fGSpEWLFjVrHUlJSW6xeOedd/Svf/3r5HS2jh//+Me6+eabJfkmbfnrX/960rcJAH5PP/20+3t2drYcx2m0veM4eu655wIu3xJnnnmmJOnYsWMqLS1t1ToAIBhuv/12jR07VpL0wAMPaNeuXZJ8t5T5v7ybM2eOevfu7S6zceNG972rr75a48ePb3I748eP19VXXy3J96Xiv//971b1118/JZ1wCx5CF8E2jPjD6w9/+EONGDFCSUlJ+sUvfiFJWrFihQ4ePNis9UyfPl2nnnqqJLXLWVtJ7tllSfrggw/aZZsAcPDgQeXm5kqSxo4dW+9gqTFnnnmmxowZI0l6//33m11f66p7dUpycnKLlweAYHEcRwsWLFBcXJy8Xq9uu+027dq1S/fdd58k3/2u06ZNq7fM8uXL3d9/85vfNHtbv/71rwOuoyXq1s+UlJRWrQP2IdiGiS+++ELr16+X5Dtb63fTTTdJkg4cONDs4pGQkKAHHnhAkvS3v/1Na9asCXJvTzRs2DDFxcVJkj7++OOTvj0AkKR//etf7iMu/BPaNZe/fVVVldatW9fibfsfG+TxeNyrawCgowwcOFC///3vJfm+sDv//PPl9XoVFxenhQsXnnA1S35+viTfLRrnnXdes7czevRoRUVF1VtHS9V97NrZZ5/dqnXAPgTbMOE/W+s4jm644Qb39Ysuukg9e/as16Y5brvtNvXq1UuSAj6rNtiio6PVt29fSdLevXtP+vYAQJK2b9/u/n7WWWe1aNm67YuKilq07LJly/Tll19K8l0lAwCdwT333KOhQ4dKknbu3ClJ+v3vf6+BAwee0NZfP/v37++enGiOuLg49e/fX1LLa6fku33jySeflCSdcsopAZ+pi9BEsA0D1dXVeu211yT5ziDUvf8hMjJS119/vSQpLy/PvWeiKbGxsW6gXbt2rXup3snkv5Rk3759J31bACCp3r2t3bp1a9GydduXlJQEbHPo0CEdOHBABw4c0P79+7V582Y9/vjj7tU01157rWbMmNGKngNA8EVFRemXv/yl++dTTz1V99xzT8C2/vrZ0tpZd5mGaqfkC7D++nngwAHt3r1b77//vkaPHq3//Oc/6tq1q958880WhWrYjWAbBv7+97/rq6++klT/MmQ//wGUMUavvvpqs9d7yy23uN+oBeu5to0xxkhSkxO3AEBn0JxademllyopKUlJSUlKSUnR0KFD9cgjj+jYsWNasmSJlixZosjIyHboLQA07euvv9YTTzzh/vmbb77RW2+9FfTtNKd+5ufnu/UzKSlJvXv31iWXXKJ169ZpwoQJ2rZtm84999yg9w2dF8E2DPgvMY6Pjw94OcagQYM0ZMgQSVJOTk6z1xsVFeXea7F+/fqTUtjq2r9/vyQpNTX1pG4HAPzqnmkoKytr0bJ1ry5pad2qrKzUb3/7W23btq1FywHAyTRt2jTt379fsbGx6tevnyTf7RL+Y7S6/PWzpbVTqq2frT3my8vLa/WM9LAXwTbEeb1erVy5UpI0YsQIbd26VRs3bjzhx39T/9atW1v0CJ/rr79eP/zhDyX5ztr6z6oGW2VlpXufxWmnnXZStgEAx/MfuEnS//3f/7Vo2brt666nrry8PBljZIzRgQMHtHHjRl133XWSpD179ujyyy/X4cOHW9FzAAiuN954Q3/5y18k+Y75Fi1aJMdxtHfvXv32t789ob2/7hUWFurIkSPN3s6RI0fc+3Mbqp2S7/Y6f/08duyY9uzZo4ULF7pzxzz11FMtuhIR9iPYhrhly5apoqJCkrRmzRqNGDEi4E/dh223ZBKpyMhIPfroo5KkTz75RH/+85+D+wFq/Pvf/3aLYktm1gOAtjj33HPdS4FXr17domX97aOiopp1OVyXLl10zjnnaPHixZoyZYok6bPPPuOsA4AOV1JSopkzZ0qShgwZonvvvVdjx47VbbfdJklasGCB+8xaP/8jzyorK/XPf/6z2dv6xz/+ocrKynrraEpERIROO+003XL
2023-01-17 12:26:02 +01:00
"text/plain": [
2023-05-11 10:37:43 +02:00
"<Figure size 1200x300 with 3 Axes>"
2023-01-17 12:26:02 +01:00
]
},
2023-05-11 10:37:43 +02:00
"metadata": {},
2023-01-17 12:26:02 +01:00
"output_type": "display_data"
}
],
"source": [
"plot_perceptron()"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Perceptron –
"\n",
"Zamiast funkcji bipolarnej możemy zastosować funkcję sigmoidalną jako funkcję aktywacji."
]
},
{
"cell_type": "code",
2023-05-11 10:37:43 +02:00
"execution_count": 6,
2023-01-17 12:26:02 +01:00
"metadata": {
"slideshow": {
"slide_type": "notes"
}
},
"outputs": [],
"source": [
"def plot_activation_functions():\n",
" plt.figure(figsize=(16,7))\n",
" plt.subplot(121)\n",
" x = [-2,-.23,2] \n",
" y = [-1, -1, 1]\n",
" plt.ylim(-1.2,1.2)\n",
" plt.xlim(-2.2,2.2)\n",
" plt.plot([-2,2],[1,1], color='black', ls=\"dashed\")\n",
" plt.plot([-2,2],[-1,-1], color='black', ls=\"dashed\")\n",
" plt.step(x, y, lw=3)\n",
" ax = plt.gca()\n",
" ax.spines['right'].set_color('none')\n",
" ax.spines['top'].set_color('none')\n",
" ax.xaxis.set_ticks_position('bottom')\n",
" ax.spines['bottom'].set_position(('data',0))\n",
" ax.yaxis.set_ticks_position('left')\n",
" ax.spines['left'].set_position(('data',0))\n",
"\n",
" plt.annotate(r'$\\theta_0$',\n",
" xy=(-.23,0), xycoords='data',\n",
" xytext=(-50, +50), textcoords='offset points', fontsize=26,\n",
" arrowprops=dict(arrowstyle=\"->\"))\n",
"\n",
" plt.subplot(122)\n",
" x2 = np.linspace(-2,2,100)\n",
" y2 = np.tanh(x2+ 0.23)\n",
" plt.ylim(-1.2,1.2)\n",
" plt.xlim(-2.2,2.2)\n",
" plt.plot([-2,2],[1,1], color='black', ls=\"dashed\")\n",
" plt.plot([-2,2],[-1,-1], color='black', ls=\"dashed\")\n",
" plt.plot(x2, y2, lw=3)\n",
" ax = plt.gca()\n",
" ax.spines['right'].set_color('none')\n",
" ax.spines['top'].set_color('none')\n",
" ax.xaxis.set_ticks_position('bottom')\n",
" ax.spines['bottom'].set_position(('data',0))\n",
" ax.yaxis.set_ticks_position('left')\n",
" ax.spines['left'].set_position(('data',0))\n",
"\n",
" plt.annotate(r'$\\theta_0$',\n",
" xy=(-.23,0), xycoords='data',\n",
" xytext=(-50, +50), textcoords='offset points', fontsize=26,\n",
" arrowprops=dict(arrowstyle=\"->\"))\n",
"\n",
" plt.show()"
]
},
{
"cell_type": "code",
2023-05-11 10:37:43 +02:00
"execution_count": 7,
2023-01-17 12:26:02 +01:00
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
2023-05-11 10:37:43 +02:00
"image/png": "iVBORw0KGgoAAAANSUhEUgAABOsAAAIvCAYAAAAlNV/cAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAByuElEQVR4nO3dd3RU9dbG8WcmvUMICQmEDgm9SLDQiwUUVFCv8lqvBbsCFpoFQbAr2PGiooL9giioiFgoXnrvnRBaCul9Mu8fyMghhQSSnMnM97NWlsk+vznZMWSy88wpFrvdbhcAAAAAAAAA01nNbgAAAAAAAADASYR1AAAAAAAAgJMgrAMAAAAAAACcBGEdAAAAAAAA4CQI6wAAAAAAAAAnQVgHAAAAAAAAOAnCOgAAAAAAAMBJENYBAAAAAAAAToKwDoBTsNvtSk9Pl91uN7sVAAAAuCHmUQDOgrAOgFPIyMhQSEiIMjIyzG4FAAAAboh5FICzIKwDAAAAAAAAnARhHQAAAAAAAOAkCOsAAAAAAAAAJ0FYBwAAAAAAADgJwjoAAAAAAADASRDWAQAAAAAAAE6CsA4AAAAAAABwEoR1AAAAAAAAgJMgrAMAAAAAAACcBGEdAAAAAAAA4CQI6wAAAAAAAAAnQVgHAAAAAAAAOAnCOgAAAAAAAMBJENYBAAAAAAAAToKwDgAAAAAAAHAShHUAAAAAAACAkyCsAwAAAAAAAJwEYR0AAAAAAADgJAjrAAAAAAAAACdBWAcAAAAAAAA4CcI6AAAAAAAAwEkQ1gEAAAAAAABOgrAOAAAAAAAAcBKEdQAAAAAAAICTIKwDAAAAAAAAnARhHQAAAAAAAOAkCOsAAAAAAAAAJ0FYBwAAAAAAADgJwjoAAAAAAADASRDWAQAAAAAAAE6CsA4AAAAAAABwEoR1AAAAAAAAgJMgrAMAAAAAAACcBGEdAAAAAAAA4CQI6wAAAAAAAAAnQVgHAAAAAAAAOAnCOsDF7dixQ2+++aZuv/12tWvXTp6enrJYLJo0adJ57XfRokUaOHCgwsLC5Ofnp9jYWI0bN06ZmZmV1DkAAABcBTMpAJSfp9kNAKha7777rqZOnVqp+3z99dc1cuRIWSwW9ejRQxEREVqyZIkmT56sb7/9VkuXLlVYWFilfk4AAADUXMykAFB+HFkHuLi2bdvqscce06xZs7Rt2zbdcsst57W/devWadSoUfLw8ND8+fP1xx9/6KuvvtKePXvUr18/7dixQ/fee28ldQ8AAABXwEwKAOXHkXWAi7vrrrsMH1ut55fRT5kyRXa7XXfccYcGDBjgqPv7+2vGjBlq2rSpvv32W23fvl2xsbHn9bkAAADgGphJAaD8OLIOQLnl5+dr/vz5kqRhw4YV296oUSN169ZNkjRnzpxq7Q0AAADugZkUgKvjyDoABoW2IqXlFJS4bfv27cq1e8rqF6ymrdorOTOv2Jp2XS7WstUbtHLD1hK3lybj77UpmXkqsJb/cQBcS21/b1mtFrPbAAA4sZ07dyo7O1uS1KVLlxLXdOnSRUuWLNG6deuqszUAqBROF9ZlZWWVus3Dw0O+vr7lWmu1WuXn53dOa7Ozs2W320tca7FY5O/vf05rc3JyVFRUVGofAQEB57Q2NzdXNputUtb6+/vLYjn5R1JeXp4KCwsrZa2fn5/jUPf8/HwVFJQcBlV0ra+vrzw8PCq8tqCgQPn5+aWu9fHxkaenZ4XXFhYWKi+v9KDJ29tbXl5eFV5rs9mUm5tb6lovLy95e3uXa+3p3/+ioiLl5OQ4Pv5+41E9/9MuZeSV/m8k+uHZkqReb/yvlMZ7Kvrhnlon6YJJi0rdz5mK8k4OXD1f/l1WH/+zrAbgqtaM7686gT5mtwE3xzxa8bXMo8yjFZlHT+1TOjmPlvWzcfp+T82u27ZtkySFhITIarU6Hu/p6Skfn5O/Qxo0aCBJ2r17d4n7P32t3W5Xdna2Y11WVpbj+yRV7Oee54iS1/IcwXNERZ8jzvy5r4y1Jf3cn+70f4dmc7rTYAMDA0t9Gzp0qGFteHh4qWtPv26BJDVu3LjUtT179jSsbd26dalr4+LiDGvj4uJKXdu6dWvD2p49e5a6tnHjxoa1AwYMKHVteHi4Ye3QoUPL/P92ultuuaXMtaf/Yx0+fHiZa5OSkhxrR44cWebagwcPOtaOGzeuzLWnfvlK0uTJk8tcu3btWsfaqVOnlrl2yZIljrXTp08vc+3PP//sWDtr1qwy155+aP2cOXPKXDtr1izH2p9//rnMtdOnT3esXbJkSZlrT7+z1tq1a8tcu2nTJsfabdu2/bMtKFhPfLm6zKAOAAB3UNbvUeZR5tEz35hHKz6PTp482bE2MTGxzLXjxo1zrD148KACAwN1ww03SJLS0tIMa0eOHKkzrVu3rsT9Dh8+XHl5eUpPT9fRo0cVGBioqKgoSVJUVJRh7Zk3wyirX54jeI44843niPN7jjD8zVrCW0nPEaW9nf4ckZSUVOa/Q7M5XVgHwBxW30BZfZ3rCQoAAACoKlOmTFFISIgjpAMAZ2Gxl3Y8rEk4pLjiazmkmEOKK3JI8b333qvPPvtMEydO1NixYx2HCadk5avHa8tLfVxVK8rLVvwbNyj60a84DRZwY5wGC2fAPFrxtcyjzKMVPcXtnnvu0cyZMzVhwgSNGjWqXPs9dYrbggULdMMNNygkJEQJCQmOtaef4vbaa69p1KhR6tSpk+FopdPXSif/3Zw6HS4jI0MxMTE6fPiwgoODHWs5DfYfPEdUfC3PESdxGmzFON016yryP6eq1p7+ZFWZa09/cq3Mtaf/MqjMtT4+Po5/yJW51tvb2/HDZNZaLy8vw7UyKmutp6en40mwMtd6eHiU+9/w2daefv0Nq9XqWJtrL97LopE9Vdv/n/+n27ZtU69evSRJe/fuLfFQ4aefflrvvfeeBg0apBkzZpSrZ0nKSE9XkzekPx/vraDThiMA7uX05xzALMyjFV/LPFrxte48j57u9Hm0vGtjY2MlnTwNtqioSEFBQcXWHjp0SJLUrFmzMvd/6t9MSEiIY11AQECZj+E5ouJreY6o+FqeI046l+eI8rBYLIa1drtdmXmFSsnMV+0ALwX5lu//Z1VxurAOgPOo7e9tOMKla4fW8rUUKjs7W3u3bVSfPn2KPWbT6r9UlJOuCzu2qdDRMV5FJ9eGBvoomKNqAAAAUIqYmBj5+/srOztbq1evLnEmXb16tSSpc+fO1d0eACdRYCtScma+EjPylJR56i1fSZl5Ss7MU3JWvlL+fkvOyld+4ckjRafd1EmDO5h7ejxhHYBy8/b21pVXXqmvv/5as2fPLjYYHThwQMuXnzyV9tprrzWjRQAAALg4ZlLAvdmK7ErKzNPRtFwdSz/1lqej6bk6npGn4+m5SszIU0p2vs7lwm8pmaWf+ltdCOsAlOqBBx/UFx9/YKiNHj1a33zzjT766CMNHTpUV1xxhaST18K48847ZbPZNHToUMfpCQAAAMC5eOutt/TWW2+pa9eu+uSTTwzbmEkB15WZV6iEEzlKSM1WwokcHUrN0ZHUXB1Jy9Hh1JPhXGFR1d1+ISWr9Ov/VRfCOsDFrV27Vvfff7/j4z179kiS3n//ff3www+O+oezvir22IS/r/Vxus6dO+vVV1/VyJEjNXDgQPXq1Uvh4eFasmSJjhw5opiYGL333ntV8JUAAACgpirvTDpnzhxFRkZKkpKSkrRjxw7Vq1ev2P6YSYGaq9BWpITUHB1IztbBlGzFn8hWfMrJ9w+dyFFqduk34agOJ0z+/BJhHeDy0tPTtWLFimL1Q4cOOS68K6nMuwedacSIEWrXrp1effVVrVy5UllZWWrYsKHGjBmjMWPGlHiRXwAAALiv8s6kZd158kzMpIDzKiqy62h6rvYkZmpvYpb2JWVpf3KWDiSfDOaq8si4M3lYLaoT4K2wQB+FBfkoLMBbdQJPXp89NMBbdQK8//6vj0IDvRXg7XH2nVYxi720ezgDcCvJmXm6YNIiQ23N+P4VuknE+UhPT1dISIjS0tIUzN1gAQAAUM2YR4G
2023-01-17 12:26:02 +01:00
"text/plain": [
2023-05-11 10:37:43 +02:00
"<Figure size 1600x700 with 2 Axes>"
2023-01-17 12:26:02 +01:00
]
},
2023-05-11 10:37:43 +02:00
"metadata": {},
2023-01-17 12:26:02 +01:00
"output_type": "display_data"
}
],
"source": [
"plot_activation_functions()"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Perceptron a regresja liniowa\n",
"\n",
"<img src=\"reglin.png\" alt=\"Rys. 11.2. Perceptron a regresja liniowa\" style=\"height: 60%\">"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"#### Uczenie regresji liniowej:\n",
"* Model: $$h_{\\theta}(x) = \\sum_{i=0}^n \\theta_ix_i$$\n",
"* Funkcja kosztu (błąd średniokwadratowy): $$J(\\theta) = \\frac{1}{m} \\sum_{i=1}^{m} (h_{\\theta}(x^{(i)}) - y^{(i)})^2$$\n",
"* Po obliczeniu $\\nabla J(\\theta)$ - zwykły SGD."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Perceptron a dwuklasowa regresja logistyczna\n",
"\n",
"<img src=\"reglog.png\" alt=\"Rys. 11.3. Perceptron a dwuklasowa regresja logistyczna\" style=\"height:60%\">"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"#### Uczenie dwuklasowej regresji logistycznej:\n",
"* Model: $h_{\\theta}(x) = \\sigma(\\sum_{i=0}^n \\theta_ix_i) = P(1|x,\\theta)$\n",
"* Funkcja kosztu (entropia krzyżowa): $$\\begin{eqnarray} J(\\theta) &=& -\\frac{1}{m} \\sum_{i=1}^{m} \\big( y^{(i)}\\log P(1|x^{(i)},\\theta) \\\\ && + (1-y^{(i)})\\log(1-P(1|x^{(i)},\\theta)) \\big) \\end{eqnarray}$$\n",
"* Po obliczeniu $\\nabla J(\\theta)$ - zwykły SGD."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Perceptron a wieloklasowa regresja logistyczna\n",
"\n",
"<img src=\"multireglog.png\" alt=\"Rys. 11.4. Perceptron a wieloklasowa regresja logistyczna\" style=\"height:60%\">"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"#### Wieloklasowa regresja logistyczna\n",
"* Model (dla $c$ klasyfikatorów binarnych): \n",
"$$\\begin{eqnarray}\n",
"h_{(\\theta^{(1)},\\dots,\\theta^{(c)})}(x) &=& \\mathrm{softmax}(\\sum_{i=0}^n \\theta_{i}^{(1)}x_i, \\ldots, \\sum_{i=0}^n \\theta_i^{(c)}x_i) \\\\ \n",
"&=& \\left[ P(k|x,\\theta^{(1)},\\dots,\\theta^{(c)}) \\right]_{k=1,\\dots,c} \n",
"\\end{eqnarray}$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* Funkcja kosztu (**przymując model regresji binarnej**): $$\\begin{eqnarray} J(\\theta^{(k)}) &=& -\\frac{1}{m} \\sum_{i=1}^{m} \\big( y^{(i)}\\log P(k|x^{(i)},\\theta^{(k)}) \\\\ && + (1-y^{(i)})\\log P(\\neg k|x^{(i)},\\theta^{(k)}) \\big) \\end{eqnarray}$$\n",
"* Po obliczeniu $\\nabla J(\\theta)$, **c-krotne** uruchomienie SGD, zastosowanie $\\mathrm{softmax}(X)$ do niezależnie uzyskanych klasyfikatorów binarnych."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* Przyjmijmy: \n",
"$$ \\Theta = (\\theta^{(1)},\\dots,\\theta^{(c)}) $$\n",
"\n",
"$$h_{\\Theta}(x) = \\left[ P(k|x,\\Theta) \\right]_{k=1,\\dots,c}$$\n",
"\n",
"$$\\delta(x,y) = \\left\\{\\begin{array}{cl} 1 & \\textrm{gdy } x=y \\\\ 0 & \\textrm{wpp.}\\end{array}\\right.$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* Wieloklasowa funkcja kosztu $J(\\Theta)$ (kategorialna entropia krzyżowa):\n",
"$$ J(\\Theta) = -\\frac{1}{m}\\sum_{i=1}^{m}\\sum_{k=1}^{c} \\delta({y^{(i)},k}) \\log P(k|x^{(i)},\\Theta) $$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* Gradient $\\nabla J(\\Theta)$:\n",
"$$ \\dfrac{\\partial J(\\Theta)}{\\partial \\Theta_{j,k}} = -\\frac{1}{m}\\sum_{i = 1}^{m} (\\delta({y^{(i)},k}) - P(k|x^{(i)}, \\Theta)) x^{(i)}_j \n",
"$$\n",
"\n",
"* Liczymy wszystkie wagi jednym uruchomieniem SGD"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Perceptron –
"\n",
"* W przypadku jednowarstowej sieci neuronowej wystarczy znać gradient funkcji kosztu.\n",
"* Wtedy liczymy tak samo jak w przypadku regresji liniowej, logistycznej, wieloklasowej logistycznej itp. (wymienione modele to szczególne przypadki jednowarstwowych sieci neuronowych).\n",
"* Regresja liniowa i binarna regresja logistyczna to jeden neuron.\n",
"* Wieloklasowa regresja logistyczna to tyle neuronów, ile klas."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Funkcja aktywacji i funkcja kosztu są **dobierane do problemu**."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
2023-04-14 10:47:15 +02:00
"## 10.2. Funkcje aktywacji"
2023-01-17 12:26:02 +01:00
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* Operacje, które każdy neuron wykonuje, to dodawanie i mnożenie. Są to **funkcje liniowe**."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* Złożenie funkcji liniowych jest funkcją liniową."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* Gdyby nie było funkcji aktywacji, cała sieć neuronowa działałaby jak regresja liniowa."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* **Najważniejszym celem** stosowania funkcji aktywacji jest **wprowadzenie nieliniowości**, dzięki której sieć może uczyć się zależności bardziej skomplikowanych niż tylko liniowe."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Jakie cechy powinna mieć dobra funkcja aktywacji?"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* **różniczkowalna** (co najmniej **przedziałami różniczkowalna**), aby można było obliczyć jej gradient"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* **prosta obliczeniowo**, ponieważ podczas uczenia sieci jej wartość obliczana jest wiele razy"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* odporna na **problem zanikającego gradientu** (wyjaśnienie kilka slajdów dalej)"
]
},
{
"cell_type": "code",
2023-05-11 10:37:43 +02:00
"execution_count": 8,
2023-01-17 12:26:02 +01:00
"metadata": {
"slideshow": {
"slide_type": "notes"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
2023-05-11 10:37:43 +02:00
"2023-05-04 10:23:21.183242: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA\n",
"To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
"2023-05-04 10:23:22.176521: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory\n",
"2023-05-04 10:23:22.176596: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n",
"2023-05-04 10:23:24.881062: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory\n",
"2023-05-04 10:23:24.881370: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory\n",
"2023-05-04 10:23:24.881390: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n"
2023-01-17 12:26:02 +01:00
]
}
],
"source": [
"%matplotlib inline\n",
"\n",
"import math\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"import random\n",
"\n",
"import keras\n",
"from keras.datasets import mnist\n",
"from keras.models import Sequential\n",
"from keras.layers import Dense, Dropout, SimpleRNN, LSTM\n",
"from keras.optimizers import Adagrad, Adam, RMSprop, SGD\n",
"\n",
"from IPython.display import YouTubeVideo"
]
},
{
"cell_type": "code",
2023-05-11 10:37:43 +02:00
"execution_count": 9,
2023-01-17 12:26:02 +01:00
"metadata": {
"slideshow": {
"slide_type": "notes"
}
},
"outputs": [],
"source": [
"def plot(fun):\n",
" x = np.arange(-3.0, 3.0, 0.01)\n",
" y = [fun(x_i) for x_i in x]\n",
" fig = plt.figure(figsize=(14, 7))\n",
" ax = fig.add_subplot(111)\n",
" fig.subplots_adjust(left=0.1, right=0.9, bottom=0.1, top=0.9)\n",
" ax.set_xlim(-3.0, 3.0)\n",
" ax.set_ylim(-1.5, 1.5)\n",
" ax.grid()\n",
" ax.plot(x, y)\n",
" plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Funkcja logistyczna"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"$$ g(x) = \\frac{1}{1 + e^{-x}} $$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"#### Funkcja logistyczna –
]
},
{
"cell_type": "code",
2023-05-11 10:37:43 +02:00
"execution_count": 10,
2023-01-17 12:26:02 +01:00
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"data": {
2023-05-11 10:37:43 +02:00
"image/png": "iVBORw0KGgoAAAANSUhEUgAABLoAAAJpCAYAAACq4lvbAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABYv0lEQVR4nO3de5zddWHn//dM5n6fScj9yl3lYsJFukDLRSteoGhXu4vW0tV1t6xVxK6Cl12sVuyu7Ra1uN3VVldl7cVisVJXVOpPpMZAEFEkCCQkIZDbJHOfyczk/P6YycCYBBIyyUm+83w+Hudx5nzP93zP5/j4OE5efr7fU1EqlUoBAAAAgGNcZbkHAAAAAABTQegCAAAAoBCELgAAAAAKQegCAAAAoBCELgAAAAAKQegCAAAAoBCELgAAAAAKQegCAAAAoBCELgAAAAAKoayha82aNfnUpz6Vq6++OqeffnqqqqpSUVGRj370oy/oeDfeeGMqKiqe8/bwww9P8acAAAAA4GhQVc43/8xnPpObb755yo975pln5qUvfek+n2ttbZ3y9wMAAACg/Moauk477bT8wR/8QZYvX54VK1bkYx/7WL74xS8e8nGvvPLK3HjjjYc+QAAAAACOGWUNXW9729smPa6sdMkwAAAAAF4YZQkAAACAQijriq7DZfXq1bn++uvT2dmZ1tbWLF++PJdffnmam5vLPTQAAAAADpNChq6vf/3r+frXvz5pW2traz75yU/mLW95y/O+fmhoKENDQxOPd+/enc7OzsycOTMVFRVTPl4AAACA6a5UKqWnpyfz589/wZe3KlToOuGEE/Kxj30sr3rVq7JkyZIkyUMPPZSPf/zj+cd//Mf8zu/8TmbMmJE3velNz3mcm266KR/+8IePxJABAAAAeJYNGzZk4cKFL+i1FaVSqTTF43nBrr766nzhC1/IRz7ykXzwgx+c0mO/853vzKc+9akcd9xx2bhxY2pqava77y+v6Orq6srixYvzyCOPpKOjY0rHBUeT4eHh3HXXXbn44otTXV1d7uHAYWGeM12Y60wX5jrTgXnOdNHZ2ZmTTz45O3fuTGtr6ws6RqFWdD2XG2+8Mbfccku2bt2alStX5sILL9zvvrW1tamtrd1re0dHR2bOnHk4hwllNTw8nIaGhsycOdP/gFJY5jnThbnOdGGuMx2Y50w3h3LZqGnzrYsdHR2ZPXt2kmTjxo1lHg0AAAAAU23ahK7R0dF0dXUliW9fBAAAACigaRO6br/99vT396eioiJnn312uYcDAAAAwBQ75kLXpz/96Zx66ql5y1veMmn7+vXr86UvfSmDg4N7veZrX/ta3va2tyVJ3vSmN2Xu3LlHZKwAAAAAHDllvRj96tWrc80110w8fuyxx5Ikf/EXf5F//Md/nNh+2223Zd68eUmSbdu2Zc2aNXvFqs7Ozvz2b/92fu/3fi/Lly/PggULMjAwkIceeii/+MUvkiQXX3xxPvOZzxzujwUAAABAGZQ1dHV3d2flypV7bd+4ceOkC8YPDQ0977EWLVqU973vfVm1alUeffTRrF69Ort27cqsWbPy2te+NldddVV+67d+K5WVx9wiNgAAAAAOQFlD10UXXZRSqXRQr7nxxhtz44037rV95syZ+fjHPz5FIwMAAADgWGN5EwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhlDV1r1qzJpz71qVx99dU5/fTTU1VVlYqKinz0ox89pON++9vfzqtf/erMmjUr9fX1OfXUU/OBD3wgvb29UzRyAAAAAI42VeV888985jO5+eabp/SY/+N//I9cd911qaioyIUXXpg5c+bk+9//fj72sY/lq1/9au6+++7MmjVrSt8TAAAAgPIr64qu0047LX/wB3+QL3/5y/n5z3+e3/7t3z6k491///15z3vekxkzZuQb3/hGvve97+Vv/uZv8thjj+XSSy/NmjVr8h//43+cotEDAAAAcDQp64qut73tbZMeV1YeWne76aabUiqV8ru/+7t51ateNbG9oaEhn/vc53L88cfnq1/9ah5++OGceuqph/ReAAAAABxdCnMx+l27duUb3/hGkuSqq67a6/klS5bk/PPPT5LcdtttR3RsAAAAABx+hQldjzzySPr7+5MkZ5999j732bP9/vvvP2LjAgAAAODIKEzoWrt2bZKkra0tzc3N+9xn0aJFk/YFAAAAoDjKeo2uqdTT05MkaWxs3O8+TU1NSZLu7u7nPNbQ0FCGhoYmHu/Zf3h4OMPDw4c6VDhq7Znf5jlFZp4zXZjrTBfmOtOBec50MRVzvDChayrddNNN+fCHP7zX9rvuuisNDQ1lGBEcWXfeeWe5hwCHnXnOdGGuM12Y60wH5jlFt+eSVIeiMKFrz+mKfX19+92nt7c3SdLS0vKcx7rhhhty3XXXTTzu7u7OokWLcvHFF2fmzJlTMFo4Og0PD+fOO+/MK17xilRXV5d7OHBYmOdMF+Y604W5znRgnjNdbN++/ZCPUZjQtXTp0iTJzp0709PTs8/rdG3YsGHSvvtTW1ub2travbZXV1f7pcK0YK4zHZjnTBfmOtOFuc50YJ5TdFMxvwtzMfpTTjll4rTCe++9d5/77Nm+YsWKIzYuAAAAAI6MwoSumpqavOY1r0mS3HrrrXs9/8QTT+See+5Jkrzuda87omMDAAAA4PA75kLXpz/96Zx66ql5y1vestdz119/fSoqKvJXf/VX+eY3vzmxvb+/P29961szOjqa3/zN38ypp556JIcMAAAAwBFQ1mt0rV69Otdcc83E48ceeyxJ8hd/8Rf5x3/8x4ntt912W+bNm5ck2bZtW9asWZO5c+fudbwVK1bkT/7kT3Ldddfl1a9+dX7t134ts2fPzve///089dRTOeWUU/I//+f/PMyfCgAAAIByKGvo6u7uzsqVK/favnHjxmzcuHHi8dDQ0AEf893vfndOP/30/Mmf/El+9KMfpa+vL4sXL84NN9yQG264YZ8XqQcAAABgapVKpQwO707frpH0DY2kb2g0/btG0js0kv5do2P3QyPp2zWaweHRXH3WrEN+z7KGrosuuiilUumgXnPjjTfmxhtvfM59Xv7yl+flL3/5IYwMAAAAYHoplUoZGtmdnsGxGNU7OJKeoeH0Do6kb9dIeodGJ8JU39DIeLTas20sZO2JWv3jP+8+iOzzppd2HPJnKGvoAgAAAODQlEqlDAyPjoep8UA1OJLeoeFJ0ap36JnnnwlZI+kZHJ54PHIwZeogNNTMSGNtVRprZqShpipNtVVpqJ28bfcUvLfQBQAAAFBGQyOj6R4YSffgcLoHhtM9OJLugeF0DQy
2023-01-17 12:26:02 +01:00
"text/plain": [
2023-05-11 10:37:43 +02:00
"<Figure size 1400x700 with 1 Axes>"
2023-01-17 12:26:02 +01:00
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plot(lambda x: 1 / (1 + math.exp(-x)))"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* Funkcja logistyczna przyjmuje wartości z przedziału $(0, 1)$."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Tangens hiperboliczny"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"$$ g(x) = \\tanh x = \\frac{e^{x} - e^{-x}}{e^{x} + e^{-x}} $$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"#### Tangens hiperboliczny –
]
},
{
"cell_type": "code",
2023-05-11 10:37:43 +02:00
"execution_count": 11,
2023-01-17 12:26:02 +01:00
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"data": {
2023-05-11 10:37:43 +02:00
"image/png": "iVBORw0KGgoAAAANSUhEUgAABLoAAAJpCAYAAACq4lvbAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABpnUlEQVR4nO3dd3hUVf7H8c/MZGbSOwklCaGDUkNVQBFQsWFbdRcbCrqr608RXcW2i6sruqvuYsF1RUUF1o6CYsG6AgqBIEgLnST0JKQnk8nM/P5IkUgxIeUmd96v55ln7j333DPfcc8Ow4c751p8Pp9PAAAAAAAAQCtnNboAAAAAAAAAoDEQdAEAAAAAAMAUCLoAAAAAAABgCgRdAAAAAAAAMAWCLgAAAAAAAJgCQRcAAAAAAABMgaALAAAAAAAApkDQBQAAAAAAAFMg6AIAAAAAAIApGBp0paen69lnn9XEiRPVp08fBQQEyGKx6NFHHz2p8aZPny6LxXLCx+bNmxv5XQAAAAAAAKAlCDDyxV944QXNnDmz0cft16+f+vfvf8xjERERjf56AAAAAAAAMJ6hQVfv3r119913a8CAAUpJSdFjjz2mN954o8HjXnLJJZo+fXrDCwQAAAAAAECrYWjQNXny5Fr7VitLhgEAAAAAAODkkCwBAAAAAADAFAy9oquppKWladq0acrNzVVERIQGDBigiy66SGFhYUaXBgAAAAAAgCZiyqBr0aJFWrRoUa22iIgIPfPMM7ruuut+9XyXyyWXy1Wz7/V6lZubq5iYGFkslkavFwAAAAAAwN/5fD4VFhaqffv2J728lamCri5duuixxx7Teeedp44dO0qSNm7cqMcff1wfffSRrr/+etlsNl199dUnHGfGjBl6+OGHm6NkAAAAAAAAHCEzM1MJCQknda7F5/P5GrmekzZx4kS99tpreuSRR/Tggw826ti33367nn32WbVp00ZZWVlyOBzH7fvLK7ry8/OVlJSkLVu2KDo6ulHrAloSt9utr7/+WmeddZbsdrvR5QBNgnkOf8Fch79grsMfMM/hL3Jzc9W9e3fl5eUpIiLipMYw1RVdJzJ9+nTNmjVLhw4d0ooVKzRy5Mjj9nU6nXI6nUe1R0dHKyYmpinLBAzldrsVHBysmJgY/gCFaTHP4S+Y6/AXzHX4A+Y5/E1Dlo3ym7suRkdHKy4uTpKUlZVlcDUAAAAAAABobH4TdHk8HuXn50sSd18EAAAAAAAwIb8JuhYuXKiSkhJZLBYNGjTI6HIAAAAAAADQyFpd0PXcc8+pZ8+euu6662q1Z2RkaO7cuSorKzvqnA8++ECTJ0+WJF199dVq27Zts9QKAAAAAACA5mPoYvRpaWm69dZba/a3b98uSXrxxRf10Ucf1bQvWLBA7dq1kyRlZ2crPT39qLAqNzdX1157rW655RYNGDBAHTp0UGlpqTZu3KitW7dKks466yy98MILTf22AAAAAAAAYABDg66CggKtWLHiqPasrKxaC8a7XK5fHSsxMVH33nuvUlNTtW3bNqWlpam8vFyxsbG68MILNWHCBF111VWyWlvdRWwAAAAAAACoA0ODrlGjRsnn89XrnOnTp2v69OlHtcfExOjxxx9vpMoAAAAAAADQ2nB5EwAAAAAAAEyBoAsAAAAAAACmQNAFAAAAAAAAUyDoAgAAAAAAgCkQdAEAAAAAAMAUCLoAAAAAAABgCgRdAAAAAAAAMAWCLgAAAAAAAJgCQRcAAAAAAABMgaALAAAAAAAApkDQBQAAAAAAAFMg6AIAAAAAAIApEHQBAAAAAADAFAi6AAAAAAAAYAoEXQAAAAAAADAFgi4AAAAAAACYAkEXAAAAAAAATIGgCwAAAAAAAKZA0AUAAAAAAABTIOgCAAAAAACAKRB0AQAAAAAAwBQIugAAAAAAAGAKBF0AAAAAAAAwBYIuAAAAAAAAmAJBFwAAAAAAAEyBoAsAAAAAAACmQNAFAAAAAAAAUyDoAgAAAAAAgCkQdAEAAAAAAMAUCLoAAAAAAABgCgRdAAAAAAAAMAWCLgAAAAAAAJgCQRcAAAAAAABMgaALAAAAAAAApkDQBQAAAAAAAFMg6AIAAAAAAIApEHQBAAAAAADAFAi6AAAAAAAAYAoEXQAAAAAAADAFgi4AAAAAAACYAkEXAAAAAAAATIGgCwAAAAAAAKZA0AUAAAAAAABTIOgCAAAAAACAKRB0AQAAAAAAwBQIugAAAAAAAGAKBF0AAAAAAAAwBYIuAAAAAAAAmAJBFwAAAAAAAEyBoAsAAAAAAACmQNAFAAAAAAAAUyDoAgAAAAAAgCkQdAEAAAAAAMAUCLoAAAAAAABgCgRdAAAAAAAAMAWCLgAAAAAAAJgCQRcAAAAAAABMgaALAAAAAAAApkDQBQAAAAAAAFMg6AIAAAAAAIApEHQBAAAAAADAFAi6AAAAAAAAYAoEXQAAAAAAADAFgi4AAAAAAACYAkEXAAAAAAAATIGgCwAAAAAAAKZA0AUAAAAAAABTIOgCAAAAAACAKRB0AQAAAAAAwBQIugAAAAAAAGAKBF0AAAAAAAAwBYIuAAAAAAAAmAJBFwAAAAAAAEyBoAsAAAAAAACmQNAFAAAAAAAAUyDoAgAAAAAAgCkYGnSlp6fr2Wef1cSJE9WnTx8FBATIYrHo0UcfbdC4X3zxhc4//3zFxsYqKChIPXv21AMPPKCioqJGqhwAAAAAAAAtTYCRL/7CCy9o5syZjTrmP//5T02dOlUWi0UjR45UfHy8vvvuOz322GN67733tHTpUsXGxjbqawIAAAAAAMB4hl7R1bt3b919992aN2+eNm3apGuvvbZB461Zs0Z33XWXbDabPv74Y3377bd6++23tX37do0ZM0bp6en6wx/+0EjVAwAAAAAAoCUx9IquyZMn19q3WhuWu82YMUM+n0833HCDzjvvvJr24OBgvfzyy+rcubPee+89bd68WT179mzQawEAAAAAAKBlMc1i9OXl5fr4448lSRMmTDjqeMeOHTV8+HBJ0oIFC5q1NgAAAAAAADQ90wRdW7ZsUUlJiSRp0KBBx+xT3b5mzZpmqwsAAAAAAADNwzRB186dOyVJkZGRCgsLO2afxMTEWn0BAAAAAABgHoau0dWYCgsLJUkhISHH7RMaGipJKigoOOFYLpdLLperZr+6v9vtltvtbmipQItVPb+Z5zAz5jn8BXMd/oK5Dn/APEdj8Xh9qvD65PF65fFKXp9PHq+v5tnj9cnj88nr1RHblc/Vx70+HfMcj7fqvGOe45PniGMVR5xf2Vfyen36bd+IBr9H0wRdjWnGjBl6+OGHj2r/+uuvFRwcbEBFQPNasmSJ0SUATY55Dn/BXIe/YK7DHzDPjeXxSR6vVOGTKrxV+1XbR7ZV+Cw/Hz/qmCqDoppnS1XII3lU9Vzr+M/PtdssdexXu80ni9H/GU8oLvvEFybVhWmCruqfKxYXFx+3T1FRkSQpPDz8hGPdd999mjp1as1+QUGBEhMTddZZZykmJqYRqgVaJrfbrSVLlujss8+W3W43uhygSTDP4S+Y6/AXzHX4A3+f5z6fT26PT64Kr1wVHrkqvCpz/7ztqtqubPMes195zbZXZe7KY+VV/dwen9yeyv1yj0/lHq/cVY/yCl/Nttdn9H+JpmWzWiofFslqtchmqdy3WiwKsFqq2n4+dmQfm9Uiq1XHOefnY0efU70vWS0WnTmsTYPfh2mCruTkZElSXl6eCgsLj7lOV2ZmZq2+x+N0OuV0Oo9qt9vtfvmhAv/DXIc/YJ7DXzDX4S+Y6/AHLXWe+3w+lbm9KimvUEm5R6Vuj0rLPVXbVW1V7SVV7WVuz8/9a/pWb1fU9K8OqlpiyOSwWWW3WeQIsMpuq3w4q7Yr2yw1246q5wCbVfaqkCeg6nybtbKfzWqRvaq9ss2iAKtVAdXPVkvltq1qu3r/yD42i+zWn8+vHjugettqlc1WFUJVB05VAVRLkJOT0+AxTBN09ejRQ8HBwSopKdGqVat01llnHdVn1apVkqSUlJTmLg8AAAAAgBbD4/WpuLxCRWUVKnZ
2023-01-17 12:26:02 +01:00
"text/plain": [
2023-05-11 10:37:43 +02:00
"<Figure size 1400x700 with 1 Axes>"
2023-01-17 12:26:02 +01:00
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plot(lambda x: math.tanh(x))"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* Tangens hiperboliczny przyjmuje wartości z przedziału $(-1, 1)$."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* Powstaje z funkcji logistytcznej przez przeskalowanie i przesunięcie."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* Daje szybszą zbieżność niż funkcja logistyczna dzięki temu, że przedział wartości jest symetryczny względem zera."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* Ze względu na ograniczony przedział wartości, obie funkcje (logistyczna i $\\tanh$) są podatne na problem zanikającego gradientu."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Problem zanikającego gradientu (*vanishing gradient problem*)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* Sigmoidalne funkcje aktywacji ograniczają wartości na wyjściach neuronów do niewielkich przedziałów ($(-1, 1)$, $(0, 1)$ itp.).\n",
"* Jeżeli sieć ma wiele warstw, to podczas propagacji wstecznej mnożymy przez siebie wiele małych wartości → obliczony gradient jest mały.\n",
"* Im więcej warstw, tym silniejszy efekt zanikania."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"#### Sposoby na zanikający gradient\n",
"\n",
"* Modyfikacja algorytmu optymalizacji (*RProp*, *RMSProp*)\n",
"* Użycie innej funckji aktywacji (ReLU, softplus)\n",
"* Dodanie warstw *dropout*\n",
"* Nowe architektury (LSTM itp.)\n",
"* Więcej danych, zwiększenie mocy obliczeniowej"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### ReLU (*Rectifier Linear Unit*)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"$$ g(x) = \\max(0, x) $$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"#### ReLU –
]
},
{
"cell_type": "code",
2023-05-11 10:37:43 +02:00
"execution_count": 12,
2023-01-17 12:26:02 +01:00
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"data": {
2023-05-11 10:37:43 +02:00
"image/png": "iVBORw0KGgoAAAANSUhEUgAABLoAAAJpCAYAAACq4lvbAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA7iklEQVR4nO3de5zVdb3v8fcgwwiD3I8XBKS8YUnGiB7bYCFoXkoiO7r3ITPcmrvIrYRtBS/nYLrFOtYJNdydo6Wp7G5uDJM84iWPlzaB0NUtJJmA7jQgGW6No875w8PsiIsMs2bWzG89n4/HPB6s3/qt7/qsHt9GfLnWb1U1NTU1BQAAAAA6uS7lHgAAAAAASkHoAgAAAKAQhC4AAAAACkHoAgAAAKAQhC4AAAAACkHoAgAAAKAQhC4AAAAACkHoAgAAAKAQhC4AAAAACqGsoWvZsmW56aabMmnSpAwfPjxdu3ZNVVVVrr322j1ab8aMGamqqtrlz7PPPlviVwEAAABAR9C1nE9+yy23ZNasWSVf96ijjsp73/veHd7Xu3fvkj8fAAAAAOVX1tB15JFH5vOf/3xGjBiRurq6XHfddbnzzjtbve6ECRMyY8aM1g8IAAAAQKdR1tB1/vnnb3O7SxeXDAMAAABgzyhLAAAAABRCWd/R1VaWLFmSadOmZd26dendu3dGjBiR008/Pfvss0+5RwMAAACgjRQydN1333257777tjnWu3fv3HjjjTnnnHPe9vENDQ1paGhovv3mm29m3bp16d+/f6qqqko+LwAAAECla2pqyoYNGzJw4MA9vrxVoULXwQcfnOuuuy6nnnpqDjrooCTJM888k+uvvz4//OEP88lPfjJ77bVXPv7xj+9ynZkzZ+bqq69uj5EBAAAA+DOrVq3KoEGD9uixVU1NTU0lnmePTZo0KXfccUeuueaaXHnllSVd+6KLLspNN92U//Sf/lNWr16dbt267fTcv3xH1/r16zNkyJAsX748/fr1K+lc0JE0Njbm0UcfzQknnJDq6upyjwNtwj6nUtjrVAp7vX1s2NKYT8/5WZ759w3p071rvn72iBy2n0vDtBf7nEqxbt26HHbYYXn11VfTu3fvPVqjUO/o2pUZM2Zk9uzZ+cMf/pCFCxfm+OOP3+m5NTU1qamp2e54v3790r9//7YcE8qqsbExPXr0SP/+/f0DlMKyz6kU9jqVwl5ve+u3NOaif16YZ9e9kf59e2fOp47LEQf0KvdYFcU+p9K05rJRFfOti/369cu+++6bJFm9enWZpwEAAOj41m9pzCduW5hfrF6fvj2qRS6gw6uY0PXGG29k/fr1SeLbFwEAAN6GyAV0RhUTuubNm5fNmzenqqoqI0eOLPc4AAAAHZbIBXRWnS503XzzzRk2bFjOOeecbY6vXLkyd911V/70pz9t95h77703559/fpLk4x//ePbff/92mRUAAKCzEbmAzqysF6NfsmRJJk+e3Hx7xYoVSZKvf/3r+eEPf9h8fO7cuTnggAOSJGvWrMmyZcu2i1Xr1q3LJz7xiXzmM5/JiBEjcuCBB2bLli155pln8pvf/CZJcsIJJ+SWW25p65cFAADQKYlcQGdX1tBVX1+fhQsXbnd89erV21wwvqGh4W3XGjx4cC677LIsWrQozz33XJYsWZLXXnstAwYMyIc//OFMnDgxf/3Xf50uXTrdm9gAAADanMgFFEFZQ9eYMWPS1NTUosfMmDEjM2bM2O54//79c/3115doMgAAgMohcgFF4e1NAAAAFUzkAopE6AIAAKhQIhdQNEIXAABABRK5gCISugAAACqMyAUUldAFAABQQUQuoMiELgAAgAohcgFFJ3QBAABUAJELqARCFwAAQMGJXEClELoAAAAKTOQCKonQBQAAUFAiF1BphC4AAIACErmASiR0AQAAFIzIBVQqoQsAAKBARC6gkgldAAAABSFyAZVO6AIAACgAkQtA6AIAAOj0RC6AtwhdAAAAnZjIBfAfhC4AAIBOSuQC2JbQBQAA0AmJXADbE7oAAAA6GZELYMeELgAAgE5E5ALYOaELAACgkxC5AHZN6AIAAOgERC6Atyd0AQAAdHAiF8DuEboAAAA6MJELYPcJXQAAAB2UyAXQMkIXAABAByRyAbSc0AUAANDBiFwAe0boAgAA6EBELoA9J3QBAAB0ECIXQOsIXQAAAB2AyAXQekIXAABAmYlcAKUhdAEAAJSRyAVQOkIXAABAmYhcAKUldAEAAJSByAVQekIXAABAOxO5ANqG0AUAANCORC6AtiN0AQAAtBORC6BtCV0AAADtQOQCaHtCFwAAQBsTuQDah9AFAADQhkQugPYjdAEAALQRkQugfQldAAAAbUDkAmh/QhcAAECJiVwA5SF0AQAAlJDIBVA+QhcAAECJiFwA5SV0AQAAlIDIBVB+QhcAAEAriVwAHYPQBQAA0AoiF0DHIXQBAADsIZELoGMRugAAAPaAyAXQ8QhdAAAALSRyAXRMQhcAAEALiFwAHZfQBQAAsJtELoCOTegCAADYDSIXQMcndAEAALwNkQugcxC6AAAAdkHkAug8hC4AAICdELkAOhehCwAAYAdELoDOR+gCAAD4CyIXQOckdAEAAPwZkQug8xK6AAAA/j+RC6BzE7oAAAAicgEUgdAFAABUPJELoBiELgAAoKKJXADFIXQBAAAVS+QCKBahCwAAqEgiF0DxCF0AAEDFEbkAiknoAgAAKorIBVBcQhcAAFAxRC6AYhO6AACAiiByARSf0AUAABSeyAVQGYQuAACg0EQugMohdAEAAIUlcgFUFqELAAAoJJELoPIIXQAAQOGIXACVqayha9myZbnpppsyadKkDB8+PF27dk1VVVWuvfbaVq370EMP5bTTTsuAAQPSvXv3DBs2LFdccUU2btxYoskBAICOSuQCqFxdy/nkt9xyS2bNmlXSNf/n//yfmTp1aqqqqnL88cdnv/32y+OPP57rrrsu99xzT5544okMGDCgpM8JAAB0DCIXQGUr6zu6jjzyyHz+85/P3XffnX/7t3/LJz7xiVatt3Tp0lxyySXZa6+9cv/99+exxx7Ld7/73axYsSLjxo3LsmXL8ulPf7pE0wMAAB2JyAVAWd/Rdf75529zu0uX1nW3mTNnpqmpKeeee25OPfXU5uM9evTIbbfdlne+852555578uyzz2bYsGGtei4AAKDjqN/SmHO/tUTkAqhwhbkY/WuvvZb7778/STJx4sTt7j/ooIMyatSoJMncuXPbdTYAAKDtbH49mXTH0yIXAMUJXcuXL8/mzZuTJCNHjtzhOVuPL126tN3mAgAA2k79lsbMfmav/PLFepELgOKErueffz5J0qdPn+yzzz47PGfw4MHbnAsAAHRe67c0ZtIdT2fVpiqRC4AkZb5GVylt2LAhSVJbW7vTc3r27Jkkqa+v3+VaDQ0NaWhoaL699fzGxsY0Nja2dlTosLbub/ucIrPPqRT2OkVX//8j1y9frE9t16Z84xPvzSEDutvzFJLf6VSKUuzxwoSuUpo5c2auvvrq7Y4/+uij6dGjRxkmgva1YMGCco8Abc4+p1LY6xTR5teT2c/slVWbqlLbtSmffdcbWfmLn2TlL8o9GbQtv9Mpuq2XpGqNwoSurR9X3LRp007P2bhxY5KkV69dv515+vTpmTp1avPt+vr6DB48OCeccEL69+9fgmmhY2psbMyCBQty0kknpbq6utzjQJuwz6kU9jpFVd/8ccW3rsn1jU+8Nyt/8RN7nULzO51KsXbt2lavUZjQNXTo0CTJq6++mg0bNuzwOl2rVq3a5tydqampSU1NzXbHq6ur/VKhItjrVAL7nEphr1Mk67c05txvLdnmwvOHDOielb+w16kM9jlFV4r9XZiL0R9++OHNHytcvHjxDs/Zeryurq7d5gIAAFpv/ZbGfOK2hfnF6vUuPA/AThUmdHXr1i0f+tCHkiRz5szZ7v4XXnghTz31VJLkox/9aLvOBgAA7DmRC4Dd1elC180335xhw4blnHPO2e6+adOmpaqqKt/85jfzwAMPNB/fvHlzzjvvvLz
2023-01-17 12:26:02 +01:00
"text/plain": [
2023-05-11 10:37:43 +02:00
"<Figure size 1400x700 with 1 Axes>"
2023-01-17 12:26:02 +01:00
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plot(lambda x: max(0, x))"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"#### ReLU –
"* Mniej podatna na problem zanikającego gradientu (*vanishing gradient*) niż funkcje sigmoidalne, dzięki czemu SGD jest szybciej zbieżna.\n",
"* Prostsze obliczanie gradientu.\n",
"* Dzięki zerowaniu ujemnych wartości, wygasza neurony, „rozrzedzając” sieć (*sparsity*), co przyspiesza obliczenia."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"#### ReLU –
"* Dla dużych wartości gradient może „eksplodować”.\n",
"* Za dużo „wygaszonych” neuronów może mieć negatywny wpływ na przepływ gradientu do poprzedzających warstw."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Inne funkcje aktywacji\n",
"\n",
"W literaturze zaproponowano też inne funkcje aktywacji, które są lepsze od ReLU pod niektórymi względami:\n",
"* softplus\n",
"* ELU\n",
"* leaky ReLU\n",
"* SiLU / swish\n",
"\n",
"W praktyce jednak ReLU pozostaje najpopularniejszą funkcją aktywacji ze względu na jej prostotę, skuteczność i szybkość obliczania."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
2023-04-14 10:47:15 +02:00
"## 10.3. Wielowarstwowe sieci neuronowe\n",
2023-01-17 12:26:02 +01:00
"\n",
"czyli _Artificial Neural Networks_ (ANN) lub _Multi-Layer Perceptrons_ (MLP)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"![Rys. 9.5. Wielowarstwowa sieć neuronowa](nn1.png \"Rys. 9.5. Wielowarstwowa sieć neuronowa\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"#### Architektura sieci\n",
"\n",
"* Sieć neuronowa jako graf neuronów. \n",
"* Organizacja sieci przez warstwy.\n",
"* Najczęściej stosowane są sieci jednokierunkowe i gęste.\n",
"* $n$-warstwowa sieć neuronowa ma $n+1$ warstw (nie liczymy wejścia).\n",
"* Rozmiary sieci określane poprzez liczbę neuronów lub parametrów."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Sieć neuronowa jednokierunkowa (*feedforward*)\n",
"\n",
"* Mając daną $n$-warstwową sieć neuronową oraz jej parametry $\\Theta^{(1)}, \\ldots, \\Theta^{(L)} $ oraz $\\beta^{(1)}, \\ldots, \\beta^{(L)} $, liczymy:<br/><br/> \n",
"$$a^{(l)} = g^{(l)}\\left( a^{(l-1)} \\Theta^{(l)} + \\beta^{(l)} \\right). $$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"![Rys. 11.6. Wielowarstwowa sieć neuronowa - feedforward](nn2.png \"Rys. 11.6. Wielowarstwowa sieć neuronowa - feedforward\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* Funkcje $g^{(l)}$ to tzw. **funkcje aktywacji**.<br/>\n",
"Dla $i = 0$ przyjmujemy $a^{(0)} = \\mathrm{x}$ (wektor wierszowy cech) oraz $g^{(0)}(x) = x$ (identyczność)."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* Parametry $\\Theta$ to wagi na połączeniach miedzy neuronami dwóch warstw.<br/>\n",
"Rozmiar macierzy $\\Theta^{(l)}$, czyli macierzy wag na połączeniach warstw $a^{(l-1)}$ i $a^{(l)}$, to $\\dim(a^{(l-1)}) \\times \\dim(a^{(l)})$."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* Parametry $\\beta$ zastępują tutaj dodawanie kolumny z jedynkami do macierzy cech.<br/>Macierz $\\beta^{(l)}$ ma rozmiar równy liczbie neuronów w odpowiedniej warstwie, czyli $1 \\times \\dim(a^{(l)})$."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* **Klasyfikacja**: dla ostatniej warstwy $L$ (o rozmiarze równym liczbie klas) przyjmuje się $g^{(L)}(x) = \\mathop{\\mathrm{softmax}}(x)$.\n",
"* **Regresja**: pojedynczy neuron wyjściowy jak na obrazku. Funkcją
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* Pozostałe funkcje aktywacji najcześciej mają postać sigmoidy, np. sigmoidalna, tangens hiperboliczny.\n",
"* Mogą mieć też inny kształt, np. ReLU, leaky ReLU, maxout."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Uczenie wielowarstwowych sieci neuronowych"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Mając algorytm SGD oraz gradienty wszystkich wag, moglibyśmy trenować każdą sieć."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* Niech:\n",
"$$\\Theta = (\\Theta^{(1)},\\Theta^{(2)},\\Theta^{(3)},\\beta^{(1)},\\beta^{(2)},\\beta^{(3)})$$\n",
"\n",
"* Funkcja sieci neuronowej z grafiki:\n",
"\n",
"$$\\small h_\\Theta(x) = \\tanh(\\tanh(\\tanh(x\\Theta^{(1)}+\\beta^{(1)})\\Theta^{(2)} + \\beta^{(2)})\\Theta^{(3)} + \\beta^{(3)})$$\n",
"* Funkcja kosztu dla regresji:\n",
"$$J(\\Theta) = \\dfrac{1}{2m} \\sum_{i=1}^{m} (h_\\Theta(x^{(i)})- y^{(i)})^2 $$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Jak obliczymy gradienty?\n",
"\n",
"$$\\nabla_{\\Theta^{(l)}} J(\\Theta) = ? \\quad \\nabla_{\\beta^{(l)}} J(\\Theta) = ?$$\n",
"\n",
"* Postać funkcji kosztu zależna od wybranej architektury sieci oraz funkcji aktywacji."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"$$\\small J(\\Theta) = \\frac{1}{2}(a^{(L)} - y)^2 $$\n",
"$$\\small \\dfrac{\\partial}{\\partial a^{(L)}} J(\\Theta) = a^{(L)} - y$$\n",
"\n",
"$$\\small \\tanh^{\\prime}(x) = 1 - \\tanh^2(x)$$"
]
}
],
"metadata": {
"celltoolbar": "Slideshow",
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
2023-12-14 18:04:47 +01:00
"version": "3.10.12"
2023-01-17 12:26:02 +01:00
},
"livereveal": {
"start_slideshow_at": "selected",
"theme": "white"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
