2021-03-02 08:32:40 +01:00
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
2021-04-14 08:03:54 +02:00
"## Uczenie maszynowe –
2021-05-26 08:08:01 +02:00
"# 11. Wielowarstwowe sieci neuronowe w praktyce"
2021-03-02 08:32:40 +01:00
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## 11.1. Funkcje aktywacji"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* Każda funkcja aktywacji ma swoje zalety i wady.\n",
"* Różne rodzaje funkcji aktywacji nadają się do różnych zastosowań."
]
},
{
"cell_type": "code",
2021-05-26 11:25:35 +02:00
"execution_count": 3,
2021-03-02 08:32:40 +01:00
"metadata": {
"slideshow": {
"slide_type": "notes"
}
},
2021-05-26 11:25:35 +02:00
"outputs": [],
2021-03-02 08:32:40 +01:00
"source": [
"%matplotlib inline\n",
"\n",
"import math\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"import random\n",
"\n",
"from IPython.display import YouTubeVideo"
]
},
{
"cell_type": "code",
2021-05-26 11:25:35 +02:00
"execution_count": 4,
2021-03-02 08:32:40 +01:00
"metadata": {
"slideshow": {
"slide_type": "notes"
}
},
"outputs": [],
"source": [
"def plot(fun):\n",
" x = np.arange(-3.0, 3.0, 0.01)\n",
" y = [fun(x_i) for x_i in x]\n",
" fig = plt.figure(figsize=(14, 7))\n",
" ax = fig.add_subplot(111)\n",
" fig.subplots_adjust(left=0.1, right=0.9, bottom=0.1, top=0.9)\n",
" ax.set_xlim(-3.0, 3.0)\n",
" ax.set_ylim(-1.5, 1.5)\n",
" ax.grid()\n",
" ax.plot(x, y)\n",
" plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Funkcja logistyczna"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"$$ g(x) = \\frac{1}{1 + e^{-x}} $$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* Przyjmuje wartości z przedziału $(0, 1)$."
]
},
{
"cell_type": "code",
2021-05-26 11:25:35 +02:00
"execution_count": 5,
2021-03-02 08:32:40 +01:00
"metadata": {
"slideshow": {
2021-05-26 11:25:35 +02:00
"slide_type": "subslide"
2021-03-02 08:32:40 +01:00
}
},
"outputs": [
{
"data": {
2021-05-26 11:25:35 +02:00
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA1cAAAG2CAYAAACTRXz+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deaxe52Hf+d9D3n1feLmvkihqtWRZomQ4C90kjmO4cdOpW3s6TdrOjCadBtP5o0CDyaDBdFCggwEG0046TYVp0AboNBOgdWNM3CZNO4zTArZk2ZJtbZbMRaQoicvdeDfe7cwf78srUrqUKN8j3kvy8wFevO9533PveWg8pvjFOed5S1VVAQAAYG02rfcAAAAAbgXiCgAAoAbiCgAAoAbiCgAAoAbiCgAAoAbiCgAAoAa1xFUp5bdKKWdLKd+/xudHSikTpZTnmo+/XcdxAQAANoqWmn7PP03yG0l++332+ZOqqj5f0/EAAAA2lFrOXFVV9fUko3X8LgAAgJvRjbzn6pOllOdLKf+mlHL/DTwuAADAR66uywI/yLeT7KuqaqqU8rkk/zrJwdV2LKU8meTJJOno6PjE3r17b9AQuVksLy9n0yZrsXA184JrMTdYjXnBtZgbrOYHP/jB+aqqRj5ov1JVVS0HLKXsT/L/VlX1wHXseyLJo1VVnX+//Q4dOlS98sortYyPW8fRo0dz5MiR9R4GG4x5wbWYG6zGvOBazA1WU0p5tqqqRz9ovxuS5aWU7aWU0nx9uHncCzfi2AAAADdCLZcFllL+RZIjSbaUUk4n+fUkrUlSVdVvJvlzSf5aKWUxyWySL1V1nTIDAADYAGqJq6qqvvwBn/9GGku1AwAA3JLcrQcAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFCDWuKqlPJbpZSzpZTvX+PzUkr5B6WU10op3y2lPFLHcQEAADaKus5c/dMkn32fz38uycHm48kk/6im4wIAAGwItcRVVVVfTzL6Prt8IclvVw3fSDJQStlRx7EBAAA2ght1z9WuJKeu2D7dfA8AAOCW0HKDjlNWea9adcdSnkzj0sGMjIzk6NGjH+GwuBlNTU2ZF7yHecG1mBusxrzgWswN1uJGxdXpJHuu2N6d5MxqO1ZV9VSSp5Lk0KFD1ZEjRz7ywXFzOXr0aMwL3s284FrMDVZjXnAt5gZrcaMuC/xqkl9srhr4RJKJqqrevEHHBgAA+MjVcuaqlPIvkhxJsqWUcjrJrydpTZKqqn4zydeSfC7Ja0lmkvyVOo4LAACwUdQSV1VVffkDPq+S/PU6jgUAALAR3ah7rgAAANZdVVW5tLic6UuLmZlfyvT8YqYvLWb60lJm5hvP0/PvbP/s/duv+3eLKwAAYMO7tLiUqbnFTF1azMXm89TcYqbnr95+5/OFq96bmV9aeV5aXnXh8vcoJdkz1HXdYxRXAADAR+bymaLJ2YVMzC5kcm4hk7OLmZxrbs8u5OIVETQ1t3j1dvP1/NLyBx5r86aSnvaW9LS3pLej8TzQ1ZbdQ13pbtucrrbGe13tm9Pd1pKuts3N7ZaVz7vbN6e7vSXdbS3paN2UUkr+wnX+WcUVAADwvhaWGnE0Obf4vpE0Obd4xevm57MLHxhG7S2bVmKop/m8c6DzPe+tPDpa0nvl+x0t6W1vXYmh9SKuAADgNrG8XOXi3GLGZ+czNrOQsZn5TDSfx2YWMj4zn/Hm9vgVn1+8tPi+v7dlU0l/Z2v6Lj86WrJrsLPxXkdr+jpb0tfR+s4+HS0rr3s7WtLesvkG/S/w0RJXAABwE6qqKpNzi7kwdSmj0/O5MD2fC1PzGZ2+1AyjRiyNzcxnfHYh483ta91uVErS19Gawa7W9He1ZbinLXdt7clAV2sGOtsy0LVaJDVer/cZo41CXAEAwAZwOZZGp+dzYepSLkzPr/L6ckRdytjMfBaWVi+lztbNGexqzUBXWwa7W7OjvzMDXa0Z7GpE0kBX2zufN5/7O1uzeZNAWgtxBQAAH5GqqjI+s5BzU5dy7mLjcfbi3MrrC9PzOd882zQ6fe1Y6mlvyXBPW4a627JroCMf29WfoZ62DHe3Nd9vX3k92NWWjtZb4zK7m424AgCAD2luYakZSs1omrqUc5NzV0XU5fdXC6aO1k0Z6W3Plp727BroyIO7+jLc0wikoe62lddi6eYirgAAoGlxucob47N5a2Iub0/OvfPcfN2IqEurLvBQSjLc3Z6R3vZs7W3PwW29Geltz0jPO++NNB897S3uUboFiSsAAG55VVXl4qXFvD3xTii9E02X8vbkXN6cmMuFqUup/vA/XPWzbZs3ZVt/e7b3deTe7X35iYPvRNLleNra156hrra0bN60Tn9CNgJxBQDATW9uYSlnxmdzZnwuZ8Zn88b4bGN7YjZvjjciamZ+6T0/N9DVmu19HdnW15H7dvRlbuytPPHQPSvvbe/vyGBXq7NMXBdxBQDAhra8XOXC9Pw7wXRlPDVj6sL0/FU/U0qytbc9O/o7c8+O3vzkoZFsb8bS5edtfR3vuZfp6NHRHDm890b+8biFiCsAANbV0nKVtyfncmp0JqfGZnNqdOaqkDozMZf5xeWrfqarbXN2DXRm50BnHtjVn10DHdnZ3N410JltfR1pa3GJHjeWuAIA4CN1eTnyU2MzOTU6m9dHZ5qvZ3J6bDZvjM1mfumdeCol2dbbkV2DnXlw90B+9oGORkj1vxNPfZ0WhGDjEVcAAKzZ3MJSTo3ONMLpijNQrzcDaupdq+sNdrVmz1BX7tvRl8/cvy17h7qyZ7Are4a6snOgI+0tlh7n5iOuAAC4LrPzSzk5Op0T52dy4sJ0Tl6YzvHz0zl5YSZvTsxdtW9H66aVWHr8wFD2DDVeN97rTG9H6zr9KeCjI64AAFgxM7+YkxdmcuL8dE5cmLkqoN6avDqghrrbsn+4K5+8Yzj7t3Rn33BXdjfjaaSn3WV73HbEFQDAbWZxaTmvj87k2Lnp/PDcVI6dm87xC9M5cX46Zy9eumrfLT1t2TfcnU/dtSX7h7uyb0t3Dgx3Z+9wV/o7nX2CK4krAIBb1MTMQl47N5Vj56byw3PTzeepvD46k4WlamW/LT1tObClOz9x90j2D3dl/5bu7B9unIly+R5cP3EFAHATW1xazumx2ZUzUFc+X/ndT62bS/YNd+eurT35zP3bc+dIT+4Y6c6dW3rS3yWgoA7iCgDgJrC4tJwTF2by6tsX84O3p/Lq2Yt59e2pHD8/fdUy5sPdbbljpDs/c9+2RjyN9OSOkZ7sGexMy2bf+wQfJXEFALCBXI6o1842IuoHbzci6tj5qasu5dsz1Jm7t/bmyD0juXOkp/nozkBX2zqOHm5v4goAYB0sLi3n5Og7Z6J+8PbFvHa2cUnflWeiLkfUp+/ZmoNbe3L3tt7cubU7XW3+GQcbjf9XAgB8xM5PXcrLb17My29N5qXm86tnpzK/+N6I+slDI7l7a6+IgpuQ/7cCANRkbmEpr52dystvXczLb042nt+azPmpdxaW2Nrbnnt29OVTd23JoW29ObitJ3dt7RFRcAvw/2IAgA+pqqqcmZi7IqAaMXXs/HSWlhv3RbW3bMqh7b359KGtuWdHX+7d3pt
2021-03-02 08:32:40 +01:00
"text/plain": [
2021-05-26 11:25:35 +02:00
"<Figure size 1008x504 with 1 Axes>"
2021-03-02 08:32:40 +01:00
]
},
2021-05-26 11:25:35 +02:00
"metadata": {
"needs_background": "light"
},
2021-03-02 08:32:40 +01:00
"output_type": "display_data"
}
],
"source": [
2021-05-26 11:25:35 +02:00
"# Wykres funkcji logistycznej\n",
2021-03-02 08:32:40 +01:00
"plot(lambda x: 1 / (1 + math.exp(-x)))"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Tangens hiperboliczny"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"$$ g(x) = \\tanh x = \\frac{e^{x} - e^{-x}}{e^{x} + e^{-x}} $$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* Przyjmuje wartości z przedziału $(-1, 1)$.\n",
"* Powstaje z funkcji logistycznej przez przeskalowanie i przesunięcie."
]
},
{
"cell_type": "code",
2021-05-26 11:25:35 +02:00
"execution_count": 6,
2021-03-02 08:32:40 +01:00
"metadata": {
"slideshow": {
2021-05-26 11:25:35 +02:00
"slide_type": "subslide"
2021-03-02 08:32:40 +01:00
}
},
"outputs": [
{
"data": {
2021-05-26 11:25:35 +02:00
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA1cAAAG2CAYAAACTRXz+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3Rc1b238e+eGfVqWcVNcrfcccM23aYaQgKEEiB0Bwcu3EB679yQSl4SSIAQiqlxQjPBdBAdV9yr3GXZVu9tyn7/0NgII9mydaQzkp7PWrOmHc35kbWX5Cdn5oyx1goAAAAA0DEetwcAAAAAgJ6AuAIAAAAABxBXAAAAAOAA4goAAAAAHEBcAQAAAIADiCsAAAAAcIAjcWWMecgYU2SMWdvG87OMMZXGmJXhy8+c2C8AAAAARAqfQ6/ziKR7JM0/zDbvWWvPd2h/AAAAABBRHDlyZa19V1KZE68FAAAAAN1RV37m6gRjzCpjzMvGmHFduF8AAAAA6HROvS3wSFZIGmytrTHGnCfpeUkjW9vQGDNP0jxJio2NnZqTk9NFI6K7CIVC8ng4Fws+i3WBtrA20BrWBdrC2kBrNm/eXGKtzTjSdsZa68gOjTFDJP3XWju+HdvukDTNWltyuO1yc3Ptpk2bHJkPPUdeXp5mzZrl9hiIMKwLtIW1gdawLtAW1gZaY4xZbq2ddqTtuiTLjTH9jDEmfHt6eL+lXbFvAAAAAOgKjrwt0BjzlKRZktKNMQWSfi4pSpKstfdJukTSzcaYgKR6SZdbpw6ZAQAAAEAEcCSurLVXHOH5e9R8qnYAAAAA6JH4tB4AAAAAOIC4AgAAAAAHEFcAAAAA4ADiCgAAAAAcQFwBAAAAgAOIKwAAAABwAHEFAAAAAA4grgAAAADAAcQVAAAAADiAuAIAAAAABxBXAAAAAOAA4goAAAAAHEBcAQAAAIADiCsAAAAAcABxBQAAAAAOIK4AAAAAwAHEFQAAAAA4gLgCAAAAAAcQVwAAAADgAOIKAAAAABxAXAEAAACAA4grAAAAAHAAcQUAAAAADiCuAAAAAMABxBUAAAAAOIC4AgAAAAAHEFcAAAAA4ADiCgAAAAAcQFwBAAAAgAOIKwAAAABwAHEFAAAAAA4grgAAAADAAcQVAAAAADiAuAIAAAAABxBXAAAAAOAA4goAAAAAHEBcAQAAAIADiCsAAAAAcABxBQAAAAAOIK4AAAAAwAHEFQAAAAA4gLgCAAAAAAcQVwAAAADgAOIKAAAAABxAXAEAAACAA4grAAAAAHAAcQUAAAAADiCuAAAAAMABxBUAAAAAOIC4AgAAAAAHEFcAAAAA4ADiCgAAAAAcQFwBAAAAgAOIKwAAAABwAHEFAAAAAA4grgAAAADAAcQVAAAAADiAuAIAAAAABzgSV8aYh4wxRcaYtW08b4wxfzHG5BtjVhtjpjixXwAAAACIFE4duXpE0pzDPH+upJHhyzxJf3dovwAAAAAQERyJK2vtu5LKDrPJBZLm22YfS0o1xvR3Yt8AAAAAEAl8XbSfgZJ2t7hfEH5sbxftHwAAAEAPZK1VyErBkFXIWgVDVkFrFQqFb4fvB0NWoZA+vW0/fT70mcc++1qjspLaPUtXxZVp5THb6obGzFPzWweVkZGhvLy8ThwL3VFNTQ3rAp/DukBbWBtoDesCbemNa8Naq6CVgiHJH5IC1ioQUotL831/OEz8rTwXCEkBK/lD4TixzY+FDrx2+PVDNhw34fsHnjsYNQfvKxxF+sy2oUN+NhR+vjPNHR/d7m27Kq4KJGW3uD9IUmFrG1prH5D0gCTl5ubaWbNmdfpw6F7y8vLEusChWBdoC2sDrWFdoC2RsDZCIat6f1B1TUHVNQVU2xhUQyCoBn9Qjf6QGvwH7odv+0Oq9wfV6A8evH9g+4PbBEKfed4fDKkpEFJjsPm2dThQfB4jn9fI5/GEr5tvez1GUV4Tvm6+7/N6FOMxn/2ZQ37e6zGK8njk9RpFeYy8Hs/B1/GF73s9ksdj5DXNj3vC1wcvxjQ/79Gnzx147MDPHHxMBx8bkp6gn7X3v9vZ/xnbtFDSrcaYpyXNkFRpreUtgQAAAOj2rLWqawqquiGg6ga/qhoCqmrwq7ohoLrGgGqbgp9eh2Oprqntx+uagsc0R7TPo1ifR7FRXsVGeRUX5VVslEcxUV6lxEUpNilGsVFexfg8ionyKMrrUbTPo2hv8yXqwO0W11EHr42ifR7FfOax1reP8hoZ09ob13o+R+LKGPOUpFmS0o0xBZJ+LilKkqy190laJOk8SfmS6iRd78R+AQAAACf4gyGV1zVpT3VIi7eVqrzOr8r6JlXVfz6Yqhv8zY83HrgfUDB05EM/0V6P4mO8Soj2KT7aq/gYnxKivUqNj1ZCjFfx0c33Dzx+4DouHEux4Vj6zG2f92AweTy9M2giiSNxZa294gjPW0m3OLEvAAAA4HCCIauy2iYVVzeqpKZR5XVNKq9tUnmdXxV1zdfldU2qaHFd0xj49AU++Pgzr2eMlBjjU3JslJJim6/7p8RqVGyikuOaH0uK/fQ6ucV1QoxPCdE+xUV7Fe1z6luQEKm66m2BAAAAwDELhaxKWwTTgUvz/aYWtxtVVtuktg4kJcf61CchWqnx0eqbGK0RmYlKjY9Sn/ho9YmPUuGOfJ00bVLzYwnRzYEU7eOoENqFuAIAAICr/MGQiqobta+yQfsqG7S3sr75uqpB+ysbtLeyQfurGhRopZhifB5lJMUoPTFGg/rEa3JOqjISY5Qefiw9MUZpCc3hlBIXJZ/38EeP8hp36OSR6Z31n4oejrgCAABAp2rwB1VQXqfd5fUqKKtTQXm9dpfXaU95vfZWNqi4pvFzZ6uLjfKof0qc+iXHasbQNPVLiVVWcqwykmIOxlR6YrQSY3y99uQJiDzEFQAAADrEWqvi6kZtK6nVjpJa7WoRUAXl9SqubvzM9tE+jwb1idPA1DiN7pesfimxBy/9U2LVPzlOyXFEE7of4goAAADtUlnv1/aSWm0vqdH24lptK6nV9nBQ1bY4fbjXYzQgNVbZfeI1OzdD2X3ilZ0Wr+y0OA3qE6+MxBg+w4QeibgCAADAZ5TUNGrzvmpt3l+tzUU12rK/WtuKa1Va23RwG4+RBvWJ19D0BB0/JE3DMhI0pG+ChqYnqH9K7BE/2wT0RMQVAABAL1XV4NfGveGIOnipUVmLiEqJi9KorESdPS5LQ9MTNDQ9UUPTm49Exfi8Lk4PRB7iCgAAoBcorm7UusJKrSusOni9s7Tu4POJMT6NzErU2WOzNDIrSblZSRqVlaiMpBg++wS0E3EFAADQwxRVN2jlrgqt2fNpTO2v+vSkEjlp8Ro3IFmXTh2kcQNSNKpfkgakxBJRQAcRVwAAAN1YYyCo9YVV+mRXhT7ZXaFPdpWroLxeUvPnokZkJuqk4ekaOyBZ4wakaOyAZKXERbk8NdAzEVcAAADdyP6qBi3ZXqYVu8r1ya4KrS+sUlMwJEkakBKryTl9dN2JQzQ5J1XjBqQoNorPRQFdhbgCAACIUNZaFZTXa/H2Mi3ZXqrF28sOfk4qNsqjiYNSdf3JQzQ5u48m56QqKznW5YmB3o24AgAAiCC7y+r0QX6JFm8v0+JtpSqsbJAkpcZH6fghabp65mBNH5qmMf2TFcXpzoGIQlwBAAC4qKrBrw/zS/V+frHe31KiHeEjU+mJ0ZoxtK9uGpam6UPTNCoziS/eBSIccQUAANCFAsGQVhVU6N3NJXpvS7FWFVQqGLKKj/bqhGF9de2JQ3TKyHQNz0jk7H1AN0NcAQAAdLKqBr/e2VSstzYW6e1NRaqo88sYaeKgVN182nCdMjJdk3P6KNrH2/yA7oy4AgAA6AQ7S2v1xoYivblhv5ZsL1MgZNUnPkqn52bq9DGZOnlEulLjo90eE4CDiCsAAAAHWGu1fm+VFq3Zq1fW7tPW4lpJzd8zNfeUoTpzTJam5PSRl89NAT0WcQUAAHCMrLVaV9gcVIvW7NWO0jp5jDRjaF99dcZgnTEmU4P7Jrg9JoA
2021-03-02 08:32:40 +01:00
"text/plain": [
2021-05-26 11:25:35 +02:00
"<Figure size 1008x504 with 1 Axes>"
2021-03-02 08:32:40 +01:00
]
},
2021-05-26 11:25:35 +02:00
"metadata": {
"needs_background": "light"
},
2021-03-02 08:32:40 +01:00
"output_type": "display_data"
}
],
"source": [
2021-05-26 11:25:35 +02:00
"# Wykres funkcji tangensa hiperbolicznego\n",
2021-03-02 08:32:40 +01:00
"plot(lambda x: math.tanh(x))"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
2021-05-26 08:08:01 +02:00
"### ReLU (*Rectifier Linear Unit*)"
2021-03-02 08:32:40 +01:00
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"$$ g(x) = \\max(0, x) $$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"#### ReLU –
2021-05-26 11:25:35 +02:00
"* Mniej podatna na problem zanikającego gradientu (*vanishing gradient*) niż funkcje sigmoidalne, dzięki czemu SGD jest szybciej zbieżna.\n",
2021-03-02 08:32:40 +01:00
"* Prostsze obliczanie gradientu.\n",
2021-05-26 11:25:35 +02:00
"* Dzięki zerowaniu ujemnych wartości, wygasza neurony, „rozrzedzając” sieć (*sparsity*), co przyspiesza obliczenia."
2021-03-02 08:32:40 +01:00
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
2021-05-26 11:25:35 +02:00
"slide_type": "subslide"
2021-03-02 08:32:40 +01:00
}
},
"source": [
"#### ReLU –
"* Dla dużych wartości gradient może „eksplodować”.\n",
"* „Wygaszanie” neuronów."
]
},
{
"cell_type": "code",
2021-05-26 11:25:35 +02:00
"execution_count": 7,
2021-03-02 08:32:40 +01:00
"metadata": {
"slideshow": {
2021-05-26 11:25:35 +02:00
"slide_type": "subslide"
2021-03-02 08:32:40 +01:00
}
},
"outputs": [
{
"data": {
2021-05-26 11:25:35 +02:00
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA1cAAAG2CAYAAACTRXz+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAYtklEQVR4nO3da6xld3nf8d+DL3WF05KKCRjbBFq509KoSYPlgFK1x4ohxkVxiKAyqhqUqpoSBTV9ESlWUUG9ILWqVFUpKGakWAGSQi1RBys44RL3YJDq4oAM8QVTi9DOdAALcwkjaOjET1/MnuQw3mfmzOz/OXvtvT8f6cj7smb//y8ejv1lrb2mujsAAAAs5lnL3gAAAMA6EFcAAAADiCsAAIABxBUAAMAA4goAAGAAcQUAADDAkLiqqjur6smqeniX97eq6ptV9dDs5y0j1gUAAJiKSwd9zq8neXuSd5/jmI9396sHrQcAADApQ85cdff9Sb424rMAAABW0UF+5+rlVfWZqvqdqvobB7guAADAvht1WeD5fDrJD3b3yaq6JclvJblu3oFVdSTJkSS54oorXvrCF77wgLbIqnj66afzrGe5Fwvfy1ywG7PBPOYieeo7nW/9v85fvLzy/VfUsrczGWaDeT7/+c9/tbsPne+46u4hC1bVi5L8dnf/0B6O/WKS67v7q+c67vDhw/34448P2R/rY3t7O1tbW8veBhNjLtiN2WCeTZ6L7s6/+MDD+Y0H/nf+yd/9y7n95r+WKnF1xibPBrurqk919/XnO+5Asryqnl+z/9VW1Q2zdZ86iLUBADhNWMH+GnJZYFW9N8lWkudW1fEkb01yWZJ09x1JXpvk56vqVJLvJLmtR50yAwDgvIQV7L8hcdXdrz/P+2/P6Vu1AwBwwIQVHAzf1gMAWGPCCg6OuAIAWFPCCg6WuAIAWEPCCg6euAIAWDPCCpZDXAEArBFhBcsjrgAA1oSwguUSVwAAa0BYwfKJKwCAFSesYBrEFQDAChNWMB3iCgBgRQkrmBZxBQCwgoQVTI+4AgBYMcIKpklcAQCsEGEF0yWuAABWhLCCaRNXAAArQFjB9IkrAICJE1awGsQVAMCECStYHeIKAGCihBWsFnEFADBBwgpWj7gCAJgYYQWrSVwBAEyIsILVJa4AACZCWMFqE1cAABMgrGD1iSsAgCUTVrAexBUAwBIJK1gf4goAYEmEFawXcQUAsATCCtaPuAIAOGDCCtaTuAIAOEDCCtaXuAIAOCDCCtabuAIAOADCCtafuAIA2GfCCjaDuAIA2EfCCjaHuAIA2CfCCjaLuAIA2AfCCjaPuAIAGExYwWYSVwAAAwkr2FziCgBgEGEFm01cAQAMIKwAcQUAsCBhBSTiCgBgIcIKOENcAQBcJGEF7CSuAAAugrACziauAAAukLAC5hFXAAAXQFgBuxFXAAB7JKyAcxFXAAB7IKyA8xFXAADnIayAvRBXAADnIKyAvRJXAAC7EFbAhRBXAABzCCvgQokrAICzCCvgYogrAIAdhBVwscQVAMCMsAIWIa4AACKsgMWJKwBg4wkrYARxBQBsNGEFjCKuAICNJayAkYbEVVXdWVVPVtXDu7xfVfUrVfVEVX22qn50xLoAABdLWAGjjTpz9etJbj7H+69Kct3s50iSXx20LgDABRNWwH64dMSHdPf9VfWicxxya5J3d3cneaCqnlNVV3X3l0asDwCwV92d9zz63dx3TFgBYx3Ud66uTnJsx/Pjs9cAAA7MmTNW9x07JayA4YacudqDeb+1eu6BVUdy+tLBHDp0KNvb2/u4LVbRyZMnzQXPYC7YjdngjD87Y3UqN13dedkVX87HPvaVZW+LifE7g0UcVFwdT3LtjufXJDkx78DuPprkaJIcPny4t7a29n1zrJbt7e2YC85mLtiN2SDZecbq9KWAL7viy7nxxhuXvS0myO8MFnFQlwXek+RnZ3cNfFmSb/q+FQBwENy8AjgoQ85cVdV7k2wleW5VHU/y1iSXJUl335Hk3iS3JHkiybeT/NyIdQEAzkVYAQdp1N0CX3+e9zvJL4xYCwBgL4QVcNAO6rJAAIADI6yAZRBXAMBaEVbAsogrAGBtCCtgmcQVALAWhBWwbOIKAFh5wgqYAnEFAKw0YQVMhbgCAFaWsAKmRFwBACtJWAFTI64AgJUjrIApElcAwEoRVsBUiSsAYGUIK2DKxBUAsBKEFTB14goAmDxhBawCcQUATJqwAlaFuAIAJktYAatEXAEAkySsgFUjrgCAyRFWwCoSVwDApAgrYFWJKwBgMoQVsMrEFQAwCcIKWHXiCgBYOmEFrANxBQAslbAC1oW4AgCWRlgB60RcAQBLIayAdSOuAIADJ6yAdSSuAIADJayAdSWuAIADI6yAdSauAIADIayAdSeuAIB9J6yATSCuAIB9JayATSGuAIB9I6yATSKuAIB9IayATSOuAIDhhBWwicQVADCUsAI2lbgCAIYRVsAmE1cAwBDCCth04goAWJiwAhBXAMCChBXAaeIKALhowgrgz4grAOCiCCuA7yWuAIALJqwAnklcAQAXRFgBzCeuAIA9E1YAuxNXAMCeCCuAcxNXAMB5CSuA8xNXAMA5CSuAvRFXAMCuhBXA3okrAGAuYQVwYcQVAPAMwgrgwokrAOB7CCuAiyOuAIA/JawALp64AgCSCCuARYkrAEBYAQwgrgBgwwkrgDHEFQBsMGEFMI64AoANJawAxhJXALCBhBXAeOIKADaMsALYH+IKADaIsALYP0PiqqpurqrHq+qJqrp9zvtbVfXNqnpo9vOWEesCAHsnrAD216WLfkBVXZLkHUlekeR4kger6p7ufvSsQz/e3a9edD0A4MIJK4D9N+LM1Q1JnujuL3T3d5O8L8mtAz4XABhAWAEcjOruxT6g6rVJbu7ufzx7/g+T/Fh3v2nHMVtJ3p/TZ7ZOJPml7n5kl887kuRIkhw6dOild91110L7Y/2cPHkyV1555bK3wcSYC3az6bPR3XnPo9/NfcdO5ZYXX5bX/dXLhFXMBbszG8xz4403fqq7rz/fcQtfFphk3m/os4vt00l+sLtPVtUtSX4ryXXzPqy7jyY5miSHDx/ura2tAVtknWxvb8dccDZzwW42eTbOnLG675gzVmfb5Lng3MwGixhxWeDxJNfueH5NTp+d+lPd/UfdfXL2+N4kl1XVcwesDQDM4VJAgIM3Iq4eTHJdVb24qi5PcluSe3YeUFXPr9lv9Kq6YbbuUwPWBgDOIqwAlmPhywK7+1RVvSnJh5JckuTO7n6kqt44e/+OJK9N8vNVdSrJd5Lc1ot+2QsAeAZhBbA8I75zdeZSv3vPeu2OHY/fnuTtI9YCAOYTVgDLNeQvEQYAlktYASyfuAKAFSesAKZBXAHAChNWANMhrgBgRQkrgGkRVwCwgoQVwPSIKwBYMcIKYJrEFQCsEGEFMF3iCgBWhLACmDZxBQArQFgBTJ+4AoCJE1YAq0FcAcCECSuA1SGuAGCihBXAahFXADBBwgpg9YgrAJgYYQWwmsQVAEyIsAJYXeIKACZCWAGsNnEFABMgrABWn7gCgCUTVgDrQVwBwBIJK4D1Ia4AYEmEFcB6EVcAsATCCmD9iCsAOGDCCmA9iSsAOEDCCmB9iSsAOCDCCmC9iSsAOADCCmD9iSsA2GfCCmAziCsA2EfCCmBziCsA2CfCCmCziCsA2AfCCmDziCsAGExYAWwmcQUAAwkrgM0lrgBgEGEFsNnEFQAMIKwAEFcAsCBhBUAirgBgIcIKgDPEFQBcJGEFwE7iCgAugrAC4GziCgAukLACYB5xBQAXQFgBsBtxBQB7JKwAOBdxBQB7IKwAOB9xBQDnIawA2AtxBQDnIKwA2CtxBQC7EFYAXAhxBQBzCCsALpS4AoCzCCsALoa4AoAdhBUAF0tcAcCMsAJgEeIKACKsAFicuAJg4wkrAEYQVwBsNGEFwCjiCoCNJawAGElcAbCRhBUAo4krADaOsAJgP4grADaKsAJgv4grADaGsAJgP4krADaCsAJgv4krANa
2021-03-02 08:32:40 +01:00
"text/plain": [
2021-05-26 11:25:35 +02:00
"<Figure size 1008x504 with 1 Axes>"
2021-03-02 08:32:40 +01:00
]
},
2021-05-26 11:25:35 +02:00
"metadata": {
"needs_background": "light"
},
2021-03-02 08:32:40 +01:00
"output_type": "display_data"
}
],
"source": [
2021-05-26 11:25:35 +02:00
"# Wykres fukncji ReLU\n",
2021-03-02 08:32:40 +01:00
"plot(lambda x: max(0, x))"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Softplus"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"$$ g(x) = \\log(1 + e^{x}) $$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* Wygładzona wersja ReLU."
]
},
{
"cell_type": "code",
2021-05-26 11:25:35 +02:00
"execution_count": 8,
2021-03-02 08:32:40 +01:00
"metadata": {
"slideshow": {
2021-05-26 11:25:35 +02:00
"slide_type": "subslide"
2021-03-02 08:32:40 +01:00
}
},
"outputs": [
{
"data": {
2021-05-26 11:25:35 +02:00
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA1cAAAG2CAYAAACTRXz+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxV9Z3/8fcn+74vhBD2EPZFENdqqDsyUq1W7WrbGcYuM7adLk6d1hn7c8budpmpdaa2OqNVq7VuKG5N3UFB9l0SSEggZM/Nntzv749cKGICkRxybpLX8/Hgwb03h5xPH49vAy/Pud9rzjkBAAAAAAYnwu8BAAAAAGAkIK4AAAAAwAPEFQAAAAB4gLgCAAAAAA8QVwAAAADgAeIKAAAAADzgSVyZ2T1mVm1mm/v5erGZNZrZ+tCv73pxXgAAAAAIF1EefZ/fSfqlpPuOc8wrzrllHp0PAAAAAMKKJ1eunHMvS6rz4nsBAAAAwHA0lO+5OsvMNpjZM2Y2awjPCwAAAACnnFe3BZ7IOkkTnHMBM1sq6U+SCvs60MxWSFohSXFxcQvHjx8/RCNiuAgGg4qIYC8WvBfrAv1hbaAvfq2LrqB0qDWozqCUFmtKi7UhnwHHx88M9GXnzp01zrnsEx1nzjlPTmhmEyU95ZybPYBjyyQtcs7VHO+4oqIit2PHDk/mw8hRUlKi4uJiv8dAmGFdoD+sDfTFj3XxzKYqfeORjYqKNP302vlaUpQzpOfHwPAzA30xs7XOuUUnOm5IrlyZ2RhJB51zzswWq/d2xNqhODcAAICfunqCuuOZ7frNq6WaV5Cm//rEacpPi/d7LACngCdxZWa/l1QsKcvMKiTdKilakpxzd0m6WtIXzKxbUpuk65xXl8wAAADC1IHGdn3pgXVau7deN5w9Ud9eOkMxUdxyBoxUnsSVc+76E3z9l+rdqh0AAGBUeHVXjW568B21d/XoF9cv0N/MG+v3SABOsaHa0AIAAGBUCAadfvHSbt354k4V5iTpvz6xUFNzkvweC8AQIK4AAAA8UtfSqa88tF4v7zykKxfk6/YrZyshhn9uAaMF/28HAADwwDv76vWl+9epJtCpf79yjq5fXCAztloHRhPiCgAAYBCcc7r39TLdvnKbxqTG6dEvnK0541L9HguAD4grAACAkxTo6Na3Ht2opzdW6cIZOfrxNfOVmhDt91gAfEJcAQAAnIQdB5r1hfvXqqymRTdfNl0rPjRZERHcBgiMZsQVAADAB/SHt8v13ce3KCkuSg/83Zk6c3Km3yMBCAPEFQAAwAC1dHTrO49v1h/X7ddZkzP1s+vnKyc5zu+xAIQJ4goAAGAAth9o0pfuX6c9NS36yoWF+ocPFyqS2wABHIW4AgAAOA7nnB56q1y3PrFFKfHRuv9vz9DZU7L8HgtAGCKuAAAA+hHo6NYtj23S4+srde7ULP302vnKTo71eywAYYq4AgAA6MOWykb9wwPvqKy2Rf900TR9cclUbgMEcFzEFQAAwFGcc/q/1fv0vae2Kj0hmt0AAQwYcQUAABDS1N6lf/7jJj29sUrnTcvWTz82T5lJ3AYIYGCIKwAAAEmbKhr15d+vU0V9m755aZFuPG8KHwoM4AMhrgAAwKjmnNN9b+zV7U9vU2ZSjB5ccaZOn5jh91gAhiHiCgAAjFoNrZ361qMbtWrLQX14eo5+dM08ZSTG+D0WgGGKuAIAAKPS6j21+spD61UT6NC3l07X3547mdsAAQwKcQUAAEaV7p6gHtvVqSdXvanxGQl69Atna+64NL/HAjACEFcAAGDU2N/Qpq8+uF5ryrp01YJ83faR2UqK5Z9DALzBTxMAADAqPLu5St96dJO6e4L6uzkxuuXa+X6PBGCEIa4AAMCI1t7Vo+89tVX3r96nueNS9fPrFqhs81t+jwVgBCKuAADAiLXjQLP+4ffrtPNgQCvOm6yvX1ykmKgIlfk9GIARibgCAAAjjnNO/7d6n/7fU1uVHBelez+3WOdPy/Z7LAAjHHEFAABGlKM/u+q8adn68TXzlJ0c6/dYAEYB4goAAIwYa0rrdNOD76gm0KFbls7Q58+dxGdXARgyxBUAABj2unqCuvOFnfpVybt8dhUA3xBXAABgWNtzKKCvPrReGyoadc3Ccbr1ill8dhUAX/CTBwAADEvOOT34Vrlue3KrYqIi9KtPnKbL5uT5PRaAUYy4AgAAw05toEM3/3GTnt96UOdMzdSPr5mvMalxfo8FYJQjrgAAwLBSsqNa33hkoxpbu/Qvl8/Q585h0woA4YG4AgAAw0J7V4/ueGa7fvd6mYpyk3Xf5xZrRl6K32MBwBHEFQAACHtbK5v0lYfe0c6DAX32nIn61qXTFRcd6fdYAPAexBUAAAhbwaDTb14t1Q9X7VBqQrTu/dxinT8t2++xAKBPxBUAAAhLVY1t+vofNui13bW6eGau7vjoXGUkxvg9FgD0i7gCAABh56mNlbrlsc3q7A7qjqvm6NrTC2TGphUAwhtxBQAAwkZDa6e+8/gWPbmhUvMK0nTntfM1KSvR77EAYECIKwAAEBZKdlTrm49sVF1Lp/7pomn6QvEURUVG+D0WAAwYcQUAAHzV0tGt21du0wOr92labpLuueF0zc5P9XssAPjAiCsAAOCbt8rq9E8Pb1B5fav+/rzJ+upF09hiHcCwRVwBAIAh197Vo58+v1N3v7JH49Lj9dCKs7R4UobfYwHAoBBXAABgSG2pbNTXHtqgHQebdf3i8brl8hlKiuWfJACGP36SAQCAIdHdE9Rdf3lXd76wSxmJMfrtDadryfQcv8cCAM8QVwAA4JTbcyigrz28QevLG7Rsbp6+t3y20vlAYAAjDHEFAABOmWDQ6b43ynTHs9sVGxWpn1+/QFfMG+v3WABwShBXAADglNhb26JvPLJRa0rrdP60bP3g6rnKTYnzeywAOGWIKwAA4Klg0OneN8r0g2d3KCrC9IOPztU1i8bJzPweDQBOKeIKAAB4pqymRd98ZKPWlPVerbrjo3OUlxrv91gAMCSIKwAAMGjBoNNvXy/TD1dtV3RkhH549VxdvZCrVQBGF+IKAAAMSmlNi775yAa9VVavJUXZ+o+r5mpMKu+tAjD6EFcAAOCk9ASdfvtaqX64aodioiL0o2vm6aOn5XO1CsCoRVwBAIAPbM+hgL7xyEat3VuvD0/P0b9fOYerVQBGPeIKAAAMWE/Q6Z5XS/Wj53YoNipCP/nYPF25gKtVACARVwAAYIC2VTXp5kc3akNFoy6YnqN/v2oOn1sFAEchrgAAwHF1dPfoly/t1q9K3lVKfLR+dt18XTFvLFerAOAYxBUAAOjX22V1+tajG/XuoRZdtSBf/7JspjISY/weCwDCEnEFAADeJ9DRrR88u13/++ZejU2N1+8+e7qKi3L8HgsAwhpxBQAA3uPP26t1y2ObVNXUrs+cNVFfv6RISbH8kwEATsSTn5Rmdo+kZZKqnXOz+/i6SfqZpKWSWiXd4Jxb58W5AQCAN2oDHbrtqa16fH2lpuYk6ZEbz9bCCel+jwUAw4ZX/xnqd5J+Kem+fr5+maTC0K8zJP0q9DsAAPCZc06Pr6/UbU9tVXN7l266oFBfXDJFsVGRfo8GAMOKJ3HlnHvZzCYe55Dlku5zzjlJb5pZmpnlOeeqvDg/AAA4OeV1rfru45v15x2HNL8gTd//6FwVjUn2eywAGJaG6gbqfEnlRz2vCL1GXAEA4IOunqDuebVUd76wS2bSd5bN1A1nT1RkBNurA8DJGqq46usntevzQLMVklZIUnZ2tkpKSk7hWBiOAoEA6wLvw7pAf1gb77e7oUf3bulUeXNQC3Ii9ckZMcrs3qtXXt7r92hDhnWB/rA2MBhDFVcVkgqOej5OUmVfBzrn7pZ0tyQVFRW54uLiUz4chpeSkhKxLnAs1gX6w9r4q8a2Lv1w1Xbdv3qfcpPj9OtPzdIls8b4PZYvWBfoD2sDgzFUcfWEpC+b2YPq3ciikfdbAQAwNJxzempjlW57aqtqAx264eyJ+qeL2V4dALzm1Vb
2021-03-02 08:32:40 +01:00
"text/plain": [
2021-05-26 11:25:35 +02:00
"<Figure size 1008x504 with 1 Axes>"
2021-03-02 08:32:40 +01:00
]
},
2021-05-26 11:25:35 +02:00
"metadata": {
"needs_background": "light"
},
2021-03-02 08:32:40 +01:00
"output_type": "display_data"
}
],
"source": [
2021-05-26 11:25:35 +02:00
"# Wykres funkcji softplus\n",
2021-03-02 08:32:40 +01:00
"plot(lambda x: math.log(1 + math.exp(x)))"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
2021-05-26 08:08:01 +02:00
"### Problem zanikającego gradientu (*vanishing gradient problem*)"
2021-03-02 08:32:40 +01:00
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* Sigmoidalne funkcje aktywacji ograniczają wartości na wyjściach neuronów do niewielkich przedziałów ($(-1, 1)$, $(0, 1)$ itp.).\n",
"* Jeżeli sieć ma wiele warstw, to podczas propagacji wstecznej mnożymy przez siebie wiele małych wartości → obliczony gradient jest mały.\n",
"* Im więcej warstw, tym silniejszy efekt zanikania."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"#### Sposoby na zanikający gradient\n",
"\n",
2021-05-26 11:25:35 +02:00
"* Modyfikacja algorytmu optymalizacji (*RProp*, *RMSProp*)\n",
2021-03-02 08:32:40 +01:00
"* Użycie innej funckji aktywacji (ReLU, softplus)\n",
2021-05-26 11:25:35 +02:00
"* Dodanie warstw *dropout*\n",
2021-03-02 08:32:40 +01:00
"* Nowe architektury (LSTM itp.)\n",
"* Więcej danych, zwiększenie mocy obliczeniowej"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
2021-05-26 08:08:01 +02:00
"## 11.2. Odmiany metody gradientu prostego\n",
2021-03-02 08:32:40 +01:00
"\n",
"* Batch gradient descent\n",
"* Stochastic gradient descent\n",
"* Mini-batch gradient descent"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
2021-05-26 11:25:35 +02:00
"### *Batch gradient descent*\n",
2021-03-02 08:32:40 +01:00
"\n",
"* Klasyczna wersja metody gradientu prostego\n",
"* Obliczamy gradient funkcji kosztu względem całego zbioru treningowego:\n",
" $$ \\theta := \\theta - \\alpha \\cdot \\nabla_\\theta J(\\theta) $$\n",
"* Dlatego może działać bardzo powoli\n",
2021-05-26 11:25:35 +02:00
"* Nie można dodawać nowych przykładów na bieżąco w trakcie trenowania modelu (*online learning*)"
2021-03-02 08:32:40 +01:00
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
2021-05-26 11:25:35 +02:00
"### *Stochastic gradient descent* (SGD)\n",
2021-03-02 08:32:40 +01:00
"\n",
"* Aktualizacja parametrów dla każdego przykładu:\n",
" $$ \\theta := \\theta - \\alpha \\cdot \\nabla_\\theta \\, J \\! \\left( \\theta, x^{(i)}, y^{(i)} \\right) $$\n",
"* Dużo szybszy niż _batch gradient descent_\n",
2021-05-26 11:25:35 +02:00
"* Można dodawać nowe przykłady na bieżąco w trakcie trenowania (*online learning*)"
2021-03-02 08:32:40 +01:00
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* Częsta aktualizacja parametrów z dużą wariancją:\n",
"\n",
2021-05-26 11:25:35 +02:00
"<img src=\"http://ruder.io/content/images/2016/09/sgd_fluctuation.png\" style=\"margin: auto;\" width=\"50%\" />"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* Z jednej strony dzięki temu uczenie nie \"utyka\" w złych minimach lokalnych, ale z drugiej strony może „wyskoczyć” z dobrego minimum"
2021-03-02 08:32:40 +01:00
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### _Mini-batch gradient descent_\n",
"\n",
"* Kompromis między _batch gradient descent_ i SGD\n",
" $$ \\theta := \\theta - \\alpha \\cdot \\nabla_\\theta \\, J \\left( \\theta, x^{(i : i+n)}, y^{(i : i_n)} \\right) $$\n",
"* Stabilniejsza zbieżność dzięki redukcji wariancji aktualizacji parametrów\n",
"* Szybszy niż klasyczny _batch gradient descent_\n",
"* Typowa wielkość batcha: między 50 a 256 przykładów"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Wady klasycznej metody gradientu prostego, czyli dlaczego potrzebujemy optymalizacji\n",
"\n",
2021-05-26 11:25:35 +02:00
"* Trudno dobrać właściwą szybkość uczenia (*learning rate*)\n",
2021-03-02 08:32:40 +01:00
"* Jedna ustalona wartość stałej uczenia się dla wszystkich parametrów\n",
"* Funkcja kosztu dla sieci neuronowych nie jest wypukła, więc uczenie może utknąć w złym minimum lokalnym lub punkcie siodłowym"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
2021-05-26 08:08:01 +02:00
"## 11.3. Algorytmy optymalizacji metody gradientu\n",
2021-03-02 08:32:40 +01:00
"\n",
"* Momentum\n",
"* Nesterov Accelerated Gradient\n",
"* Adagrad\n",
"* Adadelta\n",
"* RMSprop\n",
"* Adam\n",
"* Nadam\n",
"* AMSGrad"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Momentum\n",
"\n",
"* SGD źle radzi sobie w „wąwozach” funkcji kosztu\n",
"* Momentum rozwiązuje ten problem przez dodanie współczynnika $\\gamma$, który można trakować jako „pęd” spadającej piłki:\n",
" $$ v_t := \\gamma \\, v_{t-1} + \\alpha \\, \\nabla_\\theta J(\\theta) $$\n",
" $$ \\theta := \\theta - v_t $$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
2021-05-26 11:25:35 +02:00
"### Przyspiesony gradient Nesterova (*Nesterov Accelerated Gradient*, NAG)\n",
2021-03-02 08:32:40 +01:00
"\n",
"* Momentum czasami powoduje niekontrolowane rozpędzanie się piłki, przez co staje się „mniej sterowna”\n",
"* Nesterov do piłki posiadającej pęd dodaje „hamulec”, który spowalnia piłkę przed wzniesieniem:\n",
" $$ v_t := \\gamma \\, v_{t-1} + \\alpha \\, \\nabla_\\theta J(\\theta - \\gamma \\, v_{t-1}) $$\n",
" $$ \\theta := \\theta - v_t $$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Adagrad\n",
"\n",
"* “<b>Ada</b>ptive <b>grad</b>ient”\n",
2021-05-26 11:25:35 +02:00
"* Adagrad dostosowuje współczynnik uczenia (*learning rate*) do parametrów: zmniejsza go dla cech występujących częściej, a zwiększa dla występujących rzadziej\n",
"* Świetny do trenowania na rzadkich (*sparse*) zbiorach danych\n",
2021-03-02 08:32:40 +01:00
"* Wada: współczynnik uczenia może czasami gwałtownie maleć"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Adadelta i RMSprop\n",
"* Warianty algorytmu Adagrad, które radzą sobie z problemem gwałtownych zmian współczynnika uczenia"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Adam\n",
"\n",
"* “<b>Ada</b>ptive <b>m</b>oment estimation”\n",
"* Łączy zalety algorytmów RMSprop i Momentum\n",
"* Można go porównać do piłki mającej ciężar i opór\n",
"* Obecnie jeden z najpopularniejszych algorytmów optymalizacji"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Nadam\n",
"* “<b>N</b>esterov-accelerated <b>ada</b>ptive <b>m</b>oment estimation”\n",
"* Łączy zalety algorytmów Adam i Nesterov Accelerated Gradient"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### AMSGrad\n",
"* Wariant algorytmu Adam lepiej dostosowany do zadań takich jak rozpoznawanie obiektów czy tłumaczenie maszynowe"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
2021-05-26 11:25:35 +02:00
"<img src=\"contours_evaluation_optimizers.gif\" style=\"margin: auto;\" width=\"60%\" />"
2021-03-02 08:32:40 +01:00
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
2021-05-26 11:25:35 +02:00
"<img src=\"saddle_point_evaluation_optimizers.gif\" style=\"margin: auto;\" width=\"50%\" />"
2021-03-02 08:32:40 +01:00
]
}
],
"metadata": {
"celltoolbar": "Slideshow",
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.3"
},
"livereveal": {
"start_slideshow_at": "selected",
2021-04-14 08:03:54 +02:00
"theme": "white"
2021-03-02 08:32:40 +01:00
}
},
"nbformat": 4,
"nbformat_minor": 4
}
