umz21/wyk/11_Wielowarstwowe_sieci_neuronowe.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Uczenie maszynowe – zastosowania\n",
    "# 11. Wielowarstwowe sieci neuronowe w praktyce"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 11.1. Funkcje aktywacji"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Złożenie funkcji liniowych jest funkcją liniową.\n",
    "* Głównym zadaniem funkcji aktywacji jest wprowadzenie nieliniowości do sieci neuronowej, żeby model mógł odwzorowywać nie tylko liniowe zależności między danymi.\n",
    "* Każda funkcja aktywacji ma swoje zalety i wady.\n",
    "* Różne rodzaje funkcji aktywacji nadają się do różnych zastosowań."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import math\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import random\n",
    "\n",
    "from IPython.display import YouTubeVideo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "def plot(fun):\n",
    "    x = np.arange(-3.0, 3.0, 0.01)\n",
    "    y = [fun(x_i) for x_i in x]\n",
    "    fig = plt.figure(figsize=(14, 7))\n",
    "    ax = fig.add_subplot(111)\n",
    "    fig.subplots_adjust(left=0.1, right=0.9, bottom=0.1, top=0.9)\n",
    "    ax.set_xlim(-3.0, 3.0)\n",
    "    ax.set_ylim(-1.5, 1.5)\n",
    "    ax.grid()\n",
    "    ax.plot(x, y)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Funkcja logistyczna"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$$ g(x) = \\frac{1}{1 + e^{-x}} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Przyjmuje wartości z przedziału $(0, 1)$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1cAAAG2CAYAAACTRXz+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deaxe52Hf+d9D3n1feLmvkihqtWRZomQ4C90kjmO4cdOpW3s6TdrOjCadBtP5o0CDyaDBdFCggwEG0046TYVp0AboNBOgdWNM3CZNO4zTArZk2ZJtbZbMRaQoicvdeDfe7cwf78srUrqUKN8j3kvy8wFevO9533PveWg8pvjFOed5S1VVAQAAYG02rfcAAAAAbgXiCgAAoAbiCgAAoAbiCgAAoAbiCgAAoAbiCgAAoAa1xFUp5bdKKWdLKd+/xudHSikTpZTnmo+/XcdxAQAANoqWmn7PP03yG0l++332+ZOqqj5f0/EAAAA2lFrOXFVV9fUko3X8LgAAgJvRjbzn6pOllOdLKf+mlHL/DTwuAADAR66uywI/yLeT7KuqaqqU8rkk/zrJwdV2LKU8meTJJOno6PjE3r17b9AQuVksLy9n0yZrsXA184JrMTdYjXnBtZgbrOYHP/jB+aqqRj5ov1JVVS0HLKXsT/L/VlX1wHXseyLJo1VVnX+//Q4dOlS98sortYyPW8fRo0dz5MiR9R4GG4x5wbWYG6zGvOBazA1WU0p5tqqqRz9ovxuS5aWU7aWU0nx9uHncCzfi2AAAADdCLZcFllL+RZIjSbaUUk4n+fUkrUlSVdVvJvlzSf5aKWUxyWySL1V1nTIDAADYAGqJq6qqvvwBn/9GGku1AwAA3JLcrQcAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFCDWuKqlPJbpZSzpZTvX+PzUkr5B6WU10op3y2lPFLHcQEAADaKus5c/dMkn32fz38uycHm48kk/6im4wIAAGwItcRVVVVfTzL6Prt8IclvVw3fSDJQStlRx7EBAAA2ght1z9WuJKeu2D7dfA8AAOCW0HKDjlNWea9adcdSnkzj0sGMjIzk6NGjH+GwuBlNTU2ZF7yHecG1mBusxrzgWswN1uJGxdXpJHuu2N6d5MxqO1ZV9VSSp5Lk0KFD1ZEjRz7ywXFzOXr0aMwL3s284FrMDVZjXnAt5gZrcaMuC/xqkl9srhr4RJKJqqrevEHHBgAA+MjVcuaqlPIvkhxJsqWUcjrJrydpTZKqqn4zydeSfC7Ja0lmkvyVOo4LAACwUdQSV1VVffkDPq+S/PU6jgUAALAR3ah7rgAAANZdVVW5tLic6UuLmZlfyvT8YqYvLWb60lJm5hvP0/PvbP/s/duv+3eLKwAAYMO7tLiUqbnFTF1azMXm89TcYqbnr95+5/OFq96bmV9aeV5aXnXh8vcoJdkz1HXdYxRXAADAR+bymaLJ2YVMzC5kcm4hk7OLmZxrbs8u5OIVETQ1t3j1dvP1/NLyBx5r86aSnvaW9LS3pLej8TzQ1ZbdQ13pbtucrrbGe13tm9Pd1pKuts3N7ZaVz7vbN6e7vSXdbS3paN2UUkr+wnX+WcUVAADwvhaWGnE0Obf4vpE0Obd4xevm57MLHxhG7S2bVmKop/m8c6DzPe+tPDpa0nvl+x0t6W1vXYmh9SKuAADgNrG8XOXi3GLGZ+czNrOQsZn5TDSfx2YWMj4zn/Hm9vgVn1+8tPi+v7dlU0l/Z2v6Lj86WrJrsLPxXkdr+jpb0tfR+s4+HS0rr3s7WtLesvkG/S/w0RJXAABwE6qqKpNzi7kwdSmj0/O5MD2fC1PzGZ2+1AyjRiyNzcxnfHYh483ta91uVErS19Gawa7W9He1ZbinLXdt7clAV2sGOtsy0LVaJDVer/cZo41CXAEAwAZwOZZGp+dzYepSLkzPr/L6ckRdytjMfBaWVi+lztbNGexqzUBXWwa7W7OjvzMDXa0Z7GpE0kBX2zufN5/7O1uzeZNAWgtxBQAAH5GqqjI+s5BzU5dy7mLjcfbi3MrrC9PzOd882zQ6fe1Y6mlvyXBPW4a627JroCMf29WfoZ62DHe3Nd9vX3k92NWWjtZb4zK7m424AgCAD2luYakZSs1omrqUc5NzV0XU5fdXC6aO1k0Z6W3Plp727BroyIO7+jLc0wikoe62lddi6eYirgAAoGlxucob47N5a2Iub0/OvfPcfN2IqEurLvBQSjLc3Z6R3vZs7W3PwW29Geltz0jPO++NNB897S3uUboFiSsAAG55VVXl4qXFvD3xTii9E02X8vbkXN6cmMuFqUup/vA/XPWzbZs3ZVt/e7b3deTe7X35iYPvRNLleNra156hrra0bN60Tn9CNgJxBQDATW9uYSlnxmdzZnwuZ8Zn88b4bGN7YjZvjjciamZ+6T0/N9DVmu19HdnW15H7dvRlbuytPPHQPSvvbe/vyGBXq7NMXBdxBQDAhra8XOXC9Pw7wXRlPDVj6sL0/FU/U0qytbc9O/o7c8+O3vzkoZFsb8bS5edtfR3vuZfp6NHRHDm890b+8biFiCsAANbV0nKVtyfncmp0JqfGZnNqdOaqkDozMZf5xeWrfqarbXN2DXRm50BnHtjVn10DHdnZ3N410JltfR1pa3GJHjeWuAIA4CN1eTnyU2MzOTU6m9dHZ5qvZ3J6bDZvjM1mfumdeCol2dbbkV2DnXlw90B+9oGORkj1vxNPfZ0WhGDjEVcAAKzZ3MJSTo3ONMLpijNQrzcDaupdq+sNdrVmz1BX7tvRl8/cvy17h7qyZ7Are4a6snOgI+0tlh7n5iOuAAC4LrPzSzk5Op0T52dy4sJ0Tl6YzvHz0zl5YSZvTsxdtW9H66aVWHr8wFD2DDVeN97rTG9H6zr9KeCjI64AAFgxM7+YkxdmcuL8dE5cmLkqoN6avDqghrrbsn+4K5+8Yzj7t3Rn33BXdjfjaaSn3WV73HbEFQDAbWZxaTmvj87k2Lnp/PDcVI6dm87xC9M5cX46Zy9eumrfLT1t2TfcnU/dtSX7h7uyb0t3Dgx3Z+9wV/o7nX2CK4krAIBb1MTMQl47N5Vj56byw3PTzeepvD46k4WlamW/LT1tObClOz9x90j2D3dl/5bu7B9unIly+R5cP3EFAHATW1xazumx2ZUzUFc+X/ndT62bS/YNd+eurT35zP3bc+dIT+4Y6c6dW3rS3yWgoA7iCgDgJrC4tJwTF2by6tsX84O3p/Lq2Yt59e2pHD8/fdUy5sPdbbljpDs/c9+2RjyN9OSOkZ7sGexMy2bf+wQfJXEFALCBXI6o1842IuoHbzci6tj5qasu5dsz1Jm7t/bmyD0juXOkp/nozkBX2zqOHm5v4goAYB0sLi3n5Og7Z6J+8PbFvHa2cUnflWeiLkfUp+/ZmoNbe3L3tt7cubU7XW3+GQcbjf9XAgB8xM5PXcrLb17My29N5qXm86tnpzK/+N6I+slDI7l7a6+IgpuQ/7cCANRkbmEpr52dystvXczLb042nt+azPmpdxaW2Nrbnnt29OVTd23JoW29ObitJ3dt7RFRcAvw/2IAgA+pqqqcmZi7IqAaMXXs/HSWlhv3RbW3bMqh7b359KGtuWdHX+7d3ptD23sz3NO+zqMHPiriCgDgfSwuLee1c1N54Y3JvHBmMi+cmchLb05mcm5xZZ/dg525Z3tfPvvA9tyzvS/37OjN/uHubN5U1nHkwI0mrgAAmuYWlvLSm5cjajIvnpnIy29dzKXmvVEdrZtyz/a+fP6hnbm3eTbq7u296fNFu0DEFQBwm5qYWcgLb07kxTPvnJF67exUmlf1pa+jJffv7M9femJfHtjVn/t39uXAlm7fFQVck7gCAG555y5eyvfeGF+5tO/ZYzM592//cOXz7X0duX9nXz57//bct7MRUrsHO1OKy/qA6yeuAIBbyvjMfL73xkS+e3oi3z09nu+dnsiZibmVzw9s6c6B/k35Kz95V+5vhtQWi0wANRBXAMBNa+rSYr7/RiOivnt6It97YyInL8ysfH5gS3ceOzCUB3f152O7B3Lfzr70tLfk6NGjOXLkrnUcOXArElcAwE1hdn4pL77ZOCP1vdMTef70eI6dn07VvEdq10BnHtrTny89tjcP7e7P/bv6099poQngxhFXAMCGs7i0nJffupjnTo2vhNSrZ6dWvkNqW197Htw1kC88vCsf292fB3f1+/4oYN2JKwBg3b01MZfvvD6W506N5zuvj+d7b0xkdmEpSTLY1ZqP7R7IZ+7blgd3D+Rju/uzra9jnUcM8F7iCgC4oWbmF/O90xMrIfXcqfG8NdlYcKJt86bcv6svXzq8Jx/fO5iP7xmwah9w0xBXAMBHZnm5yrHz01edlXrl7Ysrl/ftG+7K43cM5eN7BvLw3sHcu6M37S2b13nUAD8acQUA1GZ0ej7PnRrLc6+P5zunGmelLs4tJkl621vy8N6B/Lf33pmP7x3IQ7sH3CcF3FLEFQDwI1lervLDc1P51smxPHtyLN8+OZZj56eTJJtKcmh7X/70Qzvz8J6BPLJ3IHds6cmmTS7vA25d4goAuC4z84t57tR4vn05pl4fz8TsQpJkqLstj+wdzBcf3ZOP7x3Ig7v6093unxnA7cXfegDAqs6Mz+bZZkg9e3IsL745uXKv1MGtPfncg9vzyN7BfGLfYA5s6bboBHDbE1cAQBaXlvPSmxfzrZOjK5f4nZlorODX2bo5D+8ZyF/7yTvziX2DeWTvYPq7fDkvwLuJKwC4DU3MLuTbr4/l2RONs1LPnRpf+V6pnf0deWTfYP7rfYN5dN9Q7tnRm9bNm9Z5xAAbn7gCgNvAWxNzefrEaJ45PppnTozmlbcvpqqSzZtK7tvRl7/w2J58Yl/jEr+dA53rPVyAm5K4AoBbTFVVOX5+Os+cGM3Tx8fyzInRvD46kyTpatucT+wbzOce3JFH9w/m4T0D6WrzzwGAOvjbFABuckvLVV56czJPN89KPXNiNOen5pM0VvF7bP9gfvGT+3L4wFDu29GXFpf4AXwkxBUA3GTmFpby/KnxxpmpE43FJ6YuNb6od/dgZ37i4EgeOzCUx/YP5c4Rq/gB3CjiCgA2uMm5hTx7Ymzlnqnvnp7I/NJykuTubT35wsM7c/jAUA4fGMqOfvdLAawXcQUAG8zZyXcWn3j6xFhefmsyVZW0bCp5YFd//vKn9uex/UN5dN9gBrvb1nu4ADSJKwBYZ29OzOabx0bzzeMX8o1jozl+fjpJ4/ulHtk3kL/xUwdzeP9QHt5r8QmAjczf0ABwg70xPptv/PBCvnn8Qr55fDQnLzRW8uvtaMnh/UP58uE9OXxgOPfv7PP9UgA3EXEFAB+hqqpyemw23zjWOCv1zeMXcnpsNknS39mawweG8ouf3J/HDwzl3h192bzJ4hMANytxBQA1qqoqr4/O5BvHLjQv9RvNG+ONmBrsas3jB4bzX/7YgTx+YDj3bO/NJjEFcMsQVwCwBpe/sPebx0dXguqtybkkyXB3Wx6/Yyj/zU/ekccPDOfg1h4xBXALE1cA8CFUVZUfnptqXuI3mm8eu5CzFy8lSUZ62/P4gaE8fsdwPnnHUO4c6fEdUwC3EXEFAO+jqqq8enbqisv8LuT81HySZFtfez5553AePzCcx+8Yyh1bfGEvwO1MXAHAFZaXq7zy9sV881hjJb9vHh/N6HQjpnb2d+THD47kiTuG8viB4ewb7hJTAKwQVwDc1paXq7z01mTjMr9jF/L0idGMzywkSXYPdubTh7bm8TuG8sk7hrN7sFNMAXBNtcRVKeWzSf5+ks1J/q+qqv7euz4/kuT3khxvvvWvqqr6O3UcGwA+jKXlKi+emWx+Ye+FPH18NJNzi0mSvUNd+cx921Yu89s92LXOowXgZrLmuCqlbE7yD5P8TJLTSZ4ppXy1qqoX37Xrn1RV9fm1Hg8APoyl5SrPnxpv3DN1fDTPHB/NxUuNmDqwpTufe3BHnrijEVM7+jvXebQA3MzqOHN1OMlrVVUdS5JSyu8k+UKSd8cVAHzkFpeW8703JlaWRv/GazOZ+8P/lCS5Y6Q7f/rhnXn8wFCeuGM42/o61nm0ANxK6oirXUlOXbF9Osnjq+z3yVLK80nOJPmbVVW9sNovK6U8meTJJBkZGcnRo0drGCK3kqmpKfOC9zAvbl+Ly1VOTC7n5dGlvDK6nFfHljK31PhsZ3fJYyNVHtzWkbuHNmWgPUkuJOMX8tK3X81L6zlw1pW/M7gWc4O1qCOuVruzt3rX9reT7KuqaqqU8rkk/zrJwdV+WVVVTyV5KkkOHTpUHTlypIYhcis5evRozAvezby4fSwsLee7pycaZ6WOXcizJ8cyM9+oqbu39eSLjw3niTuGc/jAUEZ6280NVmVecC3mBmtRR1ydTrLniu3daZydWlFV1eQVr79WSvk/Sylbqqo6X8PxAbiFzS8u57unx1cu8/vWibHMLjRi6tC23nzxE7tXYmq4p32dRwvA7ayOuHomycFSyoEkbyT5UpL//ModSinbk7xdVVVVSjmcZFOSCzUcG4BbzKXFpcaZqR9eyDeON85MzS0sJ0nu2d6bv/DYnjxxx1AOHxjOUHfbOo8WAN6x5riqqmqxlPIrSf4gjaXYf6uqqhdKKb/c/Pw3k/y5JH+tlLKYZDbJl6qqevelgwDchi4tLuW51985M/XsybFcWlxOKck92/vy5cN7G0ujHxjKoJgCYAOr5Xuuqqr6WpKvveu937zi9W8k+Y06jgXAzW1uYSnPNZdG/8axC/nO6+MrMXXv9r78xcf3Nc9MDWWgS0wBcPOoJa4A4FrmFpby7dfH8s1jjTNT3zk1nvlmTN2/sy//xRP7GvdM7R9Kf1freg8XAH5k4gqAWs3OL+U7r481z0yN5rlT45lfWs6mkty/sz+/9MlGTD26fyj9nWIKgFuHuAJgTaYuLebZk2N5+viFPH28EVMLS1U2leTBXf35y5/anyfuGMqj+4fS1yGmALh1iSsAPpSx6fk8c2I0Tx8fzdMnRvPCmcksLVfZvKnkgV39+as/diBPHBjOo/sH0yumALiNiCsA3tfbk3P55vHRPHO8EVSvvH0xSdLWsikf3zOQv37kzhw+MJyP7x1Id7v/rABw+/JfQQBWVFWVU6Oz+WbzEr+nT4zm5IWZJEl32+Z8Yv9Qfv7hnTl8YCgf292f9pbN6zxiANg4xBXAbWx5ucpr56auOjP11uRckmSwqzWP7R/KX3piXx4/MJx7d/SmZfOmdR4xAGxc4grgNrK4tJyX3ry4cmbqmROjGZtZSJJs62vP4weG89iBoTx+YCh3jfRk06ayziMGgJuHuAK4hV3+wt7Gan6jefbkWKYuLSZJ9g135afv3ZbDB4by+IHh7BnqTCliCgB+VOIK4BZyfupSvnViLM+eHM0zJ8bywpmJLCxVSZK7t/XkFz6+K48dGMrh/UPZ3t+xzqMFgFuLuAK4SVVVlWPnp/OtE6P51omxfOvkWI6fn07SWMnvod39+a9+/I48tn8wj+wdzEBX2zqPGABubeIK4CZxaXEp339jshFTJ8fy7MmxjE7PJ2ksPvGJfUP50mN78uj+wTywy0p+AHCjiSuADWpiZiHPvt64vO/ZE2N57vR45heXkyQHtnTnT92zNY/tH8wn9g3lzpFu90sBwDoTVwAbwOVL/L59cizffn08z54czQ/enkqStGwquX9Xf37xiX15tBlTI73t6zxiAODdxBXAOpi6tJjnT403Y2os3zk1nvHmkui97S15ZN9gfv6hnfnEvqE8vGcgnW0u8QOAjU5cAXzE3n1W6juvj+WVty+maizil7u29uQz923LI3sH88i+Qd8vBQA3KXEFULOLcwt5/tREvvP6KmelOlry8J6B/Oz92/PIvsE8vGcg/Z2t6zxiAKAO4gpgDZaXqxy/cO2zUge39uRn79ueR/YN5JG9g7nTWSkAuGWJK4AP4ezkXJ4/PZHnT43n+dPjef7UeCbnFpM0zkp9fO9gPvvA9nx8r7NSAHC7EVcA13BxbiHfe2Miz59qxNR3T4/nzMRckmTzppJ7tvfm8w/tzEO7+52VAgDEFUCSzC8u5+W3JptnpBox9dq5qZXL+/YNd+XR/UN5aM9AHt7Tn/t29FvBDwC4irgCbjuX75P67unxPH9qIs+dGs+LZyYzv9T4gt7h7rY8tGcgn//Yzjy0pz8P7R7IYHfbOo8aANjoxBVwS1tebiyD/v03JvK95uPFM5OZutS4T6qzdXMe3N2fv/yp/Xlo90Ae2tOfXQOdKcXlfQDAhyOugFvG0nKVY+emViLqhTcm88KZiUzPLyVJ2lo25d4dffkzH9+ZB3f156E9A7lrpCctmzet88gBgFuBuAJuSotLy/nhuel8742JfP+NifynF2fzxn/4g8w0Q6qjdVPu29GX/+wTu/PArv48uKs/d23tSauQAgA+IuIK2PAuLS7l1ben8tKbkyuX97345mTmFhr3SHW2bs7u7uTPP7pnJaTuHOl2RgoAuKHEFbChjE7P56U3J/PSm5N58cxkXnxzMq+dncricmPZvu62zbl/Z3++fHhvHmyG1B0jPfmTr/9xjhy5f51HDwDczsQVsC6Wl6ucHJ3Ji2eaIdWMqbcm51b22dbXnnt39OVP3bM19+3sy707+rJ/uDubfZcUALABiSvgIzczv5hX3rqYF684I/XyWxdX7o/avKnkrpGefPLO4dy7ozf37ejPvTt6M9zTvs4jBwC4fuIKqM3C0nKOn5/Oy29dzA/euth4fvtiXh+dWdmnt70l9+7sy59/dE/u29GX+3b25a6tPelo9YW8AMDNTVwBH9rycpU3xmfzylsX88rbF/NKM6J+eG4qC0uNe6M2byo5sKU7D+7uz5/7xO4c2t6b+3b0Zfeg75ACAG5N4gq4pqqqcmF6/qqzUC+/dTGvvn1x5bujkmTXQGcObe/NkUNbc8/23ty9rTd3bu1Oe4uzUQDA7UNcAVlernJmYjavnZ26+nFuKuMzCyv7DXa15tD23nzx0T25e1tvDm3vzd3betLb0bqOowcA2BjEFdxGFpaWc/LCTF47O5UfnmsE1KtnL+aHZ6czu/DOmajBrtYc3Nqbn3tgR+7a2pND23pz9/aejPS0u6QPAOAaxBXcgqYuLebE+emVgLr8OHFheuWeqCTZ0d+Ru7b25EuHh3LX1p7cNdKTu7b2WKUPAOBHIK7gJjW/uJxTYzM5fm46x89P59j56Rw7N5Xj56dz9uKllf02lWTfcHfuHOnJT927LXdt7cnBrT25c2tPetr9FQAAUBf/soINbHm5yluTcyvx1AipRkCdGpvN0vI7Z6GGuttyYEt3fuLukRzY0p07tnTnwEh39g93W+YcAOAGEFewzpaWq5wZn83rozM5eWEmJ0enc2p0JsfOTefEhenMLSyv7NvZujkHtnTn/l39+dMP7cyBLd0rj4GutnX8UwAAIK7gBpidX2rG0/QVETWTU6MzOT02c9V9UK2bS/YMdmX/lu586q4tjbNQI925Y0tPtvVZUAIAYKMSV1CDqqpyfmo+p8Zm8vqFd85Avd6MqHNX3AOVJL0dLdk33JX7dvTlsw9sz76hruwd7sreoa7s6O/M5k0CCgDgZiOu4DosL1c5P3Upp8Zmc3psJm+Mz+b0WOPxRnP7ysv3ksZKfHuGunLk7pHsG+7K3uHu7Bvqyr7hrvR3tjoDBQBwixFXkMZ9T2cvzjWDaSZvjF0RT+OzeWNsNvNLV8fTUHdbdg105u5tvfn0oa3ZPdiZPc142j3YZREJAIDbjLjilldVVcZmFvLmxGzeHJ9rPE/MNR+zOTM+lzPjs1m8YuW9JNnS055dg525b2dfPnP/tuwe6Mzuwa7sGuzMroHOdFvGHACAK/jXITe1qqoyPrOQMxOzeWtiLmcm5vLWSkS9E1KXFq8+69SyqWRbX0d2DnTk4T0D+fzHdmTXYCOedjfjyZknAAA+DHHFhjU7v5SzF+fy9uSlq56/++pcnnr1Gyvx9O57nTZvKtne15Ht/R15YFd/PnP/9mxvhtT2/s7s7O/IcE+7RSMAAKiVuOKGuzKa3p6cy9mLl3K2+fz2Fc8X5xbf87Ntmzelt7XKvq1LuW9nX37qnq3ZMdCZHf0d2dHfkZ0DndkinAAAWAfiilpcWlzKhan5nJ+61Hw0X1+cz4XpSzl3RThdK5q29rVnW19HDm7tyY/dtSUjvY3trVc8D3S15o//+I9z5Min1uFPCQAA1yauWFVVVZmZX1qJpXPNSDp/8Z2AuhxT56ZWD6Yk6W7bnOGe9oz0tufQ9t78+MGRbO1rz9bejmy74tnS5AAA3OzE1W1iYWk54zMLGZuZz+j0fMam5zM6M5/RqcZzY3uh8TzdCKl338t0WX9na7b0tGVLT3vu3dmXH+9uvN7S254tPe0Z7mnLSE/jdWebRSEAALg9iKub0KXFpUzMLmRydiHjMwuNWJqZz+j0e+PpcixNXuPMUpL0tLdksLs1Q11tGe5py8GtPRnqbls1loa629LWsukG/mkBAODmIK7WydJylcnZhUzMLmS8+Twxu5CJmfnGezPvvDd+RUhNzC5kdmHpmr+3o3VThrraMtjdlqHutuwZ7MpQd1sGu9oy1N3aeP+Kzwe6WtPe4uwSAACslSdlHa8AAA8OSURBVLj6Ec0vLufi3EIuzi02HwuZbD5f+d7FucVcvHRFPDUj6Vr3KF3W1bY5/Z2tK499w1352O7WDHS1pb+zNX2drRlofjbU3bYSTS7DAwCA9XHbxdXi0nKm55cyfWmx8Wi+fieOrgyk94bS5YB695fSrqazdXN6O1rS29GS/s7WbO3tyMGtvSvBNNB19XPj0Ygnl94BAMDNZcPH1fzicjOCFjN9aSlTlxYzM98Io6lLS5mZX8zU5VC6tHTVvo19GtszzZ+9nihKGmeOGmHU2oijrrbsHupK3+X32luu+vzyc1/zuaejJa2bBRIAANwuNnRcnZxczt3/47+5rn03laS7vSU97S3pbm9Jd9vmdLe3ZHdXV3raG68b77ek+4rtnvbN6WpruTqM2lvSIowAAIAPYUPHVV9byd/8zN1XRNEVYdTWiKCu9s3paW9Je8sm35MEAACsmw0dV4MdJb/ypw6u9zAAAAA+kGvfAAAAalBLXJVSPltKeaWU8lop5VdX+byUUv5B8/PvllIeqeO4AAAAG8Wa46qUsjnJP0zyc0nuS/LlUsp979rt55IcbD6eTPKP1npcAACAjaSOM1eHk7xWVdWxqqrmk/xOki+8a58vJPntquEbSQZKKTtqODYAAMCGUMeCFruSnLpi+3SSx69jn11J3nz3LyulPJnG2a2MjIzk6NGjNQyRW8nU1JR5wXuYF1yLucFqzAuuxdxgLeqIq9XWP69+hH0ab1bVU0meSpJDhw5VR44cWdPguPUcPXo05gXvZl5wLeYGqzEvuBZzg7Wo47LA00n2XLG9O8mZH2EfAACAm1YdcfVMkoOllAOllLYkX0ry1Xft89Ukv9hcNfCJJBNVVb3nkkAAAICb1ZovC6yqarGU8itJ/iDJ5iS/VVXVC6WUX25+/ptJvpbkc0leSzKT5K+s9bgAAAAbSR33XKWqqq+lEVBXvvebV7yukvz1Oo4FAACwEdXyJcIAAAC3O3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQg5a1/HApZSjJ/5Nkf5ITSf58VVVjq+x3IsnFJEtJFquqenQtxwUAANho1nrm6leT/Puqqg4m+ffN7Wv5dFVVDwsrAADgVrTWuPpCkn/WfP3PkvyZNf4+AACAm1KpqupH/+FSxquqGrhie6yqqsFV9jueZCxJleQfV1X11Pv8zieTPJkkIyMjn/jd3/3dH3l83JqmpqbS09Oz3sNggzEvuBZzg9WYF1yLucFqPv3pTz97PVfgfeA9V6WUP0qyfZWPfu1DjOdTVVWdKaVsTfLvSikvV1X19dV2bIbXU0ly6NCh6siRIx/iMNwOjh49GvOCdzMvuBZzg9WYF1yLucFafGBcVVX109f6rJTydillR1VVb5ZSdiQ5e43fcab5fLaU8pUkh5OsGlcAAAA3o7Xec/XVJL/UfP1LSX7v3TuUUrpLKb2XXyf5TJLvr/G4AAAAG8pa4+rvJfmZUsqrSX6muZ1Sys5Sytea+2xL8h9LKc8neTrJ71dV9W/XeFwAAIANZU3fc1VV1YUkP7XK+2eSfK75+liSh9ZyHAAAgI1urWeuAAAAiLgCAACohbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACogbgCAACowZriqpTyxVLKC6WU5VLKo++z32dLKa+UUl4rpfzqWo4JAACwEa31zNX3k/zZJF+/1g6llM1J/mGSn0tyX5Ivl1LuW+NxAQAANpSWtfxwVVUvJUkp5f12O5zktaqqjjX3/Z0kX0jy4lqODQAAsJHciHuudiU5dcX26eZ7AAAAt4wPPHNVSvmjJNtX+ejXqqr6ves4xmqntar3Od6TSZ5MkpGRkRw9evQ6DsHtZGpqyrzgPcwLrsXcYDXmBddibrAWHxhXVVX99BqPcTrJniu2dyc58z7HeyrJU0ly6NCh6siRI2s8PLeao0ePxrzg3cwLrsXcYDXmBddibrAWN+KywGeSHCylHCiltCX5UpKv3oDjAgAA3DBrXYr9F0opp5N8Msnvl1L+oPn+zlLK15KkqqrFJL+S5A+SvJTkd6uqemFtwwYAANhY1rpa4FeSfGWV988k+dwV219L8rW1HAsAAGAjuxGXBQIAANzyxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEANxBUAAEAN1hRXpZQvllJeKKUsl1IefZ/9TpRSvldKea6U8q21HBMAAGAjalnjz38/yZ9N8o+vY99PV1V1fo3HAwAA2JDWFFdVVb2UJKWUekYDAABwk7pR91xVSf6wlPJsKeXJG3RMAACAG+YDz1yVUv4oyfZVPvq1qqp+7zqP86mqqs6UUrYm+XellJerqvr6NY73ZJLLAXaplPL96zwGt48tSVxiyruZF1yLucFqzAuuxdxgNYeuZ6cPjKuqqn56rSOpqupM8/lsKeUrSQ4nWTWuqqp6KslTSVJK+VZVVddcKIPbk3nBaswLrsXcYDXmBddibrCa612U7yO/LLCU0l1K6b38Osln0lgIAwAA4Jax1qXYf6GUcjrJJ5P8finlD5rv7yylfK2527Yk/7GU8nySp5P8flVV/3YtxwUAANho1rpa4FeSfGWV988k+Vzz9bEkD/2Ih3jqRx8dtzDzgtWYF1yLucFqzAuuxdxgNdc1L0pVVR/1QAAAAG55N2opdgAAgFvaho6rUsr/XEr5binluVLKH5ZSdq73mNgYSin/aynl5eb8+EopZWC9x8T6K6V8sZTyQilluZRipafbXCnls6WUV0opr5VSfnW9x8PGUEr5rVLKWV/1wpVKKXtKKf9fKeWl5n9H/sZ6j4mNoZTSUUp5upTyfHNu/E/vu/9GviywlNJXVdVk8/V/l+S+qqp+eZ2HxQZQSvlMkv9QVdViKeV/SZKqqv7WOg+LdVZKuTfJcpJ/nORvVlV1XcumcusppWxO8oMkP5PkdJJnkny5qqoX13VgrLtSyk8kmUry21VVPbDe42FjKKXsSLKjqqpvN1e5fjbJn/F3BqWUkqS7qqqpUkprkv+Y5G9UVfWN1fbf0GeuLodVU3eSjVuC3FBVVf1hVVWLzc1vJNm9nuNhY6iq6qWqql5Z73GwIRxO8lpVVceqqppP8jtJvrDOY2IDqKrq60lG13scbCxVVb1ZVdW3m68vJnkpya71HRUbQdUw1dxsbT6u2SQbOq6SpJTyd0spp5L8xSR/e73Hw4b0V5P8m/UeBLCh7Epy6ort0/EPJeA6lFL2J/l4km+u70jYKEopm0spzyU5m+TfVVV1zbmx7nFVSvmjUsr3V3l8IUmqqvq1qqr2JPnnSX5lfUfLjfRBc6O5z68lWUxjfnAbuJ55AUnKKu+5+gF4X6WUniT/Msl//64rqLiNVVW1VFXVw2lcKXW4lHLNS4rX9D1Xdaiq6qevc9f/O8nvJ/n1j3A4bCAfNDdKKb+U5PNJfqrayDcPUqsP8XcGt7fTSfZcsb07yZl1GgtwE2jeT/Mvk/zzqqr+1XqPh42nqqrxUsrRJJ9NsuqiOOt+5ur9lFIOXrH580leXq+xsLGUUj6b5G8l+fmqqmbWezzAhvNMkoOllAOllLYkX0ry1XUeE7BBNRct+CdJXqqq6n9b7/GwcZRSRi6vSl1K6Uzy03mfJtnoqwX+yySH0lj962SSX66q6o31HRUbQSnltSTtSS403/qGlSQppfxCkv8jyUiS8STPVVX1s+s7KtZLKeVzSf73JJuT/FZVVX93nYfEBlBK+RdJjiTZkuTtJL9eVdU/WddBse5KKT+W5E+SfC+Nf3cmyf9QVdXX1m9UbASllI8l+Wdp/LdkU5Lfrarq71xz/40cVwAAADeLDX1ZIAAAwM1CXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANTg/wf8ZnB57RCIVwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1008x504 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Wykres funkcji logistycznej\n",
    "plot(lambda x: 1 / (1 + math.exp(-x)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Tangens hiperboliczny"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$$ g(x) = \\tanh x = \\frac{e^{x} - e^{-x}}{e^{x} + e^{-x}} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Przyjmuje wartości z przedziału $(-1, 1)$.\n",
    "* Powstaje z funkcji logistycznej przez przeskalowanie i przesunięcie."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1cAAAG2CAYAAACTRXz+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3Rc1b238e+eGfVqWcVNcrfcccM23aYaQgKEEiB0Bwcu3EB679yQSl4SSIAQiqlxQjPBdBAdV9yr3GXZVu9tyn7/0NgII9mydaQzkp7PWrOmHc35kbWX5Cdn5oyx1goAAAAA0DEetwcAAAAAgJ6AuAIAAAAABxBXAAAAAOAA4goAAAAAHEBcAQAAAIADiCsAAAAAcIAjcWWMecgYU2SMWdvG87OMMZXGmJXhy8+c2C8AAAAARAqfQ6/ziKR7JM0/zDbvWWvPd2h/AAAAABBRHDlyZa19V1KZE68FAAAAAN1RV37m6gRjzCpjzMvGmHFduF8AAAAA6HROvS3wSFZIGmytrTHGnCfpeUkjW9vQGDNP0jxJio2NnZqTk9NFI6K7CIVC8ng4Fws+i3WBtrA20BrWBdrC2kBrNm/eXGKtzTjSdsZa68gOjTFDJP3XWju+HdvukDTNWltyuO1yc3Ptpk2bHJkPPUdeXp5mzZrl9hiIMKwLtIW1gdawLtAW1gZaY4xZbq2ddqTtuiTLjTH9jDEmfHt6eL+lXbFvAAAAAOgKjrwt0BjzlKRZktKNMQWSfi4pSpKstfdJukTSzcaYgKR6SZdbpw6ZAQAAAEAEcCSurLVXHOH5e9R8qnYAAAAA6JH4tB4AAAAAOIC4AgAAAAAHEFcAAAAA4ADiCgAAAAAcQFwBAAAAgAOIKwAAAABwAHEFAAAAAA4grgAAAADAAcQVAAAAADiAuAIAAAAABxBXAAAAAOAA4goAAAAAHEBcAQAAAIADiCsAAAAAcABxBQAAAAAOIK4AAAAAwAHEFQAAAAA4gLgCAAAAAAcQVwAAAADgAOIKAAAAABxAXAEAAACAA4grAAAAAHAAcQUAAAAADiCuAAAAAMABxBUAAAAAOIC4AgAAAAAHEFcAAAAA4ADiCgAAAAAcQFwBAAAAgAOIKwAAAABwAHEFAAAAAA4grgAAAADAAcQVAAAAADiAuAIAAAAABxBXAAAAAOAA4goAAAAAHEBcAQAAAIADiCsAAAAAcABxBQAAAAAOIK4AAAAAwAHEFQAAAAA4gLgCAAAAAAcQVwAAAADgAOIKAAAAABxAXAEAAACAA4grAAAAAHAAcQUAAAAADiCuAAAAAMABxBUAAAAAOIC4AgAAAAAHEFcAAAAA4ADiCgAAAAAcQFwBAAAAgAOIKwAAAABwAHEFAAAAAA4grgAAAADAAcQVAAAAADiAuAIAAAAABzgSV8aYh4wxRcaYtW08b4wxfzHG5BtjVhtjpjixXwAAAACIFE4duXpE0pzDPH+upJHhyzxJf3dovwAAAAAQERyJK2vtu5LKDrPJBZLm22YfS0o1xvR3Yt8AAAAAEAl8XbSfgZJ2t7hfEH5sbxftHwAAAEAPZK1VyErBkFXIWgVDVkFrFQqFb4fvB0NWoZA+vW0/fT70mcc++1qjspLaPUtXxZVp5THb6obGzFPzWweVkZGhvLy8ThwL3VFNTQ3rAp/DukBbWBtoDesCbemNa8Naq6CVgiHJH5IC1ioQUotL831/OEz8rTwXCEkBK/lD4TixzY+FDrx2+PVDNhw34fsHnjsYNQfvKxxF+sy2oUN+NhR+vjPNHR/d7m27Kq4KJGW3uD9IUmFrG1prH5D0gCTl5ubaWbNmdfpw6F7y8vLEusChWBdoC2sDrWFdoC2RsDZCIat6f1B1TUHVNQVU2xhUQyCoBn9Qjf6QGvwH7odv+0Oq9wfV6A8evH9g+4PbBEKfed4fDKkpEFJjsPm2dThQfB4jn9fI5/GEr5tvez1GUV4Tvm6+7/N6FOMxn/2ZQ37e6zGK8njk9RpFeYy8Hs/B1/GF73s9ksdj5DXNj3vC1wcvxjQ/79Gnzx147MDPHHxMBx8bkp6gn7X3v9vZ/xnbtFDSrcaYpyXNkFRpreUtgQAAAOj2rLWqawqquiGg6ga/qhoCqmrwq7ohoLrGgGqbgp9eh2Oprqntx+uagsc0R7TPo1ifR7FRXsVGeRUX5VVslEcxUV6lxEUpNilGsVFexfg8ionyKMrrUbTPo2hv8yXqwO0W11EHr42ifR7FfOax1reP8hoZ09ob13o+R+LKGPOUpFmS0o0xBZJ+LilKkqy190laJOk8SfmS6iRd78R+AQAAACf4gyGV1zVpT3VIi7eVqrzOr8r6JlXVfz6Yqhv8zY83HrgfUDB05EM/0V6P4mO8Soj2KT7aq/gYnxKivUqNj1ZCjFfx0c33Dzx+4DouHEux4Vj6zG2f92AweTy9M2giiSNxZa294gjPW0m3OLEvAAAA4HCCIauy2iYVVzeqpKZR5XVNKq9tUnmdXxV1zdfldU2qaHFd0xj49AU++Pgzr2eMlBjjU3JslJJim6/7p8RqVGyikuOaH0uK/fQ6ucV1QoxPCdE+xUV7Fe1z6luQEKm66m2BAAAAwDELhaxKWwTTgUvz/aYWtxtVVtuktg4kJcf61CchWqnx0eqbGK0RmYlKjY9Sn/ho9YmPUuGOfJ00bVLzYwnRzYEU7eOoENqFuAIAAICr/MGQiqobta+yQfsqG7S3sr75uqpB+ysbtLeyQfurGhRopZhifB5lJMUoPTFGg/rEa3JOqjISY5Qefiw9MUZpCc3hlBIXJZ/38EeP8hp36OSR6Z31n4oejrgCAABAp2rwB1VQXqfd5fUqKKtTQXm9dpfXaU95vfZWNqi4pvFzZ6uLjfKof0qc+iXHasbQNPVLiVVWcqwykmIOxlR6YrQSY3y99uQJiDzEFQAAADrEWqvi6kZtK6nVjpJa7WoRUAXl9SqubvzM9tE+jwb1idPA1DiN7pesfimxBy/9U2LVPzlOyXFEE7of4goAAADtUlnv1/aSWm0vqdH24lptK6nV9nBQ1bY4fbjXYzQgNVbZfeI1OzdD2X3ilZ0Wr+y0OA3qE6+MxBg+w4QeibgCAADAZ5TUNGrzvmpt3l+tzUU12rK/WtuKa1Va23RwG4+RBvWJ19D0BB0/JE3DMhI0pG+ChqYnqH9K7BE/2wT0RMQVAABAL1XV4NfGveGIOnipUVmLiEqJi9KorESdPS5LQ9MTNDQ9UUPTm49Exfi8Lk4PRB7iCgAAoBcorm7UusJKrSusOni9s7Tu4POJMT6NzErU2WOzNDIrSblZSRqVlaiMpBg++wS0E3EFAADQwxRVN2jlrgqt2fNpTO2v+vSkEjlp8Ro3IFmXTh2kcQNSNKpfkgakxBJRQAcRVwAAAN1YYyCo9YVV+mRXhT7ZXaFPdpWroLxeUvPnokZkJuqk4ekaOyBZ4wakaOyAZKXERbk8NdAzEVcAAADdyP6qBi3ZXqYVu8r1ya4KrS+sUlMwJEkakBKryTl9dN2JQzQ5J1XjBqQoNorPRQFdhbgCAACIUNZaFZTXa/H2Mi3ZXqrF28sOfk4qNsqjiYNSdf3JQzQ5u48m56QqKznW5YmB3o24AgAAiCC7y+r0QX6JFm8v0+JtpSqsbJAkpcZH6fghabp65mBNH5qmMf2TFcXpzoGIQlwBAAC4qKrBrw/zS/V+frHe31KiHeEjU+mJ0ZoxtK9uGpam6UPTNCoziS/eBSIccQUAANCFAsGQVhVU6N3NJXpvS7FWFVQqGLKKj/bqhGF9de2JQ3TKyHQNz0jk7H1AN0NcAQAAdLKqBr/e2VSstzYW6e1NRaqo88sYaeKgVN182nCdMjJdk3P6KNrH2/yA7oy4AgAA6AQ7S2v1xoYivblhv5ZsL1MgZNUnPkqn52bq9DGZOnlEulLjo90eE4CDiCsAAAAHWGu1fm+VFq3Zq1fW7tPW4lpJzd8zNfeUoTpzTJam5PSRl89NAT0WcQUAAHCMrLVaV9gcVIvW7NWO0jp5jDRjaF99dcZgnTEmU4P7Jrg9JoAuQlwBAAAcBWut1u6p0ktr9urltXu1s7ROXo/RCcP6at6pw3XOuCz1TYxxe0wALiCuAAAA2qGgvE7Pf7JHz67Yo20ltfJ6jE4c3lc3nTZc54zrp7QEPj8F9HbEFQAAQBuqGvx6ec1ePbtijxZvL5MkTR+aphtPHaY54/qpD0EFoAXiCgAAoIVAMKT3tpTomRUFen39fjUGQhqWnqBvnzVKF04eqOy0eLdHBBChiCsAAABJu8vqtGDZbi1Ytlv7qxqVGh+lrxyfrYsmD9Sk7FS+0BfAERFXAACg1/IHQ3pzQ5GeWrJL724pliTNGpWhX34pR6ePzuRLfQEcFeIKAAD0OqX1If3h1Y1asKxAxdWN6pccq/89faQumzZIg/rwtj8Ax4a4AgAAvYK1Vku2l+nRj3bolbX1krZqdm6mrpieo1m5GfJ5OUoFoGOIKwAA0KM1+INauLJQD3+4Qxv2ViklLkpzhkTpR5edzFEqAI4irgAAQI9UXN2oRz/coScW71R5nV+5WUm688sTdOGkgVr84XuEFQDHEVcAAKBH2V5Sqwfe3aZnVhTIHwzpzDFZuv6kITphWF/O+AegUxFXAACgR1i1u0L3vbNVr6zbpyivRxdPGaR5pw7T0PQEt0cD0EsQVwAAoNuy1ur9/BLd+3a+Pt5WpqRYn24+bbiuO2mIMpNi3R4PQC9DXAEAgG7HWqt3t5To7jc2a8WuCvVLjtWPzxujK2bkKDGGf94AcAe/fQAAQLdhrdU7m4v1/97YopW7KzQgJVb/d9F4XTJ1kGJ8XrfHA9DLEVcAACDiWWuVF46qVbsrNDA1Tr+5aIIumTpI0T6+nwpAZCCuAABARFuyvUy/e2Wjlu8s16A+cbrzyxN08RSiCkDkIa4AAEBE2rC3Sn94dZPe2likrOQYjlQBiHjEFQAAiCi7y+p01+ub9fzKPUqK8ekH547WtScMUVw0n6kCENmIKwAAEBHKapv0lze36InFO+X1GN102nDddOpwpcRHuT0aALQLcQUAAFzVFAjpsY936u43Nqu2KajLpmXr9jNHKiuZ76kC0L0QVwAAwBXWWr21sUj/99IGbSup1amjMvTTL4zRyKwkt0cDgGNCXAEAgC63aV+17nhpvd7bUqJhGQl6+LrjNSs3Q8YYt0cDgGNGXAEAgC5TWefXH1/bpCcW71RSbJR+/sWxumrmYEV5OQMggO6PuAIAAJ3OWqtnVuzRnYs2qLyuSVfPHKzbzxylPgnRbo8GAI4hrgAAQKfatK9aP31+rZbsKNPknFTNnztd4wakuD0WADiOuAIAAJ2ipjGgu9/YrIc+2KGkWJ9+d/EEXTo1Wx4Pn6sC0DMRVwAAwHGvrN2rny9cp/1Vjbr8+Gx9b85opfEWQAA9HHEFAAAcU1TVoJ+9sE6vrNunMf2T9ferpmpKTh+3xwKALkFcAQCADrPW6t/LCnTHS+vVEAjpe3NydeMpwzgLIIBehbgCAAAdsqu0Tj96bo3ezy/R9CFpuvPiCRqekej2WADQ5YgrAABwTIIhq4c/2K4/vbZZXo/RHReO15XTczhhBYBei7gCAABHbWdprb69YJWW7SzX6aMzdceF4zUgNc7tsQDAVcQVAABoN2utnli8S79ZtEFej9Fdlx2niyYPlDEcrQIA4goAALTLvsoGff+Z1Xpnc7FOHpGu318ykaNVANACcQUAAA7LWquFqwr10+fXqikY0q8uGKerZgzms1UAcAjiCgAAtKmyzq8fPb9GL63eqyk5qfrTZZM0ND3B7bEAICIRVwAAoFVLtpfp9qc/UVF1o757Tq5uOm24vBytAoA2EVcAAOAzAsGQ/vJWvu55a4ty0uL17P+cqImDUt0eCwAiHnEFAAAOKiiv0+1Pr9SyneW6eMog/fKCcUqM4Z8LANAeHidexBgzxxizyRiTb4z5QSvPzzLGVBpjVoYvP3NivwAAwDn/XV2oc+9+Txv3VevuyyfpT5cdR1gBwFHo8G9MY4xX0r2SzpJUIGmpMWahtXb9IZu+Z609v6P7AwAAzmrwB/WLhev09NLdmpSdqr9cPlk5fePdHgsAuh0n/u+o6ZLyrbXbJMkY87SkCyQdGlcAACDCbC+p1f88sUIb9lbp5lnD9a2zRinK68gbWwCg1zHW2o69gDGXSJpjrf1a+P7VkmZYa29tsc0sSc+o+chWoaTvWGvXtfF68yTNk6SMjIypCxYs6NB86HlqamqUmJjo9hiIMKwLtIW10bal+wL655pGeT3S1yfGaGJG73kLIOsCbWFtoDWzZ89ebq2ddqTtnPgt2to5WQ8tthWSBltra4wx50l6XtLI1l7MWvuApAckKTc3186aNcuBEdGT5OXliXWBQ7Eu0BbWxuc1BUL6zaINemTlDk3KTtW9X52igalxbo/VpVgXaAtrAx3hRFwVSMpucX+Qmo9OHWStrWpxe5Ex5m/GmHRrbYkD+wcAAO20p6JetzyxQit3V+j6k4boh+eOUbSPtwECgBOciKulkkYaY4ZK2iPpcklXttzAGNNP0n5rrTXGTFfzWQpLHdg3AABop3c2F+u2pz9RIGj1t69O0XkT+rs9EgD0KB2OK2ttwBhzq6RXJXklPWStXWeMuSn8/H2SLpF0szEmIKle0uW2ox/2AgAA7WKt1d/ytuqPr21SblaS/n7VVA1NT3B7LADocRz55Kq1dpGkRYc8dl+L2/dIuseJfQEAgParaQzou/9epZfX7tMXjxug3108QfHRvefEFQDQlfjtCgBAD7W9pFbz5i/T1uIa/eQLYzT35KEyprXzUAEAnEBcAQDQA721cb9ue3qlfB6jx+bO0Ekj0t0eCQB6POIKAIAeJBSyuvftfN31xmaN7Z+s+66aquy0eLfHAoBegbgCAKCHqG8K6jv/WaWXVu/VRZMH6s4vT1BslNftsQCg1yCuAADoAfZXNejG+cu0Zk+lfnTeaN14yjA+XwUAXYy4AgCgm1tTUKmvzV+qmoaA/nH1NJ05NsvtkQCgVyKuAADoxhat2atvLVipvgkx+s/NJ2pM/2S3RwKAXou4AgCgG7LW6q9v5euu1zdr6uA+uv/qqUpPjHF7LADo1YgrAAC6mQZ/UN/9z2q9uKpQX548UL/hxBUAEBGIKwAAupHi6kbdOH+ZVhVU6PtzRuum0zhxBQBECuIKAIBuIr+oRtc9vESlNU2676qpOmdcP7dHAgC0QFwBANANLN5WqnmPLVeU1+hfX5+piYNS3R4JAHAI4goAgAi3cFWhvrNglbLT4vTI9dOVnRbv9kgAgFYQVwAARChrre5/d5t++/JGTR+apgeunqrU+Gi3xwIAtIG4AgAgAgWCIf3ixXV6/ONd+uJxA/THSycqxscZAQEgkhFXAABEmLqmgP73yU/05sYi3XTacH3vnFx5PJwREAAiHXEFAEAEKa5u1NxHl2rtnkrdceF4XTVzsNsjAQDaibgCACBC7Cqt09UPLVZRVaP+cc00nTEmy+2RAABHgbgCACACrCus1HUPL5U/GNKTN87Q5Jw+bo8EADhKxBUAAC77aGup5s1fpqRYn5668QSNyExyeyQAwDEgrgAAcNEra/fqG0+v1OC0eM2fO139U+LcHgkAcIyIKwAAXPLk4l36yfNrNDmnj/557TS+wwoAujniCgCALmat1V/fytddr2/W6aMzde+VUxQXzXdYAUB3R1wBANCFgiGrX764TvM/2qmLpwzSby+eoCivx+2xAAAOIK4AAOgi/mBI316wSgtXFerrpw7TD84dLWP4cmAA6CmIKwAAukCDP6hbnlihNzcW6ftzRuvmWcPdHgkA4DDiCgCATlbTGNCNjy7Tx9tL9esLx+vqmYPdHgkA0AmIKwAAOlFFXZOufXip1u6p1J8vm6QLJw90eyQAQCchrgAA6CRF1Q26+sEl2l5aq/uumqqzxma5PRIAoBMRVwAAdIKC8jpd9eBiFVU36uHrjtdJI9LdHgkA0MmIKwAAHLa1uEZXPbhYtY0BPTZ3hqYO7uP2SACALkBcAQDgoHWFlbrmn0tkjPT0vBM0dkCy2yMBALoIcQUAgEOW7yzTdQ8vVVKMT49/bYaGZSS6PRIAoAsRVwAAOGDxtlJd/8hSZSXH6vGvzdDA1Di3RwIAdDHiCgCADvpoa6lueGSpBqTG6qkbZyozOdbtkQAALiCuAADogA/zS3TDo0uV3SdeT944UxlJMW6PBABwCXEFAMAx+iC/RHMfXaqctOawSk8krACgN/O4PQAAAN3Ru5uLdcMjSzWkb4KeIqwAAOLIFQAARy1vU5HmPbZcwzMS9cTXZigtIdrtkQAAEYAjVwAAHIW3NxZp3vzlGpmZqCcJKwBACxy5AgCgnd7csF83P75Co/ol6vG5M5QaT1gBAD7FkSsAANrh9fX7ddPjyzW6f5KemDuTsAIAfA5HrgAAOIJX1+3TrU+u0NgBKZp/w3SlxEW5PRIAIAJx5AoAgMN4ec1e3fLECo0fmKLH5hJWAIC2EVcAALThpdV7detTn+i47FTNv2G6kmMJKwBA23hbIAAArXhxVaFu/9dKTc5O1SM3TFdiDH8yAQCHx5ErAAAO8cLKPbrt6U80NacPYQUAaDf+WgAA0MJznxTo2wtW6fghaXrouuOVQFgBANqJvxgAAIQ9s7xA3/nPKs0c2lf/vG6a4qP5MwkAaD/+agAAIGnBst36/jOrddLwdP3jmmmKi/a6PRIAoJshrgAAvd6/lu7SD55do5NHNIdVbBRhBQA4epzQAgDQqz21ZJe+/8wanTIyg7ACAHQIcQUA6LUe/3infvjsGs3OzdADV08lrAAAHcLbAgEAvdL8j3boZy+s0xmjM/W3q6YoxkdYAQA6hrgCAPQ6j3ywXb94cb3OHJOle786mbACADiCuAIA9Cr/fH+7fv3f9TpnXJb+esUURft4hzwAwBnEFQCg13jwvW2646UNOnd8P/3lismK8hJWAADnEFcAgF7h/ne26s6XN+oLE/rr/10+ibACADjOkb8sxpg5xphNxph8Y8wPWnneGGP+En5+tTFmihP7BQCgPf6Wl687X96o8yf2192EFQCgk3T4yJUxxivpXklnSSqQtNQYs9Bau77FZudKGhm+zJD09/A1AACdauHWJj27ZZMumDRAf7r0OPkIKwBAJ3HiL8x0SfnW2m3W2iZJT0u64JBtLpA03zb7WFKqMaa/A/sGAKBNd7+xRc9u8euiyQN112WTCCsAQKdy4jNXAyXtbnG/QJ8/KtXaNgMl7T30xYwx8yTNk6SMjAzl5eU5MCJ6kpqaGtYFPod1gUM9t6VJL2z1a3qm1Rczy/Xeu++4PRIiCL8z0BbWBjrCibgyrTxmj2Gb5getfUDSA5KUm5trZ82a1aHh0PPk5eWJdYFDsS5wgLVWf359s17Ymq9Lpg7SeellOn32bLfHQoThdwbawtpARzjx/ogCSdkt7g+SVHgM2wAA0CHWWv3xtU36y1v5+sq0bP3+4onymNb+/z0AAJznRFwtlTTSGDPUGBMt6XJJCw/ZZqGka8JnDZwpqdJa+7m3BAIAcKystfrdK5t079tbdcX0HN355QnyeAgrAEDX6fDbAq21AWPMrZJeleSV9JC1dp0x5qbw8/dJWiTpPEn5kuokXd/R/QIAcIC1Vne+vFEPvLtNV83M0a++NJ6wAgB0OUe+RNhau0jNAdXysfta3LaSbnFiXwAAtGSt1f+9tEEPvr9d15wwWL/80jgZ3goIAHCBI3EFAIAbrLX61X/X6+EPdui6E4fo518cS1gBAFxDXAEAuiVrrX754no98uEO3XDSUP30/DGEFQDAVcQVAKDbCYWsfr5wnR77eKduPGWofnQeYQUAcB9xBQDoVkIhq5+8sFZPLt6lr582TD+YM5qwAgBEBOIKANBthEJWP35+jZ5asls3zxqu752TS1gBACIGcQUA6BZCIasfPrtG/1q2W7fOHqFvnz2KsAIARBTiCgAQ8YIhq+/+Z5WeXbFH3zh9hL55FmEFAIg8xBUAIKIFgiF9+9+r9MLKQn3zzFG67cyRbo8EAECriCsAQMTyB0O6/V8r9dLqvfruObm6ZfYIt0cCAKBNxBUAICI1BUK67elP9PLaffrReaM179Thbo8EAMBhEVcAgIjTGAjq1ic/0evr9+un54/V3JOHuj0SAABHRFwBACJKgz+o/3lihd7aWKRfXTBO15wwxO2RAABoF+IKABAxGvxBzXtsud7dXKzfXDRBV87IcXskAADajbgCAESE+qagbpy/TB9sLdHvL56oy47PdnskAACOCnEFAHBdbWNAcx9dqiXby/THS47TxVMHuT0SAABHjbgCALiqpjGg6x9eouU7y/Xnr0zSBZMGuj0SAADHhLgCALimqsGv6x5aolUFlfrLFZN1/sQBbo8EAMAxI64AAK6orPPrmoeXaN2eSt1zxWSdO6G/2yMBANAhxBUAoMuV1jTq6n8uUX5Rjf721Sk6e1w/t0cCAKDDiCsAQJfaV9mgrz74sfZU1Osf107TaaMy3B4JAABHEFcAgC6zu6xOX31wsUprGvXo9dM1Y1hft0cCAMAxxBUAoEtsK67RVx9crNrGgJ64caYmZae6PRIAAI4irgAAnW7jvipd9eASWWv19LwTNHZAstsjAQDgOOIKANCpVhdU6JqHlijG59ETXztBIzIT3R4JAIBOQVwBADrN0h1luv7hpUqNj9KTX5upnL7xbo8EAECnIa4AAJ3i/S0lunH+MvVPidUTN85Q/5Q4t0cCAKBTEVcAAMe9vn6/bnlihYZlJOixuTOUkRTj9kgAAHQ64goA4Khnlhfoe8+s1vgByXr0hulKjY92eyQAALoEcQUAcMyD723THS9t0Ekj+ur+q6cpMYY/MwCA3oO/egCADrPW6k+vbdY9b+fr3PH99P8un6QYn9ftsQAA6FLEFQCgQ4Ihq5++sFZPLt6lK6Zn644LJ8jrMW6PBQBAlyOuAADHrCkQ0jcXrNRLq/fq5lnD9b1zcmUMYQUA6J2IKwDAMaltDOimx5frvS0l+vF5Y3TjqcPcHgo5EwwAABm+SURBVAkAAFcRVwCAo1Ze26TrH1mq1QUV+v0lE3XZtGy3RwIAwHXEFQDgqOyrbNDV/1ysnWV1+vtVU3XOuH5ujwQAQEQgrgAA7ZZfVKNrH1qiynq/Hrn+eJ04PN3tkQAAiBjEFQCgXZbtKNPX5i+Tz+PRUzfO1IRBKW6PBABARCGuAABH9MravfrG0ys1KDVOj94wXdlp8W6PBABAxCGuAACH9cgH2/XL/67X5OxUPXjt8UpLiHZ7JAAAIhJxBQBoVShk9btXNur+d7fp7LFZuvvyyYqL9ro9FgAAEYu4AgB8TmMgqO/9Z7VeWFmoq2cO1i++NE5eD18ODADA4RBXAIDPqGrw6+vzl+ujbaX63pxc3XzacBlDWAEAcCTEFQDgoD0V9Zr7yFLlF9XorsuO05enDHJ7JAAAug3iCgAgSVq1u0JzH12mRn9QD19/vE4ZmeH2SAAAdCvEFQBAi9bs1bcWrFR6YoyeunGGRmYluT0SAADdDnEFAL2YtVZ/y9uqP7y6SVNyUvXANdOUnhjj9lgAAHRLxBUA9FJNgZB++OwaPbOiQF86boB+f8lExUZxqnUAAI4VcQUAvVB5bZO+/vhyLdleptvPHKnbzhjJGQEBAOgg4goAepltxTW64ZGlKqxo0N2XT9IFkwa6PRIAAD0CcQUAvcj7W0p0y5Mr5PMYPTVvhqYOTnN7JAAAegziCgB6AWut/vn+dv1m0QaNzEzSP66Zppy+8W6PBQBAj0JcAUAP1+AP6kfPrdGzK/bonHFZuuuySUqI4dc/AABO468rAPRg+yob9PXHlmlVQaW+eeYo/e/pI+TxcOIKAAA6A3EFAD3U8p3luunx5aprDOj+q6fqnHH93B4JAIAejbgCgB5owdLd+snza9UvJVaPz52h3H5Jbo8EAECPR1wBQA/S4A/qFwvX6emlu3XyiHTdc+VkpcZHuz0WAAC9AnEFAD3E7rI63fzEcq3dU6VbZg/Xt87KlZfPVwEA0GWIKwDoAd7eVKTbn16pkLX6xzXTdNbYLLdHAgCg1+lQXBlj0iT9S9IQSTskXWatLW9lux2SqiUFJQWstdM6sl8AQLNgyOruN7for29t0eh+ybrvqika3DfB7bEAAOiVPB38+R9IetNaO1LSm+H7bZltrZ1EWAGAM8pqm3Tdw0v0lze36OIpg/Tc/5xIWAEA4KKOvi3wAkmzwrcflZQn6fsdfE0AwBEs2V6mbzz1icpqm3Tnlyfo8uOzZQyfrwIAwE3GWnvsP2xMhbU2tcX9cmttn1a22y6pXJKVdL+19oHDvOY8SfMkKSMjY+qCBQuOeT70TDU1NUpMTHR7DESY3rIuQtbqxa1+PZ/vV2a80c3HxWhIitftsSJab1kbODqsC7SFtYHWzJ49e3l73oF3xCNXxpg3JLX2zZM/Pop5TrLWFhpjMiW9bozZaK19t7UNw+H1gCTl5ubaWbNmHcVu0Bvk5eWJdYFD9YZ1UVTVoNv/tVIfbq3TBZMG6P8umqDEGM5LdCS9YW3g6LEu0BbWBjriiH+VrbVntvWcMWa/Maa/tXavMaa/pKI2XqMwfF1kjHlO0nRJrcYVAODz3tlcrG/9a6XqmoL6/SUTdenUQbwNEACACNPRE1oslHRt+Pa1kl44dANjTIIxJunAbUlnS1rbwf0CQK/gD4b025c36tqHlig9MUYLbz1Jl03j81UAAESijr6f5LeSFhhj5kraJelSSTLGDJD0oLX2PElZkp4L/0PAJ+lJa+0rHdwvAPR4+UU1+ua/VmrNnkpdMT1HP//iWMVG8fkqAAAiVYfiylpbKumMVh4vlHRe+PY2Scd1ZD8A0JtYa/XYxzv1m0UbFBfl1X1XTdGc8f3dHgsAABwBn4QGgAhSVNWg7/5ntd7ZXKzTRmXoD5dMVGZyrNtjAQCAdiCuACBCvLJ2n3747GrVNQX1qwvG6eqZg/lsFQAA3QhxBQAuq2rw69cvrte/lxdowsAU/fkrkzQik+9YAQCguyGuAMBFb28s0g+fXaOi6gbdOnuEvnHGSEX7OnoiVwAA4AbiCgBcUFnn16/+u17PrCjQqKxE3X/1STouO9XtsQAAQAcQVwDQxV5fv18/fm6NSmub9L+nj9Ctp49QjI9TrAMA0N0RVwDQRcprm/TLF9fp+ZWFGt0vSQ9dd7zGD0xxeywAAOAQ4goAOpm1Vs+v3KM7/rtBlfV+3XbGSN0yewSfrQIAoIchrgCgE20rrtFPX1irD/JLNSk7VY9dNEFjByS7PRYAAOgExBUAdILGQFD35W3TvXn5ivF59OsLx+vK6TnyevjeKgAAeiriCgAc9tHWUv34+TXaVlyr8yf218/OH6vM5Fi3xwIAAJ2MuAIAh+yvatBvX96o5z7Zo+y0OD1y/fGalZvp9lgAAKCLEFcA0EGNgaAeen+H/vrWFgWCVrfMHq5bZ49UXDSnVwcAoDchrgCgA97auF+/enG9dpTW6ayxWfrJF8ZocN8Et8cCAAAuIK4A4BhsK67Rr/+7Xm9vKtawjAQ9esN0nTYqw+2xAACAi4grADgKZbVN+utbW/T4xzsV4/PqJ18Yo2tOGMJ3VgEAAOIKANqjwR/UQx9s19/f3qrapoC+cny2vnnWKGUmcRZAAADQjLgCgMMIhqyeXVGgu17frL2VDTpzTKa+P2e0RmYluT0aAACIMMQVALTCWqt3Nhfrty9v1MZ91TpuUIr+/JVJmjmsr9ujAQCACEVcAcAhPtxaorte26xlO8uVkxave66crC9M6C9jjNujAQCACEZcAUDYsh1l+tNrm/XRtlL1S47Vry8cr69My+ZkFQAAoF2IKwC93srdFbrr9c16d3Ox0hNj9LPzx+rKGTmKjeJLgAEAQPsRVwB6rWU7ynTv2/l6e1Ox+sRH6YfnjtY1JwxRXDRRBQAAjh5xBaBXsdbq3S0luvftfC3ZXqa0hGh995xcXXviECXG8CsRAAAcO/4lAaBXCIWsXl23T/fm5Wvtnir1T4nVz784Vpcfn8ORKgAA4AjiCkCP1uAP6rlP9ujB97Zpa3GthqYn6PcXT9SFkwdyogoAAOAo4gpAj1RU3aDHP9qpxxfvUlltk8b2T9Y9V07WueP7y+vhlOoAAMB5xBWAHmXD3ir98/3tWriyUP5QSGeMztLck4dq5rA0vqcKAAB0KuIKQLfnD4a0dF9AD/zjY324tVRxUV5dPj1b1580VEPTE9weDwAA9BLEFYBua29lvZ5asltPL9mloupGDUz16PtzRuvK6TlKiY9yezwAANDLEFcAupVQyOr9/BI9/vFOvbmxSCFrddqoDF2RUKVvXDKbz1MBAADXEFcAuoU9FfV6ZnmB/rO8QLvK6pSWEK0bTxmmr87IUXZavPLy8ggrAADgKuIKQMRq8Af16rp9+veyAn2wtUTWSicO76tvnz1Kc8b3U4yP76cCAACRg7gCEFGstVqxq1zPrtijhasKVd0Q0KA+cbrtjJG6eMogZafFuz0iAABAq4grAK6z1mrD3motXFWoF1cVak9FvWKjPDp3fH9dOnWQZg7rKw9v+QMAABGOuALgmh0ltVq4qlALVxUqv6hGXo/RKSPT9e2zR+mssVlKiuWMfwAAoPsgrgB0qR0ltXp13T4tWrNXqwoqJUnTh6bpjgvH67wJ/ZWWEO3yhAAAAMeGuALQqay1WrunSq+t36dX1+3T5v01kqQJA1P04/PG6Pzj+qt/SpzLUwIAAHQccQXAcYFgSEt2lOm1dfv1+vr92lNRL49pPkL18y+O1dnj+mlgKkEFAAB6FuIKgCOKqhqUt7lY72wq1ntbilXVEFCMz6NTRmbo9jNH6owxWbzlDwAA9GjEFYBjEgiGtGJXhfI2FSlvU7HW762SJGUmxWjO+H6anZupU0dlKCGGXzMAAKB34F89ANrFWqutxTX6cGupPswv1YdbS1TVEJDXYzQ1p4++e06uZudmakz/JBnDadMBAEDvQ1wBaJW1VrvL6vXh1hJ9tK1UH24tVXF1oyRpYGqc5ozvp1m5mTppRLpS4jhlOgAAAHEFQJIUCjUfmVq2s1zLdpTr422l2lNRL0nKSIrRCcP66sThfXXi8HRlp8VxdAoAAOAQxBXQSzX4g1qzp1LLdpRr2Y4yLd9Vroo6vyQpLSFaxw/po6+fNkwnDu+r4RmJxBQAAMAREFdALxAKWW0rqdXqggqtLqjUqoIKrdtTpaZgSJI0LCNBZ4/N0rQhaZo2uI+GpicQUwAAAEeJuAJ6GGut9lTUH4yo1bsrtXZPpaobA5Kk+Givxg9I0XUnDdG0wX00dXAf9U2McXlqAACA7o+4ArqxpkBI+UU12rivShv2VmnD3mpt2Ful0tomSVKU12hM/2RdMHmAJg5K1XGDUjUiM1FeD0elAAAAnEZcAd2AtVZF1Y3avL/6MxG1tbhG/qCVJEX7PBqVlajTR2dq4qAUTRyUqtH9kxTj87o8PQAAQO9AXAERJBAMaXd5vfKLag5ethbXaGtRzcG39UlSVnKMxvRP1uzRmRrdL0lj+ydraHqCfF6Pi9MDAAD0bsQV0MWstSqpadKuslrtKKnTztJa5RfXaGtRrbaX1B48yYQkZSbFaHhGoi6cPFAjMhM1MjNRo/snKy0h2sX/AgAAALSGuAI6QSAYUmFFg3aW1WpnaZ12lTVH1IHbdU3Bg9t6jJSTFq8RmYmalZuh4ZmJGpGZqOEZiXw5LwAAQDdCXAHHoLrBr8KKBhVW1GtPRb0KD14atKeiXvuqGhQM2YPbR/s8yu4TpyF9E3TC8L4anBavwX0TlNM3XoP6xPG5KAAAgB6AuAJaCIWsSmubtL+qQcXVjSqqblBRVaP2VTVob+WnMVXdEPjMz/k8Rv1SYjUgNU7Th6ZpQGqsctLilZOWoMF949UvOVYeztAHAADQoxFX6PGstaqqD6i0tlFltU0qrW1SaU2TiqobtL+qUcXVDSqqbtT+qgaV1DR95ojTAanxURqQEqdBfeI1Y2iaBqTGHbwMTI1TRlIMpzcHAADo5YgrdCvWWjUEmr8kt7LOr8p6v8rrmoOprKZJZbWNzbfDl9LaJpXXNinQSjBJUt+EaGUmxyozKUa5WUnKTI5RVvh+RtKB6xjFRvG2PQAAABwecYUuZa1VYyCkmsaAahoCzdeNAVU3BFRR16TKer+q6pujqSJ8XVnvPxhSlfX+5lB6461WXz851qe+iTFKS4hWdlq8JmWnKi0hWmkJ0eqbGK2+CTEHb6cnxiiKU5cDAADAIcQVDisYsqprCqjeH1RDU0h1/oDqm4LNF39QdU1B1TU1x9GBYKo95P6BgDpwv62jSAcYIyXF+JQaH62UuCilxEVpQGrcwdule3dpyrjRB++nxjfHUp/4aEX7iCUAAAC4o0NxZYy5VNIvJI2RNN1au6yN7eZIuluSV9KD1trfdmS/vZG1VoGQVVMgpMZASI2B4MHbTeH7jf6QGoMhNfpDagqG1OgPtnj+05+p9382jhrC1/Utb4e3afmdS+0RH+1VYoyv+RLbfJ2dEK+k8P2E8HNJsT4lRDc/duC5lLgopcZFKzHWd9jPL+Xl7dOs6Tkd/Z8UAAAAcFRHj1ytlfRlSfe3tYExxivpXklnSSqQtNQYs9Bau76D+26VtVYh23zEJRQOkmDIKhRqvh2yzfcPXuynz7X8mdAh2wSCVv5gSP6gVSAUvg6GPveYPxhq3jYU+uzPBEPNcRQMNd8OWvlDn3+NQLB5m6YWUXQgno5wwKddorxGsVFexUd7FRflVVy0T3FRHsVH+9QnPlpx0V7FR3kVFx2+RB3Yrvk6Ptqr2Ba346K8zcEUjiVO6gAAAIDeqkNxZa3dIEnGHPYf1NMl5Vtrt4W3fVrSBZKOGFd7akI6/U95zaFjrYLB8HVICoZC4RiSAqGQQiGFn3OgQDrIYySf16Moj1GUzyOfx6Mor5HPaxTl9SjK45HPa+TzehTtNfJ5PIqNCj8XfjzG61FMlEfRXo9ioryK8R247VGMz6ton0cxvkNve8K3vS1+1qMY76f3OR04AAAA0Dm64jNXAyXtbnG/QNKM9vxglMdoTP9keY2Rz2Pk8Rh5jZHXG772fHrxGCOvR/J6POHnwrc9kif8896Wr+E55GKan/O1sY3PcyB+msPoQCC1DKID23D0BgAAAOh9jhhXxpg3JPVr5akfW2tfaMc+WiuNNg8vGWPmSZonSRkZGbp0QFU7dnEYBz4yFDz6HwtJ8nds7+gENTU1ysvLc3sMRBjWBdrC2kBrWBdoC2sDHXHEuLLWntnBfRRIym5xf5CkwsPs7wFJD0hSbm6unTVrVgd3j54mLy9PrAscinWBtrA20BrWBdrC2kBHdMV5q5dKGmmMGWqMiZZ0uaSFXbBfAAAAAOgyHYorY8xFxpgCSSdIeskY82r48QHGmEWSZK0NSLpV0quSNkhaYK1d17GxAQAAACCydPRsgc9Jeq6Vxwslndfi/iJJizqyLwAAAACIZF3xtkAAAAAA6PGIKwAAAABwAHEFAAAAAA4grgAAAADAAcQVAAAAADiAuAIAAAAABxBXAAAAAOAA4goAAAAAHEBcAQAAAIADiCsAAAAAcABxBQAAAAAOIK4AAAAAwAHEFQAAAAA4gLgCAAAAAAcQVwAAAADgAOIKAAAAABxAXAEAAACAA4grAAAAAHAAcQUAAAAADiCuAAAAAMABxBUAAAAAOIC4AgAAAAAHEFcAAAAA4ADiCgAAAAAcQFwBAAAAgAOIKwAAAABwAHEFAAAAAA4grgAAAADAAcQVAAAAADiAuAIAAAAABxBXAAAAAOAA4goAAAAAHEBcAQAAAIADiCsAAAAAcABxBQAAAAAOIK4AAAAAwAHEFQAAAAA4gLgCAAAAAAcQVwAAAADgAOIKAAAAABxAXAEAAACAA4grAAAAAHAAcQUAAAAADiCuAAAAAMABxBUAAAAAOIC4AgAAAAAHEFcAAAAA4ADiCgAAAAAcQFwBAAAAgAOIKwAAAABwAHEFAAAAAA4grgAAAADAAcQVAAAAADiAuAIAAAAABxBXAAAAAOAA4goAAAAAHEBcAQAAAIADiCsAAAAAcECH4soYc6kxZp0xJmSMmXaY7XYYY9YYY1YaY5Z1ZJ8AAAAAEIl8Hfz5tZK+LOn+dmw721pb0sH9AQAAAEBE6lBcWWs3SJIxxplpAAAAAKCb6qrPXFlJrxljlhtj5nXRPgEAAACgyxzxyJUx5g1J/Vp56sfW2hfauZ+TrLWFxphMSa8bYzZaa99tY3/zJB0IsEZjzNp27gO9R7ok3mKKQ7Eu0BbWBlrDukBbWBtoTW57NjpiXFlrz+zoJNbawvB1kTHmOUnTJbUaV9baByQ9IEnGmGXW2jZPlIHeiXWB1rAu0BbWBlrDukBbWBtoTXtPytfpbws0xiQYY5IO3JZ0tppPhAEAAAAAPUZHT8V+kTGmQNIJkl4yxrwafnyAMWZReLMsSe8bY1ZJWiLpJWvtKx3ZLwAAAABEmo6eLfA5Sc+18nihpPPCt7dJOu4Yd/HAsU+HHox1gdawLtAW1gZaw7pAW1gbaE271oWx1nb2IAAAAADQ43XVqdgBAAAAoEeL6LgyxvzaGLPaGLPSGPOaMWaA2zMhMhhj/mCM2RheH88ZY1LdngnuM8ZcaoxZZ4wJGWM401MvZ4yZY4zZZIzJN8b8wO15EBmMMQ8ZY4r4qhe0ZIzJNsa8bYzZEP47cpvbMyEyGGNijTFLjDGrwmvjl4fdPpLfFmiMSbbWVoVvf0PSWGvtTS6PhQhgjDlb0lvW2oAx5neSZK39vstjwWXGmDGSQpLul/Qda227TpuKnscY45W0WdJZkgokLZV0hbV2vauDwXXGmFMl1Uiab60d7/Y8iAzGmP6S+ltrV4TPcr1c0oX8zoAxxkhKsNbWGGOiJL0v6TZr7cetbR/RR64OhFVYgqTILUF0KWvta9baQPjux5IGuTkPIoO1doO1dpPbcyAiTJeUb63dZq1tkvS0pAtcngkRwFr7rqQyt+dAZLHW7rXWrgjfrpa0QdJAd6dCJLDNasJ3o8KXNpvk/7d3/6A6hmEcx78/p0gZmTjFIJtYTAZFOUlHNmVQJoPBpihKWaXMRxlQ6hgUBgZlIYuiGCzypyh1JhNdhveok845jjzc9+l8P/UO99M9/Ia7532u3uu53q6LK4AkF5O8A44C51rnUZeOA/dbh5DUlY3Auznr9/igJGkJkmwGdgJP2yZRL5KMJXkOfAYeVNWCZ6N5cZXkYZKX83wOAVTV2aoaB64DJ9um1f/0u7Mxu+cs8I3R+dAKsJRzIQGZ55rdD5IWlWQdMA2c+qWDSitYVX2vqh2MOqV2JVmwpfiv/udqCFW1b4lbbwB3gfP/MI468ruzkeQYcBDYWz2/PKhB/cE9Qyvbe2B8znoT8LFRFknLwOz7NNPA9aq63TqP+lNVM0keARPAvENxmv9ytZgkW+csJ4HXrbKoL0kmgNPAZFV9bZ1HUneeAVuTbEmyGjgC3GmcSVKnZocWTAGvqupS6zzqR5INP6dSJ1kL7GORmqT3aYHTwDZG07/eAieq6kPbVOpBkjfAGuDL7KUnTpJUksPAFWADMAM8r6r9bVOplSQHgMvAGHC1qi42jqQOJLkJ7AHWA5+A81U11TSUmkuyG3gMvGD03AlwpqrutUulHiTZDlxj9F2yCrhVVRcW3N9zcSVJkiRJy0XXbYGSJEmStFxYXEmSJEnSACyuJEmSJGkAFleSJEmSNACLK0mSJEkagMWVJEmSJA3A4kqSJEmSBmBxJUmSJEkD+AFDCuTx1BiOjgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1008x504 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Wykres funkcji tangensa hiperbolicznego\n",
    "plot(lambda x: math.tanh(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### ReLU (*Rectifier Linear Unit*)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$$ g(x) = \\max(0, x) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### ReLU – zalety\n",
    "* Mniej podatna na problem zanikającego gradientu (*vanishing gradient*) niż funkcje sigmoidalne, dzięki czemu SGD jest szybciej zbieżna.\n",
    "* Prostsze obliczanie gradientu.\n",
    "* Dzięki zerowaniu ujemnych wartości, wygasza neurony, „rozrzedzając” sieć (*sparsity*), co przyspiesza obliczenia."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### ReLU – wady\n",
    "* Dla dużych wartości gradient może „eksplodować”.\n",
    "* „Wygaszanie” neuronów."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1cAAAG2CAYAAACTRXz+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAYtklEQVR4nO3da6xld3nf8d+DL3WF05KKCRjbBFq509KoSYPlgFK1x4ohxkVxiKAyqhqUqpoSBTV9ESlWUUG9ILWqVFUpKGakWAGSQi1RBys44RL3YJDq4oAM8QVTi9DOdAALcwkjaOjET1/MnuQw3mfmzOz/OXvtvT8f6cj7smb//y8ejv1lrb2mujsAAAAs5lnL3gAAAMA6EFcAAAADiCsAAIABxBUAAMAA4goAAGAAcQUAADDAkLiqqjur6smqeniX97eq6ptV9dDs5y0j1gUAAJiKSwd9zq8neXuSd5/jmI9396sHrQcAADApQ85cdff9Sb424rMAAABW0UF+5+rlVfWZqvqdqvobB7guAADAvht1WeD5fDrJD3b3yaq6JclvJblu3oFVdSTJkSS54oorXvrCF77wgLbIqnj66afzrGe5Fwvfy1ywG7PBPOYieeo7nW/9v85fvLzy/VfUsrczGWaDeT7/+c9/tbsPne+46u4hC1bVi5L8dnf/0B6O/WKS67v7q+c67vDhw/34448P2R/rY3t7O1tbW8veBhNjLtiN2WCeTZ6L7s6/+MDD+Y0H/nf+yd/9y7n95r+WKnF1xibPBrurqk919/XnO+5Asryqnl+z/9VW1Q2zdZ86iLUBADhNWMH+GnJZYFW9N8lWkudW1fEkb01yWZJ09x1JXpvk56vqVJLvJLmtR50yAwDgvIQV7L8hcdXdrz/P+2/P6Vu1AwBwwIQVHAzf1gMAWGPCCg6OuAIAWFPCCg6WuAIAWEPCCg6euAIAWDPCCpZDXAEArBFhBcsjrgAA1oSwguUSVwAAa0BYwfKJKwCAFSesYBrEFQDAChNWMB3iCgBgRQkrmBZxBQCwgoQVTI+4AgBYMcIKpklcAQCsEGEF0yWuAABWhLCCaRNXAAArQFjB9IkrAICJE1awGsQVAMCECStYHeIKAGCihBWsFnEFADBBwgpWj7gCAJgYYQWrSVwBAEyIsILVJa4AACZCWMFqE1cAABMgrGD1iSsAgCUTVrAexBUAwBIJK1gf4goAYEmEFawXcQUAsATCCtaPuAIAOGDCCtaTuAIAOEDCCtaXuAIAOCDCCtabuAIAOADCCtafuAIA2GfCCjaDuAIA2EfCCjaHuAIA2CfCCjaLuAIA2AfCCjaPuAIAGExYwWYSVwAAAwkr2FziCgBgEGEFm01cAQAMIKwAcQUAsCBhBSTiCgBgIcIKOENcAQBcJGEF7CSuAAAugrACziauAAAukLAC5hFXAAAXQFgBuxFXAAB7JKyAcxFXAAB7IKyA8xFXAADnIayAvRBXAADnIKyAvRJXAAC7EFbAhRBXAABzCCvgQokrAICzCCvgYogrAIAdhBVwscQVAMCMsAIWIa4AACKsgMWJKwBg4wkrYARxBQBsNGEFjCKuAICNJayAkYbEVVXdWVVPVtXDu7xfVfUrVfVEVX22qn50xLoAABdLWAGjjTpz9etJbj7H+69Kct3s50iSXx20LgDABRNWwH64dMSHdPf9VfWicxxya5J3d3cneaCqnlNVV3X3l0asDwCwV92d9zz63dx3TFgBYx3Ud66uTnJsx/Pjs9cAAA7MmTNW9x07JayA4YacudqDeb+1eu6BVUdy+tLBHDp0KNvb2/u4LVbRyZMnzQXPYC7YjdngjD87Y3UqN13dedkVX87HPvaVZW+LifE7g0UcVFwdT3LtjufXJDkx78DuPprkaJIcPny4t7a29n1zrJbt7e2YC85mLtiN2SDZecbq9KWAL7viy7nxxhuXvS0myO8MFnFQlwXek+RnZ3cNfFmSb/q+FQBwENy8AjgoQ85cVdV7k2wleW5VHU/y1iSXJUl335Hk3iS3JHkiybeT/NyIdQEAzkVYAQdp1N0CX3+e9zvJL4xYCwBgL4QVcNAO6rJAAIADI6yAZRBXAMBaEVbAsogrAGBtCCtgmcQVALAWhBWwbOIKAFh5wgqYAnEFAKw0YQVMhbgCAFaWsAKmRFwBACtJWAFTI64AgJUjrIApElcAwEoRVsBUiSsAYGUIK2DKxBUAsBKEFTB14goAmDxhBawCcQUATJqwAlaFuAIAJktYAatEXAEAkySsgFUjrgCAyRFWwCoSVwDApAgrYFWJKwBgMoQVsMrEFQAwCcIKWHXiCgBYOmEFrANxBQAslbAC1oW4AgCWRlgB60RcAQBLIayAdSOuAIADJ6yAdSSuAIADJayAdSWuAIADI6yAdSauAIADIayAdSeuAIB9J6yATSCuAIB9JayATSGuAIB9I6yATSKuAIB9IayATSOuAIDhhBWwicQVADCUsAI2lbgCAIYRVsAmE1cAwBDCCth04goAWJiwAhBXAMCChBXAaeIKALhowgrgz4grAOCiCCuA7yWuAIALJqwAnklcAQAXRFgBzCeuAIA9E1YAuxNXAMCeCCuAcxNXAMB5CSuA8xNXAMA5CSuAvRFXAMCuhBXA3okrAGAuYQVwYcQVAPAMwgrgwokrAOB7CCuAiyOuAIA/JawALp64AgCSCCuARYkrAEBYAQwgrgBgwwkrgDHEFQBsMGEFMI64AoANJawAxhJXALCBhBXAeOIKADaMsALYH+IKADaIsALYP0PiqqpurqrHq+qJqrp9zvtbVfXNqnpo9vOWEesCAHsnrAD216WLfkBVXZLkHUlekeR4kger6p7ufvSsQz/e3a9edD0A4MIJK4D9N+LM1Q1JnujuL3T3d5O8L8mtAz4XABhAWAEcjOruxT6g6rVJbu7ufzx7/g+T/Fh3v2nHMVtJ3p/TZ7ZOJPml7n5kl887kuRIkhw6dOild91110L7Y/2cPHkyV1555bK3wcSYC3az6bPR3XnPo9/NfcdO5ZYXX5bX/dXLhFXMBbszG8xz4403fqq7rz/fcQtfFphk3m/os4vt00l+sLtPVtUtSX4ryXXzPqy7jyY5miSHDx/ura2tAVtknWxvb8dccDZzwW42eTbOnLG675gzVmfb5Lng3MwGixhxWeDxJNfueH5NTp+d+lPd/UfdfXL2+N4kl1XVcwesDQDM4VJAgIM3Iq4eTHJdVb24qi5PcluSe3YeUFXPr9lv9Kq6YbbuUwPWBgDOIqwAlmPhywK7+1RVvSnJh5JckuTO7n6kqt44e/+OJK9N8vNVdSrJd5Lc1ot+2QsAeAZhBbA8I75zdeZSv3vPeu2OHY/fnuTtI9YCAOYTVgDLNeQvEQYAlktYASyfuAKAFSesAKZBXAHAChNWANMhrgBgRQkrgGkRVwCwgoQVwPSIKwBYMcIKYJrEFQCsEGEFMF3iCgBWhLACmDZxBQArQFgBTJ+4AoCJE1YAq0FcAcCECSuA1SGuAGCihBXAahFXADBBwgpg9YgrAJgYYQWwmsQVAEyIsAJYXeIKACZCWAGsNnEFABMgrABWn7gCgCUTVgDrQVwBwBIJK4D1Ia4AYEmEFcB6EVcAsATCCmD9iCsAOGDCCmA9iSsAOEDCCmB9iSsAOCDCCmC9iSsAOADCCmD9iSsA2GfCCmAziCsA2EfCCmBziCsA2CfCCmCziCsA2AfCCmDziCsAGExYAWwmcQUAAwkrgM0lrgBgEGEFsNnEFQAMIKwAEFcAsCBhBUAirgBgIcIKgDPEFQBcJGEFwE7iCgAugrAC4GziCgAukLACYB5xBQAXQFgBsBtxBQB7JKwAOBdxBQB7IKwAOB9xBQDnIawA2AtxBQDnIKwA2CtxBQC7EFYAXAhxBQBzCCsALpS4AoCzCCsALoa4AoAdhBUAF0tcAcCMsAJgEeIKACKsAFicuAJg4wkrAEYQVwBsNGEFwCjiCoCNJawAGElcAbCRhBUAo4krADaOsAJgP4grADaKsAJgv4grADaGsAJgP4krADaCsAJgv4krANaesALgIAyJq6q6uaoer6onqur2Oe9XVf3K7P3PVtWPjlgXAM5HWAFwUC5d9AOq6pIk70jyiiTHkzxYVfd096M7DntVkutmPz+W5Fdn/wSAfdPdec+j3819x4QVAPtv4bhKckOSJ7r7C0lSVe9LcmuSnXF1a5J3d3cneaCqnlNVV3X3l871wV/7v51/89uPnusQNtCx43+cT5w0F3wvc8E8X3zq27nv2ClhBcCBGBFXVyc5tuP58TzzrNS8Y65O8oy4qqojSY4kyeXP+yv5jf/+hwO2yDrpdOqYueB7mQvmqUpuvrbzsiu+nI997CvL3g4TcvLkyWxvby97G0yQ2WARI+Jq3v8N2BdxzOkXu48mOZokhw8f7s+97e8ttjvWzvb2dra2tpa9DSbGXLAbs8E85oLdmA0WMeKGFseTXLvj+TVJTlzEMQAAACtrRFw9mOS6qnpxVV2e5LYk95x1zD1JfnZ218CXJfnm+b5vBQAAsEoWviywu09V1ZuSfCjJJUnu7O5HquqNs/fvSHJvkluSPJHk20l+btF1AQAApmTEd67S3ffmdEDtfO2OHY87yS+MWAsAAGCKhvwlwgAAAJtOXAEAAAwgrgAAAAYQVwAAAAOIKwAAgAHEFQAAwADiCgAAYABxBQAAMIC4AgAAGEBcAQAADCCuAAAABhBXAAAAA4grAACAAcQVAADAAOIKAABgAHEFAAAwgLgCAAAYQFwBAAAMIK4AAAAGEFcAAAADiCsAAIABxBUAAMAA4goAAGAAcQUAADCAuAIAABhAXAEAAAwgrgAAAAYQVwAAAAOIKwAAgAHEFQAAwADiCgAAYABxBQAAMIC4AgAAGEBcAQAADCCuAAAABhBXAAAAA4grAACAAcQVAADAAOIKAABgAHEFAAAwgLgCAAAYQFwBAAAMIK4AAAAGEFcAAAADiCsAAIABxBUAAMAA4goAAGAAcQUAADCAuAIAABhAXAEAAAwgrgAAAAYQVwAAAAOIKwAAgAHEFQAAwADiCgAAYABxBQAAMIC4AgAAGEBcAQAADHDpIn+4qv5Skv+S5EVJvpjk73f31+cc98Uk30ryJ0lOdff1i6wLAAAwNYueubo9ye9193VJfm/2fDc3dvePCCsAAGAdLRpXtyZ51+zxu5L89IKfBwAAsJKquy/+D1d9o7ufs+P517v7++cc94dJvp6kk7yzu4+e4zOPJDmSJIcOHXrpXXfdddH7Yz2dPHkyV1555bK3wcSYC3ZjNpjHXLAbs8E8N95446f2cgXeeb9zVVUfTfL8OW+9+QL28+PdfaKqfiDJR6rqc919/7wDZ+F1NEkOHz7cW1tbF7AMm2B7ezvmgrOZC3ZjNpjHXLAbs8EizhtX3X3Tbu9V1Veq6qru/lJVXZXkyV0+48Tsn09W1d1JbkgyN64AAABW0aLfubonyRtmj9+Q5ANnH1BVz66q7zvzOMkrkzy84LoAAACTsmhc/dskr6iq/5nkFbPnqaoXVNW9s2Oel+QTVfWZJJ9M8sHu/t0F1wUAAJiUhf6eq+5+KslPzHn9RJJbZo+/kOSHF1kHAABg6hY9cwUAAEDEFQAAwBDiCgAAYABxBQAAMIC4AgAAGEBcAQAADCCuAAAABhBXAAAAA4grAACAAcQVAADAAOIKAABgAHEFAAAwgLgCAAAYQFwBAAAMIK4AAAAGEFcAAAADiCsAAIABxBUAAMAA4goAAGAAcQUAADCAuAIAABhAXAEAAAwgrgAAAAYQVwAAAAOIKwAAgAHEFQAAwADiCgAAYABxBQAAMIC4AgAAGEBcAQAADCCuAAAABhBXAAAAA4grAACAAcQVAADAAOIKAABgAHEFAAAwgLgCAAAYQFwBAAAMIK4AAAAGEFcAAAADiCsAAIABxBUAAMAA4goAAGAAcQUAADCAuAIAABhAXAEAAAwgrgAAAAYQVwAAAAOIKwAAgAHEFQAAwADiCgAAYABxBQAAMIC4AgAAGEBcAQAADCCuAAAABhBXAAAAA4grAACAAcQVAADAAAvFVVW9rqoeqaqnq+r6cxx3c1U9XlVPVNXti6wJAAAwRYueuXo4yc8kuX+3A6rqkiTvSPKqJC9J8vqqesmC6wIAAEzKpYv84e5+LEmq6lyH3ZDkie7+wuzY9yW5Ncmji6wNAAAwJQfxnaurkxzb8fz47DUAAIC1cd4zV1X10STPn/PWm7v7A3tYY95prT7HekeSHEmSQ4cOZXt7ew9LsElOnjxpLngGc8FuzAbzmAt2YzZYxHnjqrtvWnCN40mu3fH8miQnzrHe0SRHk+Tw4cO9tbW14PKsm+3t7ZgLzmYu2I3ZYB5zwW7MBos4iMsCH0xyXVW9uKouT3JbknsOYF0AAIADs+it2F9TVceTvDzJB6vqQ7PXX1BV9yZJd59K8qYkH0ryWJK7uvuRxbYNAAAwLYveLfDuJHfPef1Eklt2PL83yb2LrAUAADBlB3FZIAAAwNoTVwAAAAOIKwAAgAHEFQAAwADiCgAAYABxBQAAMIC4AgAAGEBcAQAADCCuAAAABhBXAAAAA4grAACAAcQVAADAAOIKAABgAHEFAAAwgLgCAAAYQFwBAAAMIK4AAAAGEFcAAAADiCsAAIABxBUAAMAA4goAAGAAcQUAADCAuAIAABhAXAEAAAwgrgAAAAYQVwAAAAOIKwAAgAHEFQAAwADiCgAAYABxBQAAMIC4AgAAGEBcAQAADCCuAAAABhBXAAAAA4grAACAAcQVAADAAOIKAABgAHEFAAAwgLgCAAAYQFwBAAAMIK4AAAAGEFcAAAADiCsAAIABxBUAAMAA4goAAGAAcQUAADCAuAIAABhAXAEAAAwgrgAAAAYQVwAAAAOIKwAAgAHEFQAAwADiCgAAYABxBQAAMIC4AgAAGEBcAQAADCCuAAAABhBXAAAAAywUV1X1uqp6pKqerqrrz3HcF6vqD6rqoar6/UXWBAAAmKJLF/zzDyf5mSTv3MOxN3b3VxdcDwAAYJIWiqvufixJqmrMbgAAAFbUQX3nqpN8uKo+VVVHDmhNAACAA3PeM1dV9dEkz5/z1pu7+wN7XOfHu/tEVf1Ako9U1ee6+/5d1juS5EyA/XFVPbzHNdgcz03iElPOZi7YjdlgHnPBbswG8xzey0HnjavuvmnRnXT3idk/n6yqu5PckGRuXHX30SRHk6Sqfr+7d71RBpvJXDCPuWA3ZoN5zAW7MRvMs9eb8u37ZYFV9eyq+r4zj5O8MqdvhAEAALA2Fr0V+2uq6niSlyf5YFV9aPb6C6rq3tlhz0vyiar6TJJPJvlgd//uIusCAABMzaJ3C7w7yd1zXj+R5JbZ4y8k+eGLXOLoxe+ONWYumMdcsBuzwTzmgt2YDebZ01xUd+/3RgAAANbeQd2KHQAAYK1NOq6q6l9X1Wer6qGq+nBVvWDZe2IaqurfV9XnZvNxd1U9Z9l7Yvmq6nVV9UhVPV1V7vS04arq5qp6vKqeqKrbl70fpqGq7qyqJ/1VL+xUVddW1X+rqsdm/x75xWXviWmoqiuq6pNV9ZnZbPzLcx4/5csCq+ovdPcfzR7/0yQv6e43LnlbTEBVvTLJfd19qqr+XZJ09y8veVssWVX99SRPJ3lnkl/q7j3dNpX1U1WXJPl8klckOZ7kwSSv7+5Hl7oxlq6q/k6Sk0ne3d0/tOz9MA1VdVWSq7r707O7XH8qyU/7nUFVVZJnd/fJqrosySeS/GJ3PzDv+EmfuToTVjPPTjLdEuRAdfeHu/vU7OkDSa5Z5n6Yhu5+rLsfX/Y+mIQbkjzR3V/o7u8meV+SW5e8Jyagu+9P8rVl74Np6e4vdfenZ4+/leSxJFcvd1dMQZ92cvb0stnPrk0y6bhKkqp6W1UdS/IPkrxl2fthkv5Rkt9Z9iaASbk6ybEdz4/HfygBe1BVL0ryt5L8j+XuhKmoqkuq6qEkTyb5SHfvOhtLj6uq+mhVPTzn59Yk6e43d/e1SX4zyZuWu1sO0vlmY3bMm5Ocyun5YAPsZS4gSc15zdUPwDlV1ZVJ3p/kn511BRUbrLv/pLt/JKevlLqhqna9pHihv+dqhO6+aY+H/uckH0zy1n3cDhNyvtmoqjckeXWSn+gpf3mQoS7gdwab7XiSa3c8vybJiSXtBVgBs+/TvD/Jb3b3f132fpie7v5GVW0nuTnJ3JviLP3M1blU1XU7nv5Uks8tay9MS1XdnOSXk/xUd3972fsBJufBJNdV1Yur6vIktyW5Z8l7AiZqdtOCX0vyWHf/h2Xvh+moqkNn7kpdVX8+yU05R5NM/W6B709yOKfv/vW/kryxu//PcnfFFFTVE0n+XJKnZi894E6SVNVrkvynJIeSfCPJQ939k8vdFctSVbck+Y9JLklyZ3e/bclbYgKq6r1JtpI8N8lXkry1u39tqZti6arqbyf5eJI/yOn/7kySf97d9y5vV0xBVf3NJO/K6X+XPCvJXd39r3Y9fspxBQAAsComfVkgAADAqhBXAAAAA4grAACAAcQVAADAAOIKAABgAHEFAAAwgLgCAAAYQFwBAAAM8P8BvTqGDv/IGCkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1008x504 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Wykres fukncji ReLU\n",
    "plot(lambda x: max(0, x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Softplus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$$ g(x) = \\log(1 + e^{x}) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Wygładzona wersja ReLU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1cAAAG2CAYAAACTRXz+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxV9Z3/8fcn+74vhBD2EPZFENdqqDsyUq1W7WrbGcYuM7adLk6d1hn7c8budpmpdaa2OqNVq7VuKG5N3UFB9l0SSEggZM/Nntzv749cKGICkRxybpLX8/Hgwb03h5xPH49vAy/Pud9rzjkBAAAAAAYnwu8BAAAAAGAkIK4AAAAAwAPEFQAAAAB4gLgCAAAAAA8QVwAAAADgAeIKAAAAADzgSVyZ2T1mVm1mm/v5erGZNZrZ+tCv73pxXgAAAAAIF1EefZ/fSfqlpPuOc8wrzrllHp0PAAAAAMKKJ1eunHMvS6rz4nsBAAAAwHA0lO+5OsvMNpjZM2Y2awjPCwAAAACnnFe3BZ7IOkkTnHMBM1sq6U+SCvs60MxWSFohSXFxcQvHjx8/RCNiuAgGg4qIYC8WvBfrAv1hbaAvfq2LrqB0qDWozqCUFmtKi7UhnwHHx88M9GXnzp01zrnsEx1nzjlPTmhmEyU95ZybPYBjyyQtcs7VHO+4oqIit2PHDk/mw8hRUlKi4uJiv8dAmGFdoD+sDfTFj3XxzKYqfeORjYqKNP302vlaUpQzpOfHwPAzA30xs7XOuUUnOm5IrlyZ2RhJB51zzswWq/d2xNqhODcAAICfunqCuuOZ7frNq6WaV5Cm//rEacpPi/d7LACngCdxZWa/l1QsKcvMKiTdKilakpxzd0m6WtIXzKxbUpuk65xXl8wAAADC1IHGdn3pgXVau7deN5w9Ud9eOkMxUdxyBoxUnsSVc+76E3z9l+rdqh0AAGBUeHVXjW568B21d/XoF9cv0N/MG+v3SABOsaHa0AIAAGBUCAadfvHSbt354k4V5iTpvz6xUFNzkvweC8AQIK4AAAA8UtfSqa88tF4v7zykKxfk6/YrZyshhn9uAaMF/28HAADwwDv76vWl+9epJtCpf79yjq5fXCAztloHRhPiCgAAYBCcc7r39TLdvnKbxqTG6dEvnK0541L9HguAD4grAACAkxTo6Na3Ht2opzdW6cIZOfrxNfOVmhDt91gAfEJcAQAAnIQdB5r1hfvXqqymRTdfNl0rPjRZERHcBgiMZsQVAADAB/SHt8v13ce3KCkuSg/83Zk6c3Km3yMBCAPEFQAAwAC1dHTrO49v1h/X7ddZkzP1s+vnKyc5zu+xAIQJ4goAAGAAth9o0pfuX6c9NS36yoWF+ocPFyqS2wABHIW4AgAAOA7nnB56q1y3PrFFKfHRuv9vz9DZU7L8HgtAGCKuAAAA+hHo6NYtj23S4+srde7ULP302vnKTo71eywAYYq4AgAA6MOWykb9wwPvqKy2Rf900TR9cclUbgMEcFzEFQAAwFGcc/q/1fv0vae2Kj0hmt0AAQwYcQUAABDS1N6lf/7jJj29sUrnTcvWTz82T5lJ3AYIYGCIKwAAAEmbKhr15d+vU0V9m755aZFuPG8KHwoM4AMhrgAAwKjmnNN9b+zV7U9vU2ZSjB5ccaZOn5jh91gAhiHiCgAAjFoNrZ361qMbtWrLQX14eo5+dM08ZSTG+D0WgGGKuAIAAKPS6j21+spD61UT6NC3l07X3547mdsAAQwKcQUAAEaV7p6gHtvVqSdXvanxGQl69Atna+64NL/HAjACEFcAAGDU2N/Qpq8+uF5ryrp01YJ83faR2UqK5Z9DALzBTxMAADAqPLu5St96dJO6e4L6uzkxuuXa+X6PBGCEIa4AAMCI1t7Vo+89tVX3r96nueNS9fPrFqhs81t+jwVgBCKuAADAiLXjQLP+4ffrtPNgQCvOm6yvX1ykmKgIlfk9GIARibgCAAAjjnNO/7d6n/7fU1uVHBelez+3WOdPy/Z7LAAjHHEFAABGlKM/u+q8adn68TXzlJ0c6/dYAEYB4goAAIwYa0rrdNOD76gm0KFbls7Q58+dxGdXARgyxBUAABj2unqCuvOFnfpVybt8dhUA3xBXAABgWNtzKKCvPrReGyoadc3Ccbr1ill8dhUAX/CTBwAADEvOOT34Vrlue3KrYqIi9KtPnKbL5uT5PRaAUYy4AgAAw05toEM3/3GTnt96UOdMzdSPr5mvMalxfo8FYJQjrgAAwLBSsqNa33hkoxpbu/Qvl8/Q585h0woA4YG4AgAAw0J7V4/ueGa7fvd6mYpyk3Xf5xZrRl6K32MBwBHEFQAACHtbK5v0lYfe0c6DAX32nIn61qXTFRcd6fdYAPAexBUAAAhbwaDTb14t1Q9X7VBqQrTu/dxinT8t2++xAKBPxBUAAAhLVY1t+vofNui13bW6eGau7vjoXGUkxvg9FgD0i7gCAABh56mNlbrlsc3q7A7qjqvm6NrTC2TGphUAwhtxBQAAwkZDa6e+8/gWPbmhUvMK0nTntfM1KSvR77EAYECIKwAAEBZKdlTrm49sVF1Lp/7pomn6QvEURUVG+D0WAAwYcQUAAHzV0tGt21du0wOr92labpLuueF0zc5P9XssAPjAiCsAAOCbt8rq9E8Pb1B5fav+/rzJ+upF09hiHcCwRVwBAIAh197Vo58+v1N3v7JH49Lj9dCKs7R4UobfYwHAoBBXAABgSG2pbNTXHtqgHQebdf3i8brl8hlKiuWfJACGP36SAQCAIdHdE9Rdf3lXd76wSxmJMfrtDadryfQcv8cCAM8QVwAA4JTbcyigrz28QevLG7Rsbp6+t3y20vlAYAAjDHEFAABOmWDQ6b43ynTHs9sVGxWpn1+/QFfMG+v3WABwShBXAADglNhb26JvPLJRa0rrdP60bP3g6rnKTYnzeywAOGWIKwAA4Klg0OneN8r0g2d3KCrC9IOPztU1i8bJzPweDQBOKeIKAAB4pqymRd98ZKPWlPVerbrjo3OUlxrv91gAMCSIKwAAMGjBoNNvXy/TD1dtV3RkhH549VxdvZCrVQBGF+IKAAAMSmlNi775yAa9VVavJUXZ+o+r5mpMKu+tAjD6EFcAAOCk9ASdfvtaqX64aodioiL0o2vm6aOn5XO1CsCoRVwBAIAPbM+hgL7xyEat3VuvD0/P0b9fOYerVQBGPeIKAAAMWE/Q6Z5XS/Wj53YoNipCP/nYPF25gKtVACARVwAAYIC2VTXp5kc3akNFoy6YnqN/v2oOn1sFAEchrgAAwHF1dPfoly/t1q9K3lVKfLR+dt18XTFvLFerAOAYxBUAAOjX22V1+tajG/XuoRZdtSBf/7JspjISY/weCwDCEnEFAADeJ9DRrR88u13/++ZejU2N1+8+e7qKi3L8HgsAwhpxBQAA3uPP26t1y2ObVNXUrs+cNVFfv6RISbH8kwEATsSTn5Rmdo+kZZKqnXOz+/i6SfqZpKWSWiXd4Jxb58W5AQCAN2oDHbrtqa16fH2lpuYk6ZEbz9bCCel+jwUAw4ZX/xnqd5J+Kem+fr5+maTC0K8zJP0q9DsAAPCZc06Pr6/UbU9tVXN7l266oFBfXDJFsVGRfo8GAMOKJ3HlnHvZzCYe55Dlku5zzjlJb5pZmpnlOeeqvDg/AAA4OeV1rfru45v15x2HNL8gTd//6FwVjUn2eywAGJaG6gbqfEnlRz2vCL1GXAEA4IOunqDuebVUd76wS2bSd5bN1A1nT1RkBNurA8DJGqq46usntevzQLMVklZIUnZ2tkpKSk7hWBiOAoEA6wLvw7pAf1gb77e7oUf3bulUeXNQC3Ii9ckZMcrs3qtXXt7r92hDhnWB/rA2MBhDFVcVkgqOej5OUmVfBzrn7pZ0tyQVFRW54uLiUz4chpeSkhKxLnAs1gX6w9r4q8a2Lv1w1Xbdv3qfcpPj9OtPzdIls8b4PZYvWBfoD2sDgzFUcfWEpC+b2YPq3ciikfdbAQAwNJxzempjlW57aqtqAx264eyJ+qeL2V4dALzm1Vbsv5dULCnLzCok3SopWpKcc3dJWqnebdh3q3cr9s96cV4AAHB8+2pb9Z3HN+svOw9pdn6K7vnM6ZozLtXvsQBgRPJqt8DrT/B1J+lLXpwLAACcWFdPUP/9yh797IVdioowfXfZTH36rAmKiozwezQAGLG4HwAAgBFm7d46ffuPm7XjYLMumZWrf71ilvJS4/0eCwBGPOIKAIARojbQoe8/u10Pv12hsalx+u9PL9JFM3P9HgsARg3iCgCAYa4n6PT7Nfv0w1U71NLRrRXnTdZNFxQqkQ0rAGBI8VMXAIBhbH15g77zp83atL9RZ07O0PeWz1ZhbrLfYwHAqERcAQAwDNW3dOoHq7brwbfKlZ0Uq59dN19XzBsrM/N7NAAYtYgrAACGkWDQ6aG3y/X9Z7erub1bnz9nkm66sFDJcdF+jwYAox5xBQDAMLGxokHfeXyLNpQ3aPGk3lsAi8ZwCyAAhAviCgCAMNfQ2qkfrtqhB9bsU2ZirO68dr6Wz+cWQAAIN8QVAABhqifo9NBb5frRczvU0NqpG86eqK9eNE0p3AIIAGGJuAIAIAytKa3Tvz6xRVurmrR4Yob+9YpZmjk2xe+xAADHQVwBABBG9je06T9WbtNTG6s0NjVOv/z4Al0+J49bAAFgGCCuAAAIA22dPfr1y+/qrr+8K+ekmy4o1I3nT1F8TKTfowEABoi4AgDAR845Pb2pSv+xcrv2N7Tp8rl5+vbSGcpPi/d7NADAB0RcAQDgk62VTfq3J7dodWmdZuSl6Ccfm6czJmf6PRYA4CQRVwAADLG6lk796LkdenDNPqXGR+v2K2frutPHKzKC91UBwHBGXAEAMEQ6unv0v2/s1c9f3KWWzh595uyJ+soF05SawNbqADASEFcAAJxizjk9s/mA7nhmu/bVter8adn6l8tnqDA32e/RAAAeIq4AADiF3tlXr9uf3qa399arKDdZ931usc6blu33WACAU4C4AgDgFCiva9UPVu3QkxsqlZ0cqzuumqNrFhXwvioAGMGIKwAAPNTY1qX/+vNu/fa1MkVESP/44an6+/OnKDGWv3IBYKTjJz0AAB7o6gnqgdX7dOcLO9XQ1qWrFozT1y+ZprxUPq8KAEYL4goAgEFwzumFbdX6j2e2ac+hFp01OVO3XD5Ds/NT/R4NADDEiCsAAE7Sun31uuOZ7VpTWqfJ2Yn6n08v0gUzcmTG+6oAYDQirgAA+IB2VzfrB8/u0HNbDyorKUa3LZ+l6xePV3RkhN+jAQB8RFwBADBAVY1tuvP5XfrD2nIlxETpaxdN0+fPncRmFQAAScQVAAAn1NDaqV+VvKvfvV4m56Qbzp6kLy2ZosykWL9HAwCEEeIKAIB+tHf16LevlelXJbvV3NGtK+fn66sXTVNBRoLfowEAwhBxBQDAMbp7gnpkbYXufGGXDjS168PTc/SNS4o0Iy/F79EAAGGMuAIAICQYdHp2ywH9+LkdevdQixaMT9PPrpuvMyZn+j0aAGAYIK4AAKOec04vbqvWT57fqa1VTZqak6Rff2qhLp6Zy7bqAIABI64AAKOWc06v7KrRj5/fqQ3lDZqQmaCfXjtPV8zLV2QEUQUA+GCIKwDAqPTmnlr95LmdWlNWp/y0eH3/o3N01Wnj+KwqAMBJI64AAKPKun31+slzO/Xq7hrlJMfqtuWzdO3pBYqNivR7NADAMEdcAQBGhc37G/WT53fqpe3VykyM0b9cPkOfPHOC4qKJKgCAN4grAMCIVt4c1I3/u1bPbjmg1PhofeOSIt1w9kQlxvJXIADAW/zNAgAYkTbvb9QvXtqlVVvalBTbpZsuKNTnPzRJKXHRfo8GABihiCsAwIiyobxBv3hpl17YVq3kuChdMSVat32iWGkJMX6PBgAY4YgrAMCIsHZvvX7x0i6V7Dik1Phofe2iafrM2RP1zurXCCsAwJAgrgAAw9qa0jr9/MVdenV3jdITovXNS4v0qTMnKJnb/wAAQ4y4AgAMO845vbGnVj9/cZfe3FOnrKQYfXvpdH3ijAlsVAEA8A1/AwEAhg3nnF7eVaNfvrRLb5XVKyc5Vt9dNlPXLx6v+Bi2VAcA+Iu4AgCEvZ6g08pNVfpVybvaWtWkvNQ43bZ8lj62qIDPqQIAhA3iCgAQttq7evTougrd/fIe7a1t1eTsRP3g6rn6yPx8xURF+D0eAADvQVwBAMJOU3uX7n9zn37zaqlqAh2aNy5V//zJhbp4Zq4iIszv8QAA6BNxBQAIG9XN7frta2X6vzf2qrmjWx8qzNIXiufrrMmZMiOqAADhjbgCAPhub22L7n55j/6wtkLdPUFdNidPXzh/imbnp/o9GgAAA0ZcAQB8s768Qf/zyh6t3FSlqIgIfXThOK04b7ImZSX6PRoAAB8YcQUAGFI9Qafntx7Ub17do7fK6pUcG6W/+9Bkff7cScpJifN7PAAAThpxBQAYEi0d3XpkbYXuea1Ue2tbNS49Xt9dNlMfO71ASXzwLwBgBOBvMwDAKVXV2KZ7X9+rB1bvVVN7t04bn6ZvXTpdF8/MVVQk26kDAEYO4goAcEps3t+o37xaqic3VCronC6bnafPnTtJCyek+z0aAACnBHEFAPBMT9Dpz9ur9T+v7tGbe+qUGBOpT581UZ89Z6IKMhL8Hg8AgFOKuAIADFpja5cefrtc971ZpvK6NuWlxunbS6frusXjlRIX7fd4AAAMCeIKAHDStlU16b43yvTYO/vV3hXU4okZuvnSGbp4Vq6ieT8VAGCUIa4AAB9Id09Qz209qN+9XqY1pXWKi47QR+bn69NnTdTMsSl+jwcAgG+IKwDAgNQEOvTgmn26f/U+VTW2a1x6vP75sum69vQCpSXE+D0eAAC+I64AAMe1obxB975Rpqc2VKmzJ6hzp2bptuWz9eHpOYqMML/HAwAgbBBXAID3CXR064n1lXpgzV5t3t+kxJhIXbe4QJ8+a4Km5iT7PR4AAGGJuAIAHLF5f6MeWLNPj7+zXy2dPZo+Jlm3LZ+ljyzIZ9c/AABOwJO4MrNLJf1MUqSk/3HO3XHM14slPS6pNPTSH51zt3lxbgDA4LR2duvJDZV6YPU+bahoVGxUhJbNHauPnzFep41Pkxm3/gEAMBCDjiszi5T0n5IuklQh6S0ze8I5t/WYQ19xzi0b7PkAAN7YVtWkB1bv05/e2a/mjm4V5iTp1r+ZqasWjFNqAlepAAD4oLy4crVY0m7n3B5JMrMHJS2XdGxcAQB81trZrac3VumBNfv0zr4GxURF6PI5efr4GeO1aEI6V6kAABgEc84N7huYXS3pUufc34aef0rSGc65Lx91TLGkR9V7ZatS0tedc1v6+X4rJK2QpOzs7IUPP/zwoObDyBMIBJSUlOT3GAgzrIv+Oee0uyGoV/Z3a01Vt9p7pDGJpiUF0TpnbJSSYkZ2ULE20BfWBfrD2kBflixZstY5t+hEx3lx5aqvv5WPLbZ1kiY45wJmtlTSnyQV9vXNnHN3S7pbkoqKilxxcbEHI2IkKSkpEesCx2JdvN/Bpnb9cd1+/WFdufYcaldCTKSWzR+naxaO0+JJGaPmKhVrA31hXaA/rA0MhhdxVSGp4Kjn49R7deoI51zTUY9Xmtl/mVmWc67Gg/MDAEI6u4N6aftBPfx2hUp2VCvopNMnpuvG86Zo6dw8JcWySSwAAKeKF3/LviWp0MwmSdov6TpJHz/6ADMbI+mgc86Z2WJJEZJqPTg3AEC9m1P84e0K/Wn9ftW1dCo3JVY3nj9FVy8cp8nZ3N4CAMBQGHRcOee6zezLklapdyv2e5xzW8zsxtDX75J0taQvmFm3pDZJ17nBvtkLAEa5Q80denJDpR57Z7827W9UdKTpopm5umZRgT40NUtRkRF+jwgAwKjiyf0hzrmVklYe89pdRz3+paRfenEuABjNWju79fzWg3rsnf16ZVeNeoJOs8am6Na/manl8/OVkRjj94gAAIxa3HwPAGGuJ+j02u4a/emd/Xp2ywG1dvYoPy1ef3/eZH1kQb6m5Sb7PSIAABBxBQBhyTmnLZVN+tM7+/XEhkpVN3coOS5Ky+eP1Ufm5+v0iRmKiBgdu/0BADBcEFcAEEbK61r15MZKPbZuv3ZVBxQdaVpSlKMrF+RryfQcxUVH+j0iAADoB3EFAD6rbGjT0xur9NTGSm2oaJTUu3367VfO1uVz8pSWwPuoAAAYDogrAPBBdVO7nt5Upac2Vmnt3npJ0uz8FN182XRdPidPBRkJPk8IAAA+KOIKAIZITaBDz2w+oKc2VGpNWZ2ck6aPSdbXL56mZXPHamJWot8jAgCAQSCuAOAUqmvp1HNbDuipjVV6/d0aBZ00JTtRN11QqGVz8zQ1h53+AAAYKYgrAPBYVWObnttyUM9uPqDVpbUKOmliZoK+WDxVy+blqSg3WWbs9AcAwEhDXAGAB0prWvTs5gN6dssBbShvkCQV5iTpS0um6pJZYzRrbApBBQDACEdcAcBJcM5pW1Wznt1yQKs2H9COg82SpLnjUvWNS4p0yawxmpqT5POUAABgKBFXADBAPUGndfvq9fzW3lv+9tW1KsKk0ydm6Na/mamLZ41Rflq832MCAACfEFcAcBxN7V16eechvbStWn/eUa361i5FR5rOmZqlLxZP0YUzc5WVFOv3mAAAIAwQVwBwjH21rXph20G9uP2gVu+pU3fQKT0hWkuKcnTBjFydNy1LyXHRfo8JAADCDHEFYNTrCTq9s69eL2yr1ovbDmpXdUCSNDUnSZ//0CRdOCNXp41PV2QEG1IAAID+EVcARqXaQIde2VWjv+w8pL/sPKS6lk5FRZgWT8rQdYvH68IZOZqQyYf6AgCAgSOuAIwK3T1BbahoUMmO3pjatL9RzkkZiTE6rzBLF8zI1flF2Urhdj8AAHCSiCsAI9aBxna9HLoy9cquQ2pq71aESQvGp+trF07T+UXZmj02VRHc7gcAADxAXAEYMdq7erR2b71e3nVIf9lxSNsP9H72VG5KrC6dPUbnT8vRuVOzlJrA1SkAAOA94grAsNUTdNq8v1GvvVujp95q07svPKeO7qCiI02LJmTo5sum6/xp2Zo+JllmXJ0CAACnFnEFYNhwzqm0pkWv7a7Ra7tr9fq7NWpq75YkjUsyfeKMiTq3MFOLJ2UqKZYfbwAAYGjxrw8AYa26qV2vv1urV3fX6LXdNapqbJck5afF69LZY3TO1CydPSVLW9a+oeLimT5PCwAARjPiCkBYqWxo0+rSWq3eU6fVpXUqrWmRJKUlROvsKZn68tQsnTMlSxMyE7jVDwAAhBXiCoBvnHOqqG/Tm3tqtbq0TqtLa1Ve1yZJSomL0uJJGfr44vE6c3KmZo1NYVc/AAAQ1ogrAEPGOaey2latPhxTe2pVGbrNLz0hWosnZeizZ0/SGZMzNH1MiiKJKQAAMIwQVwBOmc7uoLZWNWnt3nqt3Vunt8vqVd3cIUnKSorRGZMydePkDJ0xKVOFOUlcmQIAAMMacQXAM/Utnb0hta9ea/fWa0N5gzq6g5J6N6A4c3KmzgjF1JTsRN4zBQAARhTiCsBJCQad9tQEQlel6vX23nrtOdS7+URUhGlWfqo+ccYELZqYrtPGp2tMapzPEwMAAJxaxBWAAakNdGhjRaPWlzdoQ0WD1pc3qKG1S1Lv+6UWTkjX1QvHaeH4dM0rSFNcdKTPEwMAAAwt4grA+7R19mhzZaM2lDccianDu/iZSdNyknXJzDFaODFdCyeka3IWt/gBAAAQV8Ao190T1K7qgDYcuSLVqJ0Hm9UTdJJ63ys1ryBVnzxjguYVpGl2fqqSYvnRAQAAcCz+hQSMIp3dQe2qbtaW/U3aXNmozfsbta2qWW1dPZKk1PhozStI00UzcjSvIE1zx6UpOznW56kBAACGB+IKGKHaOnu0/UCTNlc2acv+Rm2ubNTOAwF19vTu3pcUG6WZY1N03eICzRuXpvkFaZqQmcDtfQAAACeJuAJGgPqWTm070KRtVc1HQmp3dUChO/uUnhCt2fmp+ty5kzRrbIpm56dqQkYCnysFAADgIeIKGEY6u4N691BA2w80aXtVs7YfaNb2A0062NRx5JjclFjNHpuqS2fnHQmpsalxXJECAAA4xYgrIAw553SwqUPbjkRUk3YcaNbu6oC6Q5ejYiIjNDUnSedMzdKMMSmanpes6WNSeI8UAACAT4grwEfOOR0KdGj3wYB2VQe0uzqgnQd7r0g1tnUdOW5sapym56Xow9NzND0vRTPGJGtiVqKiIyN8nB4AAABHI66AIeCcU1Vju3ZVB7TrYO8VqMMxdXREJcdGaWpukpbOydOM0JWootxkpSZE+zg9AAAABoK4AjzU2R1UeX2rSg+1aPehgHYdDGh3dW9MtXT2HDkuPSFahbnJunxungpzklSYk6zC3CTlJMfy3igAAIBhirgCPqBg0OlAU7tKa1q0p6ZFpYdaVFoTUGlNi8rr2458+K4k5STHqjA3SdcsKtDUnCRNzUlSYU6SMpN4XxQAAMBIQ1wBfXDOqb61S6U1Ae051KLSmvf+6ugOHjk2LjpCk7KSNGtsqpbNHatJWYmamJWoqTlJSo3ndj4AAIDRgrjCqNXdE1RVY7v21bW+91dtq/bWtqipvfvIsVERpvEZCZqUlahzp2ZpYlaiJmclalJ2onKT4/i8KAAAABBXGNma2ru0r/a98VQe+n1/fduRbc2l3oAalx6vgowEzR3XewVqcnaiJmUlaVx6PDvzAQAA4LiIKwxbzjk1tnWpor5N+xvaVNnQpv2hx/sb2rSvrlUNrV3v+TPpCdEan5GgOfmpWjY3T+MzElSQkaDxGQnKS41XJFegAAAAcJKIK4StnqDTwab23mhqaFNFfW9AbXq3Xbev+4sqG9reswOf1Pv+p7Fp8cpPi9flc/I0ITPhSEAVZCQoJY73QAEAAODUIK7gi56gU02gQ1WN7TrQ2K6DTe2qCv1++CrUgcb299y2J/VeeUqJcioal6hzC7OUHwqp/PTe3zMSY9jKHAAAAL4grn2GQL8AABF5SURBVOC5ts4eHWj6azQdfnyg8a+PDwU63rNludT7nqfclDiNTYvTwgnp74mm/LR4jU2LV2JslEpKSlRcvMin/3UAAABA34grDIhzTk3t3TrU3K7q5g4dOvwr0KFDTb2/Vzd16EBTuxrbut7355Njo5SbGqcxKXE6tzBLY1LilJsap7yUOI1JjVNuSpwyE2PYdQ8AAADDFnE1ynV096gm0Knqpva/xlIonI6NqM6jPtvpsJjICGUnxyo7OVbjMxO0eFKGxoQi6nA0jUmNU1IsSw0AAAAjG//iHWHau3pU19KpupZO1QQ6jjyubelUbeh5bei1ukCnmju6+/w+GYkxyk6KVU5KrCZnJR4JqMO/cpJjlZ0Up5T4KN7jBAAAAIi4CmvdPUE1tXerobVT9a1damzrVH1LV7+xVBvoeN/ueYdFRZgyEmOUkRijzKQYFaSn9T5OjDkqmOKUnRyrzKQYPtMJAAAA+ICIqyHQE3RqautSQ1uXGlo71dDapYZQKDW0dakxFE/vedzaqab2vq8qSVJ0pIXiqDeGJmQmHImlzKTYI48zQs9T4rjCBAAAAJxKxNUAOOfU0tmjprYuNbV3qbm9+8jjprZuNbd3qemo1w5/vTemel9zru/vbSalxEUrLSFaaQkxSkuI0cSsRKUnxCg1PlrpoddTE6KVnhCjtPhoZSTFKDmWWAIAAADCyYiPq+6eoFo6etTc0aWWjh4FOroV6OhWS+j3QHv3kUjqDaO/Pj46noL9xNFhcdERSomLVkp8tFLiopSWEKMJmYlKT4hWaiiK0hOjlRYf89eQiu89PpId8gAAAIBhL6zjqjsoba1sUkvnX0PoSBQdCaSe98VSS2fv8+b2bnX0scNdX5Jio5QSFxWKo2iNSYnTtNzkI68lx0UdFU+h56GQSo6LVkwU71ECAAAARrOwjquKQFBLf/5Kv1+Pj45UUlyUkmKjlBgbqaTYKI1Ni1NibO9rva8f8zguSkmxkUqMjVJiTJSSQ3HE1SMAAAAAgxHWcZUVb7rrk6f1GUiJMVEEEQAAAICwEdZxlRRtunR2nt9jAAAAAMAJ8UYhAAAAAPCAJ3FlZpea2Q4z221mN/fxdTOzn4e+vtHMTvPivAAAAAAQLgYdV2YWKek/JV0maaak681s5jGHXSapMPRrhaRfDfa8AAAAABBOvLhytVjSbufcHudcp6QHJS0/5pjlku5zvd6UlGZmvJkKAAAAwIjhxYYW+ZLKj3peIemMARyTL6nq2G9mZivUe3VL2dnZKikp8WBEjCSBQIB1gfdhXaA/rA30hXWB/rA2MBhexFVf+6G7kzim90Xn7pZ0tyQVFRW54uLiQQ2HkaekpESsCxyLdYH+sDbQF9YF+sPawGB4cVtghaSCo56Pk1R5EscAAAAAwLDlRVy9JanQzCaZWYyk6yQ9ccwxT0j6dGjXwDMlNTrn3ndLIAAAAAAMV4O+LdA5121mX5a0SlKkpHucc1vM7MbQ1++StFLSUkm7JbVK+uxgzwsAAAAA4cSL91zJObdSvQF19Gt3HfXYSfqSF+cCAAAAgHDkyYcIAwAAAMBoR1wBAAAAgAeIKwAAAADwAHEFAAAAAB4grgAAAADAA8QVAAAAAHiAuAIAAAAADxBXAAAAAOAB4goAAAAAPEBcAQAAAIAHiCsAAAAA8ABxBQAAAAAeIK4AAAAAwAPEFQAAAAB4gLgCAAAAAA8QVwAAAADgAeIKAAAAADxAXAEAAACAB4grAAAAAPAAcQUAAAAAHiCuAAAAAMADxBUAAAAAeIC4AgAAAAAPEFcAAAAA4AHiCgAAAAA8QFwBAAAAgAeIKwAAAADwAHEFAAAAAB4grgAAAADAA8QVAAAAAHiAuAIAAAAADxBXAAAAAOAB4goAAAAAPEBcAQAAAIAHiCsAAAAA8ABxBQAAAAAeIK4AAAAAwAPEFQAAAAB4gLgCAAAAAA8QVwAAAADgAeIKAAAAADxAXAEAAACAB4grAAAAAPAAcQUAAAAAHiCuAAAAAMADxBUAAAAAeIC4AgAAAAAPEFcAAAAA4AHiCgAAAAA8QFwBAAAAgAeIKwAAAADwAHEFAAAAAB4grgAAAADAA8QVAAAAAHiAuAIAAAAADxBXAAAAAOAB4goAAAAAPEBcAQAAAIAHogbzh80sQ9JDkiZKKpP0MedcfR/HlUlqltQjqds5t2gw5wUAAACAcDPYK1c3S3rROVco6cXQ8/4scc7NJ6wAAAAAjESDjavlku4NPb5X0kcG+f0AAAAAYFgy59zJ/2GzBudc2lHP651z6X0cVyqpXpKT9Gvn3N3H+Z4rJK2QpOzs7IUPP/zwSc+HkSkQCCgpKcnvMRBmWBfoD2sDfWFdoD+sDfRlyZIlawdyB94J33NlZi9IGtPHl275APOc45yrNLMcSc+b2Xbn3Mt9HRgKr7slqaioyBUXF3+A02A0KCkpEesCx2JdoD+sDfSFdYH+sDYwGCeMK+fchf19zcwOmlmec67KzPIkVffzPSpDv1eb2WOSFkvqM64AAAAAYDga7HuunpD0mdDjz0h6/NgDzCzRzJIPP5Z0saTNgzwvAAAAAISVwcbVHZIuMrNdki4KPZeZjTWzlaFjciW9amYbJK2R9LRz7tlBnhcAAAAAwsqgPufKOVcr6YI+Xq+UtDT0eI+keYM5DwAAAACEu8FeuQIAAAAAiLgCAAAAAE8QVwAAAADgAeIKAAAAADxAXAEAAACAB4grAAAAAPAAcQUAAAAAHiCuAAAAAMADxBUAAAAAeIC4AgAAAAAPEFcAAAAA4AHiCgAAAAA8QFwBAAAAgAeIKwAAAADwAHEFAAAAAB4grgAAAADAA8QVAAAAAHiAuAIAAAAADxBXAAAAAOAB4goAAAAAPEBcAQAAAIAHiCsAAAAA8ABxBQAAAAAeIK4AAAAAwAPEFQAAAAB4gLgCAAAAAA8QVwAAAADgAeIKAAAAADxAXAEAAACAB4grAAAAAPAAcQUAAAAAHiCuAAAAAMADxBUAAAAAeIC4AgAAAAAPEFcAAAAA4AHiCgAAAAA8QFwBAAAAgAeIKwAAAADwAHEFAAAAAB4grgAAAADAA8QVAAAAAHiAuAIAAAAADxBXAAAAAOAB4goAAAAAPEBcAQAAAIAHiCsAAAAA8ABxBQAAAAAeIK4AAAAAwAPEFQAAAAB4gLgCAAAAAA8QVwAAAADgAeIKAAAAADxAXAEAAACAB4grAAAAAPAAcQUAAAAAHiCuAAAAAMADxBUAAAAAeIC4AgAAAAAPDCquzOwaM9tiZkEzW3Sc4y41sx1mttvMbh7MOQEAAAAgHA32ytVmSVdJerm/A8wsUtJ/SrpM0kxJ15vZzEGeFwAAAADCStRg/rBzbpskmdnxDlssabdzbk/o2AclLZe0dTDnBgAAAIBwMhTvucqXVH7U84rQawAAAAAwYpzwypWZvSBpTB9fusU59/gAztHXZS13nPOtkLRCkrKzs1VSUjKAU2A0CQQCrAu8D+sC/WFtoC+sC/SHtYHBOGFcOecuHOQ5KiQVHPV8nKTK45zvbkl3S1JRUZErLi4e5Okx0pSUlIh1gWOxLtAf1gb6wrpAf1gbGIyhuC3wLUmFZjbJzGIkXSfpiSE4LwAAAAAMmcFuxX6lmVVIOkvS02a2KvT6WDNbKUnOuW5JX5a0StI2SQ8757YMbmwAAAAACC+D3S3wMUmP9fF6paSlRz1fKWnlYM4FAAAAAOFsKG4LBAAAAIARj7gCAAAAAA8QVwAAAADgAeIKAAAAADxAXAEAAACAB4grAAAAAPAAcQUAAAAAHiCuAAAAAMADxBUAAAAAeIC4AgAAAAAPEFcAAAAA4AHiCgAAAAA8QFwBAAAAgAeIKwAAAADwAHEFAAAAAB4grgAAAADAA8QVAAAAAHiAuAIAAAAADxBXAAAAAOAB4goAAAAAPEBcAQAAAIAHiCsAAAAA8ABxBQAAAAAeIK4AAAAAwAPEFQAAAAB4gLgCAAAAAA8QVwAAAADgAeIKAAAAADxAXAEAAACAB4grAAAAAPAAcQUAAAAAHiCuAAAAAMADxBUAAAAAeIC4AgAAAAAPEFcAAAAA4AHiCgAAAAA8QFwBAAAAgAeIKwAAAADwAHEFAAAAAB4grgAAAADAA8QVAAAAAHiAuAIAAAAADxBXAAAAAOAB4goAAAAAPEBcAQAAAIAHiCsAAAAA8ABxBQAAAAAeIK4AAAAAwAPEFQAAAAB4gLgCAAAAAA8QVwAAAADgAeIKAAAAADxAXAEAAACAB4grAAAAAPAAcQUAAAAAHiCuAAAAAMADxBUAAAAAeIC4AgAAAAAPDCquzOwaM9tiZkEzW3Sc48rMbJOZrTeztwdzTgAAAAAIR1GD/PObJV0l6dcDOHaJc65mkOcDAAAAgLA0qLhyzm2TJDPzZhoAAAAAGKaG6j1XTtJzZrbWzFYM0TkBAAAAYMic8MqVmb0gaUwfX7rFOff4AM9zjnOu0sxyJD1vZtudcy/3c74Vkg4HWIeZbR7gOTB6ZEniFlMci3WB/rA20BfWBfrD2kBfigZy0Anjyjl34WAncc5Vhn6vNrPHJC2W1GdcOefulnS3JJnZ2865fjfKwOjEukBfWBfoD2sDfWFdoD+sDfRloJvynfLbAs0s0cySDz+WdLF6N8IAAAAAgBFjsFuxX2lmFZLOkvS0ma0KvT7WzFaGDsuV9KqZbZC0RtLTzrlnB3NeAAAAAAg3g90t8DFJj/XxeqWkpaHHeyTNO8lT3H3y02EEY12gL6wL9Ie1gb6wLtAf1gb6MqB1Yc65Uz0IAAAAAIx4Q7UVOwAAAACMaGEdV2b2PTPbaGbrzew5Mxvr90wID2b2QzPbHlofj5lZmt8zwX9mdo2ZbTGzoJmx09MoZ2aXmtkOM9ttZjf7PQ/Cg5ndY2bVfNQLjmZmBWb2ZzPbFvp75Ca/Z0J4MLM4M1tjZhtCa+Pfjnt8ON8WaGYpzrmm0ON/lDTTOXejz2MhDJjZxZJecs51m9n3Jck59y2fx4LPzGyGpKCkX0v6unNuQNumYuQxs0hJOyVdJKlC0luSrnfObfV1MPjOzM6TFJB0n3Nutt/zIDyYWZ6kPOfcutAu12slfYSfGTAzk5TonAuYWbSkVyXd5Jx7s6/jw/rK1eGwCkmUFL4liCHlnHvOOdcdevqmpHF+zoPw4Jzb5pzb4fccCAuLJe12zu1xznVKelDScp9nQhhwzr0sqc7vORBenHNVzrl1ocfNkrZJyvd3KoQD1ysQehod+tVvk4R1XEmSmd1uZuWSPiHpu37Pg7D0OUnP+D0EgLCSL6n8qOcV4h9KAAbAzCZKWiBptb+TIFyYWaSZrZdULel551y/a8P3uDKzF8xscx+/lkuSc+4W51yBpPslfdnfaTGUTrQ2QsfcIqlbvesDo8BA1gUgyfp4jbsfAByXmSVJelTSV465gwqjmHOuxzk3X713Si02s35vKR7U51x5wTl34QAPfUDS05JuPYXjIIycaG2Y2WckLZN0gQvnNw/CUx/gZwZGtwpJBUc9Hyep0qdZAAwDoffTPCrpfufcH/2eB+HHOddgZiWSLpXU56Y4vl+5Oh4zKzzq6RWStvs1C8KLmV0q6VuSrnDOtfo9D4Cw85akQjObZGYxkq6T9ITPMwEIU6FNC34jaZtz7id+z4PwYWbZh3elNrN4SRfqOE0S7rsFPiqpSL27f+2VdKNzbr+/UyEcmNluSbGSakMvvclOkjCzKyX9QlK2pAZJ651zl/g7FfxiZksl3SkpUtI9zrnbfR4JYcDMfi+pWFKWpIOSbnXO/cbXoeA7MztX0iuSNqn3352S9G3n3Er/pkI4MLO5ku5V798lEZIeds7d1u/x4RxXAAAAADBchPVtgQAAAAAwXBBXAAAAAOAB4goAAAAAPEBcAQAAAIAHiCsAAAAA8ABxBQAAAAAeIK4AAAAAwAPEFQAAAAB44P8DBsD9j/dJYQQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1008x504 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Wykres funkcji softplus\n",
    "plot(lambda x: math.log(1 + math.exp(x)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Problem zanikającego gradientu (*vanishing gradient problem*)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Sigmoidalne funkcje aktywacji ograniczają wartości na wyjściach neuronów do niewielkich przedziałów ($(-1, 1)$, $(0, 1)$ itp.).\n",
    "* Jeżeli sieć ma wiele warstw, to podczas propagacji wstecznej mnożymy przez siebie wiele małych wartości → obliczony gradient jest mały.\n",
    "* Im więcej warstw, tym silniejszy efekt zanikania."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Sposoby na zanikający gradient\n",
    "\n",
    "* Modyfikacja algorytmu optymalizacji (*RProp*, *RMSProp*)\n",
    "* Użycie innej funckji aktywacji (ReLU, softplus)\n",
    "* Dodanie warstw *dropout*\n",
    "* Nowe architektury (LSTM itp.)\n",
    "* Więcej danych, zwiększenie mocy obliczeniowej"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 11.2. Odmiany metody gradientu prostego\n",
    "\n",
    "* Batch gradient descent\n",
    "* Stochastic gradient descent\n",
    "* Mini-batch gradient descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### *Batch gradient descent*\n",
    "\n",
    "* Klasyczna wersja metody gradientu prostego\n",
    "* Obliczamy gradient funkcji kosztu względem całego zbioru treningowego:\n",
    "  $$ \\theta := \\theta - \\alpha \\cdot \\nabla_\\theta J(\\theta) $$\n",
    "* Dlatego może działać bardzo powoli\n",
    "* Nie można dodawać nowych przykładów na bieżąco w trakcie trenowania modelu (*online learning*)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### *Stochastic gradient descent* (SGD)\n",
    "\n",
    "* Aktualizacja parametrów dla każdego przykładu:\n",
    "  $$ \\theta := \\theta - \\alpha \\cdot \\nabla_\\theta \\, J \\! \\left( \\theta, x^{(i)}, y^{(i)} \\right) $$\n",
    "* Dużo szybszy niż _batch gradient descent_\n",
    "* Można dodawać nowe przykłady na bieżąco w trakcie trenowania (*online learning*)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Częsta aktualizacja parametrów z dużą wariancją:\n",
    "\n",
    "<img src=\"http://ruder.io/content/images/2016/09/sgd_fluctuation.png\" style=\"margin: auto;\" width=\"50%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Z jednej strony dzięki temu uczenie nie \"utyka\" w złych minimach lokalnych, ale z drugiej strony może „wyskoczyć” z dobrego minimum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### _Mini-batch gradient descent_\n",
    "\n",
    "* Kompromis między _batch gradient descent_ i SGD\n",
    "  $$ \\theta := \\theta - \\alpha \\cdot \\nabla_\\theta \\, J \\left( \\theta, x^{(i : i+n)}, y^{(i : i_n)} \\right) $$\n",
    "* Stabilniejsza zbieżność dzięki redukcji wariancji aktualizacji parametrów\n",
    "* Szybszy niż klasyczny _batch gradient descent_\n",
    "* Typowa wielkość batcha: między 50 a 256 przykładów"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Wady klasycznej metody gradientu prostego, czyli dlaczego potrzebujemy optymalizacji\n",
    "\n",
    "* Trudno dobrać właściwą szybkość uczenia (*learning rate*)\n",
    "* Jedna ustalona wartość stałej uczenia się dla wszystkich parametrów\n",
    "* Funkcja kosztu dla sieci neuronowych nie jest wypukła, więc uczenie może utknąć w złym minimum lokalnym lub punkcie siodłowym"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 11.3. Algorytmy optymalizacji metody gradientu\n",
    "\n",
    "* Momentum\n",
    "* Nesterov Accelerated Gradient\n",
    "* Adagrad\n",
    "* Adadelta\n",
    "* RMSprop\n",
    "* Adam\n",
    "* Nadam\n",
    "* AMSGrad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Momentum\n",
    "\n",
    "* SGD źle radzi sobie w „wąwozach” funkcji kosztu\n",
    "* Momentum rozwiązuje ten problem przez dodanie współczynnika $\\gamma$, który można trakować jako „pęd” spadającej piłki:\n",
    "  $$ v_t := \\gamma \\, v_{t-1} + \\alpha \\, \\nabla_\\theta J(\\theta) $$\n",
    "  $$ \\theta := \\theta - v_t $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Przyspiesony gradient Nesterova (*Nesterov Accelerated Gradient*, NAG)\n",
    "\n",
    "* Momentum czasami powoduje niekontrolowane rozpędzanie się piłki, przez co staje się „mniej sterowna”\n",
    "* Nesterov do piłki posiadającej pęd dodaje „hamulec”, który spowalnia piłkę przed wzniesieniem:\n",
    "  $$ v_t := \\gamma \\, v_{t-1} + \\alpha \\, \\nabla_\\theta J(\\theta - \\gamma \\, v_{t-1}) $$\n",
    "  $$ \\theta := \\theta - v_t $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Adagrad\n",
    "\n",
    "* “<b>Ada</b>ptive <b>grad</b>ient”\n",
    "* Adagrad dostosowuje współczynnik uczenia (*learning rate*) do parametrów: zmniejsza go dla cech występujących częściej, a zwiększa dla występujących rzadziej\n",
    "* Świetny do trenowania na rzadkich (*sparse*) zbiorach danych\n",
    "* Wada: współczynnik uczenia może czasami gwałtownie maleć"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Adadelta i RMSprop\n",
    "* Warianty algorytmu Adagrad, które radzą sobie z problemem gwałtownych zmian współczynnika uczenia"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Adam\n",
    "\n",
    "* “<b>Ada</b>ptive <b>m</b>oment estimation”\n",
    "* Łączy zalety algorytmów RMSprop i Momentum\n",
    "* Można go porównać do piłki mającej ciężar i opór\n",
    "* Obecnie jeden z najpopularniejszych algorytmów optymalizacji"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Nadam\n",
    "* “<b>N</b>esterov-accelerated <b>ada</b>ptive <b>m</b>oment estimation”\n",
    "* Łączy zalety algorytmów Adam i Nesterov Accelerated Gradient"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### AMSGrad\n",
    "* Wariant algorytmu Adam lepiej dostosowany do zadań takich jak rozpoznawanie obiektów czy tłumaczenie maszynowe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"contours_evaluation_optimizers.gif\" style=\"margin: auto;\" width=\"60%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"saddle_point_evaluation_optimizers.gif\" style=\"margin: auto;\" width=\"50%\" />"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "livereveal": {
   "start_slideshow_at": "selected",
   "theme": "white"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}