umz21/wyk/11_Wielowarstwowe_sieci_neuronowe.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Uczenie maszynowe UMZ 2019/2020\n",
    "### 26 maja 2020\n",
    "# 11. Wielowarstwowe sieci neuronowe i algorytmy optymalizacji"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 11.1. Funkcje aktywacji"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Każda funkcja aktywacji ma swoje zalety i wady.\n",
    "* Różne rodzaje funkcji aktywacji nadają się do różnych zastosowań."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/pawel/.local/lib/python2.7/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n",
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import math\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import random\n",
    "\n",
    "import keras\n",
    "from keras.datasets import mnist\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, SimpleRNN, LSTM\n",
    "from keras.optimizers import Adagrad, Adam, RMSprop, SGD\n",
    "\n",
    "from IPython.display import YouTubeVideo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "def plot(fun):\n",
    "    x = np.arange(-3.0, 3.0, 0.01)\n",
    "    y = [fun(x_i) for x_i in x]\n",
    "    fig = plt.figure(figsize=(14, 7))\n",
    "    ax = fig.add_subplot(111)\n",
    "    fig.subplots_adjust(left=0.1, right=0.9, bottom=0.1, top=0.9)\n",
    "    ax.set_xlim(-3.0, 3.0)\n",
    "    ax.set_ylim(-1.5, 1.5)\n",
    "    ax.grid()\n",
    "    ax.plot(x, y)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Funkcja logistyczna"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$$ g(x) = \\frac{1}{1 + e^{-x}} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Przyjmuje wartości z przedziału $(0, 1)$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Funkcja logistyczna – wykres"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1cAAAG2CAYAAACTRXz+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XmsXudh3/nfw7vvGy/3VRu1Wou1\n2HAWOk4cx0jjpq1bezpN0mXUdhq0M0Axk04GCaaDAl2AWVNMa6RB00GRNJhOWs/UbdI0ZdJ2YFmW\nLVnWZslcxEUSl7vxbrzbmT/el1ekxCvJvke8l+TnA7x43/O+5/I8NB6R/Pqc87ylqqoAAACwPls2\negAAAAA3A3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQg1riqpTya6WUs6WUb6/x+eFSymQp\n5dnm45fqOC4AAMBm0VrTr/OPk/xKkn/yHvv8h6qqfrKm4wEAAGwqtZy5qqrqD5OM1fFrAQAA3Iiu\n5z1XHy+lPFdK+dellPuu43EBAAA+dHVdFvh+vpFkf1VV06WUzyb5F0nuvNaOpZQnkzyZJJ2dnR/d\nt2/fdRoiN4qVlZVs2WItFq5mXrAWc4NrMS9Yi7nBtXznO985X1XV6PvtV6qqquWApZQDSf7fqqru\n/wD7Hk/yaFVV599rv0OHDlWvvPJKLePj5nHkyJEcPnx4o4fBJmNesBZzg2sxL1iLucG1lFKeqarq\n0ffb77pkeSllRymlNF8/3jzuhetxbAAAgOuhlssCSym/keRwkq2llFNJfjlJW5JUVfUPkvyJJH+5\nlLKUZC7JF6q6TpkBAABsArXEVVVVX3yfz38ljaXaAQAAbkru1gMAAKiBuAIAAKiBuAIAAKiBuAIA\nAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiB\nuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIA\nAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiB\nuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIA\nAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiB\nuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKhBLXFVSvm1UsrZUsq31/i8\nlFL+t1LKa6WUb5VSHqnjuAAAAJtFXWeu/nGSz7zH5z+R5M7m48kk/0dNxwUAANgUaomrqqr+MMnY\ne+zyuST/pGr4apLBUsrOOo4NAACwGVyve652Jzl5xfap5nsAAAA3hdaNHsA7lVKeTOPSwYyOjubI\nkSMbOyA2nenpafOCdzEvWIu5wbWYF6zF3GA9rldcnU6y94rtPc333qWqqi8l+VKSHDp0qDp8+PCH\nPjhuLEeOHIl5wTuZF6zF3OBazAvWYm6wHtfrssAvJ/mZ5qqBH0syWVXVG9fp2AAAAB+6Ws5clVJ+\nI8nhJFtLKaeS/HKStiSpquofJPlKks8meS3JbJI/W8dxAQAANota4qqqqi++z+dVkr9Sx7EAAAA2\no023oAUAAMCHpaqqXFpaycylpcwuLGdmYSkzl5Yyc2k5swuN55mFt7d//L4dH/jXFlcAAMCmd2lp\nOdPzS5m+tJSLzefp+aXMLFy9/fbni1e9N7uwvPq8vFJ9oGOWkuwd7v7AYxRXAADAh+bymaKpucVM\nzi1man4xU3NLmZpvbs8t5uIVETQ9v3T1dvP1wvLK+x6rZUtJb0drejta09fZeB7sbs+e4e70tLek\nu73xXndHS3raW9Pd3tLcbl39vKejJT0drelpb01n25aUUvKnPuDvVVwBAADvaXG5EUdT80vvGUlT\n80tXvG5+Prf4vmHU0bplNYZ6m8+7Brve9d7qo7M1fVe+39mavo621RjaKOIKAABuESsrVS7OL2Vi\nbiHjs4sZn13IZPN5fHYxE7MLmWhuT1zx+cVLS+/567ZuKRnoakv/5Udna3YPdTXe62xLf1dr+jvb\n3t6ns3X1dV9nazpaW67T/wIfLnEFAAA3oKqqMjW/lAvTlzI2s5ALMwu5ML2QsZlLzTBqxNL47EIm\n5hYz0dxe63ajUpL+zrYMdbdloLs9I73tuWNbbwa72zLY1Z7B7mtFUuP1Rp8x2izEFQAAbAKXY2ls\nZiEXpi/lwszCNV5fjqhLGZ9dyOLytUupq60lQ91tGexuz1BPW3YOdGWwuy1D3Y1IGuxuf/vz5vNA\nV1tatgik9RBXAADwIamqKhOzizk3fSnnLjYeZy/Or76+MLOQ882zTWMza8dSb0drRnrbM9zTnt2D\nnfnI7oEM97ZnpKe9+X7H6uuh7vZ0tt0cl9ndaMQVAAB8j+YXl5uh1Iym6Us5NzV/VURdfv9awdTZ\ntiWjfR3Z2tuR3YOdeWB3f0Z6G4E03NO++los3VjEFQAANC2tVDk9MZc3J+fz1tT828/N142IunTN\nBR5KSUZ6OjLa15FtfR25c3tfRvs6Mtr79nujzUdvR6t7lG5C4goAgJteVVW5eGkpb02+HUpvR9Ol\nvDU1nzcm53Nh+lKq3/39q362vWVLtg90ZEd/Z+7Z0Z8fuvPtSLocT9v6OzLc3Z7Wli0b9DtkMxBX\nAADc8OYXl3NmYi5nJuZzZmIupyfmGtuTc3ljohFRswvL7/q5we627OjvzPb+zty7sz/z42/mYw/e\nvfrejoHODHW3OcvEByKuAADY1FZWqlyYWXg7mK6Mp2ZMXZhZuOpnSkm29XVk50BX7t7Zlx8+NJod\nzVi6/Ly9v/Nd9zIdOTKWw4/vu56/PW4i4goAgA21vFLlran5nBybzcnxuZwcm70qpM5MzmdhaeWq\nn+lub8nuwa7sGuzK/bsHsnuwM7ua27sHu7K9vzPtrS7R4/oSVwAAfKguL0d+cnw2J8fm8vrYbPP1\nbE6Nz+X0+FwWlt+Op1KS7X2d2T3UlQf2DObH7+9shNTA2/HU32VBCDYfcQUAwLrNLy7n5NhsI5yu\nOAP1ejOgpt+xut5Qd1v2Dnfn3p39+fR927NvuDt7h7qzd7g7uwY709Fq6XFuPOIKAIAPZG5hOSfG\nZnL8/GyOX5jJiQszOXZ+JicuzOaNyfmr9u1s27IaS08cHM7e4cbrxntd6ets26DfBXx4xBUAAKtm\nF5Zy4sJsjp+fyfELs1cF1JtTVwfUcE97Dox05+O3jeTA1p7sH+nOnmY8jfZ2uGyPW464AgC4xSwt\nr+T1sdkcPTeT756bztFzMzl2YSbHz8/k7MVLV+27tbc9+0d68ok7tubASHf2b+3JwZGe7BvpzkCX\ns09wJXEFAHCTmpxdzGvnpnP03HS+e26m+Tyd18dms7hcre63tbc9B7f25IfuGs2Bke4c2NqTAyON\nM1Eu34MPTlwBANzAlpZXcmp8bvUM1JXPV373U1tLyf6RntyxrTefvm9Hbh/tzW2jPbl9a28GugUU\n1EFcAQDcAJaWV3L8wmxefetivvPWdF49ezGvvjWdY+dnrlrGfKSnPbeN9uTH7t3eiKfR3tw22pu9\nQ11pbfG9T/BhElcAAJvI5Yh67Wwjor7zViOijp6fvupSvr3DXblrW18O3z2a20d7m4+eDHa3b+Do\n4dYmrgAANsDS8kpOjL19Juo7b13Ma2cbl/RdeSbqckR98u5tuXNbb+7a3pfbt/Wku90/42Cz8V8l\nAMCH7Pz0pbz8xsW8/OZUXmo+v3p2OgtL746oHz40mru29YkouAH5rxUAoCbzi8t57ex0Xn7zYl5+\nY6rx/OZUzk+/vbDEtr6O3L2zP5+4Y2sObe/Lndt7c8e2XhEFNwH/FQMAfI+qqsqZyfkrAqoRU0fP\nz2R5pXFfVEfrlhza0ZdPHtqWu3f2554dfTm0oy8jvR0bPHrgwyKuAADew9LySl47N50XTk/lhTNT\neeHMZF56YypT80ur++wZ6srdO/rzmft35O4d/bl7Z18OjPSkZUvZwJED15u4AgBoml9czktvXI6o\nqbx4ZjIvv3kxl5r3RnW2bcndO/rzkw/uyj3Ns1F37ehLvy/aBSKuAIBb1OTsYl54YzIvnnn7jNRr\nZ6fTvKov/Z2tuW/XQP7Mx/bn/t0DuW9Xfw5u7fFdUcCaxBUAcNM7d/FSnj89sXpp3zNHZ3Pu3/zu\n6uc7+jtz367+fOa+Hbl3VyOk9gx1pRSX9QEfnLgCAG4qE7MLef70ZL51ajLfOjWR509N5szk/Orn\nB7f25ODAlvzZH74j9zVDaqtFJoAaiCsA4IY1fWkp3z7diKhvnZrM86cnc+LC7OrnB7f25LGDw3lg\n90A+smcw9+7qT29Ha44cOZLDh+/YwJEDNyNxBQDcEOYWlvPiG40zUs+fmsxzpyZy9PxMquY9UrsH\nu/Lg3oF84bF9eXDPQO7bPZCBLgtNANePuAIANp2l5ZW8/ObFPHtyYjWkXj07vfodUtv7O/LA7sF8\n7qHd+ciegTywe8D3RwEbTlwBABvuzcn5fPP18Tx7ciLffH0iz5+ezNzicpJkqLstH9kzmE/fuz0P\n7BnMR/YMZHt/5waPGODdxBUAcF3NLizl+VOTqyH17MmJvDnVWHCivWVL7tvdny88vjcP7xvKw3sH\nrdoH3DDEFQDwoVlZqXL0/MxVZ6Veeevi6uV9+0e688Rtw3l472Ae2jeUe3b2paO1ZYNHDfD9EVcA\nQG3GZhby7MnxPPv6RL55snFW6uL8UpKkr6M1D+0bzH95z+15eN9gHtwz6D4p4KYirgCA78vKSpXv\nnpvO10+M55kT4/nGifEcPT+TJNlSkkM7+vNHHtyVh/YO5pF9g7lta2+2bHF5H3DzElcAwAcyu7CU\nZ09O5BuXY+r1iUzOLSZJhnva88i+oXz+0b15eN9gHtg9kJ4O/8wAbi3+1AMArunMxFyeaYbUMyfG\n8+IbU6v3St25rTeffWBHHtk3lI/uH8rBrT0WnQBueeIKAMjS8kpeeuNivn5ibPUSvzOTjRX8utpa\n8tDewfzlH749H90/lEf2DWWg25fzAryTuAKAW9Dk3GK+8fp4njneOCv17MmJ1e+V2jXQmUf2D+W/\n2D+UR/cP5+6dfWlr2bLBIwbY/MQVANwC3pycz9eOj+XpY2N5+vhYXnnrYqoqadlScu/O/vypx/bm\no/sbl/jtGuza6OEC3JDEFQDcZKqqyrHzM3n6+Fi+dmw8Tx8fy+tjs0mS7vaWfHT/UD77wM48emAo\nD+0dTHe7fw4A1MGfpgBwg1teqfLSG1P5WvOs1NPHx3J+eiFJYxW/xw4M5Wc+vj+PHxzOvTv70+oS\nP4APhbgCgBvM/OJynjs50Tgzdbyx+MT0pcYX9e4Z6soP3Tmaxw4O57EDw7l91Cp+ANeLuAKATW5q\nfjHPHB9fvWfqW6cms7C8kiS5a3tvPvfQrjx+cDiPHxzOzgH3SwFsFHEFAJvM2am3F5/42vHxvPzm\nVKoqad1Scv/ugfzcJw7ksQPDeXT/UIZ62jd6uAA0iSsA2GBvTM7lqaNjeerYhXz16FiOnZ9J0vh+\nqUf2D+avferOPH5gOA/ts/gEwGbmT2gAuM5OT8zlq9+9kKeOXchTx8Zy4kJjJb++ztY8fmA4X3x8\nbx4/OJL7dvX7fimAG4i4AoAPUVVVOTU+l68ebZyVeurYhZwan0uSDHS15fGDw/mZjx/IEweHc8/O\n/rRssfgEwI1KXAFAjaqqyutjs/nq0QvNS/3GcnqiEVND3W154uBI/vwPHMwTB0dy946+bBFTADcN\ncQUA63D5C3ufOja2GlRvTs0nSUZ62vPEbcP5iz98W544OJI7t/WKKYCbmLgCgO9BVVX57rnp5iV+\nY3nq6IWcvXgpSTLa15EnDg7nidtG8vHbhnP7aK/vmAK4hYgrAHgPVVXl1bPTV1zmdyHnpxeSJNv7\nO/Lx20fyxMGRPHHbcG7b6gt7AW5l4goArrCyUuWVty7mqaONlfyeOjaWsZlGTO0a6MwP3jmaj902\nnCcOjmT/SLeYAmCVuALglrayUuWlN6cal/kdvZCvHR/LxOxikmTPUFc+eWhbnrhtOB+/bSR7hrrE\nFABrqiWuSimfSfK/JmlJ8qtVVf3td3z+c0n+XpLTzbd+paqqX63j2ADwvVheqfLimanmF/ZeyNeO\njWVqfilJsm+4O5++d/vqZX57hro3eLQA3EjWHVellJYkfz/JjyU5leTpUsqXq6p68R27/rOqqn5+\nvccDgO/F8kqV505ONO6ZOjaWp4+N5eKlRkwd3NqTzz6wMx+7rRFTOwe6Nni0ANzI6jhz9XiS16qq\nOpokpZTfTPK5JO+MKwD40C0tr+T505OrS6N/9bXZzP/uf0qS3Dbakz/y0K48cXA4H7ttJNv7Ozd4\ntADcTOqIq91JTl6xfSrJE9fY74+XUn4oyXeS/NdVVZ28xj4ppTyZ5MkkGR0dzZEjR2oYIjeT6elp\n84J3MS9uXUsrVY5PreTlseW8MraSV8eXM7/c+GxXT8ljo1Ue2N6Zu4a3ZLAjSS4kExfy0jdezUsb\nOXA2lD8zWIu5wXpcrwUt/p8kv1FV1aVSyl9M8utJfuRaO1ZV9aUkX0qSQ4cOVYcPH75OQ+RGceTI\nkZgXvJN5cetYXF7Jt05NNs5KHb2QZ06MZ3ahUVN3be/N5x8bycduG8njB4cz2tdhbnBN5gVrMTdY\njzri6nSSvVds78nbC1ckSaqqunDF5q8m+bs1HBeAW8DC0kq+dWpi9TK/rx8fz9xiI6YObe/L5z+6\nZzWmRno7Nni0ANzK6oirp5PcWUo5mEZUfSHJf3blDqWUnVVVvdHc/KnElRgAXNulpeXGmanvXshX\njzXOTM0vriRJ7t7Rlz/12N587LbhPH5wJMM97Rs8WgB427rjqqqqpVLKzyf5nTSWYv+1qqpeKKX8\nzSRfr6rqy0n+ainlp5IsJRlL8nPrPS4AN4dLS8t59vW3z0w9c2I8l5ZWUkpy947+fPHxfY2l0Q8O\nZ0hMAbCJ1XLPVVVVX0nylXe890tXvP4bSf5GHccC4MY2v7icZ5tLo3/16IV88/WJ1Zi6Z0d//vQT\n+5tnpoYz2C2mALhxXK8FLQC4Rc0vLucbr4/nqaONM1PfPDmRhWZM3berP//5x/Y37pk6MJyB7raN\nHi4AfN/EFQC1mltYzjdfH2+emRrLsycnsrC8ki0luW/XQH72442YevTAcAa6xBQANw9xBcC6TF9a\nyjMnxvO1YxfytWONmFpcrrKlJA/sHsjPfeJAPnbbcB49MJz+TjEFwM1LXAHwPRmfWcjTx8fytWNj\n+drxsbxwZirLK1VatpTcv3sgf+4HDuZjB0fy6IGh9IkpAG4h4gqA9/TW1HyeOjaWp481guqVty4m\nSdpbt+ThvYP5K4dvz+MHR/LwvsH0dPhrBYBbl78FAVhVVVVOjs3lqeYlfl87PpYTF2aTJD3tLfno\ngeH81EO78vjB4Xxkz0A6Wls2eMQAsHmIK4Bb2MpKldfOTV91ZurNqfkkyVB3Wx47MJw/87H9eeLg\nSO7Z2ZfWli0bPGIA2LzEFcAtZGl5JS+9cXH1zNTTx8cyPruYJNne35EnDo7ksYPDeeLgcO4Y7c2W\nLWWDRwwANw5xBXATu/yFvY3V/MbyzInxTF9aSpLsH+nOj96zPY8fHM4TB0eyd7grpYgpAPh+iSuA\nm8j56Uv5+vHxPHNiLE8fH88LZyazuFwlSe7a3puffnh3Hjs4nMcPDGfHQOcGjxYAbi7iCuAGVVVV\njp6fydePj+Xrx8fz9RPjOXZ+JkljJb8H9wzkL/zgbXnswFAe2TeUwe72DR4xANzcxBXADeLS0nK+\nfXqqEVMnxvPMifGMzSwkaSw+8dH9w/nCY3vz6IGh3L/bSn4AcL2JK4BNanJ2Mc+83ri875nj43n2\n1EQWllaSJAe39uRH7t6Wxw4M5aP7h3P7aI/7pQBgg4krgE3g8iV+3zgxnm+8PpFnTozlO29NJ0la\nt5Tct3sgP/Ox/Xm0GVOjfR0bPGIA4J3EFcAGmL60lOdOTjRjajzfPDmRieaS6H0drXlk/1B+6sFd\n+ej+4Ty0dzBd7S7xA4DNTlwBfMjeeVbqm6+P55W3LqZqLOKXO7b15tP3bs8j+4byyP4h3y8FADco\ncQVQs4vzi3nu5GS++fo1zkp1tuahvYP58ft25JH9Q3lo72AGuto2eMQAQB3EFcA6rKxUOXZh7bNS\nd27rzY/fuyOP7B/MI/uGcruzUgBw0xJXAN+Ds1Pzee7UZJ47OZHnTk3kuZMTmZpfStI4K/XwvqF8\n5v4deXifs1IAcKsRVwBruDi/mOdPT+a5k42Y+tapiZyZnE+StGwpuXtHX37ywV15cM+As1IAgLgC\nSJKFpZW8/OZU84xUI6ZeOze9ennf/pHuPHpgOA/uHcxDewdy784BK/gBAFcRV8At5/J9Ut86NZHn\nTk7m2ZMTefHMVBaWG1/QO9LTngf3DuYnP7IrD+4dyIN7BjPU077BowYANjtxBdzUVlYay6B/+/Rk\nnm8+XjwzlelLjfukutpa8sCegfzcJw7kwT2DeXDvQHYPdqUUl/cBAN8bcQXcNJZXqhw9N70aUS+c\nnsoLZyYzs7CcJGlv3ZJ7dvbnjz68Kw/sHsiDewdzx2hvWlu2bPDIAYCbgbgCbkhLyyv57rmZPH96\nMt8+PZn/9OJcTv/+72S2GVKdbVty787+/PGP7sn9uwfywO6B3LGtN21CCgD4kIgrYNO7tLScV9+a\nzktvTK1e3vfiG1OZX2zcI9XV1pI9PcmffHTvakjdPtrjjBQAcF2JK2BTGZtZyEtvTOWlN6by4pmp\nvPjGVF47O52llcayfT3tLblv10C++Pi+PNAMqdtGe/Mf/vAPcvjwfRs8egDgViaugA2xslLlxNhs\nXjzTDKlmTL05Nb+6z/b+jtyzsz8/cve23LurP/fs7M+BkZ60+C4pAGATElfAh252YSmvvHkxL15x\nRurlNy+u3h/VsqXkjtHefPz2kdyzsy/37hzIPTv7MtLbscEjBwD44MQVUJvF5ZUcOz+Tl9+8mO+8\nebHx/NbFvD42u7pPX0dr7tnVnz/56N7cu7M/9+7qzx3betPZ5gt5AYAbm7gCvmcrK1VOT8zllTcv\n5pW3LuaVZkR999x0Fpcb90a1bCk5uLUnD+wZyJ/46J4c2tGXe3f2Z8+Q75ACAG5O4gpYU1VVuTCz\ncNVZqJffvJhX37q4+t1RSbJ7sCuHdvTl8KFtuXtHX+7a3pfbt/Wko9XZKADg1iGugKysVDkzOZfX\nzk5f/Tg3nYnZxdX9hrrbcmhHXz7/6N7ctb0vh3b05a7tvenrbNvA0QMAbA7iCm4hi8srOXFhNq+d\nnc53zzUC6tWzF/PdszOZW3z7TNRQd1vu3NaXn7h/Z+7Y1ptD2/ty147ejPZ2uKQPAGAN4gpuQtOX\nlnL8/MxqQF1+HL8ws3pPVJLsHOjMHdt684XHh3PHtt7cMdqbO7b1WqUPAOD7IK7gBrWwtJKT47M5\ndm4mx87P5Oj5mRw9N51j52dy9uKl1f22lGT/SE9uH+3Np+7Znju29ebObb25fVtvejv8EQAAUBf/\nsoJNbGWlyptT86vx1AipRkCdHJ/L8srbZ6GGe9pzcGtPfuiu0Rzc2pPbtvbk4GhPDoz0WOYcAOA6\nEFewwZZXqpyZmMvrY7M5cWE2J8ZmcnJsNkfPzeT4hZnML66s7tvV1pKDW3ty3+6B/JEHd+Xg1p7V\nx2B3+wb+LgAAEFdwHcwtLDfjaeaKiJrNybHZnBqfveo+qLaWkr1D3TmwtSefuGNr4yzUaE9u29qb\n7f0WlAAA2KzEFdSgqqqcn17IyfHZvH7h7TNQrzcj6twV90AlSV9na/aPdOfenf35zP07sn+4O/tG\nurNvuDs7B7rSskVAAQDcaMQVfAArK1XOT1/KyfG5nBqfzemJuZwabzxON7evvHwvaazEt3e4O4fv\nGs3+ke7sG+nJ/uHu7B/pzkBXmzNQAAA3GXEFadz3dPbifDOYZnN6/Ip4mpjL6fG5LCxfHU/DPe3Z\nPdiVu7b35ZOHtmXPUFf2NuNpz1C3RSQAAG4x4oqbXlVVGZ9dzBuTc3ljYr7xPDnffMzlzMR8zkzM\nZemKlfeSZGtvR3YPdeXeXf359H3bs2ewK3uGurN7qCu7B7vSYxlzAACu4F+H3NCqqsrE7GLOTM7l\nzcn5nJmcz5urEfV2SF1auvqsU+uWku39ndk12JmH9g7mJz+yM7uHGvG0pxlPzjwBAPC9EFdsWnML\nyzl7cT5vTV266vlbr87nS69+dTWe3nmvU8uWkh39ndkx0Jn7dw/k0/ftyI5mSO0Y6Mqugc6M9HZY\nNAIAgFqJK667K6Ppran5nL14KWebz29d8XxxfuldP9vesiV9bVX2b1vOvbv686m7t2XnYFd2DnRm\n50Bndg12ZatwAgBgA4granFpaTkXphdyfvpS89F8fXEhF2Yu5dwV4bRWNG3r78j2/s7cua03P3DH\n1oz2Nba3XfE82N2WP/iDP8jhw5/YgN8lAACsTVxxTVVVZXZheTWWzjUj6fzFtwPqckydm752MCVJ\nT3tLRno7MtrXkUM7+vKDd45mW39HtvV1ZvsVz5YmBwDgRieubhGLyyuZmF3M+OxCxmYWMj6zkLHZ\nhYxNN54b24uN55lGSL3zXqbLBrrasrW3PVt7O3LPrv78YE/j9da+jmzt7chIb3tGexuvu9otCgEA\nwK1BXN2ALi0tZ3JuMVNzi5mYXWzE0uxCxmbeHU+XY2lqjTNLSdLb0ZqhnrYMd7dnpLc9d27rzXBP\n+zVjabinPe2tW67j7xYAAG4M4mqDLK9UmZpbzOTcYiaaz5Nzi5mcXWi8N/v2exNXhNTk3GLmFpfX\n/HU727ZkuLs9Qz3tGe5pz96h7gz3tGeouz3DPW2N96/4fLC7LR2tzi4BAMB6iavv08LSSi7OL+bi\n/FLzsZip5vOV712cX8rFS1drvUgqAAAO+0lEQVTEUzOS1rpH6bLu9pYMdLWtPvaPdOcje9oy2N2e\nga629He1ZbD52XBP+2o0uQwPAAA2xi0XV0vLK5lZWM7MpaXGo/n67Ti6MpDeHUqXA+qdX0p7LV1t\nLenrbE1fZ2sGutqyra8zd27rWw2mwe6rnxuPRjy59A4AAG4smz6uFpZWmhG0lJlLy5m+tJTZhUYY\nTV9azuzCUqYvh9Kl5av2bezT2J5t/uwHiaKkceaoEUZtjTjqbs+e4e70X36vo/Wqzy8/9zefeztb\n09YikAAA4FaxqePqxNRK7vrv//UH2ndLSXo6WtPb0Zqejtb0tLekp6M1e7q709vReN14vzU9V2z3\ndrSku7316jDqaE2rMAIAAL4Hmzqu+ttL/vqn77oiiq4Io/ZGBHV3tKS3ozUdrVt8TxIAALBhNnVc\nDXWW/PyP3LnRwwAAAHhfrn0DAACoQS1xVUr5TCnllVLKa6WUX7jG5x2llH/W/PypUsqBOo4LAACw\nWaw7rkopLUn+fpKfSHJvki+WUu59x25/Psl4VVV3JPmfk/yd9R4XAABgM6njzNXjSV6rqupoVVUL\nSX4zyefesc/nkvx68/X/leRTxeoTAADATaSOBS12Jzl5xfapJE+stU9VVUullMkkI0nOv/MXK6U8\nmeTJJBkdHc2RI0dqGCI3k+npafOCdzEvWIu5wbWYF6zF3GA9Nt1qgVVVfSnJl5Lk0KFD1eHDhzd2\nQGw6R44ciXnBO5kXrMXc4FrMC9ZibrAedVwWeDrJ3iu29zTfu+Y+pZTWJANJLtRwbAAAgE2hjrh6\nOsmdpZSDpZT2JF9I8uV37PPlJD/bfP0nkvx+VVVVDccGAADYFNZ9WWDzHqqfT/I7SVqS/FpVVS+U\nUv5mkq9XVfXlJP8oyf9ZSnktyVgaAQYAAHDTqOWeq6qqvpLkK+9475eueD2f5PN1HAsAAGAzquVL\nhAEAAG514goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goA\nAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG\n4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goA\nAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG\n4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goA\nAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG\n4goAAKAG64qrUspwKeXfllJebT4PrbHfcinl2ebjy+s5JgAAwGa03jNXv5Dk31VVdWeSf9fcvpa5\nqqoeaj5+ap3HBAAA2HTWG1efS/Lrzde/nuSPrvPXAwAAuCGVqqq+/x8uZaKqqsHm65Jk/PL2O/Zb\nSvJskqUkf7uqqn/xHr/mk0meTJLR0dGP/tZv/db3PT5uTtPT0+nt7d3oYbDJmBesxdzgWswL1mJu\ncC2f/OQnn6mq6tH32+9946qU8ntJdlzjo19M8utXxlQpZbyqqnfdd1VK2V1V1elSym1Jfj/Jp6qq\n+u77De7QoUPVK6+88n67cYs5cuRIDh8+vNHDYJMxL1iLucG1mBesxdzgWkopHyiuWt9vh6qqfvQ9\nDvJWKWVnVVVvlFJ2Jjm7xq9xuvl8tJRyJMnDSd43rgAAAG4U673n6stJfrb5+meT/Mt37lBKGSql\ndDRfb03yiSQvrvO4AAAAm8p64+pvJ/mxUsqrSX60uZ1SyqOllF9t7nNPkq+XUp5L8u/TuOdKXAEA\nADeV970s8L1UVXUhyaeu8f7Xk/yF5uv/L8kD6zkOAADAZrfeM1cAAABEXAEAANRCXAEAANRAXAEA\nANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRA\nXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEA\nANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRA\nXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEA\nANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRA\nXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRgXXFVSvl8KeWFUspKKeXR\n99jvM6WUV0opr5VSfmE9xwQAANiM1nvm6ttJ/liSP1xrh1JKS5K/n+Qnktyb5IullHvXeVwAAIBN\npXU9P1xV1UtJUkp5r90eT/JaVVVHm/v+ZpLPJXlxPccGAADYTK7HPVe7k5y8YvtU8z0AAICbxvue\nuSql/F6SHdf46BerqvqXdQ+olPJkkieTZHR0NEeOHKn7ENzgpqenzQvexbxgLeYG12JesBZzg/V4\n37iqqupH13mM00n2XrG9p/neWsf7UpIvJcmhQ4eqw4cPr/Pw3GyOHDkS84J3Mi9Yi7nBtZgXrMXc\nYD2ux2WBTye5s5RysJTSnuQLSb58HY4LAABw3ax3KfafLqWcSvLxJP+qlPI7zfd3lVK+kiRVVS0l\n+fkkv5PkpSS/VVXVC+sbNgAAwOay3tUCfzvJb1/j/TNJPnvF9leSfGU9xwIAANjMrsdlgQAAADc9\ncQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUA\nAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFAD\ncQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUA\nAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFAD\ncQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUA\nAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFCD\ndcVVKeXzpZQXSikrpZRH32O/46WU50spz5ZSvr6eYwIAAGxGrev8+W8n+WNJ/uEH2PeTVVWdX+fx\nAAAANqV1xVVVVS8lSSmlntEAAADcoK7XPVdVkt8tpTxTSnnyOh0TAADgunnfM1ellN9LsuMaH/1i\nVVX/8gMe5weqqjpdStmW5N+WUl6uquoP1zjek0kuB9ilUsq3P+AxuHVsTeISU97JvGAt5gbXYl6w\nFnODazn0QXZ637iqqupH1zuSqqpON5/PllJ+O8njSa4ZV1VVfSnJl5KklPL1qqrWXCiDW5N5wbWY\nF6zF3OBazAvWYm5wLR90Ub4P/bLAUkpPKaXv8uskn05jIQwAAICbxnqXYv/pUsqpJB9P8q9KKb/T\nfH9XKeUrzd22J/mPpZTnknwtyb+qqurfrOe4AAAAm816Vwv87SS/fY33zyT5bPP10SQPfp+H+NL3\nPzpuYuYF12JesBZzg2sxL1iLucG1fKB5Uaqq+rAHAgAAcNO7XkuxAwAA3NQ2dVyVUv7HUsq3SinP\nllJ+t5Sya6PHxOZQSvl7pZSXm/Pjt0spgxs9JjZeKeXzpZQXSikrpRQrPd3iSimfKaW8Ukp5rZTy\nCxs9HjaHUsqvlVLO+qoXrlRK2VtK+fellBebf4/8tY0eE5tDKaWzlPK1UspzzbnxP7zn/pv5ssBS\nSn9VVVPN1381yb1VVf2lDR4Wm0Ap5dNJfr+qqqVSyt9Jkqqq/tsNHhYbrJRyT5KVJP8wyV+vquoD\nLZvKzaeU0pLkO0l+LMmpJE8n+WJVVS9u6MDYcKWUH0oyneSfVFV1/0aPh82hlLIzyc6qqr7RXOX6\nmSR/1J8ZlFJKkp6qqqZLKW1J/mOSv1ZV1Vevtf+mPnN1OayaepJs3hLkuqqq6nerqlpqbn41yZ6N\nHA+bQ1VVL1VV9cpGj4NN4fEkr1VVdbSqqoUkv5nkcxs8JjaBqqr+MMnYRo+DzaWqqjeqqvpG8/XF\nJC8l2b2xo2IzqBqmm5ttzceaTbKp4ypJSil/q5RyMsmfTvJLGz0eNqU/l+Rfb/QggE1ld5KTV2yf\nin8oAR9AKeVAkoeTPLWxI2GzKKW0lFKeTXI2yb+tqmrNubHhcVVK+b1Syrev8fhcklRV9YtVVe1N\n8k+T/PzGjpbr6f3mRnOfX0yylMb84BbwQeYFAHw/Sim9Sf55kv/qHVdQcQurqmq5qqqH0rhS6vFS\nypqXFK/re67qUFXVj37AXf9pkq8k+eUPcThsIu83N0opP5fkJ5N8qtrMNw9Sq+/hzwxubaeT7L1i\ne0/zPYBrat5P88+T/NOqqv7vjR4Pm09VVROllH+f5DNJrrkozoafuXovpZQ7r9j8XJKXN2osbC6l\nlM8k+W+S/FRVVbMbPR5g03k6yZ2llIOllPYkX0jy5Q0eE7BJNRct+EdJXqqq6n/a6PGweZRSRi+v\nSl1K6UpjoaQ1m2Szrxb4z5McSmP1rxNJ/lJVVf6fR1JKeS1JR5ILzbe+aiVJSik/neR/TzKaZCLJ\ns1VV/fjGjoqNUkr5bJL/JUlLkl+rqupvbfCQ2ARKKb+R5HCSrUneSvLLVVX9ow0dFBuulPIDSf5D\nkufT+Hdnkvx3VVV9ZeNGxWZQSvlIkl9P4++SLUl+q6qqv7nm/ps5rgAAAG4Um/qyQAAAgBuFuAIA\nAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKjB/w8cV2m05IFPywAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fdda9490fd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot(lambda x: 1 / (1 + math.exp(-x)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Tangens hiperboliczny"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$$ g(x) = \\tanh x = \\frac{e^{x} - e^{-x}}{e^{x} + e^{-x}} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Przyjmuje wartości z przedziału $(-1, 1)$.\n",
    "* Powstaje z funkcji logistycznej przez przeskalowanie i przesunięcie."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Tangens hiperboliczny – wykres"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1cAAAG2CAYAAACTRXz+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3Xd0HOW9h/Hvu7vq1bKKZVtyt9xx\nw6ZjUw0pQCgBQjc4cOEGktz0nnADaeSShCQQejVOaCZ0AqLjinuVu+Si3tuW9/6htRFGsmVrpFlJ\nz+ecPTuzO9r5kTNH8pPZnTXWWgEAAAAAOsfj9gAAAAAA0BsQVwAAAADgAOIKAAAAABxAXAEAAACA\nA4grAAAAAHAAcQUAAAAADnAkrowxDxpjio0xa9p5fpYxpsoYsyJ8+6kT+wUAAACASOFz6HUelvQX\nSY8eYpv3rLVfdGh/AAAAABBRHDlzZa19V1K5E68FAAAAAD1Rd37m6nhjzEpjzCvGmPHduF8AAAAA\n6HJOvS3wcJZLGmKtrTXGnCvpeUmj2trQGDNP0jxJio2NnZabm9tNI6KnCIVC8ni4Fgs+i+MC7eHY\nQFs4LtAejg20ZdOmTaXW2ozDbWestY7s0BgzVNK/rbUTOrDtdknTrbWlh9ouLy/Pbty40ZH50Hvk\n5+dr1qxZbo+BCMNxgfZwbKAtHBdoD8cG2mKMWWatnX647boly40xA4wxJrw8I7zfsu7YNwAAAAB0\nB0feFmiMeUrSLEnpxphCST+TFCVJ1tq/S7pI0k3GmICkBkmXWqdOmQEAAABABHAkrqy1lx3m+b+o\n5VLtAAAAANAr8Wk9AAAAAHAAcQUAAAAADiCuAAAAAMABxBUAAAAAOIC4AgAAAAAHEFcAAAAA4ADi\nCgAAAAAcQFwBAAAAgAOIKwAAAABwAHEFAAAAAA4grgAAAADAAcQVAAAAADiAuAIAAAAABxBXAAAA\nAOAA4goAAAAAHEBcAQAAAIADiCsAAAAAcABxBQAAAAAOIK4AAAAAwAHEFQAAAAA4gLgCAAAAAAcQ\nVwAAAADgAOIKAAAAABxAXAEAAACAA4grAAAAAHAAcQUAAAAADiCuAAAAAMABxBUAAAAAOIC4AgAA\nAAAHEFcAAAAA4ADiCgAAAAAcQFwBAAAAgAOIKwAAAABwAHEFAAAAAA4grgAAAADAAcQVAAAAADiA\nuAIAAAAABxBXAAAAAOAA4goAAAAAHEBcAQAAAIADiCsAAAAAcABxBQAAAAAOIK4AAAAAwAHEFQAA\nAAA4gLgCAAAAAAcQVwAAAADgAOIKAAAAABxAXAEAAACAA4grAAAAAHAAcQUAAAAADiCuAAAAAMAB\nxBUAAAAAOIC4AgAAAAAHEFcAAAAA4ADiCgAAAAAcQFwBAAAAgAOIKwAAAABwAHEFAAAAAA5wJK6M\nMQ8aY4qNMWvaed4YY/5kjCkwxqwyxkx1Yr8AAAAAECmcOnP1sKQ5h3j+HEmjwrd5kv7m0H4BAAAA\nICI4ElfW2ncllR9ik/MkPWpbfCwp1RiT7cS+AQAAACAS+LppP4Mk7Wq1Xhh+bE837R8AAABAL2St\nVchKwZBVyFoFQ1ZBaxUKhZfD68GQVSikT5ftp8+HPvPYZ19rdFZSh2fprrjqMGPMPLW8dVAZGRnK\nz893dyBEnNraWo4LfA7HBdrDsYG2cFygPX3x2LDWKmilYEjyh6SAtQqE1OrWsu4Ph4m/jecCISlg\nJX8oHCe25bHQ/tcOv37IhuMmvL7/uQNRc2Bd4SjSZ7YNHfSzofDzXWnuhOgOb9tdcVUkKafV+uDw\nY59jrb1P0n2SlJeXZ2fNmtXlw6Fnyc/PF8cFDsZxgfZwbKAtHBdoTyQcG6GQVYM/qPrmoOqbA6pr\nCqoxEFSjP6gmf0iN/v3r4WV/SA3+oJr8wQPr+7c/sE0g9Jnn/cGQmgMhNQVblq3DgeLzGPm8Rj6P\nJ3zfsuz1GEV5Tfi+Zd3n9SjGYz77Mwf9vNdjFOXxyOs1ivIYeT2eA6/jC697PZLHY+Q1LY97wvcH\nbsa0PO/Rp8/tf2z/zxx4TAceG5qeoJ929L/b2f8Z27VQ0i3GmPmSZkqqstbylkAAAAD0eNZa1TcH\nVdMYUE2jX9WNAVU3+lXTGFB9U0B1zcFP78OxVN/c/uP1zcGjmiPa51Gsz6PYKK9io7yKi/IqNsqj\nmCivUuKiFJsUo9gor2J8HsVEeRTl9Sja51G0t+UWtX+51X3UgXujaJ9HMZ95rO3to7xGxhiH/1fu\nGRyJK2PMU5JmSUo3xhRK+pmkKEmy1v5d0suSzpVUIKle0rVO7BcAAABwgj8YUkV9s4pqQlq0tUwV\n9X5VNTSruuHzwVTT6G95vGn/ekDB0OFP/UR7PYqP8Soh2qf4aK/iY3xKiPYqNT5aCTFexUe3rO9/\nfP99XDiWYsOx9Jlln/dAMHk8fTNoIokjcWWtvewwz1tJNzuxLwAAAOBQgiGr8rpmldQ0qbS2SRX1\nzaqoa1ZFvV+V9S33FfXNqmx1X9sU+PQFPvj4M69njJQY41NybJSSYlvus1NiNTo2UclxLY8lxX56\nn9zqPiHGp4Ron+KivYr2OfUtSIhUEXdBCwAAAOBgoZBVWatg2n9rWW9utdyk8rpmtXciKTnWp34J\n0UqNj1b/xGiNzExUanyU+sVHq198lHZvL9CJ0ye3PJYQ3RJI0T7OCqFDiCsAAAC4yh8MqbimSXur\nGrW3qlF7qhpa7qsbta+qUXuqGrWvulGBNoopxudRRlKM0hNjNLhfvKbkpiojMUbp4cfSE2OUltAS\nTilxUfJ5D332KL9pu04ald5V/6no5YgrAAAAdKlGf1CFFfXaVdGgwvJ6FVY0aFdFvYoqGrSnqlEl\ntU2fu1pdbJRH2SlxGpAcq5nD0jQgJVZZybHKSIo5EFPpidFKjPH12YsnIPIQVwAAAOgUa61Kapq0\ntbRO20vrtLNVQBVWNKikpukz20f7PBrcL06DUuM0ZkCyBqTEHrhlp8QqOzlOyXFEE3oe4goAAAAd\nUtXg17bSOm0rrdW2kjptLa3TtnBQ1bW6fLjXYzQwNVY5/eI1Oy9DOf3ilZMWr5y0OA3uF6+MxBg+\nw4ReibgCAADAZ5TWNmnT3hpt2lejTcW12ryvRltL6lRW13xgG4+RBveL17D0BB07NE3DMxI0tH+C\nhqUnKDsl9rCfbQJ6I+IKAACgj6pu9GvDnnBEHbjVqrxVRKXERWl0VqLOGp+lYekJGpaeqGHpLWei\nYnxeF6cHIg9xBQAA0AeU1DRp7e4qrd1dfeB+R1n9gecTY3walZWos8ZlaVRWkvKykjQ6K1EZSTF8\n9gnoIOIKAACglymuadSKnZVaXfRpTO2r/vSiErlp8Ro/MFkXTxus8QNTNHpAkgamxBJRQCcRVwAA\nAD1YUyCodbur9cnOSn2yq1Kf7KxQYUWDpJbPRY3MTNSJI9I1bmCyxg9M0biByUqJi3J5aqB3Iq4A\nAAB6kH3VjVq8rVzLd1bok52VWre7Ws3BkCRpYEqspuT20zUnDNWU3FSNH5ii2Cg+FwV0F+IKAAAg\nQllrVVjRoEXbyrV4W5kWbSs/8Dmp2CiPJg1O1bUnDdWUnH6akpuqrORYlycG+jbiCgAAIILsKq/X\nBwWlWrStXIu2lml3VaMkKTU+SscOTdOVxw3RjGFpGpudrCgudw5EFOIKAADARdWNfn1YUKb3C0r0\n/uZSbQ+fmUpPjNbMYf114/A0zRiWptGZSXzxLhDhiCsAAIBuFAiGtLKwUu9uKtV7m0u0srBKwZBV\nfLRXxw/vr6tPGKqTR6VrREYiV+8DehjiCgAAoItVN/r1zsYSvbWhWG9vLFZlvV/GSJMGp+qmU0fo\n5FHpmpLbT9E+3uYH9GTEFQAAQBfYUVanN9cX6z/r92nxtnIFQlb94qN0Wl6mThubqZNGpis1Ptrt\nMQE4iLgCAABwgLVW6/ZU6+XVe/Tqmr3aUlInqeV7puaePExnjM3S1Nx+8vK5KaDXIq4AAACOkrVW\na3e3BNXLq/doe1m9PEaaOay/vjZziE4fm6kh/RPcHhNANyGuAAAAjoC1VmuKqvXS6j16Zc0e7Sir\nl9djdPzw/pp3ygidPT5L/RNj3B4TgAuIKwAAgA4orKjX858U6dnlRdpaWievx+iEEf1146kjdPb4\nAUpL4PNTQF9HXAEAALSjutGvV1bv0bPLi7RoW7kkacawNN1wynDNGT9A/QgqAK0QVwAAAK0EgiG9\nt7lUzywv1Bvr9qkpENLw9AR9+8zROn/KIOWkxbs9IoAIRVwBAABI2lVerwVLd2nB0l3aV92k1Pgo\nffXYHF0wZZAm56Tyhb4ADou4AgAAfZY/GNJ/1hfrqcU79e7mEknSrNEZ+sWXc3XamEy+1BfAESGu\nAABAn1PWENLvXtugBUsLVVLTpAHJsfrv00bpkumDNbgfb/sDcHSIKwAA0CdYa7V4W7ke+Wi7Xl3T\nIGmLZudl6rIZuZqVlyGfl7NUADqHuAIAAL1aoz+ohSt266EPt2v9nmqlxEVpztAo/fCSkzhLBcBR\nxBUAAOiVSmqa9MiH2/XEoh2qqPcrLytJd3xlos6fPEiLPnyPsALgOOIKAAD0KttK63Tfu1v1zPJC\n+YMhnTE2S9eeOFTHD+/PFf8AdCniCgAA9Aord1Xq7+9s0atr9yrK69GFUwdr3inDNSw9we3RAPQR\nxBUAAOixrLV6v6BU97xdoI+3lisp1qebTh2ha04cqsykWLfHA9DHEFcAAKDHsdbq3c2luvvNTVq+\ns1IDkmP1o3PH6rKZuUqM4Z83ANzBbx8AANBjWGv1zqYS/d+bm7ViV6UGpsTqfy+YoIumDVaMz+v2\neAD6OOIKAABEPGut8sNRtXJXpQalxunXF0zURdMGK9rH91MBiAzEFQAAiGiLt5XrN69u0LIdFRrc\nL053fGWiLpxKVAGIPMQVAACISOv3VOt3r23UWxuKlZUcw5kqABGPuAIAABFlV3m97npjk55fUaSk\nGJ++f84YXX38UMVF85kqAJGNuAIAABGhvK5Zf/rPZj2xaIe8HqMbTx2hG08ZoZT4KLdHA4AOIa4A\nAICrmgMhPfbxDt395ibVNQd1yfQc3XbGKGUl8z1VAHoW4goAALjCWqu3NhTrf19ar62ldTpldIZ+\n8oWxGpWV5PZoAHBUiCsAANDtNu6t0e0vrdN7m0s1PCNBD11zrGblZcgY4/ZoAHDUiCsAANBtqur9\n+v3rG/XEoh1Kio3Sz740TlccN0RRXq4ACKDnI64AAECXs9bqmeVFuuPl9aqob9aVxw3RbWeMVr+E\naLdHAwDHEFcAAKBLbdxbo588v0aLt5drSm6qHp07Q+MHprg9FgA4jrgCAABdorYpoLvf3KQHP9iu\npFiffnPhRF08LUceD5+rAtA7EVcAAMBxr67Zo58tXKt91U269NgcfXfOGKXxFkAAvRxxBQAAHFNc\n3aifvrBWr67dq7HZyfrbFdM0Nbef22MBQLcgrgAAQKdZa/XPpYW6/aV1agyE9N05ebrh5OFcBRBA\nn0JcAQCATtlZVq8fPrda7xeUasbQNN1x4USNyEh0eywA6HbEFQAAOCrBkNVDH2zTH17fJK/H6Pbz\nJ+jyGblcsAJAn0VcAQCAI7ajrE7fXrBSS3dU6LQxmbr9/AkamBrn9lgA4CriCgAAdJi1Vk8s2qlf\nv7xeXo/RXZccowumDJIxnK0CAOIKAAB0yN6qRn3vmVV6Z1OJThqZrt9eNImzVQDQCnEFAAAOyVqr\nhSt36yfPr1FzMKRfnjdeV8wcwmerAOAgxBUAAGhXVb1fP3x+tV5atUdTc1P1h0sma1h6gttjAUBE\nIq4AAECbFm8r123zP1FxTZO+c3aebjx1hLycrQKAdhFXAADgMwLBkP70VoH+8tZm5abF69n/OkGT\nBqe6PRYARDziCgAAHFBYUa/b5q/Q0h0VunDqYP3ivPFKjOGfCwDQER4nXsQYM8cYs9EYU2CM+X4b\nz19jjCkxxqwI3653Yr8AAMA5/161W+fc/Z427K3R3ZdO1h8uOYawAoAj0OnfmMYYr6R7JJ0pqVDS\nEmPMQmvtuoM2fdpae0tn9wcAAJzV6A/q5wvXav6SXZqck6o/XTpFuf3j3R4LAHocJ/7vqBmSCqy1\nWyXJGDNf0nmSDo4rAAAQYbaV1um/nliu9XuqddOsEfrWmaMV5XXkjS0A0OcYa23nXsCYiyTNsdZe\nH16/UtLM1mepjDHXSLpDUomkTZK+aa3d1c7rzZM0T5IyMjKmLViwoFPzofepra1VYmKi22MgwnBc\noD0cG+1bsjegB1Y3yeuRvj4pRpMy+s5bADku0B6ODbRl9uzZy6y10w+3XXf9Fn1R0lPW2iZjzNcl\nPSLptLY2tNbeJ+k+ScrLy7OzZs3qphHRU+Tn54vjAgfjuEB7ODY+rzkQ0q9fXq+HV2zX5JxU3fO1\nqRqUGuf2WN2K4wLt4dhAZzgRV0WSclqtDw4/doC1tqzV6v2SfuvAfgEAwBEqqmzQzU8s14pdlbr2\nxKH6wTljFe3jbYAA4AQn4mqJpFHGmGFqiapLJV3eegNjTLa1dk949cuS1juwXwAAcATe2VSiW+d/\nokDQ6q9fm6pzJ2a7PRIA9CqdjitrbcAYc4uk1yR5JT1orV1rjPmlpKXW2oWSvmGM+bKkgKRySdd0\ndr8AAKBjrLX6a/4W/f71jcrLStLfrpimYekJbo8FAL2OI5+5sta+LOnlgx77aavlH0j6gRP7AgAA\nHVfbFNB3/rlSr6zZqy8dM1C/uXCi4qP7zoUrAKA78dsVAIBealtpneY9ulRbSmr14y+M1dyThskY\n4/ZYANBrEVcAAPRCb23Yp1vnr5DPY/TY3Jk6cWS62yMBQK9HXAEA0IuEQlb3vF2gu97cpHHZyfr7\nFdOUkxbv9lgA0CcQVwAA9BINzUH9z79W6qVVe3TBlEG64ysTFRvldXssAOgziCsAAHqBfdWNuuHR\npVpdVKUfnjtGN5w8nM9XAUA3I64AAOjhVhdW6fpHl6i2MaB/XDldZ4zLcnskAOiTiCsAAHqwl1fv\n0bcWrFD/hBj966YTNDY72e2RAKDPIq4AAOiBrLX681sFuuuNTZo2pJ/uvXKa0hNj3B4LAPo04goA\ngB6m0R/Ud/61Si+u3K2vTBmkX3PhCgCICMQVAAA9SElNk254dKlWFlbqe3PG6MZTuXAFAEQK4goA\ngB6ioLhW1zy0WGW1zfr7FdN09vgBbo8EAGiFuAIAoAdYtLVM8x5bpiiv0dNfP06TBqe6PRIA4CDE\nFQAAEW7hyt36nwUrlZMWp4evnaGctHi3RwIAtIG4AgAgQllrde+7W3XnKxs0Y1ia7rtymlLjo90e\nCwDQDuIKAIAIFAiG9PMX1+rxj3fqS8cM1O8vnqQYH1cEBIBIRlwBABBh6psD+u8nP9F/NhTrxlNH\n6Ltn58nj4YqAABDpiCsAACJISU2T5j6yRGuKqnT7+RN0xXFD3B4JANBBxBUAABFiZ1m9rnxwkYqr\nm/SPq6br9LFZbo8EADgCxBUAABFg7e4qXfPQEvmDIT15w0xNye3n9kgAgCNEXAEA4LKPtpRp3qNL\nlRTr01M3HK+RmUlujwQAOArEFQAALnp1zR59Y/4KDUmL16NzZyg7Jc7tkQAAR4m4AgDAJU8u2qkf\nP79aU3L76YGrp/MdVgDQwxFXAAB0M2ut/vxWge56Y5NOG5Opey6fqrhovsMKAHo64goAgG4UDFn9\n4sW1evSjHbpw6mDdeeFERXk9bo8FAHAAcQUAQDfxB0P69oKVWrhyt75+ynB9/5wxMoYvBwaA3oK4\nAgCgGzT6g7r5ieX6z4ZifW/OGN00a4TbIwEAHEZcAQDQxWqbArrhkaX6eFuZfnX+BF153BC3RwIA\ndAHiCgCALlRZ36yrH1qiNUVV+uMlk3X+lEFujwQA6CLEFQAAXaS4plFX3r9Y28rq9PcrpunMcVlu\njwQA6ELEFQAAXaCwol5X3L9IxTVNeuiaY3XiyHS3RwIAdDHiCgAAh20pqdUV9y9SXVNAj82dqWlD\n+rk9EgCgGxBXAAA4aO3uKl31wGIZI82fd7zGDUx2eyQAQDchrgAAcMiyHeW65qElSorx6fHrZ2p4\nRqLbIwEAuhFxBQCAAxZtLdO1Dy9RVnKsHr9+pgalxrk9EgCgmxFXAAB00kdbynTdw0s0MDVWT91w\nnDKTY90eCQDgAuIKAIBO+LCgVNc9skQ5/eL15A3HKSMpxu2RAAAuIa4AADhKHxSUau4jS5Sb1hJW\n6YmEFQD0ZR63BwAAoCd6d1OJrnt4iYb2T9BThBUAQJy5AgDgiOVvLNa8x5ZpREainrh+ptISot0e\nCQAQAThzBQDAEXh7Q7HmPbpMozIT9SRhBQBohTNXAAB00H/W79NNjy/X6AGJenzuTKXGE1YAgE9x\n5goAgA54Y90+3fj4Mo3JTtITc48jrAAAn8OZKwAADuO1tXt1y5PLNW5gih69boZS4qLcHgkAEIE4\ncwUAwCG8snqPbn5iuSYMStFjcwkrAED7iCsAANrx0qo9uuWpT3RMTqoevW6GkmMJKwBA+3hbIAAA\nbXhx5W7d9vQKTclJ1cPXzVBiDH8yAQCHxpkrAAAO8sKKIt06/xNNy+1HWAEAOoy/FgAAtPLcJ4X6\n9oKVOnZomh685lglEFYAgA7iLwYAAGHPLCvU//xrpY4b1l8PXDNd8dH8mQQAdBx/NQAAkLRg6S59\n75lVOnFEuv5x1XTFRXvdHgkA0MMQVwCAPu/pJTv1/WdX66SRLWEVG0VYAQCOHBe0AAD0aU8t3qnv\nPbNaJ4/KIKwAAJ1CXAEA+qzHP96hHzy7WrPzMnTfldMIKwBAp/C2QABAn/ToR9v10xfW6vQxmfrr\nFVMV4yOsAACdQ1wBAPqchz/Ypp+/uE5njM3SPV+bQlgBABxBXAEA+pQH3t+mX/17nc4en6U/XzZV\n0T7eIQ8AcAZxBQDoM+5/b6tuf2m9zpkwQH+6bIqivIQVAMA5xBUAoE+4950tuuOVDfrCxGz936WT\nCSsAgOMc+ctijJljjNlojCkwxny/jedjjDFPh59fZIwZ6sR+AQDoiL/mF+iOVzboi5OydTdhBQDo\nIp3+62KM8Uq6R9I5ksZJuswYM+6gzeZKqrDWjpT0R0m/6ex+AQDoiIVbmvXbVzfqvMkD9X9fnSwf\nYQUA6CJO/IWZIanAWrvVWtssab6k8w7a5jxJj4SX/yXpdGOMcWDfAAC06+43N+vZzX5dMGWQ7rqE\nsAIAdC0nPnM1SNKuVuuFkma2t421NmCMqZLUX1LpwS9mjJknaZ4kZWRkKD8/34ER0ZvU1tZyXOBz\nOC5wsOc2N+uFLX7NyLT6UmaF3nv3HbdHQgThdwbaw7GBzoi4C1pYa++TdJ8k5eXl2VmzZrk7ECJO\nfn6+OC5wMI4L7Get1R/f2KQXthToommDdW56uU6bPdvtsRBh+J2B9nBsoDOceH9EkaScVuuDw4+1\nuY0xxicpRVKZA/sGAOAAa61+//pG/emtAn11eo5+e+EkeXgXOgCgmzgRV0skjTLGDDPGREu6VNLC\ng7ZZKOnq8PJFkt6y1loH9g0AgKSWsPrNqxt1z9tbdNmMXN3xlYnyeAgrAED36fTbAsOfobpF0muS\nvJIetNauNcb8UtJSa+1CSQ9IeswYUyCpXC0BBgCAI6y1uuOVDbrv3a264rhc/fLLEwgrAEC3c+Qz\nV9balyW9fNBjP2213CjpYif2BQBAa9Za/e9L63X/+9t01fFD9IsvjxcXpAUAuCHiLmgBAEBHWWv1\ny3+v00MfbNc1JwzVz740jrACALiGuAIA9EjWWv3ixXV6+MPtuu7EYfrJF8cSVgAAVxFXAIAeJxSy\n+tnCtXrs4x264eRh+uG5hBUAwH3EFQCgRwmFrH78who9uWinvn7qcH1/zhjCCgAQEYgrAECPEQpZ\n/ej51Xpq8S7dNGuEvnt2HmEFAIgYxBUAoEcIhax+8OxqPb10l26ZPVLfPms0YQUAiCjEFQAg4gVD\nVt/510o9u7xI3zhtpL55JmEFAIg8xBUAIKIFgiF9+58r9cKK3frmGaN16xmj3B4JAIA2EVcAgIjl\nD4Z029Mr9NKqPfrO2Xm6efZIt0cCAKBdxBUAICI1B0K6df4nemXNXv3w3DGad8oIt0cCAOCQiCsA\nQMRpCgR1y5Of6I11+/STL47T3JOGuT0SAACHRVwBACJKoz+o/3piud7aUKxfnjdeVx0/1O2RAADo\nEOIKABAxGv1BzXtsmd7dVKJfXzBRl8/MdXskAAA6jLgCAESEhuagbnh0qT7YUqrfXjhJlxyb4/ZI\nAAAcEeIKAOC6uqaA5j6yRIu3lev3Fx2jC6cNdnskAACOGHEFAHBVbVNA1z60WMt2VOiPX52s8yYP\ncnskAACOCnEFAHBNdaNf1zy4WCsLq/Sny6boi5MGuj0SAABHjbgCALiiqt6vqx5arLVFVfrLZVN0\nzsRst0cCAKBTiCsAQLcrq23SlQ8sVkFxrf76tak6a/wAt0cCAKDTiCsAQLfaW9Wor93/sYoqG/SP\nq6fr1NEZbo8EAIAjiCsAQLfZVV6vr92/SGW1TXrk2hmaOby/2yMBAOAY4goA0C22ltTqa/cvUl1T\nQE/ccJwm56S6PRIAAI4irgAAXW7D3mpdcf9iWWs1f97xGjcw2e2RAABwHHEFAOhSqworddWDixXj\n8+iJ64/XyMxEt0cCAKBLEFcAgC6zZHu5rn1oiVLjo/Tk9ccpt3+82yMBANBliCsAQJd4f3Opbnh0\nqbJTYvXEDTOVnRLn9kgAAHQp4goA4Lg31u3TzU8s1/CMBD02d6YykmLcHgkAgC5HXAEAHPXMskJ9\n95lVmjAwWY9cN0Op8dFujwQAQLcgrgAAjrn/va26/aX1OnFkf9175XQlxvBnBgDQd/BXDwDQadZa\n/eH1TfrL2wU6Z8IA/d+lkxXj87o9FgAA3Yq4AgB0SjBk9ZMX1ujJRTt12Ywc3X7+RHk9xu2xAADo\ndsQVAOCoNQdC+uaCFXpp1R7dNGuEvnt2nowhrAAAfRNxBQA4KnVNAd34+DK9t7lUPzp3rG44Zbjb\nIwEA4CriCgBwxCrqmnXtw0tUfiR5AAAZp0lEQVS0qrBSv71oki6ZnuP2SAAAuI64AgAckb1Vjbry\ngUXaUV6vv10xTWePH+D2SAAARATiCgDQYQXFtbr6wcWqavDr4WuP1Qkj0t0eCQCAiEFcAQA6ZOn2\ncl3/6FL5PB49dcNxmjg4xe2RAACIKMQVAOCwXl2zR9+Yv0KDU+P0yHUzlJMW7/ZIAABEHOIKAHBI\nD3+wTb/49zpNyUnV/Vcfq7SEaLdHAgAgIhFXAIA2hUJWv3l1g+59d6vOGpeluy+dorhor9tjAQAQ\nsYgrAMDnNAWC+u6/VumFFbt15XFD9PMvj5fXw5cDAwBwKMQVAOAzqhv9+vqjy/TR1jJ9d06ebjp1\nhIwhrAAAOBziCgBwQFFlg+Y+vEQFxbW665Jj9JWpg90eCQCAHoO4AgBIklbuqtTcR5aqyR/UQ9ce\nq5NHZbg9EgAAPQpxBQDQy6v36FsLVig9MUZP3TBTo7KS3B4JAIAeh7gCgD7MWqu/5m/R717bqKm5\nqbrvqulKT4xxeywAAHok4goA+qjmQEg/eHa1nlleqC8fM1C/vWiSYqO41DoAAEeLuAKAPqiirllf\nf3yZFm8r121njNKtp4/iioAAAHQScQUAfczWklpd9/AS7a5s1N2XTtZ5kwe5PRIAAL0CcQUAfcj7\nm0t185PL5fMYPTVvpqYNSXN7JAAAeg3iCgD6AGutHnh/m3798nqNykzSP66artz+8W6PBQBAr0Jc\nAUAv1+gP6ofPrdazy4t09vgs3XXJZCXE8OsfAACn8dcVAHqxvVWN+vpjS7WysErfPGO0/vu0kfJ4\nuHAFAABdgbgCgF5q2Y4K3fj4MtU3BXTvldN09vgBbo8EAECvRlwBQC+0YMku/fj5NRqQEqvH585U\n3oAkt0cCAKDXI64AoBdp9Af184VrNX/JLp00Ml1/uXyKUuOj3R4LAIA+gbgCgF5iV3m9bnpimdYU\nVevm2SP0rTPz5OXzVQAAdBviCgB6gbc3Fuu2+SsUslb/uGq6zhyX5fZIAAD0OZ2KK2NMmqSnJQ2V\ntF3SJdbaija2C0paHV7daa39cmf2CwBoEQxZ3f2fzfrzW5s1ZkCy/n7FVA3pn+D2WAAA9EmdPXP1\nfUn/sdbeaYz5fnj9e21s12CtndzJfQEAWimva9at8z/Re5tLddG0wbr9/AmKjfK6PRYAAH1WZ+Pq\nPEmzwsuPSMpX23EFAHDQ4m3l+sZTn6i8rll3fGWiLj02R8bw+SoAANxkrLVH/8PGVFprU8PLRlLF\n/vWDtgtIWiEpIOlOa+3zh3jNeZLmSVJGRsa0BQsWHPV86J1qa2uVmJjo9hiIMH3luAhZqxe3+PV8\ngV+Z8UY3HROjoSmcrTqUvnJs4MhwXKA9HBtoy+zZs5dZa6cfbrvDnrkyxrwpqa1vnvxR6xVrrTXG\ntFdqQ6y1RcaY4ZLeMsasttZuaWtDa+19ku6TpLy8PDtr1qzDjYg+Jj8/XxwXOFhfOC6Kqxt129Mr\n9OGWep03eaD+94KJSozhukSH0xeODRw5jgu0h2MDnXHYv8rW2jPae84Ys88Yk22t3WOMyZZU3M5r\nFIXvtxpj8iVNkdRmXAEAPu+dTSX61tMrVN8c1G8vmqSLpw3mbYAAAEQYTyd/fqGkq8PLV0t64eAN\njDH9jDEx4eV0SSdKWtfJ/QJAn+APhnTnKxt09YOLlZ4Yo4W3nKhLpvP5KgAAIlFn309yp6QFxpi5\nknZIukSSjDHTJd1orb1e0lhJ9xpjQmqJuTuttcQVABxGQXGtvvn0Cq0uqtJlM3L1sy+N42qAAABE\nsE7FlbW2TNLpbTy+VNL14eUPJU3szH4AoC+x1uqxj3fo1y+vV1yUV3+/YqrmTMh2eywAAHAYfBIa\nACJIcXWjvvOvVXpnU4lOHZ2h3100SZnJsW6PBQAAOoC4AoAI8eqavfrBs6tU3xzUL88bryuPG8Jn\nqwAA6EGIKwBwWXWjX796cZ3+uaxQEwel6I9fnayRmXzHCgAAPQ1xBQAuentDsX7w7GoV1zTqltkj\n9Y3TRyna19kLuQIAADcQVwDggqp6v37573V6ZnmhRmcl6t4rT9QxOalujwUAADqBuAKAbvbGun36\n0XOrVVbXrP8+baRuOW2kYnxcYh0AgJ6OuAKAblJR16xfvLhWz6/YrTEDkvTgNcdqwqAUt8cCAAAO\nIa4AoItZa/X8iiLd/u/1qmrw69bTR+nm2SP5bBUAAL0McQUAXWhrSa1+8sIafVBQpsk5qXrsgoka\nNzDZ7bEAAEAXIK4AoAs0BYL6e/5W3ZNfoBifR786f4Iun5Err4fvrQIAoLcirgDAYR9tKdOPnl+t\nrSV1+uKkbP30i+OUmRzr9lgAAKCLEVcA4JB91Y2685UNeu6TIuWkxenha4/VrLxMt8cCAADdhLgC\ngE5qCgT14Pvb9ee3NisQtLp59gjdMnuU4qK5vDoAAH0JcQUAnfDWhn365YvrtL2sXmeOy9KPvzBW\nQ/onuD0WAABwAXEFAEdha0mtfvXvdXp7Y4mGZyToketm6NTRGW6PBQAAXERcAcARKK9r1p/f2qzH\nP96hGJ9XP/7CWF11/FC+swoAABBXANARjf6gHvxgm/729hbVNQf01WNz9M0zRysziasAAgCAFsQV\nABxCMGT17PJC3fXGJu2patQZYzP1vTljNCorye3RAABAhCGuAKAN1lq9s6lEd76yQRv21uiYwSn6\n41cn67jh/d0eDQAARCjiCgAO8uGWUt31+iYt3VGh3LR4/eXyKfrCxGwZY9weDQAARDDiCgDClm4v\n1x9e36SPtpZpQHKsfnX+BH11eg4XqwAAAB1CXAHo81bsqtRdb2zSu5tKlJ4Yo59+cZwun5mr2Ci+\nBBgAAHQccQWgz1q6vVz3vF2gtzeWqF98lH5wzhhddfxQxUUTVQAA4MgRVwD6FGut3t1cqnveLtDi\nbeVKS4jWd87O09UnDFViDL8SAQDA0eNfEgD6hFDI6rW1e3VPfoHWFFUrOyVWP/vSOF16bC5nqgAA\ngCOIKwC9WqM/qOc+KdL9723VlpI6DUtP0G8vnKTzpwziQhUAAMBRxBWAXqm4plGPf7RDjy/aqfK6\nZo3LTtZfLp+icyZky+vhkuoAAMB5xBWAXmX9nmo98P42LVyxW/5QSKePydLck4bpuOFpfE8VAADo\nUsQVgB7PHwxpyd6A7vvHx/pwS5niory6dEaOrj1xmIalJ7g9HgAA6COIKwA91p6qBj21eJfmL96p\n4pomDUr16HtzxujyGblKiY9yezwAANDHEFcAepRQyOr9glI9/vEO/WdDsULW6tTRGbosoVrfuGg2\nn6cCAACuIa4A9AhFlQ16Zlmh/rWsUDvL65WWEK0bTh6ur83MVU5avPLz8wkrAADgKuIKQMRq9Af1\n2tq9+ufSQn2wpVTWSieM6K9vnzVacyYMUIyP76cCAACRg7gCEFGstVq+s0LPLi/SwpW7VdMY0OB+\ncbr19FG6cOpg5aTFuz0iAABAm4grAK6z1mr9nhotXLlbL67craLKBsVGeXTOhGxdPG2wjhveXx7e\n8gcAACIccQXANdtL67Rw5W4tXLlbBcW18nqMTh6Vrm+fNVpnjstSUixX/AMAAD0HcQWgW20vrdNr\na/fq5dV7tLKwSpI0Y1iabj9/gs6dmK20hGiXJwQAADg6xBWALmWt1Zqiar2+bq9eW7tXm/bVSpIm\nDkrRj84dqy8ek63slDiXpwQAAOg84gqA4wLBkBZvL9fra/fpjXX7VFTZII9pOUP1sy+N01njB2hQ\nKkEFAAB6F+IKgCOKqxuVv6lE72ws0XubS1TdGFCMz6OTR2XotjNG6fSxWbzlDwAA9GrEFYCjEgiG\ntHxnpfI3Fit/Y4nW7amWJGUmxWjOhAGanZepU0ZnKCGGXzMAAKBv4F89ADrEWqstJbX6cEuZPiwo\n04dbSlXdGJDXYzQtt5++c3aeZudlamx2kozhsukAAKDvIa4AtMlaq13lDfpwS6k+2lqmD7eUqaSm\nSZI0KDVOcyYM0Ky8TJ04Ml0pcVwyHQAAgLgCIEkKhVrOTC3dUaGl2yv08dYyFVU2SJIykmJ0/PD+\nOmFEf50wIl05aXGcnQIAADgIcQX0UY3+oFYXVWnp9got3V6uZTsrVFnvlySlJUTr2KH99PVTh+uE\nEf01IiORmAIAADgM4groA0Ihq62ldVpVWKlVhVVaWViptUXVag6GJEnDMxJ01rgsTR+apulD+mlY\negIxBQAAcISIK6CXsdaqqLLhQESt2lWlNUVVqmkKSJLio72aMDBF15w4VNOH9NO0If3UPzHG5akB\nAAB6PuIK6MGaAyEVFNdqw95qrd9TrfV7arR+T7XK6polSVFeo7HZyTpvykBNGpyqYwanamRmorwe\nzkoBAAA4jbgCegBrrYprmrRpX81nImpLSa38QStJivZ5NDorUaeNydSkwSmaNDhVY7KTFOPzujw9\nAABA30BcAREkEAxpV0WDCoprD9y2lNRqS3Htgbf1SVJWcozGZidr9phMjRmQpHHZyRqWniCf1+Pi\n9AAAAH0bcQV0M2utSmubtbO8TttL67WjrE4FJbXaUlynbaV1By4yIUmZSTEakZGo86cM0sjMRI3K\nTNSY7GSlJUS7+F8AAACAthBXQBcIBEPaXdmoHeV12lFWr53lLRG1f7m+OXhgW4+RctPiNTIzUbPy\nMjQiM1EjMxM1IiORL+cFAADoQYgr4CjUNPq1u7JRuysbVFTZoN0Hbo0qqmzQ3upGBUP2wPbRPo9y\n+sVpaP8EHT+iv4akxWtI/wTl9o/X4H5xfC4KAACgFyCugFZCIauyumbtq25USU2TimsaVVzdpL3V\njdpT9WlM1TQGPvNzPo/RgJRYDUyN04xhaRqYGqvctHjlpiVoSP94DUiOlYcr9AEAAPRqxBV6PWut\nqhsCKqtrUnlds8rqmlVW26zimkbtq25SSU2jimuatK+6UaW1zZ8547RfanyUBqbEaXC/eM0clqaB\nqXEHboNS45SRFMPlzQEAAPo44go9irVWjYGWL8mtqverqsGvivqWYCqvbVZ5XVPLcvhWVtesirpm\nBdoIJknqnxCtzORYZSbFKC8rSZnJMcoKr2ck7b+PUWwUb9sDAADAoRFX6FbWWjUFQqptCqi2MdBy\n3xRQTWNAlfXNqmrwq7qhJZoqw/dVDf4DIVXV4G8JpTffavP1k2N96p8Yo7SEaOWkxWtyTqrSEqKV\nlhCt/onR6p8Qc2A5PTFGUVy6HAAAAA4hrnBIwZBVfXNADf6gGptDqvcH1NAcbLn5g6pvDqq+uSWO\n9gdT3UHr+wNq/3p7Z5H2M0ZKivEpNT5aKXFRSomL0sDUuAPLZXt2aur4MQfWU+NbYqlffLSifcQS\nAAAA3NGpuDLGXCzp55LGSpphrV3aznZzJN0tySvpfmvtnZ3Zb19krVUgZNUcCKkpEFJTIHhguTm8\n3uQPqSkYUpM/pOZgSE3+YKvnP/2ZBv9n46gxfN/Qejm8TevvXOqI+GivEmN8LbfYlvuchHglhdcT\nws8lxfqUEN3y2P7nUuKilBoXrcRY3yE/v5Sfv1ezZuR29n9SAAAAwFGdPXO1RtJXJN3b3gbGGK+k\neySdKalQ0hJjzEJr7bpO7rtN1lqFbMsZl1A4SIIhq1CoZTlkW9YP3Oynz7X+mdBB2wSCVv5gSP6g\nVSAUvg+GPveYPxhq2TYU+uzPBEMtcRQMtSwHrfyhz79GINiyTXOrKNofT4c54dMhUV6j2Civ4qO9\niovyKi7ap7goj+KjfeoXH624aK/io7yKiw7fovZv13IfH+1VbKvluChvSzCFY4mLOgAAAKCv6lRc\nWWvXS5Ixh/wH9QxJBdbareFt50s6T9Jh46qoNqTT/pDfEjrWKhgM34ekYCgUjiEpEAopFFL4OQcK\npJM8RvJ5PYryGEX5PPJ5PIryGvm8RlFej6I8Hvm8Rj6vR9FeI5/Ho9io8HPhx2O8HsVEeRTt9Sgm\nyqsY3/5lj2J8XkX7PIrxHbzsCS97W/2sRzHeT9e5HDgAAADQNbrjM1eDJO1qtV4oaWZHfjDKYzQ2\nO1leY+TzGHk8Rl5j5PWG7z2f3jzGyOuRvB5P+LnwskfyhH/e2/o1PAfdTMtzvna28Xn2x09LGO0P\npNZBtH8bzt4AAAAAfc9h48oY86akAW089SNr7QtOD2SMmSdpniRlZGTo4oHVnXvB/R8ZCh75j4Uk\n+Tu3d3SB2tpa5efnuz0GIgzHBdrDsYG2cFygPRwb6IzDxpW19oxO7qNIUk6r9cHhx9rb332S7pOk\nvLw8O2vWrE7uHr1Nfn6+OC5wMI4LtIdjA23huEB7ODbQGd1x3eolkkYZY4YZY6IlXSppYTfsFwAA\nAAC6TafiyhhzgTGmUNLxkl4yxrwWfnygMeZlSbLWBiTdIuk1SeslLbDWru3c2AAAAAAQWTp7tcDn\nJD3XxuO7JZ3bav1lSS93Zl8AAAAAEMm6422BAAAAANDrEVcAAAAA4ADiCgAAAAAcQFwBAAAAgAOI\nKwAAAABwAHEFAAAAAA4grgAAAADAAcQVAAAAADiAuAIAAAAABxBXAAAAAOAA4goAAAAAHEBcAQAA\nAIADiCsAAAAAcABxBQAAAAAOIK4AAAAAwAHEFQAAAAA4gLgCAAAAAAcQVwAAAADgAOIKAAAAABxA\nXAEAAACAA4grAAAAAHAAcQUAAAAADiCuAAAAAMABxBUAAAAAOIC4AgAAAAAHEFcAAAAA4ADiCgAA\nAAAcQFwBAAAAgAOIKwAAAABwAHEFAAAAAA4grgAAAADAAcQVAAAAADiAuAIAAAAABxBXAAAAAOAA\n4goAAAAAHEBcAQAAAIADiCsAAAAAcABxBQAAAAAOIK4AAAAAwAHEFQAAAAA4gLgCAAAAAAcQVwAA\nAADgAOIKAAAAABxAXAEAAACAA4grAAAAAHAAcQUAAAAADiCuAAAAAMABxBUAAAAAOIC4AgAAAAAH\nEFcAAAAA4ADiCgAAAAAcQFwBAAAAgAOIKwAAAABwAHEFAAAAAA4grgAAAADAAcQVAAAAADiAuAIA\nAAAABxBXAAAAAOCATsWVMeZiY8xaY0zIGDP9ENttN8asNsasMMYs7cw+AQAAACAS+Tr582skfUXS\nvR3Ydra1trST+wMAAACAiNSpuLLWrpckY4wz0wAAAABAD9Vdn7mykl43xiwzxszrpn0CAAAAQLc5\n7JkrY8ybkga08dSPrLUvdHA/J1lri4wxmZLeMMZssNa+287+5knaH2BNxpg1HdwH+o50SbzFFAfj\nuEB7ODbQFo4LtIdjA23J68hGh40ra+0ZnZ3EWlsUvi82xjwnaYakNuPKWnufpPskyRiz1Frb7oUy\n0DdxXKAtHBdoD8cG2sJxgfZwbKAtHb0oX5e/LdAYk2CMSdq/LOkstVwIAwAAAAB6jc5eiv0CY0yh\npOMlvWSMeS38+EBjzMvhzbIkvW+MWSlpsaSXrLWvdma/AAAAABBpOnu1wOckPdfG47slnRte3irp\nmKPcxX1HPx16MY4LtIXjAu3h2EBbOC7QHo4NtKVDx4Wx1nb1IAAAAADQ63XXpdgBAAAAoFeL6Lgy\nxvzKGLPKGLPCGPO6MWag2zMhMhhjfmeM2RA+Pp4zxqS6PRPcZ4y52Biz1hgTMsZwpac+zhgzxxiz\n0RhTYIz5vtvzIDIYYx40xhTzVS9ozRiTY4x52xizLvx35Fa3Z0JkMMbEGmMWG2NWho+NXxxy+0h+\nW6AxJtlaWx1e/oakcdbaG10eCxHAGHOWpLestQFjzG8kyVr7PZfHgsuMMWMlhSTdK+l/rLUdumwq\neh9jjFfSJklnSiqUtETSZdbada4OBtcZY06RVCvpUWvtBLfnQWQwxmRLyrbWLg9f5XqZpPP5nQFj\njJGUYK2tNcZESXpf0q3W2o/b2j6iz1ztD6uwBEmRW4LoVtba1621gfDqx5IGuzkPIoO1dr21dqPb\ncyAizJBUYK3daq1tljRf0nkuz4QIYK19V1K523Mgslhr91hrl4eXayStlzTI3akQCez/t3fHoDqF\ncRzHvz83pIxM3GKQTSwmg6LcJDebsshkMJgMblHqrlLmqww3pa7BcA2UwUIWRTFYdBkodQeZ6G94\nX3XTfd2Lw3N0v586w3N6ht/w9L7nf85z/mfg03C4fniMrEl6XVwBJJlOsgCcAi61zqNeOgPcax1C\nUq9sAxaWjN/ihZKkVUiyA9gHPGmbRH2RZCzJM+ADcL+qRq6N5sVVkgdJXixzTAJU1VRVjQOzwLm2\nafUvrbQ2hnOmgC8M1ofWgNWsC0mSfkeSzcAccP6HHVRaw6rqa1XtZbBTan+SkVuK/+g7V12oqsOr\nnDoLzAOX/2Ic9chKayPJaeAYcKj6/PKgOvULvxla294B40vG24fnJGlZw/dp5oDZqrrTOo/6p6oW\nkzwEJoBlm+I0f3L1M0l2LRlOAq9aZVG/JJkALgDHq+pz6zySeucpsCvJziQbgJPA3caZJPXUsGnB\nDPCyqq62zqP+SLL1e1fqJJsYNEoaWZP0vVvgHLCbQfevN8DZqvLOo0jyGtgIfByeemwnSSU5AVwH\ntgKLwLOqOtI2lVpJchS4BowBN6pqunEk9UCSW8BBYAvwHrhcVTNNQ6m5JAeAR8BzBtedABerar5d\nKvVBkj3ATQb/JeuA21V1ZeT8PhdXkiRJkvS/6PW2QEmSJEn6X1hcSZIkSVIHLK4kSZIkqQMWV5Ik\nSZLUAYsrSZIkSeqAxZUkSZIkdcDiSpIkSZI6YHElSZIkSR34Bikt02UAw3ESAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fdda93e9590>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot(lambda x: math.tanh(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### ReLU (_Rectifier Linear Unit_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$$ g(x) = \\max(0, x) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### ReLU – zalety\n",
    "* Mniej podatna na problem zanikającego gradientu (_vanishing gradient_) niż funkcje sigmoidalne, dzięki czemu SGD jest szybciej zbieżna.\n",
    "* Prostsze obliczanie gradientu.\n",
    "* Dzięki zerowaniu ujemnych wartości, wygasza neurony, „rozrzedzając” sieć (_sparsity_), co przyspiesza obliczenia."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#### ReLU – wady\n",
    "* Dla dużych wartości gradient może „eksplodować”.\n",
    "* „Wygaszanie” neuronów."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### ReLU – wykres"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1cAAAG2CAYAAACTRXz+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAGJhJREFUeJzt3X+sZOd91/HPN/6RRQk0AS+NYzvE\npdalpkBLLNcRVbkmTnFMFTelkWwhkQDVtgirBQmBi0UCgUitkABBItpVY9VJo6QWqZul2eIkda7d\nCNzaiZzWXmfdxQR2t4lN7CT0KmnM1g9/7Gx7vZ659+7Oc2fOzLxe0tXOj7PzPH98vfbb58zZaq0F\nAACA6bxk3hsAAABYBuIKAACgA3EFAADQgbgCAADoQFwBAAB0IK4AAAA66BJXVXVnVT1dVY9OeH+9\nqr5WVY+Mft7RY10AAIChuLDT5/x8kvckef82x/x6a+0HOq0HAAAwKF3OXLXWHkjybI/PAgAAWESz\n/M7V66vqc1X1q1X152e4LgAAwJ7rdVngTj6b5M+01jar6qYkv5zkqnEHVtWBJAeSZN++fa97zWte\nM6Mtsiief/75vOQl7sXCC5kLJjEbjGMukme+0fJ7/6/lWy6uvHJfzXs7g2E2GOeJJ574cmtt/07H\nVWuty4JV9dokv9Ja+85dHPuFJNe01r683XFra2vt6NGjXfbH8tjY2Mj6+vq8t8HAmAsmMRuMs8pz\n0VrLP//oo/mFB/93fvSvfltuv/HPpUpcnbHKs8FkVfWZ1to1Ox03kyyvqlfV6J/aqrp2tO4zs1gb\nAIDThBXsrS6XBVbVh5KsJ7mkqk4keWeSi5KktfYzSX44yd+vqlNJvpHkltbrlBkAADsSVrD3usRV\na+3WHd5/T07fqh0AgBkTVjAbvq0HALDEhBXMjrgCAFhSwgpmS1wBACwhYQWzJ64AAJaMsIL5EFcA\nAEtEWMH8iCsAgCUhrGC+xBUAwBIQVjB/4goAYMEJKxgGcQUAsMCEFQyHuAIAWFDCCoZFXAEALCBh\nBcMjrgAAFoywgmESVwAAC0RYwXCJKwCABSGsYNjEFQDAAhBWMHziCgBg4IQVLAZxBQAwYMIKFoe4\nAgAYKGEFi0VcAQAMkLCCxSOuAAAGRljBYhJXAAADIqxgcYkrAICBEFaw2MQVAMAACCtYfOIKAGDO\nhBUsB3EFADBHwgqWh7gCAJgTYQXLRVwBAMyBsILlI64AAGZMWMFyElcAADMkrGB5iSsAgBkRVrDc\nxBUAwAwIK1h+4goAYI8JK1gN4goAYA8JK1gd4goAYI8IK1gt4goAYA8IK1g94goAoDNhBatJXAEA\ndCSsYHWJKwCAToQVrDZxBQDQgbACxBUAwJSEFZCIKwCAqQgr4AxxBQBwnoQVsJW4AgA4D8IKOJu4\nAgA4R8IKGEdcAQCcA2EFTCKuAAB2SVgB2xFXAAC7IKyAnYgrAIAdCCtgN8QVAMA2hBWwW+IKAGAC\nYQWcC3EFADCGsALOlbgCADiLsALOh7gCANhCWAHnS1wBAIwIK2Aa4goAIMIKmJ64AgBWnrACehBX\nAMBKE1ZAL+IKAFhZwgroqUtcVdWdVfV0VT064f2qqv9QVceq6req6i/3WBcA4HwJK6C3Xmeufj7J\njdu8/6YkV41+DiT5T53WBQA4Z8IK2AsX9viQ1toDVfXabQ65Ocn7W2styYNV9YqqurS19sUe6wMA\n7FZrLR848lzuOy6sgL5m9Z2ry5Ic3/L8xOg1AICZOXPG6r7jp4QV0F2XM1c9VdWBnL50MPv378/G\nxsZ8N8TgbG5umgtexFwwidngjD86Y3UqN1zWct2+L+X++5+a97YYGH9mMI1ZxdXJJFdseX756LUX\naa0dTHIwSdbW1tr6+vqeb47FsrGxEXPB2cwFk5gNkq1nrE5fCnjdvi/l+uuvn/e2GCB/ZjCNWV0W\neCjJ3x7dNfC6JF/zfSsAYBbcvAKYlS5nrqrqQ0nWk1xSVSeSvDPJRUnSWvuZJIeT3JTkWJKvJ/k7\nPdYFANiOsAJmqdfdAm/d4f2W5B/0WAsAYDeEFTBrs7osEABgZoQVMA/iCgBYKsIKmBdxBQAsDWEF\nzJO4AgCWgrAC5k1cAQALT1gBQyCuAICFJqyAoRBXAMDCElbAkIgrAGAhCStgaMQVALBwhBUwROIK\nAFgowgoYKnEFACwMYQUMmbgCABaCsAKGTlwBAIMnrIBFIK4AgEETVsCiEFcAwGAJK2CRiCsAYJCE\nFbBoxBUAMDjCClhE4goAGBRhBSwqcQUADIawAhaZuAIABkFYAYtOXAEAcyesgGUgrgCAuRJWwLIQ\nVwDA3AgrYJmIKwBgLoQVsGzEFQAwc8IKWEbiCgCYKWEFLCtxBQDMjLAClpm4AgBmQlgBy05cAQB7\nTlgBq0BcAQB7SlgBq0JcAQB7RlgBq0RcAQB7QlgBq0ZcAQDdCStgFYkrAKArYQWsKnEFAHQjrIBV\nJq4AgC6EFbDqxBUAMDVhBSCuAIApCSuA08QVAHDehBXAHxFXAMB5EVYALySuAIBzJqwAXkxcAQDn\nRFgBjCeuAIBdE1YAk4krAGBXhBXA9sQVALAjYQWwM3EFAGxLWAHsjrgCACYSVgC7J64AgLGEFcC5\nEVcAwIsIK4BzJ64AgBcQVgDnR1wBAH9IWAGcP3EFACQRVgDTElcAgLAC6EBcAcCKE1YAfYgrAFhh\nwgqgH3EFACtKWAH0Ja4AYAUJK4D+xBUArBhhBbA3xBUArBBhBbB3usRVVd1YVUer6lhV3T7m/bdX\n1f+pqkdGPz/SY10AYPeEFcDeunDaD6iqC5K8N8kbk5xI8lBVHWqtHTnr0F9srd027XoAwLkTVgB7\nr8eZq2uTHGutPdlaey7Jh5Pc3OFzAYAOhBXAbEx95irJZUmOb3l+Isn3jDnub1bV9yV5Isk/aq0d\nH3NMqupAkgNJsn///mxsbHTYIstkc3PTXPAi5oJJVn02Wmv5wJHnct/xU7npyoty3b4v5f77n5r3\ntuZu1eeCycwG0+gRV7vxX5J8qLX2zar60SR3Jflr4w5srR1McjBJ1tbW2vr6+oy2yKLY2NiIueBs\n5oJJVnk2zpyxuu+4M1ZnW+W5YHtmg2n0uCzwZJIrtjy/fPTaH2qtPdNa++bo6c8leV2HdQGACVwK\nCDB7PeLqoSRXVdWVVXVxkluSHNp6QFVduuXpm5M83mFdAGAMYQUwH1NfFthaO1VVtyW5N8kFSe5s\nrT1WVe9K8nBr7VCSH6+qNyc5leTZJG+fdl0A4MWEFcD8dPnOVWvtcJLDZ732ji2PfzLJT/ZYCwAY\nT1gBzFeXv0QYAJgvYQUwf+IKABacsAIYBnEFAAtMWAEMh7gCgAUlrACGRVwBwAISVgDDI64AYMEI\nK4BhElcAsECEFcBwiSsAWBDCCmDYxBUALABhBTB84goABk5YASwGcQUAAyasABaHuAKAgRJWAItF\nXAHAAAkrgMUjrgBgYIQVwGISVwAwIMIKYHGJKwAYCGEFsNjEFQAMgLACWHziCgDmTFgBLAdxBQBz\nJKwAloe4AoA5EVYAy0VcAcAcCCuA5SOuAGDGhBXAchJXADBDwgpgeYkrAJgRYQWw3MQVAMyAsAJY\nfuIKAPaYsAJYDeIKAPaQsAJYHeIKAPaIsAJYLeIKAPaAsAJYPeIKADoTVgCrSVwBQEfCCmB1iSsA\n6ERYAaw2cQUAHQgrAMQVAExJWAGQiCsAmIqwAuAMcQUA50lYAbCVuAKA8yCsADibuAKAcySsABhH\nXAHAORBWAEwirgBgl4QVANsRVwCwC8IKgJ2IKwDYgbACYDfEFQBsQ1gBsFviCgAmEFYAnAtxBQBj\nCCsAzpW4AoCzCCsAzoe4AoAthBUA50tcAcCIsAJgGuIKACKsAJieuAJg5QkrAHoQVwCsNGEFQC/i\nCoCVJawA6ElcAbCShBUAvYkrAFaOsAJgL4grAFaKsAJgr4grAFaGsAJgL4krAFaCsAJgr4krAJae\nsAJgFrrEVVXdWFVHq+pYVd0+5v2XVtUvjt7/jap6bY91AWAnwgqAWZk6rqrqgiTvTfKmJFcnubWq\nrj7rsL+X5CuttW9P8u+S/PS06wLATlpr+cCR54QVADNxYYfPuDbJsdbak0lSVR9OcnOSI1uOuTnJ\nvxg9/s9J3lNV1Vpr233ws7/f8q9/5ch2h7CCjp/4Zj69aS54IXPBOF945uu57/gpYQXATPSIq8uS\nHN/y/ESS75l0TGvtVFV9LcmfSvLlsz+sqg4kOZAkF3/rn80v/Pf/2WGLLJOWljpuLnghc8E4VcmN\nV7Rct+9Luf/+p+a9HQZkc3MzGxsb894GA2Q2mEaPuOqqtXYwycEkWVtba59/99+Y844Ymo2Njayv\nr897GwyMuWASs8E45oJJzAbT6HFDi5NJrtjy/PLRa2OPqaoLk3xLkmc6rA0AADAIPeLqoSRXVdWV\nVXVxkluSHDrrmENJ3jZ6/MNJ7tvp+1YAAACLZOrLAkffobotyb1JLkhyZ2vtsap6V5KHW2uHkrwv\nyQeq6liSZ3M6wAAAAJZGl+9ctdYOJzl81mvv2PL495O8tcdaAAAAQ9TlLxEGAABYdeIKAACgA3EF\nAADQgbgCAADoQFwBAAB0IK4AAAA6EFcAAAAdiCsAAIAOxBUAAEAH4goAAKADcQUAANCBuAIAAOhA\nXAEAAHQgrgAAADoQVwAAAB2IKwAAgA7EFQAAQAfiCgAAoANxBQAA0IG4AgAA6EBcAQAAdCCuAAAA\nOhBXAAAAHYgrAACADsQVAABAB+IKAACgA3EFAADQgbgCAADoQFwBAAB0IK4AAAA6EFcAAAAdiCsA\nAIAOxBUAAEAH4goAAKADcQUAANCBuAIAAOhAXAEAAHQgrgAAADoQVwAAAB2IKwAAgA7EFQAAQAfi\nCgAAoANxBQAA0IG4AgAA6EBcAQAAdCCuAAAAOhBXAAAAHYgrAACADsQVAABAB+IKAACgA3EFAADQ\ngbgCAADoQFwBAAB0IK4AAAA6EFcAAAAdiCsAAIAOxBUAAEAH4goAAKCDqeKqqv5kVX2iqn5n9Osr\nJxz3B1X1yOjn0DRrAgAADNG0Z65uT/JrrbWrkvza6Pk432itfdfo581TrgkAADA408bVzUnuGj2+\nK8kPTvl5AAAAC6laa+f/m6u+2lp7xehxJfnKmednHXcqySNJTiX5qdbaL2/zmQeSHEiS/fv3v+7u\nu+8+7/2xnDY3N/Pyl7983ttgYMwFk5gNxjEXTGI2GOf666//TGvtmp2O2zGuquqTSV415q07kty1\nNaaq6iuttRd976qqLmutnayqb0tyX5I3tNb+x06bW1tba0ePHt3pMFbMxsZG1tfX570NBsZcMInZ\nYBxzwSRmg3GqaldxdeFOB7TWbthmkaeq6tLW2her6tIkT0/4jJOjX5+sqo0k351kx7gCAABYFNN+\n5+pQkreNHr8tyUfPPqCqXllVLx09viTJX0lyZMp1AQAABmXauPqpJG+sqt9JcsPoearqmqr6udEx\n35Hk4ar6XJJP5fR3rsQVAACwVHa8LHA7rbVnkrxhzOsPJ/mR0eP/luQvTLMOAADA0E175goAAICI\nKwAAgC7EFQAAQAfiCgAAoANxBQAA0IG4AgAA6EBcAQAAdCCuAAAAOhBXAAAAHYgrAACADsQVAABA\nB+IKAACgA3EFAADQgbgCAADoQFwBAAB0IK4AAAA6EFcAAAAdiCsAAIAOxBUAAEAH4goAAKADcQUA\nANCBuAIAAOhAXAEAAHQgrgAAADoQVwAAAB2IKwAAgA7EFQAAQAfiCgAAoANxBQAA0IG4AgAA6EBc\nAQAAdCCuAAAAOhBXAAAAHYgrAACADsQVAABAB+IKAACgA3EFAADQgbgCAADoQFwBAAB0IK4AAAA6\nEFcAAAAdiCsAAIAOxBUAAEAH4goAAKADcQUAANCBuAIAAOhAXAEAAHQgrgAAADoQVwAAAB2IKwAA\ngA7EFQAAQAfiCgAAoANxBQAA0IG4AgAA6EBcAQAAdCCuAAAAOhBXAAAAHYgrAACADqaKq6p6a1U9\nVlXPV9U12xx3Y1UdrapjVXX7NGsCAAAM0bRnrh5N8kNJHph0QFVdkOS9Sd6U5Ookt1bV1VOuCwAA\nMCgXTvObW2uPJ0lVbXfYtUmOtdaeHB374SQ3JzkyzdoAAABDMovvXF2W5PiW5ydGrwEAACyNHc9c\nVdUnk7xqzFt3tNY+2ntDVXUgyYEk2b9/fzY2NnovwYLb3Nw0F7yIuWASs8E45oJJzAbT2DGuWms3\nTLnGySRXbHl++ei1SesdTHIwSdbW1tr6+vqUy7NsNjY2Yi44m7lgErPBOOaCScwG05jFZYEPJbmq\nqq6sqouT3JLk0AzWBQAAmJlpb8X+lqo6keT1ST5WVfeOXn91VR1OktbaqSS3Jbk3yeNJ7m6tPTbd\ntgEAAIZl2rsF3pPknjGv/26Sm7Y8P5zk8DRrAQAADNksLgsEAABYeuIKAACgA3EFAADQgbgCAADo\nQFwBAAB0IK4AAAA6EFcAAAAdiCsAAIAOxBUAAEAH4goAAKADcQUAANCBuAIAAOhAXAEAAHQgrgAA\nADoQVwAAAB2IKwAAgA7EFQAAQAfiCgAAoANxBQAA0IG4AgAA6EBcAQAAdCCuAAAAOhBXAAAAHYgr\nAACADsQVAABAB+IKAACgA3EFAADQgbgCAADoQFwBAAB0IK4AAAA6EFcAAAAdiCsAAIAOxBUAAEAH\n4goAAKADcQUAANCBuAIAAOhAXAEAAHQgrgAAADoQVwAAAB2IKwAAgA7EFQAAQAfiCgAAoANxBQAA\n0IG4AgAA6EBcAQAAdCCuAAAAOhBXAAAAHYgrAACADsQVAABAB+IKAACgA3EFAADQgbgCAADoQFwB\nAAB0IK4AAAA6EFcAAAAdiCsAAIAOxBUAAEAH4goAAKCDqeKqqt5aVY9V1fNVdc02x32hqn67qh6p\nqoenWRMAAGCILpzy9z+a5IeS/Owujr2+tfblKdcDAAAYpKniqrX2eJJUVZ/dAAAALKhZfeeqJfl4\nVX2mqg7MaE0AAICZ2fHMVVV9Msmrxrx1R2vto7tc53tbayer6k8n+URVfb619sCE9Q4kORNg36yq\nR3e5BqvjkiQuMeVs5oJJzAbjmAsmMRuMs7abg3aMq9baDdPupLV2cvTr01V1T5Jrk4yNq9bawSQH\nk6SqHm6tTbxRBqvJXDCOuWASs8E45oJJzAbj7PamfHt+WWBVvayq/viZx0m+P6dvhAEAALA0pr0V\n+1uq6kSS1yf5WFXdO3r91VV1eHTYtyb5dFV9LslvJvlYa+2/TrMuAADA0Ex7t8B7ktwz5vXfTXLT\n6PGTSf7SeS5x8Px3xxIzF4xjLpjEbDCOuWASs8E4u5qLaq3t9UYAAACW3qxuxQ4AALDUBh1XVfWv\nquq3quqRqvp4Vb163ntiGKrq31TV50fzcU9VvWLee2L+quqtVfVYVT1fVe70tOKq6saqOlpVx6rq\n9nnvh2Goqjur6ml/1QtbVdUVVfWpqjoy+vfIT8x7TwxDVe2rqt+sqs+NZuNfbnv8kC8LrKo/0Vr7\nv6PHP57k6tbaj815WwxAVX1/kvtaa6eq6qeTpLX2T+e8Leasqr4jyfNJfjbJP26t7eq2qSyfqrog\nyRNJ3pjkRJKHktzaWjsy140xd1X1fUk2k7y/tfad894Pw1BVlya5tLX22dFdrj+T5Af9mUFVVZKX\ntdY2q+qiJJ9O8hOttQfHHT/oM1dnwmrkZUmGW4LMVGvt4621U6OnDya5fJ77YRhaa4+31o7Oex8M\nwrVJjrXWnmytPZfkw0lunvOeGIDW2gNJnp33PhiW1toXW2ufHT3+vSSPJ7lsvrtiCNppm6OnF41+\nJjbJoOMqSarq3VV1PMnfSvKOee+HQfq7SX513psABuWyJMe3PD8R/6EE7EJVvTbJdyf5jfnuhKGo\nqguq6pEkTyf5RGtt4mzMPa6q6pNV9eiYn5uTpLV2R2vtiiQfTHLbfHfLLO00G6Nj7khyKqfngxWw\nm7kAgPNRVS9P8pEk//CsK6hYYa21P2itfVdOXyl1bVVNvKR4qr/nqofW2g27PPSDSQ4neecebocB\n2Wk2qurtSX4gyRvakL88SFfn8GcGq+1kkiu2PL989BrAWKPv03wkyQdba7807/0wPK21r1bVp5Lc\nmGTsTXHmfuZqO1V11ZanNyf5/Lz2wrBU1Y1J/kmSN7fWvj7v/QCD81CSq6rqyqq6OMktSQ7NeU/A\nQI1uWvC+JI+31v7tvPfDcFTV/jN3pa6qP5bTN0qa2CRDv1vgR5Ks5fTdv/5Xkh9rrfk/j6SqjiV5\naZJnRi896E6SVNVbkvzHJPuTfDXJI621vz7fXTEvVXVTkn+f5IIkd7bW3j3nLTEAVfWhJOtJLkny\nVJJ3ttbeN9dNMXdV9b1Jfj3Jb+f0f3cmyT9rrR2e364Ygqr6i0nuyul/l7wkyd2ttXdNPH7IcQUA\nALAoBn1ZIAAAwKIQVwAAAB2IKwAAgA7EFQAAQAfiCgAAoANxBQAA0IG4AgAA6EBcAQAAdPD/AVNA\ngTNkBNHAAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fdda936c6d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot(lambda x: max(0, x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Softplus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$$ g(x) = \\log(1 + e^{x}) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Wygładzona wersja ReLU."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Softplus – wykres"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1cAAAG2CAYAAACTRXz+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xd0XOWd//HPV733YlmWuyz3go0J\nJWCHXoIDgQCppKxDyi7pIWEDu+THhkAKKbtJ2IQEshAgEEIzmBaFboONuy13W5Zly+oa9dE8vz80\nNsZItrCudUfS+3WOjmdGF93vnvOs7HfunWfMOScAAAAAQP9E+T0AAAAAAAwFxBUAAAAAeIC4AgAA\nAAAPEFcAAAAA4AHiCgAAAAA8QFwBAAAAgAc8iSszu9vMqsxsXS/fX2BmDWa2Kvx1kxfnBQAAAIBI\nEePRz/mTpF9Luvcox7zsnLvEo/MBAAAAQETx5MqVc+4lSbVe/CwAAAAAGIwG8j1Xp5rZajN72sym\nDeB5AQAAAOCE8+q2wGNZKWmMcy5gZhdJ+ruk4p4ONLPFkhZLUkJCwtzRo0cP0IgYLEKhkKKi2IsF\n78a6QG9YG+iJX+uiMyQdaAmpIyRlxJsy4m3AZ8DR8TsDPdm8eXO1cy73WMeZc86TE5rZWElPOuem\n9+HYnZLmOeeqj3ZcSUmJKysr82Q+DB2lpaVasGCB32MgwrAu0BvWBnrix7p4em2lvv3wGsVEm35+\n1WwtLMkb0POjb/idgZ6Y2Qrn3LxjHTcgV67MbISk/c45Z2bz1X07Ys1AnBsAAMBPnV0h3fb0Jv3h\nlR2aVZSh//nESSrMSPR7LAAngCdxZWZ/kbRAUo6Z7ZF0s6RYSXLO/VbSFZK+ZGZBSa2SrnZeXTID\nAACIUPsa2vSV+1dqxa46XXvaWH3/oimKi+GWM2Co8iSunHPXHOP7v1b3Vu0AAADDwitbqnX9A2+r\nrbNLv7pmjj48a6TfIwE4wQZqQwsAAIBhIRRy+tWLW3XnC5tVnJei//nEXE3MS/F7LAADgLgCAADw\nSG1zh7724Cq9tPmALptTqFsvm66kOP65BQwX/H87AACAB97eXaev3LdS1YEO/ddlM3TN/CKZsdU6\nMJwQVwAAAP3gnNM9r+3UrUs2akR6gh750mmaMSrd77EA+IC4AgAAOE6B9qC++8gaPbWmUudMydNP\nr5yt9KRYv8cC4BPiCgAA4DiU7WvSl+5boZ3Vzbrhwsla/MHxioriNkBgOCOuAAAA3qe/vlWumx5b\nr5SEGN3/Lx/QB8Zn+z0SgAhAXAEAAPRRc3tQP3hsnf62skKnjs/WL66ZrbzUBL/HAhAhiCsAAIA+\n2LSvUV+5b6W2Vzfra+cU618/VKxobgMEcBjiCgAA4Cicc3rwzXLd/Ph6pSXG6r4vnKLTJuT4PRaA\nCERcAQAA9CLQHtSNj67VY6v26oyJOfr5VbOVmxrv91gAIhRxBQAA0IP1exv0r/e/rZ01zfrmuZP0\n5YUTuQ0QwFERVwAAAIdxzun/lu3WD5/coMykWHYDBNBnxBUAAEBYY1unvve3tXpqTaXOnJSrn39s\nlrJTuA0QQN8QVwAAAJLW7mnQV/+yUnvqWvWdC0p03ZkT+FBgAO8LcQUAAIY155zufX2Xbn1qo7JT\n4vTA4g/o5LFZfo8FYBAirgAAwLBV39Kh7z6yRkvX79eHJufpJ1fOUlZynN9jARikiCsAADAsLdte\no689uErVgXZ9/6LJ+sIZ47kNEEC/EFcAAGBYCXaF9OiWDj2x9A2NzkrSI186TTNHZfg9FoAhgLgC\nAADDRkV9q77+wCot39mpy+cU6paPTFdKPP8cAuANfpsAAIBh4Zl1lfruI2sV7ArpX2bE6carZvs9\nEoAhhrgCAABDWltnl3745Abdt2y3Zo5K1y+vnqOd6970eywAQxBxBQAAhqyyfU3617+s1Ob9AS0+\nc7y+dV6J4mKitNPvwQAMScQVAAAYcpxz+r9lu/X/ntyg1IQY3fO5+TprUq7fYwEY4ogrAAAwpBz+\n2VVnTsrVT6+cpdzUeL/HAjAMEFcAAGDIWL6jVtc/8LaqA+268aIp+vwZ4/jsKgADhrgCAACDXmdX\nSHc+v1m/Kd3GZ1cB8A1xBQAABrXtBwL6+oOrtHpPg66cO0o3XzqNz64C4At+8wAAgEHJOacH3izX\nLU9sUFxMlH7ziZN04YwCv8cCMIwRVwAAYNCpCbTrhr+t1XMb9uv0idn66ZWzNSI9we+xAAxzxBUA\nABhUSsuq9O2H16ihpVP/fvEUfe50Nq0AEBmIKwAAMCi0dXbptqc36U+v7VRJfqru/dx8TSlI83ss\nADiEuAIAABFvw95Gfe3Bt7V5f0CfPX2svnvBZCXERvs9FgC8C3EFAAAiVijk9IdXduiOpWVKT4rV\nPZ+br7Mm5fo9FgD0iLgCAAARqbKhVd/662q9urVG503N120fnams5Di/xwKAXhFXAAAg4jy5Zq9u\nfHSdOoIh3Xb5DF11cpHM2LQCQGQjrgAAQMSob+nQDx5brydW79WsogzdedVsjctJ9nssAOgT4goA\nAESE0rIqfefhNapt7tA3z52kLy2YoJjoKL/HAoA+I64AAICvmtuDunXJRt2/bLcm5afo7mtP1vTC\ndL/HAoD3jbgCAAC+eXNnrb750GqV17Xoi2eO19fPncQW6wAGLeIKAAAMuLbOLv38uc266+XtGpWZ\nqAcXn6r547L8HgsA+oW4AgAAA2r93gZ948HVKtvfpGvmj9aNF09RSjz/JAEw+PGbDAAADIhgV0i/\n/ec23fn8FmUlx+mP156shZPz/B4LADxDXAEAgBNu+4GAvvHQaq0qr9clMwv0w0XTlckHAgMYYogr\nAABwwoRCTve+vlO3PbNJ8THR+uU1c3TprJF+jwUAJwRxBQAATohdNc369sNrtHxHrc6alKvbr5ip\n/LQEv8cCgBOGuAIAAJ4KhZzueX2nbn+mTDFRpts/OlNXzhslM/N7NAA4oYgrAADgmZ3VzfrOw2u0\nfGf31arbPjpDBemJfo8FAAOCuAIAAP0WCjn98bWdumPpJsVGR+mOK2bqirlcrQIwvBBXAACgX3ZU\nN+s7D6/WmzvrtLAkVz+6fKZGpPPeKgDDD3EFAACOS1fI6Y+v7tAdS8sUFxOln1w5Sx89qZCrVQCG\nLeIKAAC8b9sPBPTth9doxa46fWhynv7rshlcrQIw7BFXAACgz7pCTne/skM/ebZM8TFR+tnHZumy\nOVytAgCJuAIAAH20sbJRNzyyRqv3NOjsyXn6r8tn8LlVAHAY4goAABxVe7BLv35xq35Tuk1pibH6\nxdWzdemskVytAoAjEFcAAKBXb+2s1XcfWaNtB5p1+ZxC/fslU5WVHOf3WAAQkYgrAADwHoH2oG5/\nZpP+/MYujUxP1J8+e7IWlOT5PRYARDTiCgAAvMs/NlXpxkfXqrKxTZ85day+dX6JUuL5JwMAHIsn\nvynN7G5Jl0iqcs5N7+H7JukXki6S1CLpWufcSi/ODQAAvFETaNctT27QY6v2amJeih6+7jTNHZPp\n91gAMGh49T9D/UnSryXd28v3L5RUHP46RdJvwn8CAACfOef02Kq9uuXJDWpq69T1ZxfrywsnKD4m\n2u/RAGBQ8SSunHMvmdnYoxyySNK9zjkn6Q0zyzCzAudcpRfnBwAAx6e8tkU3PbZO/yg7oNlFGfrx\nR2eqZESq32MBwKA0UDdQF0oqP+z5nvBrxBUAAD7o7Arp7ld26M7nt8hM+sElU3XtaWMVHcX26gBw\nvCLu3almtljSYknKzc1VaWmpvwMh4gQCAdYF3oN1gd6wNt5ra32X7lnfofKmkObkReuTU+KUHdyl\nl1/a5fdoA4Z1gd6wNtAfAxVXFZKKDns+Kvzaezjn7pJ0lySVlJS4BQsWnPDhMLiUlpaKdYEjsS7Q\nG9bGOxpaO3XH0k26b9lu5acm6Hefmqbzp43weyxfsC7QG9YG+mOg4upxSV81swfUvZFFA++3AgBg\nYDjn9OSaSt3y5AbVBNp17Wlj9c3z2F4dALzm1Vbsf5G0QFKOme2RdLOkWElyzv1W0hJ1b8O+Vd1b\nsX/Wi/MCAICj213Toh88tk7/3HxA0wvTdPdnTtaMUel+jwUAQ5JXuwVec4zvO0lf8eJcAADg2Dq7\nQvrfl7frF89vUUyU6aZLpurTp45RTHSU36MBwJDF/QAAAAwxK3bV6vt/W6ey/U06f1q+/uPSaSpI\nT/R7LAAY8ogrAACGiJpAu378zCY99NYejUxP0P9+ep7OnZrv91gAMGwQVwAADHJdIae/LN+tO5aW\nqbk9qMVnjtf1ZxcrmQ0rAGBA8VsXAIBBbFV5vX7w93VaW9GgD4zP0g8XTVdxfqrfYwHAsERcAQAw\nCNU1d+j2pZv0wJvlyk2J1y+unq1LZ42Umfk9GgAMW8QVAACDSCjk9OBb5frxM5vU1BbU508fp+vP\nKVZqQqzfowHAsEdcAQAwSKzZU68fPLZeq8vrNX9c9y2AJSO4BRAAIgVxBQBAhKtv6dAdS8t0//Ld\nyk6O151Xzdai2dwCCACRhrgCACBCdYWcHnyzXD95tkz1LR269rSx+vq5k5TGLYAAEJGIKwAAItDy\nHbX6j8fXa0Nlo+aPzdJ/XDpNU0em+T0WAOAoiCsAACJIRX2rfrRko55cU6mR6Qn69cfn6OIZBdwC\nCACDAHEFAEAEaO3o0u9e2qbf/nObnJOuP7tY1501QYlx0X6PBgDoI+IKAAAfOef01NpK/WjJJlXU\nt+rimQX6/kVTVJiR6PdoAID3ibgCAMAnG/Y26j+fWK9lO2o1pSBNP/vYLJ0yPtvvsQAAx4m4AgBg\ngNU2d+gnz5bpgeW7lZ4Yq1svm66rTx6t6CjeVwUAgxlxBQDAAGkPdunPr+/SL1/YouaOLn3mtLH6\n2tmTlJ7E1uoAMBQQVwAAnGDOOT29bp9ue3qTdte26KxJufr3i6eoOD/V79EAAB4irgAAOIHe3l2n\nW5/aqLd21akkP1X3fm6+zpyU6/dYAIATgLgCAOAEKK9t0e1Ly/TE6r3KTY3XbZfP0JXzinhfFQAM\nYcQVAAAeamjt1P/8Y6v++OpORUVJ//ahifriWROUHM9fuQAw1PGbHgAAD3R2hXT/st268/nNqm/t\n1OVzRulb509SQTqfVwUAwwVxBQBAPzjn9PzGKv3o6Y3afqBZp47P1o0XT9H0wnS/RwMADDDiCgCA\n47Ryd51ue3qTlu+o1fjcZP3+0/N09pQ8mfG+KgAYjogrAADep61VTbr9mTI9u2G/clLidMuiabpm\n/mjFRkf5PRoAwEfEFQAAfVTZ0Ko7n9uiv64oV1JcjL5x7iR9/oxxbFYBAJBEXAEAcEz1LR36Tek2\n/em1nXJOuva0cfrKwgnKTon3ezQAQAQhrgAA6EVbZ5f++OpO/aZ0q5rag7psdqG+fu4kFWUl+T0a\nACACEVcAABwh2BXSwyv26M7nt2hfY5s+NDlP3z6/RFMK0vweDQAQwYgrAADCQiGnZ9bv00+fLdO2\nA82aMzpDv7h6tk4Zn+33aACAQYC4AgAMe845vbCxSj97brM2VDZqYl6KfvepuTpvaj7bqgMA+oy4\nAgAMW845vbylWj99brNWl9drTHaSfn7VLF06q1DRUUQVAOD9Ia4AAMPSG9tr9LNnN2v5zloVZiTq\nxx+doctPGsVnVQEAjhtxBQAYVlburtPPnt2sV7ZWKy81XrcsmqarTi5SfEy036MBAAY54goAMCys\nq2jQz57brBc3VSk7OU7/fvEUffIDY5QQS1QBALxBXAEAhrTyppCu+/MKPbN+n9ITY/Xt80t07Wlj\nlRzPX4EAAG/xNwsAYEhaV9GgX724RUvXtyolvlPXn12sz39wnNISYv0eDQAwRBFXAIAhZXV5vX71\n4hY9v7FKqQkxunRCrG75xAJlJMX5PRoAYIgjrgAAQ8KKXXX61YtbVFp2QOmJsfrGuZP0mdPG6u1l\nrxJWAIABQVwBAAa15Ttq9csXtuiVrdXKTIrVdy4o0ac+MEap3P4HABhgxBUAYNBxzun17TX65Qtb\n9Mb2WuWkxOn7F03WJ04Zw0YVAADf8DcQAGDQcM7ppS3V+vWLW/Tmzjrlpcbrpkum6pr5o5UYx5bq\nAAB/EVcAgIjXFXJasrZSvyndpg2VjSpIT9Ati6bpY/OK+JwqAEDEIK4AABGrrbNLj6zco7te2q5d\nNS0an5us26+YqY/MLlRcTJTf4wEA8C7EFQAg4jS2deq+N3brD6/sUHWgXbNGpet7n5yr86bmKyrK\n/B4PAIAeEVcAgIhR1dSmP766U//3+i41tQf1weIcfWnBbJ06PltmRBUAILIRVwAA3+2qadZdL23X\nX1fsUbArpAtnFOhLZ03Q9MJ0v0cDAKDPiCsAgG9Wldfr9y9v15K1lYqJitJH547S4jPHa1xOst+j\nAQDwvhFXAIAB1RVyem7Dfv3hle16c2edUuNj9C8fHK/PnzFOeWkJfo8HAMBxI64AAAOiuT2oh1fs\n0d2v7tCumhaNykzUTZdM1cdOLlIKH/wLABgC+NsMAHBCVTa06p7Xdun+ZbvU2BbUSaMz9N0LJuu8\nqfmKiWY7dQDA0EFcAQBOiHUVDfrDKzv0xOq9CjmnC6cX6HNnjNPcMZl+jwYAwAlBXAEAPNMVcvrH\npir9/pXtemN7rZLjovXpU8fqs6ePVVFWkt/jAQBwQhFXAIB+a2jp1ENvleveN3aqvLZVBekJ+v5F\nk3X1/NFKS4j1ezwAAAYEcQUAOG4bKxt17+s79ejbFWrrDGn+2CzdcMEUnTctX7G8nwoAMMwQVwCA\n9yXYFdKzG/brT6/t1PIdtUqIjdJHZhfq06eO1dSRaX6PBwCAb4grAECfVAfa9cDy3bpv2W5VNrRp\nVGaivnfhZF11cpEykuL8Hg8AAN8RVwCAo1pdXq97Xt+pJ1dXqqMrpDMm5uiWRdP1ocl5io4yv8cD\nACBiEFcAgPcItAf1+Kq9un/5Lq2raFRyXLSunl+kT586RhPzUv0eDwCAiERcAQAOWVfRoPuX79Zj\nb1eouaNLk0ek6pZF0/SROYXs+gcAwDF4EldmdoGkX0iKlvR759xtR3z/Wkl3SKoIv/Rr59zvvTg3\nAKB/WjqCemL1Xt2/bLdW72lQfEyULpk5Uh8/ZbROGp0hM279AwCgL/odV2YWLem/JZ0raY+kN83s\ncefchiMOfdA599X+ng8A4I2NlY26f9lu/f3tCjW1B1Wcl6KbPzxVl88ZpfQkrlIBAPB+eXHlar6k\nrc657ZJkZg9IWiTpyLgCAPispSOop9ZU6v7lu/X27nrFxUTp4hkF+vgpozVvTCZXqQAA6AdzzvXv\nB5hdIekC59wXws8/JemUw69ShW8L/JGkA5I2S/q6c668l5+3WNJiScrNzZ370EMP9Ws+DD2BQEAp\nKSl+j4EIw7ronXNOW+tDerkiqOWVQbV1SSOSTQuLYnX6yBilxA3toGJtoCesC/SGtYGeLFy4cIVz\nbt6xjhuoDS2ekPQX51y7mX1R0j2SPtTTgc65uyTdJUklJSVuwYIFAzQiBovS0lKxLnAk1sV77W9s\n099WVuivK8u1/UCbkuKidcnsUbpy7ijNH5c1bK5SsTbQE9YFesPaQH94EVcVkooOez5K72xcIUly\nztUc9vT3km734LwAgCN0BEN6cdN+PfTWHpWWVSnkpJPHZuq6MyfoopkFSolnk1gAAE4UL/6WfVNS\nsZmNU3dUXS3p44cfYGYFzrnK8NNLJW304LwAgLCNlY3661t79PdVFapt7lB+WryuO2uCrpg7SuNz\nub0FAICB0O+4cs4Fzeyrkpaqeyv2u51z683sFklvOecel/RvZnappKCkWknX9ve8ADDcHWhq1xOr\n9+rRtyu0tqJBsdGmc6fm68p5RfrgxBzFREf5PSIAAMOKJ/eHOOeWSFpyxGs3Hfb4e5K+58W5AGA4\na+kI6rkN+/Xo2xV6eUu1ukJO00am6eYPT9Wi2YXKSo7ze0QAAIYtbr4HgAjXFXJ6dWu1/v52hZ5Z\nv08tHV0qzEjUF88cr4/MKdSk/FS/RwQAACKuACAiOee0fm+j/v52hR5fvVdVTe1KTYjRotkj9ZHZ\nhTp5bJaioobHbn8AAAwWxBUARJDy2hY9sWavHl1ZoS1VAcVGmxaW5OmyOYVaODlPCbHRfo8IAAB6\nQVwBgM/21rfqqTWVenLNXq3e0yCpe/v0Wy+brotnFCgjifdRAQAwGBBXAOCDqsY2PbW2Uk+uqdSK\nXXWSpOmFabrhwsm6eEaBirKSfJ4QAAC8X8QVAAyQ6kC7nl63T0+u3qvlO2vlnDR5RKq+dd4kXTJz\npMbmJPs9IgAA6AfiCgBOoNrmDj27fp+eXFOp17ZVK+SkCbnJuv7sYl0ys0AT89jpDwCAoYK4AgCP\nVTa06tn1+/XMun1atqNGISeNzU7SlxdM1CWzClSSnyozdvoDAGCoIa4AwAM7qpv1zLp9emb9Pq0u\nr5ckFeel6CsLJ+r8aSM0bWQaQQUAwBBHXAHAcXDOaWNlk55Zv09L1+1T2f4mSdLMUen69vklOn/a\nCE3MS/F5SgAAMJCIKwDoo66Q08rddXpuQ/ctf7trWxRl0sljs3Tzh6fqvGkjVJiR6PeYAADAJ8QV\nABxFY1unXtp8QC9urNI/yqpU19Kp2GjT6RNz9OUFE3TO1HzlpMT7PSYAAIgAxBUAHGF3TYue37hf\nL2zar2XbaxUMOWUmxWphSZ7OnpKvMyflKDUh1u8xAQBAhCGuAAx7XSGnt3fX6fmNVXph435tqQpI\nkibmpejzHxync6bk66TRmYqOYkMKAADQO+IKwLBUE2jXy1uq9c/NB/TPzQdU29yhmCjT/HFZunr+\naJ0zJU9jsvlQXwAA0HfEFYBhIdgV0uo99Sot646ptRUNck7KSo7TmcU5OntKvs4qyVUat/sBAIDj\nRFwBGLL2NbTppfCVqZe3HFBjW1BRJs0ZnalvnDNJZ5XkavrIdEVxux8AAPAAcQVgyGjr7NKKXXV6\nacsB/bPsgDbt6/7sqfy0eF0wfYTOmpSnMybmKD2Jq1MAAMB7xBWAQasr5LSuokGvbqvWk2+2atvz\nz6o9GFJstGnemCzdcOFknTUpV5NHpMqMq1MAAODEIq4ADBrOOe2obtarW6v16tYavbatWo1tQUnS\nqBTTJ04ZqzOKszV/XLZS4vn1BgAABhb/+gAQ0aoa2/Tathq9srVar26tVmVDmySpMCNRF0wfodMn\n5ui0CTlav+J1LVgw1edpAQDAcEZcAYgoe+tbtWxHjZZtr9WyHbXaUd0sScpIitVpE7L11Yk5On1C\njsZkJ3GrHwAAiCjEFQDfOOe0p65Vb2yv0bIdtVq2o0blta2SpLSEGM0fl6WPzx+tD4zP1rSRaezq\nBwAAIhpxBWDAOOe0s6ZFyw7G1PYa7Q3f5peZFKv547L02dPG6ZTxWZo8Ik3RxBQAABhEiCsAJ0xH\nMKQNlY1asatOK3bV6q2ddapqapck5aTE6ZRx2bpufJZOGZet4rwUrkwBAIBBjbgC4Jm65o7ukNpd\npxW76rS6vF7twZCk7g0oPjA+W6eEY2pCbjLvmQIAAEMKcQXguIRCTturA+GrUnV6a1edth/o3nwi\nJso0rTBdnzhljOaNzdRJozM1Ij3B54kBAABOLOIKQJ/UBNq1Zk+DVpXXa/Weeq0qr1d9S6ek7vdL\nzR2TqSvmjtLc0ZmaVZShhNhonycGAAAYWMQVgPdo7ejSur0NWl1efyimDu7iZyZNykvV+VNHaO7Y\nTM0dk6nxOdziBwAAQFwBw1ywK6QtVQGtPnRFqkGb9zepK+Qkdb9XalZRuj55yhjNKsrQ9MJ0pcTz\nqwMAAOBI/AsJGEY6giFtqWrS+opGrdvboHUVDdpY2aTWzi5JUnpirGYVZejcKXmaVZShmaMylJsa\n7/PUAAAAgwNxBQxRrR1d2rSvUev2Nmp9RYPW7W3Q5n0BdXR1796XEh+jqSPTdPX8Is0alaHZRRka\nk53E7X0AAADHibgChoC65g5t3NeojZVNh0Jqa1VA4Tv7lJkUq+mF6frcGeM0bWSaphema0xWEp8r\nBQAA4CHiChhEOoIhbTsQ0KZ9jdpU2aRN+5q0aV+j9je2HzomPy1e00em64LpBYdCamR6AlekAAAA\nTjDiCohAzjntb2zXxkMR1aiyfU3aWhVQMHw5Ki46ShPzUnT6xBxNGZGmyQWpmjwijfdIAQAA+IS4\nAnzknNOBQLu27g9oS1VAW6sC2ry/+4pUQ2vnoeNGpidockGaPjQ5T5ML0jRlRKrG5iQrNjrKx+kB\nAABwOOIKGADOOVU2tGlLVUBb9ndfgToYU4dHVGp8jCbmp+iiGQWaEr4SVZKfqvSkWB+nBwAAQF8Q\nV4CHOoIhlde1aMeBZm09ENCW/QFtreqOqeaOrkPHZSbFqjg/VRfPLFBxXoqK81JVnJ+ivNR43hsF\nAAAwSBFXwPsUCjnta2zTjupmba9u1o4DzdpRHdCO6maV17Ue+vBdScpLjVdxfoqunFekiXkpmpiX\nouK8FGWn8L4oAACAoYa4AnrgnFNdS6d2VAe0/UCzdlS/+6s9GDp0bEJslMblpGjayHRdMnOkxuUk\na2xOsibmpSg9kdv5AAAAhgviCsNWsCukyoY27a5tefdXTYt21TSrsS146NiYKNPorCSNy0nWGRNz\nNDYnWeNzkjUuN1n5qQl8XhQAAACIKwxtjW2d2l3z7ngqD/9ZUdd6aFtzqTugRmUmqigrSTNHdV+B\nGp+brHE5KRqVmcjOfAAAADgq4gqDlnNODa2d2lPXqor6Vu2tb1VF+HFFfat217aovqXzXf9NZlKs\nRmclaUZhui6ZWaDRWUkqykrS6KwkFaQnKporUAAAADhOxBUiVlfIaX9jW3c01bdqT113QK3d1qZb\nV/5Te+tb37UDn9T9/qeRGYkqzEjUxTMKNCY76VBAFWUlKS2B90ABAADgxCCu4IuukFN1oF2VDW3a\n19Cm/Y1tqgz/efAq1L6Gtnfdtid1X3lKi3EqGZWsM4pzVBgOqcLM7j+zkuPYyhwAAAC+IK7gudaO\nLu1rfCeaDj7e1/DO4wOB9nfW/TevAAARZ0lEQVRtWS51v+cpPy1BIzMSNHdM5ruiqTAjUSMzEpUc\nH6PS0lItWDDPp//rAAAAgJ4RV+gT55wa24I60NSmqqZ2HTj4FWjXgcbuP6sa27WvsU0NrZ3v+e9T\n42OUn56gEWkJOqM4RyPSEpSfnqCCtASNSE9QflqCspPj2HUPAAAAgxZxNcy1B7tUHehQVWPbO7EU\nDqcjI6rjsM92OiguOkq5qfHKTY3X6OwkzR+XpRHhiDoYTSPSE5QSz1IDAADA0Ma/eIeYts4u1TZ3\nqLa5Q9WB9kOPa5o7VBN+XhN+rTbQoab2YI8/Jys5Trkp8cpLi9f4nORDAXXwKy81XrkpCUpLjOE9\nTgAAAICIq4gW7AqpsS2o+pYO1bV0qqG1Q3XNnb3GUk2g/T275x0UE2XKSo5TVnKcslPiVJSZ0f04\nOe6wYEpQbmq8slPi+EwnAAAA4H0irgZAV8ipsbVT9a2dqm/pUH1Lp+rDoVTf2qmGcDy963FLhxrb\ner6qJEmx0RaOo+4YGpOddCiWslPiDz3OCj9PS+AKEwAAAHAiEVd94JxTc0eXGls71djWqaa24KHH\nja1BNbV1qvGw1w5+vzumul9zruefbSalJcQqIylWGUlxykiK09icZGUmxSk9MVaZ4dfTk2KVmRSn\njMRYZaXEKTWeWAIAAAAiyZCPq2BXSM3tXWpq71Rze5cC7UEF2oNqDv8ZaAseiqTuMHrn8eHxFOol\njg5KiI1SWkKs0hJjlZYQo4ykOI3JTlZmUqzSw1GUmRyrjMS4d0Iqsfv4aHbIAwAAAAa9iI6rYEja\nsLdRzR3vhNChKDoUSF3viaXmju7nTW1Btfeww11PUuJjlJYQE46jWI1IS9Ck/NRDr6UmxBwWT+Hn\n4ZBKTYhVXAzvUQIAAACGs4iOqz2BkC765cu9fj8xNlopCTFKiY9Rcny0UuJjNDIjQcnx3a91v37E\n44QYpcRHKzk+RslxMUoNxxFXjwAAAAD0R0THVU6i6befPKnHQEqOiyGIAAAAAESMiI6rlFjTBdML\n/B4DAAAAAI6JNwoBAAAAgAc8iSszu8DMysxsq5nd0MP3483swfD3l5nZWC/OCwAAAACRot9xZWbR\nkv5b0oWSpkq6xsymHnHY5yXVOecmSvq5pB/397wAAAAAEEm8uHI1X9JW59x251yHpAckLTrimEWS\n7gk/fljS2cYn4AIAAAAYQrzY0KJQUvlhz/dIOqW3Y5xzQTNrkJQtqfrIH2ZmiyUtlqTc3FyVlpZ6\nMCKGkkAgwLrAe7Au0BvWBnrCukBvWBvoj4jbLdA5d5ekuySppKTELViwwN+BEHFKS0vFusCRWBfo\nDWsDPWFdoDesDfSHF7cFVkgqOuz5qPBrPR5jZjGS0iXVeHBuAAAAAIgIXsTVm5KKzWycmcVJulrS\n40cc87ikz4QfXyHpReec8+DcAAAAABAR+n1bYPg9VF+VtFRStKS7nXPrzewWSW855x6X9AdJfzaz\nrZJq1R1gAAAAADBkePKeK+fcEklLjnjtpsMet0m60otzAQAAAEAk8uRDhAEAAABguCOuAAAAAMAD\nxBUAAAAAeIC4AgAAAAAPEFcAAAAA4AHiCgAAAAA8QFwBAAAAgAeIKwAAAADwAHEFAAAAAB4grgAA\nAADAA8QVAAAAAHiAuAIAAAAADxBXAAAAAOAB4goAAAAAPEBcAQAAAIAHiCsAAAAA8ABxBQAAAAAe\nIK4AAAAAwAPEFQAAAAB4gLgCAAAAAA8QVwAAAADgAeIKAAAAADxAXAEAAACAB4grAAAAAPAAcQUA\nAAAAHiCuAAAAAMADxBUAAAAAeIC4AgAAAAAPEFcAAAAA4AHiCgAAAAA8QFwBAAAAgAeIKwAAAADw\nAHEFAAAAAB4grgAAAADAA8QVAAAAAHiAuAIAAAAADxBXAAAAAOAB4goAAAAAPEBcAQAAAIAHiCsA\nAAAA8ABxBQAAAAAeIK4AAAAAwAPEFQAAAAB4gLgCAAAAAA8QVwAAAADgAeIKAAAAADxAXAEAAACA\nB4grAAAAAPAAcQUAAAAAHiCuAAAAAMADxBUAAAAAeIC4AgAAAAAPEFcAAAAA4AHiCgAAAAA8QFwB\nAAAAgAeIKwAAAADwAHEFAAAAAB4grgAAAADAA/2KKzPLMrPnzGxL+M/MXo7rMrNV4a/H+3NOAAAA\nAIhE/b1ydYOkF5xzxZJeCD/vSatzbnb469J+nhMAAAAAIk5/42qRpHvCj++R9JF+/jwAAAAAGJTM\nOXf8/7FZvXMuI/zYJNUdfH7EcUFJqyQFJd3mnPv7UX7mYkmLJSk3N3fuQw89dNzzYWgKBAJKSUnx\newxEGNYFesPaQE9YF+gNawM9Wbhw4Qrn3LxjHXfMuDKz5yWN6OFbN0q65/CYMrM659x73ndlZoXO\nuQozGy/pRUlnO+e2HWu4kpISV1ZWdqzDMMyUlpZqwYIFfo+BCMO6QG9YG+gJ6wK9YW2gJ2bWp7iK\nOdYBzrlzjnKS/WZW4JyrNLMCSVW9/IyK8J/bzaxU0hxJx4wrAAAAABgs+vueq8clfSb8+DOSHjvy\nADPLNLP48OMcSadL2tDP8wIAAABAROlvXN0m6Vwz2yLpnPBzmdk8M/t9+Jgpkt4ys9WS/qHu91wR\nVwAAAACGlGPeFng0zrkaSWf38Ppbkr4QfvyapBn9OQ8AAAAARLr+XrkCAAAAAIi4AgAAAABPEFcA\nAAAA4AHiCgAAAAA8QFwBAAAAgAeIKwAAAADwAHEFAAAAAB4grgAAAADAA8QVAAAAAHiAuAIAAAAA\nDxBXAAAAAOAB4goAAAAAPEBcAQAAAIAHiCsAAAAA8ABxBQAAAAAeIK4AAAAAwAPEFQAAAAB4gLgC\nAAAAAA8QVwAAAADgAeIKAAAAADxAXAEAAACAB4grAAAAAPAAcQUAAAAAHiCuAAAAAMADxBUAAAAA\neIC4AgAAAAAPEFcAAAAA4AHiCgAAAAA8QFwBAAAAgAeIKwAAAADwAHEFAAAAAB4grgAAAADAA8QV\nAAAAAHiAuAIAAAAADxBXAAAAAOAB4goAAAAAPEBcAQAAAIAHiCsAAAAA8ABxBQAAAAAeIK4AAAAA\nwAPEFQAAAAB4gLgCAAAAAA8QVwAAAADgAeIKAAAAADxAXAEAAACAB4grAAAAAPAAcQUAAAAAHiCu\nAAAAAMADxBUAAAAAeIC4AgAAAAAPEFcAAAAA4AHiCgAAAAA8QFwBAAAAgAeIKwAAAADwAHEFAAAA\nAB4grgAAAADAA8QVAAAAAHiAuAIAAAAAD/QrrszsSjNbb2YhM5t3lOMuMLMyM9tqZjf055wAAAAA\nEIn6e+VqnaTLJb3U2wFmFi3pvyVdKGmqpGvMbGo/zwsAAAAAESWmP/+xc26jJJnZ0Q6bL2mrc257\n+NgHJC2StKE/5wYAAACASDIQ77kqlFR+2PM94dcAAAAAYMg45pUrM3te0ogevnWjc+4xrwcys8WS\nFktSbm6uSktLvT4FBrlAIMC6wHuwLtAb1gZ6wrpAb1gb6I9jxpVz7px+nqNCUtFhz0eFX+vtfHdJ\nukuSSkpK3IIFC/p5egw1paWlYl3gSKwL9Ia1gZ6wLtAb1gb6YyBuC3xTUrGZjTOzOElXS3p8AM4L\nAAAAAAOmv1uxX2ZmeySdKukpM1safn2kmS2RJOdcUNJXJS2VtFHSQ8659f0bGwAAAAAiS393C3xU\n0qM9vL5X0kWHPV8iaUl/zgUAAAAAkWwgbgsEAAAAgCGPuAIAAAAADxBXAAAAAOAB4goAAAAAPEBc\nAQAAAIAHiCsAAAAA8ABxBQAAAAAeIK4AAAAAwAPEFQAAAAB4gLgCAAAAAA8QVwAAAADgAeIKAAAA\nADxAXAEAAACAB4grAAAAAPAAcQUAAAAAHiCuAAAAAMADxBUAAAAAeIC4AgAAAAAPEFcAAAAA4AHi\nCgAAAAA8QFwBAAAAgAeIKwAAAADwAHEFAAAAAB4grgAAAADAA8QVAAAAAHiAuAIAAAAADxBXAAAA\nAOAB4goAAAAAPEBcAQAAAIAHiCsAAAAA8ABxBQAAAAAeIK4AAAAAwAPEFQAAAAB4gLgCAAAAAA8Q\nVwAAAADgAeIKAAAAADxAXAEAAACAB4grAAAAAPAAcQUAAAAAHiCuAAAAAMADxBUAAAAAeIC4AgAA\nAAAPEFcAAAAA4AHiCgAAAAA8QFwBAAAAgAeIKwAAAADwAHEFAAAAAB4grgAAAADAA8QVAAAAAHiA\nuAIAAAAADxBXAAAAAOAB4goAAAAAPEBcAQAAAIAHiCsAAAAA8ABxBQAAAAAeIK4AAAAAwAPEFQAA\nAAB4gLgCAAAAAA/0K67M7EozW29mITObd5TjdprZWjNbZWZv9eecAAAAABCJYvr536+TdLmk3/Xh\n2IXOuep+ng8AAAAAIlK/4so5t1GSzMybaQAAAABgkBqo91w5Sc+a2QozWzxA5wQAAACAAXPMK1dm\n9rykET1860bn3GN9PM8ZzrkKM8uT9JyZbXLOvdTL+RZLOhhg7Wa2ro/nwPCRI4lbTHEk1gV6w9pA\nT1gX6A1rAz0p6ctBx4wr59w5/Z3EOVcR/rPKzB6VNF9Sj3HlnLtL0l2SZGZvOed63SgDwxPrAj1h\nXaA3rA30hHWB3rA20JO+bsp3wm8LNLNkM0s9+FjSeereCAMAAAAAhoz+bsV+mZntkXSqpKfMbGn4\n9ZFmtiR8WL6kV8xstaTlkp5yzj3Tn/MCAAAAQKTp726Bj0p6tIfX90q6KPx4u6RZx3mKu45/Ogxh\nrAv0hHWB3rA20BPWBXrD2kBP+rQuzDl3ogcBAAAAgCFvoLZiBwAAAIAhLaLjysx+aGZrzGyVmT1r\nZiP9ngmRwczuMLNN4fXxqJll+D0T/GdmV5rZejMLmRk7PQ1zZnaBmZWZ2VYzu8HveRAZzOxuM6vi\no15wODMrMrN/mNmG8N8j1/s9EyKDmSWY2XIzWx1eG/951OMj+bZAM0tzzjWGH/+bpKnOuet8HgsR\nwMzOk/Sicy5oZj+WJOfcd30eCz4zsymSQpJ+J+lbzrk+bZuKocfMoiVtlnSupD2S3pR0jXNug6+D\nwXdmdqakgKR7nXPT/Z4HkcHMCiQVOOdWhne5XiHpI/zOgJmZpGTnXMDMYiW9Iul659wbPR0f0Veu\nDoZVWLKkyC1BDCjn3LPOuWD46RuSRvk5DyKDc26jc67M7zkQEeZL2uqc2+6c65D0gKRFPs+ECOCc\ne0lSrd9zILI45yqdcyvDj5skbZRU6O9UiASuWyD8NDb81WuTRHRcSZKZ3Wpm5ZI+Iekmv+dBRPqc\npKf9HgJARCmUVH7Y8z3iH0oA+sDMxkqaI2mZv5MgUphZtJmtklQl6TnnXK9rw/e4MrPnzWxdD1+L\nJMk5d6NzrkjSfZK+6u+0GEjHWhvhY26UFFT3+sAw0Jd1AQDA8TCzFEmPSPraEXdQYRhzznU552ar\n+06p+WbW6y3F/fqcKy84587p46H3SVoi6eYTOA4iyLHWhpldK+kSSWe7SH7zIDz1Pn5nYHirkFR0\n2PNR4dcAoEfh99M8Iuk+59zf/J4Hkcc5V29m/5B0gaQeN8Xx/crV0ZhZ8WFPF0na5NcsiCxmdoGk\n70i61DnX4vc8ACLOm5KKzWycmcVJulrS4z7PBCBChTct+IOkjc65n/k9DyKHmeUe3JXazBLVvVFS\nr00S6bsFPiKpRN27f+2SdJ1zjv/lETKzrZLiJdWEX3qDnSRhZpdJ+pWkXEn1klY55873dyr4xcwu\nknSnpGhJdzvnbvV5JEQAM/uLpAWSciTtl3Szc+4Pvg4F35nZGZJelrRW3f/ulKTvO+eW+DcVIoGZ\nzZR0j7r/LomS9JBz7pZej4/kuAIAAACAwSKibwsEAAAAgMGCuAIAAAAADxBXAAAAAOAB4goAAAAA\nPEBcAQAAAIAHiCsAAAAA8ABxBQAAAAAeIK4AAAAAwAP/H1Jy90TyLtqpAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fdde8452e10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot(lambda x: math.log(1 + math.exp(x)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Problem zanikającego gradientu (_vanishing gradient problem_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Sigmoidalne funkcje aktywacji ograniczają wartości na wyjściach neuronów do niewielkich przedziałów ($(-1, 1)$, $(0, 1)$ itp.).\n",
    "* Jeżeli sieć ma wiele warstw, to podczas propagacji wstecznej mnożymy przez siebie wiele małych wartości → obliczony gradient jest mały.\n",
    "* Im więcej warstw, tym silniejszy efekt zanikania."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Sposoby na zanikający gradient\n",
    "\n",
    "* Modyfikacja algorytmu optymalizacji (_RProp_, _RMSProp_)\n",
    "* Użycie innej funckji aktywacji (ReLU, softplus)\n",
    "* Dodanie warstw _dropout_\n",
    "* Nowe architektury (LSTM itp.)\n",
    "* Więcej danych, zwiększenie mocy obliczeniowej"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 11.2. Wielowarstwowe sieci neuronowe w&nbsp;praktyce"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Przykład: MNIST\n",
    "\n",
    "_Modified National Institute of Standards and Technology database_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Zbiór cyfr zapisanych pismem odręcznym\n",
    "* 60 000 przykładów uczących, 10 000 przykładów testowych\n",
    "* Rozdzielczość każdego przykładu: 28 × 28 = 784 piksele"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# źródło: https://github.com/keras-team/keras/examples/minst_mlp.py\n",
    "\n",
    "import keras\n",
    "from keras.datasets import mnist\n",
    "\n",
    "# załaduj dane i podziel je na zbiory uczący i testowy\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "def draw_examples(examples, captions=None):\n",
    "    plt.figure(figsize=(16, 4))\n",
    "    m = len(examples)\n",
    "    for i, example in enumerate(examples):\n",
    "        plt.subplot(100 + m * 10 + i + 1)\n",
    "        plt.imshow(example, cmap=plt.get_cmap('gray'))\n",
    "    plt.show()\n",
    "    if captions is not None:\n",
    "        print(6 * ' ' + (10 * ' ').join(str(captions[i]) for i in range(m)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6IAAACPCAYAAADgImbyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAHEtJREFUeJzt3XmQVNXZx/HniIAYREQIISKCggiR\nTUDB1wITwBVZJKKEPUYoUYSUUKASgzEIolLFIlEkMIKUaIVVI0EElKiEAgnmZXXAyJYRUEE2Iy96\n3z/o5T5Hpqd7uvvc2z3fT9UU9ze3u+/pnme659D93GM8zxMAAAAAAFw5J+gBAAAAAADKFiaiAAAA\nAACnmIgCAAAAAJxiIgoAAAAAcIqJKAAAAADAKSaiAAAAAACnmIgCAAAAAJxiIgoAAAAAcCqtiagx\n5hZjzA5jzE5jzOhMDQq5h1pAFLUAEeoAcdQCRKgDxFELiPE8r1RfIlJORHaJyOUiUkFEPhaRxiVc\nx+Mr574OZboWQnCf+MpCHVALZeYr488J1ELOfvH6wFdW6oBayNkvaoGvpGvB87y03hG9VkR2ep73\nqed5p0Rkvoh0TeP2EE67k7gMtZD/kqkDEWqhLOA5AVHUAkSoA8RRC4hK6u/GdCail4jIXl/eF/me\nYowZZIzZYIzZkMaxEG4l1gJ1UGZQCxDh9QFxPCdAhOcExFELiDk32wfwPG+GiMwQETHGeNk+HsKJ\nOkAUtYAoagEi1AHiqAVEUQtlQzrviO4XkUt9uXbkeyh7qAVEUQsQoQ4QRy1AhDpAHLWAmHQmoutF\npIExpp4xpoKI3CMiSzMzLOQYagFR1AJEqAPEUQsQoQ4QRy0gptQfzfU877Qx5kERWS5nzoA1y/O8\nLRkbGXIGtYAoagEi1AHiqAWIUAeIoxbgZyKnRXZzMD7jnYs+8jyvVSZvkDrISRmvAxFqIUdRC4ji\n9QEiPCcgjlpAVFK1kM5HcwEAAAAASBkTUQAAAACAU0xEAQAAAABOMREFAAAAADjFRBQAAAAA4BQT\nUQAAAACAU0xEAQAAAABOMREFAAAAADh1btADAPJVy5YtVX7wwQdV7tevn8pz5sxReerUqSpv3Lgx\ng6MDAABANk2ePFnlhx56KLa9efNmta9z584q7969O3sDCwneEQUAAAAAOMVEFAAAAADgFB/NTVK5\ncuVUvvDCC5O+rv2RzPPPP1/lhg0bqvzAAw+o/Oyzz6rcq1cvlf/73/+qPGHChNj2E088kfQ4kZ7m\nzZurvGLFCpWrVKmisud5Kvft21flLl26qHzxxRenO0TkiQ4dOqg8b948ldu3b6/yjh07sj4mZMeY\nMWNUtp/TzzlH/3/yjTfeqPJ7772XlXEByIwLLrhA5cqVK6t8++23q1yjRg2VJ02apPK3336bwdEh\nVXXr1lW5T58+Kn///fex7UaNGql9V111lcp8NBcAAAAAgAxjIgoAAAAAcIqJKAAAAADAqTLTI1qn\nTh2VK1SooPL111+v8g033KBy1apVVe7Ro0fGxrZv3z6Vp0yZonL37t1VPnbsmMoff/yxyvQEuXPt\ntdfGthcsWKD22X3Edk+o/XM8deqUynZPaJs2bVS2l3Oxr18WtGvXLrZtP16LFi1yPRxnWrdurfL6\n9esDGgkybcCAASqPGjVKZX9/0dnYzzMAgufvG7R/p9u2bavy1VdfndJt16pVS2X/8iBw79ChQyqv\nWbNGZfv8H2Ud74gCAAAAAJxiIgoAAAAAcIqJKAAAAADAqbztEbXXdFy1apXKqawDmml2j4+9Ttzx\n48dVttcILCoqUvnw4cMqs2Zg5thrvl5zzTUqv/LKK7Ftu0+jJIWFhSpPnDhR5fnz56v8wQcfqGzX\nzfjx41M6fj7wr5nYoEEDtS+fekTttSLr1aun8mWXXaayMSbrY0J22D/L8847L6CRIFXXXXedyv71\nA+21fX/2s58lvK0RI0ao/J///Edl+zwW/tciEZF169YlHiwyyl7/cfjw4Sr37t07tl2pUiW1z36+\n3rt3r8r2+STstSd79uyp8vTp01Xevn17ccNGFpw4cULlsrAWaDp4RxQAAAAA4BQTUQAAAACAU0xE\nAQAAAABO5W2P6J49e1T+8ssvVc5kj6jdi3HkyBGVf/7zn6tsr/c4d+7cjI0FmfXiiy+q3KtXr4zd\ntt1vWrlyZZXt9WD9/ZAiIk2bNs3YWHJVv379Yttr164NcCTZZfcf33fffSrb/WH0BOWOjh07qjx0\n6NCEl7d/tp07d1b5wIEDmRkYSnT33XerPHnyZJWrV68e27b7AN99912Va9SoofIzzzyT8Nj27dnX\nv+eeexJeH6mx/2Z8+umnVbZr4YILLkj6tu3zRdx8880qly9fXmX7OcBfZ2fLcKtq1aoqN2vWLKCR\n5AbeEQUAAAAAOMVEFAAAAADgFBNRAAAAAIBTedsj+tVXX6k8cuRIle2+mn/+858qT5kyJeHtb9q0\nKbbdqVMntc9eQ8heL2zYsGEJbxvBadmypcq33367yonWZ7R7Ot944w2Vn332WZXtdeHsGrTXh/3F\nL36R9FjKCnt9zXw1c+bMhPvtHiOEl73+4+zZs1Uu6fwFdu8ga9Rlz7nn6j+RWrVqpfJLL72ksr3u\n9Jo1a2LbTz75pNr3/vvvq1yxYkWVX3/9dZVvuummhGPdsGFDwv1IT/fu3VX+zW9+U+rb2rVrl8r2\n35D2OqL169cv9bHgnv08UKdOnaSv27p1a5XtfuB8fL4vG3/FAQAAAABCg4koAAAAAMCpEieixphZ\nxpiDxpjNvu9VM8asMMYURv69KLvDRBhQC4iiFiBCHSCOWkAUtQAR6gDJSaZHtEBEponIHN/3RovI\nSs/zJhhjRkfyqMwPL3MWL16s8qpVq1Q+duyYyva6P/fee6/K/n4/uyfUtmXLFpUHDRqUeLDhVSB5\nUAt+zZs3V3nFihUqV6lSRWXP81RetmxZbNteY7R9+/YqjxkzRmW77+/QoUMqf/zxxyp///33Ktv9\nq/a6pBs3bpQsKpAAasFeO7VmzZqZvPnQKqlv0K5bhwokz54Tsq1///4q//SnP014eXu9yTlz5pz9\ngsErkDyrhT59+qhcUq+2/XvoX1vy6NGjCa9rr0NZUk/ovn37VH755ZcTXt6xAsmzWrjrrrtSuvxn\nn32m8vr162Pbo0bpu233hNoaNWqU0rFDpEDyrA6SYZ//o6CgQOWxY8cWe11735EjR1SeNm1aOkML\npRLfEfU8b42IfGV9u6uIRJ/1XhaRbhkeF0KIWkAUtQAR6gBx1AKiqAWIUAdITmnPmlvT87yiyPbn\nIlLs2xLGmEEikrNvAaJESdUCdVAmUAsQ4fUBcTwnIIpagAivD7CkvXyL53meMcZLsH+GiMwQEUl0\nOeS+RLVAHZQt1AJEeH1AHM8JiKIWIMLrA84o7UT0gDGmlud5RcaYWiJyMJODcqGkfo2vv/464f77\n7rsvtv3aa6+pfXYvX57LqVq48sorVbbXl7V78b744guVi4qKVPb35Rw/flzt++tf/5owp6tSpUoq\nP/zwwyr37t07o8dLQtZr4bbbblPZfgzyhd37Wq9evYSX379/fzaHk6qcek7IturVq6v861//WmX7\n9cLuCfrjH/+YnYG5kVO1YK/1+eijj6psnyNg+vTpKtvnASjp7wy/xx57LOnLiog89NBDKtvnGAih\nnKoFm/9vPpEfnuvj7bffVnnnzp0qHzxY+rubZ+dCyOk6KA37eSVRj2hZVNrlW5aKSPSMC/1FZElm\nhoMcRC0gilqACHWAOGoBUdQCRKgDWJJZvuVVEVkrIg2NMfuMMfeKyAQR6WSMKRSRjpGMPEctIIpa\ngAh1gDhqAVHUAkSoAySnxI/mep7Xq5hdHTI8FoQctYAoagEi1AHiqAVEUQsQoQ6QnLRPVpSv7M9w\nt2zZUmX/GpEdO3ZU++xeAQSnYsWKKvvXfxX5Yc+hvZ5sv379VN6wYYPKYepRrFOnTtBDyLqGDRsW\nu89erzeX2XVq9wh98sknKtt1i2DVrVs3tr1gwYKUrjt16lSVV69enYkh4Swef/xxle2e0FOnTqm8\nfPlyle31IL/55ptij3XeeeepbK8Taj9/G2NUtnuFlyzhE40u2WtDuuzza9u2rbNjIfvOOSf+YdQy\ndk6ZsyptjygAAAAAAKXCRBQAAAAA4BQTUQAAAACAU/SIFuPEiRMq22tIbdy4Mbb90ksvqX12T4/d\nV/j888+rbK9Nhsxp0aKFynZPqK1r164qv/feexkfE7Jj/fr1QQ+hWFWqVFH5lltuUblPnz4q2/1j\nNntdMnvtSQTL//Nt2rRpwsuuXLlS5cmTJ2dlTBCpWrWqykOGDFHZfi22e0K7deuW0vHq168f2543\nb57aZ593wvaXv/xF5YkTJ6Z0bISLf93XH/3oRyldt0mTJgn3f/jhhyqvXbs2pduHW/6+UP7+5x1R\nAAAAAIBjTEQBAAAAAE7x0dwk7dq1S+UBAwbEtmfPnq329e3bN2G2P5YxZ84clYuKiko7TFgmTZqk\nsn1KfPujt2H+KK7/lN8inPbbVq1atbSu36xZM5XtWrGXaapdu7bKFSpUiG337t1b7bN/dvYyD+vW\nrVP522+/Vfncc/VT9UcffSQID/sjmxMmFL9G+/vvv69y//79Vf76668zNzAo/t9REZHq1asnvLz/\n45QiIj/+8Y9VHjhwoMpdunRR+eqrr45tV65cWe2zP5Jn51deeUVlu10IwTr//PNVbty4scq///3v\nVU7UFpTqa7u9lIxdh999913C6wNhwjuiAAAAAACnmIgCAAAAAJxiIgoAAAAAcIoe0VJatGhRbLuw\nsFDts/sSO3TooPJTTz2l8mWXXabyuHHjVN6/f3+px1nWdO7cWeXmzZurbPfhLF26NOtjyhS7b8S+\nL5s2bXI5nEDYvZX+x+CFF15Q+x599NGUbtteZsPuET19+rTKJ0+eVHnr1q2x7VmzZql99hJOdi/y\ngQMHVN63b5/KlSpVUnn79u2C4NStW1flBQsWJH3dTz/9VGX7Z4/sOXXqlMqHDh1SuUaNGir/+9//\nVjnVpRb8vXxHjx5V+2rVqqXyF198ofIbb7yR0rGQWeXLl1fZXgrO/p23f572a5W/FuzlVezlvOz+\nU5t9zoA777xTZXsJKLvugTDhHVEAAAAAgFNMRAEAAAAATjERBQAAAAA4RY9oBmzevFnlnj17qnzH\nHXeobK87OnjwYJUbNGigcqdOndIdYplh99LZ68YdPHhQ5ddeey3rY0pWxYoVVR47dmzCy69atUrl\nRx55JNNDCp0hQ4aovHv37tj29ddfn9Zt79mzR+XFixervG3bNpX/8Y9/pHU8v0GDBqls96rZfYUI\n1qhRo1ROZU3fRGuMIruOHDmisr3+65tvvqmyvTaxvZ74kiVLVC4oKFD5q6++im3Pnz9f7bN7Cu39\ncMv+W8Hu21y4cGHC6z/xxBMq26/PH3zwQWzbriv7sv71Z8/Gfn0YP368yiW9ltnrVMMt/7qxJb12\ntGvXTuVp06ZlZUxB4h1RAAAAAIBTTEQBAAAAAE4xEQUAAAAAOEWPaBbYfShz585VeebMmSrba0LZ\nnwm/8cYbVX733XfTG2AZZvdGFBUVBTSSH/aEjhkzRuWRI0eqbK8t+dxzz6l8/PjxDI4uNzz99NNB\nDyEj7LWGbamsU4nMs9cjvummm5K+rt1HuGPHjoyMCelbt26dynbvXbr8r+Xt27dX++zeMPrA3bLX\nCbV7PO3XX9uyZctUnjp1qsr234H+2nrrrbfUviZNmqhsr/s5ceJEle0e0q5du6o8b948ld955x2V\n7dfNw4cPS3HKwvrkrvl/90tam9heI7Zx48Yq+9cvz1W8IwoAAAAAcIqJKAAAAADAKSaiAAAAAACn\n6BHNgKZNm6r8y1/+UuXWrVurbPeE2uzPfK9ZsyaN0cFv6dKlgR3b7jOze1Duvvtule3esh49emRn\nYAi9RYsWBT2EMu3tt99W+aKLLkp4ef8aswMGDMjGkJAD/Ota2z2hdm8Y64hmV7ly5VR+8sknVR4x\nYoTKJ06cUHn06NEq2z8vuye0VatWKvvXf2zRooXaV1hYqPL999+v8urVq1WuUqWKyvYa2r1791a5\nS5cuKq9YsUKKs3fvXpXr1atX7GVROi+88EJse/DgwSld115zfPjw4RkZU5B4RxQAAAAA4BQTUQAA\nAACAU0xEAQAAAABO0SOapIYNG6r84IMPxrbtdX5+8pOfpHTb3333ncr22pZ2bwmKZ4xJmLt166by\nsGHDsjaW3/72tyr/7ne/U/nCCy9U2V77q1+/ftkZGICUXHzxxSqX9Jw8ffr02HZZXN8XZyxfvjzo\nISDC7q2ze0JPnjypst27Z/eJt2nTRuWBAweqfOutt6rs7xf+wx/+oPbNnj1bZbtP03b06FGV//a3\nvyXMvXr1UvlXv/pVsbdt/92CzNu+fXvQQwgV3hEFAAAAADhV4kTUGHOpMWa1MWarMWaLMWZY5PvV\njDErjDGFkX8Tn0YQOY9agAh1gDhqAVHUAkSoA8RRC0hGMu+InhaRhz3PaywibUTkAWNMYxEZLSIr\nPc9rICIrIxn5jVqACHWAOGoBUdQCRKgDxFELKFGJPaKe5xWJSFFk+5gxZpuIXCIiXUXkxsjFXhaR\nd0VkVFZG6YDd12l/pt7fEyoiUrdu3VIfa8OGDSqPGzdO5SDXukwkF2rBXpvNzvbPecqUKSrPmjVL\n5S+//FJluy+kb9++se1mzZqpfbVr11Z5z549Ktv9Q/6+sjDLhTrIdXZv85VXXqmyf53KIOVrLdg9\nW+eck1oXy4cffpjJ4eSEfK2FdNx8881BD8G5sNbB448/nnC/vc6ovc732LFjVa5fv35Kx/dff/z4\n8WqffZ6QTHv11VcT5mwJay0EberUqbHtoUOHqn1XXHFFwuva5zXx35aIyK5du9IcnXspvboaY+qK\nSAsRWSciNSNFJiLyuYjUzOjIEGrUAkSoA8RRC4iiFiBCHSCOWkBxkj5rrjGmsogsEJHhnucd9f+P\nved5njHGK+Z6g0Rk0Nn2ITeVphaog/zDcwKiqAVE8foAEZ4TEEctIJGk3hE1xpSXM0U0z/O8hZFv\nHzDG1IrsryUiB892Xc/zZnie18rzvFaZGDCCVdpaoA7yC88JiKIWEMXrA0R4TkActYCSlPiOqDnz\nXxd/FpFtnudN8u1aKiL9RWRC5N8lWRlhhtSsqd/5b9y4scrTpk1T+aqrrir1sdatW6fyM888o/KS\nJfqhypV1QvOhFuw+kCFDhqjco0cPle31uho0aJD0sew+sdWrV6tcUs9KWOVDHYSd3ducao+iK/lS\nC82bN1e5Y8eOKtvP0adOnVL5+eefV/nAgQMZHF1uyJdayKTLL7886CE4F9Y6+Pzzz1WuUaOGyhUr\nVlTZPueD7a233lJ5zZo1Ki9evFjlzz77LLad7Z7QsAhrLYTJli1bVC7pOSNX5gupSOajuf8jIn1F\n5H+NMZsi33tUzhTQ68aYe0Vkt4j0zM4QESLUAkSoA8RRC4iiFiBCHSCOWkCJkjlr7vsiYorZ3SGz\nw0GYUQsQoQ4QRy0gilqACHWAOGoByQjn570AAAAAAHkr6bPmhl21atVUfvHFF1W2e4DS7d3w9/89\n99xzap+9PuQ333yT1rGQvLVr16q8fv16lVu3bp3w+vY6o3Zvsc2/zuj8+fPVPnu9J6C02rZtq3JB\nQUEwA8lTVatWVdl+HrDt379f5REjRmR8TMh9f//732Pbdp93PvZ6hVm7du1U7tatm8rXXHONygcP\n6vPn2GuMHz58WGW7bxxIxowZM1S+4447AhpJcHhHFAAAAADgFBNRAAAAAIBTTEQBAAAAAE7lVI/o\nddddF9seOXKk2nfttdeqfMkll6R1rJMnT6o8ZcoUlZ966qnY9okTJ9I6FjJn3759Kt95550qDx48\nWOUxY8akdPuTJ09W+U9/+lNse+fOnSndFlCcM8uvAchlmzdvjm0XFhaqffZ5Kq644gqVDx06lL2B\nlUHHjh1Tee7cuQkz4MLWrVtV3rZtm8qNGjVyOZxA8I4oAAAAAMApJqIAAAAAAKdy6qO53bt3P+t2\nMuy3v998802VT58+rbK9JMuRI0dSOh7CoaioSOWxY8cmzEAQli1bpvJdd90V0EjKpu3bt6vsX55L\nROSGG25wORzkIX87j4jIzJkzVR43bpzKQ4cOVdn+GwZA7tu9e7fKTZo0CWgkweEdUQAAAACAU0xE\nAQAAAABOMREFAAAAADhlPM9zdzBj3B0MmfKR53mtMnmD1EFOyngdiFALOYpaQBSvD0mqUqWKyq+/\n/rrKHTt2VHnhwoUqDxw4UOWQLRvHcwKiqAVEJVULvCMKAAAAAHCKiSgAAAAAwCkmogAAAAAAp3Jq\nHVEAAIBcc/ToUZV79uypsr2O6P3336+yveY164oCyAe8IwoAAAAAcIqJKAAAAADAKSaiAAAAAACn\n6BEFAABwyO4ZHTp0aMIMAPmId0QBAAAAAE4xEQUAAAAAOMVEFAAAAADglOse0S9EZLeIVI9shxFj\n0y7Lwm1SB+nJlzoQoRbSRS24xdg0Xh/CJ1/qQIRaSBe14FZYxxbUuJKqBeN5XrYH8sODGrPB87xW\nzg+cBMbmTpjvD2NzK8z3ibG5Feb7xNjcCfP9YWxuhfk+MTa3wnyfwjq2sI4rio/mAgAAAACcYiIK\nAAAAAHAqqInojICOmwzG5k6Y7w9jcyvM94mxuRXm+8TY3Anz/WFsboX5PjE2t8J8n8I6trCOS0QC\n6hEFAAAAAJRdfDQXAAAAAOAUE1EAAAAAgFNOJ6LGmFuMMTuMMTuNMaNdHvssY5lljDlojNns+141\nY8wKY0xh5N+LAhrbpcaY1caYrcaYLcaYYWEaXyZQC0mPjVpwO5ZQ1gJ14HwsoayDyDioBbdjoRYC\nRC0kNS7qwO1YQlkHkXHkXC04m4gaY8qJyPMicquINBaRXsaYxq6OfxYFInKL9b3RIrLS87wGIrIy\nkoNwWkQe9jyvsYi0EZEHIo9VWMaXFmohJdSCWwUSzlqgDtwqkHDWgQi14FqBUAuBoBaSRh24VSDh\nrAORXKwFz/OcfIlIWxFZ7suPiMgjro5fzJjqishmX94hIrUi27VEZEeQ4/ONa4mIdArr+KgFaoFa\noA6oA2qBWgj8saMWqAXqgDrIqVpw+dHcS0Rkry/vi3wvTGp6nlcU2f5cRGoGORgREWNMXRFpISLr\nJITjKyVqoRSohcCE6rGmDgITuseaWghM6B5raiEwoXqsqYPAhO6xzpVa4GRFxfDO/LdBoGvbGGMq\ni8gCERnued5R/74wjK+sCMNjTS2EQ9CPNXUQDmF4rKmFcAjDY00thEPQjzV1EA5heKxzqRZcTkT3\ni8ilvlw78r0wOWCMqSUiEvn3YFADMcaUlzNFNM/zvIVhG1+aqIUUUAuBC8VjTR0ELjSPNbUQuNA8\n1tRC4ELxWFMHgQvNY51rteByIrpeRBoYY+oZYyqIyD0istTh8ZOxVET6R7b7y5nPVjtnjDEi8mcR\n2eZ53iTfrlCMLwOohSRRC6EQ+GNNHYRCKB5raiEUQvFYUwuhEPhjTR2EQige65ysBcdNs7eJyCci\nsktEHguyOVZEXhWRIhH5PznzefN7ReRiOXM2qUIReUdEqgU0thvkzNvm/xKRTZGv28IyPmqBWqAW\nqAPqgOcEaoFaoBaCf6ypA+ogl2vBRAYOAAAAAIATnKwIAAAAAOAUE1EAAAAAgFNMRAEAAAAATjER\nBQAAAAA4xUQUAAAAAOAUE1EAAAAAgFNMRAEAAAAATv0/VRGGEPckXi4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fdda922aad0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      5          0          4          1          9          2          1\n"
     ]
    }
   ],
   "source": [
    "draw_examples(x_train[:7], captions=y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "60000 przykładów uczących\n",
      "10000 przykładów testowych\n"
     ]
    }
   ],
   "source": [
    "num_classes = 10\n",
    "\n",
    "x_train = x_train.reshape(60000, 784)  # 784 = 28 * 28\n",
    "x_test = x_test.reshape(10000, 784)\n",
    "x_train = x_train.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "print('{} przykładów uczących'.format(x_train.shape[0]))\n",
    "print('{} przykładów testowych'.format(x_test.shape[0]))\n",
    "\n",
    "# przekonwertuj wektory klas na binarne macierze klas\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_1 (Dense)              (None, 512)               401920    \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 512)               262656    \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 10)                5130      \n",
      "=================================================================\n",
      "Total params: 669,706\n",
      "Trainable params: 669,706\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = Sequential()\n",
    "model.add(Dense(512, activation='relu', input_shape=(784,)))\n",
    "model.add(Dropout(0.2))\n",
    "model.add(Dense(512, activation='relu'))\n",
    "model.add(Dropout(0.2))\n",
    "model.add(Dense(num_classes, activation='softmax'))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "((60000, 784), (60000, 10))\n"
     ]
    }
   ],
   "source": [
    "print(x_train.shape, y_train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/5\n",
      "60000/60000 [==============================] - 9s 153us/step - loss: 0.2489 - acc: 0.9224 - val_loss: 0.1005 - val_acc: 0.9706\n",
      "Epoch 2/5\n",
      "60000/60000 [==============================] - 9s 151us/step - loss: 0.1042 - acc: 0.9683 - val_loss: 0.0861 - val_acc: 0.9740\n",
      "Epoch 3/5\n",
      "60000/60000 [==============================] - 9s 153us/step - loss: 0.0742 - acc: 0.9782 - val_loss: 0.0733 - val_acc: 0.9796\n",
      "Epoch 4/5\n",
      "60000/60000 [==============================] - 9s 154us/step - loss: 0.0603 - acc: 0.9824 - val_loss: 0.0713 - val_acc: 0.9800\n",
      "Epoch 5/5\n",
      "60000/60000 [==============================] - 9s 157us/step - loss: 0.0512 - acc: 0.9848 - val_loss: 0.0749 - val_acc: 0.9795\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fdda4f97110>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(loss='categorical_crossentropy', optimizer=RMSprop(), metrics=['accuracy'])\n",
    "\n",
    "model.fit(x_train, y_train, batch_size=128, epochs=5, verbose=1,\n",
    "          validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test loss: 0.074858742202\n",
      "Test accuracy: 0.9795\n"
     ]
    }
   ],
   "source": [
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "\n",
    "print('Test loss: {}'.format(score[0]))\n",
    "print('Test accuracy: {}'.format(score[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Warstwa _dropout_ to metoda regularyzacji, służy zapobieganiu nadmiernemu dopasowaniu sieci. Polega na tym, że część węzłów sieci jest usuwana w sposób losowy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_4 (Dense)              (None, 512)               401920    \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 512)               262656    \n",
      "_________________________________________________________________\n",
      "dense_6 (Dense)              (None, 10)                5130      \n",
      "=================================================================\n",
      "Total params: 669,706\n",
      "Trainable params: 669,706\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/5\n",
      "60000/60000 [==============================] - 8s 139us/step - loss: 0.2237 - acc: 0.9303 - val_loss: 0.0998 - val_acc: 0.9676\n",
      "Epoch 2/5\n",
      "60000/60000 [==============================] - 8s 136us/step - loss: 0.0818 - acc: 0.9748 - val_loss: 0.0788 - val_acc: 0.9770\n",
      "Epoch 3/5\n",
      "60000/60000 [==============================] - 8s 136us/step - loss: 0.0538 - acc: 0.9831 - val_loss: 0.1074 - val_acc: 0.9695\n",
      "Epoch 4/5\n",
      "60000/60000 [==============================] - 10s 161us/step - loss: 0.0397 - acc: 0.9879 - val_loss: 0.0871 - val_acc: 0.9763\n",
      "Epoch 5/5\n",
      "60000/60000 [==============================] - 12s 195us/step - loss: 0.0299 - acc: 0.9910 - val_loss: 0.0753 - val_acc: 0.9812\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fdda3dcad50>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Bez warstw Dropout\n",
    "\n",
    "num_classes = 10\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "\n",
    "x_train = x_train.reshape(60000, 784)  # 784 = 28 * 28\n",
    "x_test = x_test.reshape(10000, 784)\n",
    "x_train = x_train.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes)\n",
    "\n",
    "model_no_dropout = Sequential()\n",
    "model_no_dropout.add(Dense(512, activation='relu', input_shape=(784,)))\n",
    "model_no_dropout.add(Dense(512, activation='relu'))\n",
    "model_no_dropout.add(Dense(num_classes, activation='softmax'))\n",
    "model_no_dropout.summary()\n",
    "\n",
    "model_no_dropout.compile(loss='categorical_crossentropy',\n",
    "                         optimizer=RMSprop(),\n",
    "                         metrics=['accuracy'])\n",
    "\n",
    "model_no_dropout.fit(x_train, y_train,\n",
    "                     batch_size=128,\n",
    "                     epochs=5,\n",
    "                     verbose=1,\n",
    "                     validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test loss (no dropout): 0.0753162465898\n",
      "Test accuracy (no dropout): 0.9812\n"
     ]
    }
   ],
   "source": [
    "# Bez warstw Dropout\n",
    "\n",
    "score = model_no_dropout.evaluate(x_test, y_test, verbose=0)\n",
    "\n",
    "print('Test loss (no dropout): {}'.format(score[0]))\n",
    "print('Test accuracy (no dropout): {}'.format(score[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_7 (Dense)              (None, 2500)              1962500   \n",
      "_________________________________________________________________\n",
      "dense_8 (Dense)              (None, 2000)              5002000   \n",
      "_________________________________________________________________\n",
      "dense_9 (Dense)              (None, 1500)              3001500   \n",
      "_________________________________________________________________\n",
      "dense_10 (Dense)             (None, 1000)              1501000   \n",
      "_________________________________________________________________\n",
      "dense_11 (Dense)             (None, 500)               500500    \n",
      "_________________________________________________________________\n",
      "dense_12 (Dense)             (None, 10)                5010      \n",
      "=================================================================\n",
      "Total params: 11,972,510\n",
      "Trainable params: 11,972,510\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/10\n",
      "60000/60000 [==============================] - 145s 2ms/step - loss: 1.4242 - acc: 0.5348 - val_loss: 0.4426 - val_acc: 0.8638\n",
      "Epoch 2/10\n",
      "60000/60000 [==============================] - 140s 2ms/step - loss: 0.3245 - acc: 0.9074 - val_loss: 0.2231 - val_acc: 0.9360\n",
      "Epoch 3/10\n",
      "60000/60000 [==============================] - 137s 2ms/step - loss: 0.1993 - acc: 0.9420 - val_loss: 0.1694 - val_acc: 0.9485\n",
      "Epoch 4/10\n",
      "60000/60000 [==============================] - 136s 2ms/step - loss: 0.1471 - acc: 0.9571 - val_loss: 0.1986 - val_acc: 0.9381\n",
      "Epoch 5/10\n",
      "60000/60000 [==============================] - 132s 2ms/step - loss: 0.1189 - acc: 0.9650 - val_loss: 0.1208 - val_acc: 0.9658\n",
      "Epoch 6/10\n",
      "60000/60000 [==============================] - 131s 2ms/step - loss: 0.0983 - acc: 0.9711 - val_loss: 0.1260 - val_acc: 0.9637\n",
      "Epoch 7/10\n",
      "60000/60000 [==============================] - 129s 2ms/step - loss: 0.0818 - acc: 0.9753 - val_loss: 0.0984 - val_acc: 0.9727\n",
      "Epoch 8/10\n",
      "60000/60000 [==============================] - 129s 2ms/step - loss: 0.0710 - acc: 0.9784 - val_loss: 0.1406 - val_acc: 0.9597\n",
      "Epoch 9/10\n",
      "60000/60000 [==============================] - 129s 2ms/step - loss: 0.0611 - acc: 0.9811 - val_loss: 0.0987 - val_acc: 0.9727\n",
      "Epoch 10/10\n",
      "60000/60000 [==============================] - 136s 2ms/step - loss: 0.0533 - acc: 0.9837 - val_loss: 0.1070 - val_acc: 0.9718\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fdd95c86610>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Więcej warstw, inna funkcja aktywacji\n",
    "\n",
    "num_classes = 10\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "\n",
    "x_train = x_train.reshape(60000, 784)  # 784 = 28 * 28\n",
    "x_test = x_test.reshape(10000, 784)\n",
    "x_train = x_train.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes)\n",
    "\n",
    "model3 = Sequential()\n",
    "model3.add(Dense(2500, activation='tanh', input_shape=(784,)))\n",
    "model3.add(Dense(2000, activation='tanh'))\n",
    "model3.add(Dense(1500, activation='tanh'))\n",
    "model3.add(Dense(1000, activation='tanh'))\n",
    "model3.add(Dense(500, activation='tanh'))\n",
    "model3.add(Dense(num_classes, activation='softmax'))\n",
    "model3.summary()\n",
    "\n",
    "model3.compile(loss='categorical_crossentropy',\n",
    "               optimizer=RMSprop(),\n",
    "               metrics=['accuracy'])\n",
    "\n",
    "model3.fit(x_train, y_train,\n",
    "           batch_size=128,\n",
    "           epochs=10,\n",
    "           verbose=1,\n",
    "           validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test loss: 0.107020105763\n",
      "Test accuracy: 0.9718\n"
     ]
    }
   ],
   "source": [
    "# Więcej warstw, inna funkcja aktywacji\n",
    "\n",
    "score = model3.evaluate(x_test, y_test, verbose=0)\n",
    "\n",
    "print('Test loss: {}'.format(score[0]))\n",
    "print('Test accuracy: {}'.format(score[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Przykład: 4-pikselowy aparat fotograficzny"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkz\nODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2MBERISGBUYLxoaL2NCOEJjY2NjY2NjY2Nj\nY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY//AABEIAWgB4AMBIgACEQED\nEQH/xAAbAAEAAgMBAQAAAAAAAAAAAAAAAwUCBAYBB//EAE0QAAEDAgIGBgYGCAMHAwUAAAEAAgME\nEQUhEhMxQZHRFBVRUmFxBiIygZKxIzNCVKHBNDVTYnJzk6IXJPAWVWN0gtLxQ7LhB2SDlML/xAAZ\nAQEAAwEBAAAAAAAAAAAAAAAAAQIDBAX/xAAlEQEBAAICAwEBAAEFAQAAAAAAAQIRAxITIVExMlIE\nFCJBgWH/2gAMAwEAAhEDEQA/APn6IiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIi\nICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAi\nIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiIC\nKz6jqe/DxPJOo6rvw8TyWnjz+K9orEVn1HVd+HieSdR1Xfh4nknjz+HaKxFZ9R1Xfh4nknUdV34e\nJ5J48/h2isRWfUdV34eJ5J1HVd+HieSePP4dorEVn1HVd+HieSdR1Xfh4nknjz+HaKxFZ9R1Xfh4\nnknUdV34eJ5J48/h2isRWfUdV34eJ5J1HVd+HieSePP4dorEVn1HVd+HieSdR1Xfh4nknjz+HaKx\nFZ9R1Xfh4nknUdV34eJ5J48/h2isRWfUdV34eJ5J1HVd+HieSePP4dorEVn1HVd+HieSdR1Xfh4n\nknjz+HaKxFZ9R1Xfh4nknUdV34eJ5J48/h2isRWfUdV34eJ5J1HVd+HieSePP4dorEVn1HVd+Hie\nSdR1Xfh4nknjz+HaKxFZ9R1Xfh4nknUdV34eJ5J48/h2isRWfUdV34eJ5J1HVd+HieSePP4dorEV\nn1HVd+HieSdR1Xfh4nknjz+HaKxFZ9R1Xfh4nknUdV34eJ5J48/h2isRWfUdV34eJ5J1HVd+HieS\nePP4dorEVn1HVd+HieSdR1Xfh4nknjz+HaKxFZ9R1Xfh4nknUdV34eJ5J48/h2isRWfUdV34eJ5J\n1HVd+HieSePP4dorEVn1HVd+HieSdR1Xfh4nknjz+HaKxFZ9R1Xfh4nknUdV34eJ5J48/h2isRWf\nUdV34eJ5J1HVd+HieSePP4dorEVn1HVd+HieSdR1Xfh4nknjz+HaKxFZ9R1Xfh4nknUdV34eJ5J4\n8/h2isRWfUdV34eJ5J1HVd+HieSePP4dorEVn1HVd+HieSdR1Xfh4nknjz+HaKxFZ9R1Xfh4nknU\ndT34eJ5J48/h2joERF3sBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERARE\nQEREBERARE3gAEk7ABclQCI4OZbWRvZfZptIvxV76MYfS13SjVQiQsLQ25OW1Uy5JJtMx3dKJF2t\nXgWGMpZHNpWghuR0jzUvUGF/c2/Eeay/3E+L+NwqKwx+lho8ZdDTs1cepY7RBO27uSr1vjl2m1LN\nXQiIrIEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQ\nEREBERAREQERSshLm6byGR947/LtQRgFxAAJJ2AK4wqBsNPrXgCR5NydwBtZVhmDGlsALQci4+0e\nS2MOrmU8eoqB9He7XWvbwKw5pbj6Xw1tav0JwYi0PjIs7s8ln6LMmpn1rBC6RoeAHXAva605sUpo\n4zqXCV25rdnvK3PQ0k9Nc43c5zST4m65dXVrXc2ua2Wbokv+WcPV7wU2um+6v+NvNe1gvSSj91TK\nqXDekznOx1xfGWHUMyJB3u7FWK29Kv1+7/l4/m9VK7uH+Iwz/RERaqiIiAiIgIiICIiAiIgIiICI\niAiIgIiICIiAiIgIiILv0boKWtFSamISFpaG3Jy2q0rsIwyChnm6K0athdcE7h5qv9F5XQxVrxE5\n7W6JNiBuPat0YzFiWEVc0dNKYo2OD9PR7Lnf2Lh5Le1b4z0p5aClxKWOuw5mhRU/17XEtLrZnLfk\ntCtnop8QecPY5kIjb6pFvWu6/wCStGyvrR0rCojT4fD+kRXDdO2ZyG3LJVNXVUlXiEklDT6iMRsB\nbogZ3dnkp4re8RnPTBERdrEREQF60Bzmg7CQDxXi9b7bcr+sPmov4T9dyMBwz7oziVT4nT4RS1z6\nI01ppo26q17AkkdvbZdIJZbfoz/ibzXP4xW0zMV1E9G41csbRC+zTom7gDe+Wa87ddGlPPS0uDYZ\nPT4izTrZGudE9hLrZWGfmtFWOJTdBw+amxmE1FbIwmKXJ2iNgz81XLp4LvbPkERF0sxERAV96MYf\nS1oqTUwiQtLQ25OW1UK6T0Re9ravRiL827CB29qx5vWK2H6savA8NZSSubSsDg0kG5U3UOGfdGcS\npa2WXoc3+Xd7B+03mpddN91k+JvNce6204v0gpYaPF3RU7AyPVNdog5XuVoMY6R2ixpJVv6RN08Z\ndJP9ENU0BtwXHN3Yqt8xLdCMaEfYN/md67eK/wDCMc/1l9FDttLJ/aOaifI6R2k83KxRaKiIm8AA\nknYALkqQXrHyRkmOSRl9ug8tvwRzXMtrI3svs0mkKixueWKqaI5XtGhsa629Z5549d/q0l2vzPO4\nWNROQdxldzUsdbUNaWOmmew7jK6/uN1xXS6n7xL8ZTpdT94l+MrHy4fF+l+u0dEZnGSOR8riLEPc\nS8D37fcoFV4FNLIyUvkc4gixJvZXWubJlOCT327ff2rfC7x3GeU9oVLSxNmqGRvJDScyBcrx8Lmt\n02kPj7zfz7FgCQQQbEbwrfqFk7DIwbXk9YkNIsQ2zQbnippaCHWaWi4tD7aEYz9q3BVGtkJI0nuL\n9oFyXc16XzREF+tjJyBcCLrP3v3Vv/GVS0MqpWtFmh5AHvUSHM3KLSKiIikEREBERAREQEREBERA\nREQEREBERBHNPFA3SleGjxUUNfTTu0Y5QXdhyuufxGd09ZIXHJp0QOwLWBsbjauW89l9RpOP0+gY\nTS1s0k01PPq6ePRMrNIjTGZP4LcncK+nfVYXanooAekRW0dZbM5DblkqLDBVS4fHKXOZrG52l0dI\nC47VOylkjjdGwlrH+00TWB8xdReO53tKmZa9LoHp3+bws9HoIfr4baOnbM5DI5ZKmrqmjq8SkkoY\ndVEI2At0Q3O7uxI6aWKN0cbixjtrWzWB/FYR0JjvoMYL7fXHNWw4rjlvaMstzTFFL0aTsb8Y5p0a\nTsb8Y5ro2zRIpejSdjfjHNSU9E+WojbIBq7+tZwO7Zkoyykm0ybrXa17xpMjke3tawkJGQZG+Dhf\niukcdWGtY0XOQGwBVuL0wLY5vVEhdouN9EHsXPOa26rS4adyNi57GaqjjxU080OlVSxNEL9EHRJL\ngM92a57/ADv3mX/9g81DJSyyyiWRxfI3Y501yPfdV8F+p7xY4k9mH4dPTYvaprJI3GKW2loi1hmd\nmaqlnNSPqHB051hGQL5b/MrLo8n7vxjmtuLDpvdUyu0SKXo8nY34xzTo8nY34xzWu4oiTeAASTsA\nFyVMKaUmwDSf4xzVphVKIYS97fpXEgnssbWWfJydYtjjtTOa5ltYx7L7NJpF103ob7FX/E35FRkC\noa5r2tMRuLHeqJpFJJINfMC1xboxOcC62y5Cx73lnVfXW7fQK0Xo5h+4VMvm81dVTZF84bawbrCB\n81j0qs/bzD/87lHgqe8WnpV+vT/IZ83KpWLta95fJKXOItd1yeJKaB3vd+C6MJccdMsru7ZISBty\nWOrbvufMlBGwbGN4K6DWM749xV1g8UbKQVBtpyE5ncL5BU63sOrmQM1E4+jvdrrXt4FY80tx9L4a\n2tX6E4MRaHsI9a+zyXDY5QTOri1hjIYC0F0rQcie0rsJsUpo4zqXCV25rdnvK4bH7mtDnG7nNuT4\nklc0l62tdzbW6tqP+D/XZzTq2o/4P9dnNaiKiXSYFRTRxzB2qzI2StPyKtejSfuf1G81Rej31c3m\nFbru4v4jDP8AWwyGeN2kxzAf5jeak6OJfa1cT+0Pbon3XyWmi00qusKpmxQGQ6JkeTcg3yBtYFbk\nkbJYyyRoc05EFVGG17aZmpnuGXu14F7eBW3PitOxhMTta/cG/mVwZY5dvbeWaUz2aqWSO99Bxbft\nXiXJJc43c4kk+JRd2O5Jthf0REVgREQEREBERAREQEREBERARFJHCXjSJDGDa4/6zQYAEkAC5O4K\nWppn0zmte5hc5ulZpvbzQzNYNGAFva8+0eShUChxTDpRO6aFpex2ZA2grSgoqid4a2Jw8SLALq0W\nN4JbteZ2MqGEMo20gzcwXYe07x71ivQSCCDYhSzgPaJmiwdk4djv9ZraTXpT9QoiKQREQFsax1KI\nTH7YOscO3cBw+ajgYJJQHeyM3HwG1YyvMkjnneVWzfqk9LiLEaIjTdIWPO0PvceH/haGI1grHNZG\nDqmG9yPaPJaiLLHhku17nbGGcey5b2bwswQRcG4RYlpB0me8dq2UZIvGuDhl7x2L1SCyjY6R2iwX\nKyiiL7uJ0WDa47l7JKNHVxAtZv7XeagZGRsI0YTd++TlzU1BWshj1FRpCO5LXC+V9xstJFXPCZTV\nTMtLqbFKaOP6F2tdb1WtGXFUtySXON3OJJPiiKuHHME5ZbERFqqIiICIiAiJvAAJJ2AC5Kj8BRS0\nsEztKWJrza1yFM5rmW02PZfZpNIVFjc8sVU0RyvaNDY1xG9UyzxmO/1aS7WfV9J93ZwToFJ93ZwV\nHh9RNLiNNG+aRzHSta4F5zBIX1/qDC/ujfiPNY+bH4v0v1wMUEUAIijDL7bKRWHpBSw0eMuhp2aE\nepY7RuTnd3JV63wsyx3GdmqIiK6BERAREQEREBERAREQEREBERAREQF61rnuDWgknYApGQkt03nQ\nj7Tv8hvXrpgGlkI0Gnad7vMqB7oxw+3aR/dByHmeSjkkdIbuN+wbgsEQERFIIiIJaLCp6+odO1w1\nEGiZGl5FxmTs8FadDpcSeJ8Mj0aJnqztc4tJN73C0cMgxGSWaWkktTR6JmZp20hncW8luPnfVQmr\nweM01FBnURghunbM5DbkuDkt71vj+Oj6gwv7m34jzXK4/SwUeMOhpmauPUsdogk53dyC6ajx6Kto\npqqGnl1cV9K5aDkL9q5LFMTixXFHzwsexoiY2z7Xvd3NTxW94jP8ayIvWtLnBozJNgu5ilb9HTOd\n9qQ6I8ht/JQqWocDJot9lg0R/rzuolEBYyNc+NzGGznCwPisljIHFhEZs85NPil/CLaDDqXDIjSY\nmzWVdQfoHNc4gXyGe7NWGA+jkcInixKNk0gLS0h7jYG/JauskotGnxiLpNbMf8vJcO0NwzOzNWmB\nMxKlNQyuBnlOiQ4SA2Gfb715266dMMbwLDosJqpoacRyxxOc1zXHaAuXia22lPdo2ho2v8vDxXbY\nzPK3Bqtxpj9U64cQRs35rh5SZjd5JPb2Lo4N3bLNnLKZLCwa0ey0bAo1gHFps/3O7VmulmIiKQVH\njc8sVW0RyPaNAZA23lXi5/H/ANNb/LHzKy5v4Ww/UWH1E0uIU0b5pHMdK0EFxzFwvr/UGF/dG/Ee\na+N4blidJYX+mZl7wvt+um+6v+NvNcW6304rH6WGjxd0VOzVx6prtEEnO5VerT0lc52NuL4yw6lm\nRIO93YqtdvF/EYZ/oimFO+15LRt7X5fhtS8EexplPa7IcFptVGxjpDosaXHsAurfCoGQ02tfYSPJ\nuTuF7WVU+eR7dG+i3utFgtrDq5lPHqJx9He7XWvbwKx5pbj6Xw1tav0JwYi0PYRZ19i4L0ipZYqu\n5Y4xtuwPtlkTv7V2k2KU0cZ1LhK7c1uz3lcXi9VUU9eJI5XNe5t3W2G5O5c0l62tdzbQwzLFKQgX\nOuZl/wBQX27XTfdX/E3mvjuG1FNUYlS66n1cuuZZ8OQJuNrdnCy+1KiXC+kznOx1xewsOoZkSDvd\n2KsVt6Vfr93/AC8fzeqld3D/ABGGf6Iikp42yzsje7RDja/ju/Faqo0VoMMj1TtKR2m27Tb7Lg25\n3eNvcsuhU4ZK1geXscQXu2ewTsVO8T1qpRWgw2MyubaYNYS0uNgHG4FxxWtiFO2mfHG3cHAnts9w\nv+CmZS3RpqIiKyBERAREQEREBEUkbGuBdI8NaOzMnyCDFjHSO0WNJPYFL9HDttLJ/aOaxfN6uhG3\nQZvA2nzKiUDJ73SO0nkkrFEUgiIgIiII5p4oG6Urw0eKjhr6ad2jHKC7sOS57EZ3T1khcTZp0QOw\nLWBINwbFcuXPZfTWcfp3+E0tdNLNNTVGrp49EzM0iNMZk5b8luTuFfTPqsKtTUMAPSIraOstmchk\ncslR4YKqXD45S5zNY3O0ujpAZZ5qdlJJHG6NhLWP9prZrA+YuovHc72lJlr0uWu6daswq1PQQ/pE\nNtHTtmchkcslVYjJSz1+uoYRFA+JtgGhtzd19nuWEdLLFG6ONxYx21rZrA+66kpqQtY6HRY0Ozad\nMZO47+SnDiuN3tFy3NNZTQeo183dFm+Z/wBEoaaUGxDbj98c1LLTyNjZEA3L1neuNp9/ZZdNsZtR\nFL0aXsb8bealp6F8k8YkaNUXetZwN8r2UZZSTaZN1rNa940mRve3ta0kLBwc9pbGbPOQOyxXTuOg\nGsY0XOQGwBV+J0YlMUl2MeXaL3XsCLFc857bqtLhptMa6iLaXFR0utm/R5ANPV7hmcxnmrfAaWqp\nBMytqDPMdEl1yQBnlmuUmZK6zWucbfbdNdx998h4LHRqwSRUS3O/pB5qvhv1PeO0x79R138h/wAl\nwS2HsqntLXzyOaRYg1FwfxWHRpOxvxjmtuLDpvamV2iIBFiLhYZx+LfxC2OjSdjfjHNOjSdjfjHN\na+lEQNxcJvAAJJ2AC5KzNLK03YG23t0281bYRA1lNriBrJCbnsF9iz5OTrFscdqd7XR21jHsvs0m\nkLnse/TW/wAA+ZX0MgVDXNe1piOVjvXNz0NIZn9KdGdBxaCRdxAWPe8k61fr1u3L4Z+tKT+cz/3B\nfdV8wZBh8Tg6npoy9puHuzIPktl9VUvcXOqqi5/4rh+ajwZHki19JhEMaLpHOJ1LPVaPF29VXSC3\n6prY/EbeKicXPeXvc57iLaTnFx/FeLowx646rPK7r0kk3JuTvXiItECIm8AAknYALkqPwFFLSwTO\n0pYmvOy5Cmc1zLaxj2X2aTSFeejGH0tb0o1MIk0C0NuTltVM85Md/q0l25ttDTMcHNgYHA3BAzBW\n50io+8z/ANV3NdVi2C4dDhNXJHTNa9kLi1wJyNl8j6VUft5fjKx8uHxfpfrtDUGU3qG602DdMn1r\nDxXmpa/6mQE912R5FUmBSySxzax7n2ItpG6tVvhZZuM76vt65rmOLXAtI3ELxTRzSWDCNY3uuF7e\nXYpRS60XYx8R7Hj1eKtv6hqh77loc4l+0C50lk4yx+2JGaXeBF1b4VTCKAyOAMrybm97AG1gVuSR\nsljLJGhzXZEFc2XPd+o1mHpzmtkH23ZC23d2LEuLtpJ80ezVSyR3voOLb+S8XTLubZX0IiKQREQE\nREBQV0joaOWRhs5oyKnWrif6vm/h/NVy/KT9UnW1b+1/tC862rf2v9oWki4e+X10ajd62rf2v9oT\nrat/a/2haSJ3y+nWN3rat/a/2hW+EVMtTTvdM7SIdYG1ty5tdB6OPdHTyOYbHT2jyWnFllcvdVzk\n0t208rhfQIb2uyHErLVRt+smbfsYNL/4UTnOebucXHtJuvF1sU2nA32Yi/xe78glRUuqNDSaxoYL\nAMaAoUTQocUw6Rs7poWF7HG5A2grTgoqiZ4a2Jw8SLALqkWN4JbteZ3TKhhDKNtIM3MF2HtO8e9Y\nr0Eggg2I2KWcB4EzRk7Jw7HLaTXpT9QoiKRtsa2ctndsb9bz9/zWs9xe9z3bXG5WxC4Qwi40tcbO\nH7v/AJ+SSRNpHkPs+Ta1u4DtPJVEbImtaJJrhp9lo2u/+PFedJkbNHK2w1Z9Vo2W7FG97nuLnEkn\nevEs3PZvS6hxGiI0zIWPO0Pvcf68FoYhWirc1sYIiYb3ItpFaiLLHhku17nbBERbsxEREiIiAt2g\nrY4Y9RUaQjuS1wvlfcbLSRUzwmc1Uy6XU2KU0UdoXCR1vVa0ZcVS3LiXON3OJJPiiKuHFME5ZbeF\nrTtaD5heatnZbyNlki00qx0Oxzh77/NNF42Pv5hZImhj9J+6fwTSdvZfyKyRBjp9rXD3X+SusHij\nZSCc205L5ncL5BU63sOrmU7NRUD6O92ute3gVjzS3H0vhZtav0JwYi0PYR63Ys/RVslO6uY1hkAe\nGghwvlfbmtObFKaOM6lwldua0Ze8rc9DST01zjdznNJPjmuXV1a13NrbGHOdgtZpsLBqXX2G2S+O\nWw5ozdVSHwDWD819lxv9SV38h/yXw5VS6TApabVy6qlsAR7chJPCytekEewyNvkwH5qi9H/q5vMK\n3XdxT/hGGf6lNTMf/VcB2A2+SjLi43cST4rxFoq38Or20zNTPcMvdrgL28CtufFadjDqna1+4N/M\nqlRY3glu15nZC5JLnG7nEknxKIi2k16UERFIIiICIiAtXE/1dN/D+a2lq4l+r5vL81XL+aT9csi+\n6wUVKYI/8tD7I+wFn0Kl+7Q/AF5zpfB0X3CSipenwjo0NtW/7A7WrY6BR/dYP6YQfB1f4B+iyfx/\nkvrHQKP7rB/TC+e1bGx43ibWNDWioyAFgPVC24f7Uz/GKIi7WIiIgIiIJaLCp6+odO0t1EGiZGl5\nFxmTb3Kxkp6SuDqvD49Cgibaoa4kE2zuB4BamGQYjJLLLSSaNPHomZmnbSGdxbfktwl+Jx63B4TT\nUTDo1ERIbrO3IZHLJcPJb3rfGenlThlPW1LaqgiEdJTG07S4gutmbDfktGd1LW4o5mFxubFoNGiR\nvu65+S6Wrw2rfPB0KPo9OCBNFpAB7d+Q8MlVYsxtFWzNhpm0zDEwANABcSXXOXYB+KcVvYynppTz\nMjkIgOkQNEP7AOzmo2fTQln22XLfEbx+fFQL1rixwc02INwV26YPEUs7QbSsFmP3dh3hRIC9aA5z\nQdhIBXi8fIIWmVwJDPWNvBL+Ed2MAwu36Gzieap8Tp8Ipa59Eaa000bdVa9gSSO3tsof8RsJt+jV\nnwt/7lAPSXC8YqnPiopjUODY4Xva27HZ2N75ZkLzt106a9RS0uDYbPT4izSrZGl0L2EuAysM/NaS\nsMRm6Dh81NjUDqitkY4xS5P0BsGZzGar108F3tlyCIi6WYiIgK+9GMOpa4VJqoRIWFobcnLaqFbm\nGek9HgD5o6qKd5l0XDVgHt7SFjzfyth+ukxbBMOhwqrkjpWteyFzmuBORt5r5H0yp/byfEV9DqvT\nvDsRpZaKGnqhLUNMTC5rQLuyF8/Fc/8A4fY3/wDbf1DyXHuttRqYHLJKyXWPc6xFrm6tFr0+EVWC\nzy01ZoawhrxoOuLZ8lsLu4v4jHP9ERFoqIibwACSdgAuSo/AWTHyRkmOSRl9ug8tvwXjmujtrGPZ\nfZpNIV56MYfS1vSjUwiTQLdG98tqpnnjMd/q0l2pXzTSMLHzzOa4WIMriCOK0+r6T7uzgu5xbBcP\nhwmrljpmteyFxa4E3BsvkXS6n9vJ8RWPlw+L9L9dTFBFACIo2svtspFVYFLJLHNrHufYi2kbq1W+\nFlx3GdmqIilpYhPUNjJIBvs25C+St+IRIrd1BE+KMtY9g0QXA2DthOaxfhcbXubpSOuHEEDJtmg5\n8VTyRPWqpFZvw+GNsr3GUNiuCDYadrZjwzUjcKjDnXc82u5t94Fv9blPeGlQikqWhlTK1osA8gD3\nqNXQIiICIiAtbEv1fN5fmtla9ex0lFKxg0nEZAKuX5Sfr6jB9RH/AAj5LNcnH6bRMja04XW5AD7P\nNZf7cRf7rrf7ea4Ot+OjcdFJ+sIf5T/m1bC5B3pnGamOXquss1jmkervI8fBS/7cRf7qrf7eadb8\nNx1S+cVbQ7H8UBcGjpG037o7Fef7bxf7rrf7ea5wzmrr62qMT4mzy6bWv22sFpwy9lc76bHRpD7G\ni/8AgcD+G1ROY5hs5pafEWXikbUStFhI63YcxwXZ7Yo0U2ua724Y3eI9U/hkkxpixmobK19vX0iC\nPcmxE1rnuDWNc5x2NaLk+5ZzQTQW10EsQOwvYQCut9FqKKHDW1OiDLPcl3YL5BXE8MdRC6KVgexw\nsQVzZc9l9RpOP04PCaWumlmmp6jV08eiZWaRGmMycvJWbGdYBlThZdR0UZ+ljA0NYfIZHLJVUsba\nGpqIhM5xY9zAxryLgHLSt4LW1knrWlkaHbQ15A4BLxXO9ont19PpS430rc84wGm+rELS3Lfd1/yV\nfTTz6zTdUTlsY0iDK7PsG3tWEb3Tl0c0jnuf7LnuJsffuU4cVxy2i5biBEIINiLEIulmlgcM4nmz\nH7+w7io3tLHFrhYg2K8WrjtTbDgWvDZiQx2eZb2/kq5XrNpk36JMSpIn6LphfwBK9qJGS0Ez43Bz\nSw5jyXLtF7k7ArHA9OWuFMwaTJQQWk+C55zW3VaXDSsXX+hlVRR0c9PLDpVcsgEMmiDokiwz3Zr3\nqen/AGEHxt5rKPDWRODomRsINwWygfmq+C/TutcSczD8OnpsYHSqySNxiltp6ItYZnMZqrWc1I6d\nwdM4SkZAvmDvmVl0eTtj/qN5rbiw6b2rldokUvR5O2P+o3mnR5O2P+o3mtdqIk3gAEk7ABclS9Hf\n2x/1G81aYPAxlNrsi+Qn1hnlfIBZ8nJ0i2OO1O5rmW1jHsvs0mkLnse/TWfyx8yvoZaKhrmva0xG\n4sd65ubDoJZ3tl1LnMcWgveL2Hgsu95Z1q+uvty+GfrSk/nM/wDcF91XzFuGUTHB14WuBuC0G4W5\nrrba2sd5PcP/AOlXwU7xuelX69d/IZ83KpUzpIS8vc2aVxFryS3TXMHswRjzufzXRhLjjpnld3aF\neta53stJ8gpekyD2dBvkwD8li6omdtlf8Suh6KaY/wDpOA7SLK0wqBkNNrX21jybm+wXtZU5JO03\nW7h1cynZqJx9He7XWva+4rHmluK+GtrV+hODEWh7CLOvsWjhOLMwKSqjlpqmoDnDRMIDjYXGYvdS\nzYpTRxnUuEr9zWjL3lcNjxd05ribuLbk+NyuaS9bWu5t3df6Y0lZSy0TaKtjkqWmJhkjDRd2Qvn4\nrm/8Pca7aX+oeSpcJrKkYnSNFRKGmZgI0za1wvt6ol8qgwipwSolpavQ1ha1/qG4sbj8lsK69J5G\nNx1zXQtd9AzO5B2uVXenO1krfJwP5Lu4v4jDP9QoCQbjIhTaEB2TEfxM5L1tK6Q2ikjeewGx/Fab\nVR66XTFnyF52WJJUoqqiCN0bmuYH5aT2WPZtKssKphDAZHAa15NztsAbWW5JGyVhZI0Oa7Igrly5\nvfqNZh6c2ZHuFi9xytmdy91slrax1uy6xezVyyR3voOLb+S8XTLLNsr6Cbm5REVgREQEREBERAUk\ncugC1zQ9h2tP+slGiCYwh4LoCXAbWn2hzUK9BLSCCQRsIU2sZN9b6r++Bt8x+agQIs5InRkaQyOw\njYVgpBERAREQX/o/jkVHD0SrJbGDeOS1wL7irWs9I6CCEmCUVElvVazt8TuXFosLwS3a8zuiUuml\nfK930r3FxcO0m5WLXZ6LsnfNZLJkQmkaw5XO3s8VtrU9KfqR30dM1v2pDpHy3fmoUllJlJd7Oxp8\nNyJBPL9LGJh7Wx/n2+9QKSGQRv8AWF2EWcO0KOVpDy1p9Tc7vDwQYlxJ0Wbd53Ba9bRippXRg+ve\n4ce1bIAAsBYL1LNz2b05SWmqGO0HxPHk1WeD4fJFMKmUFhb7A3+auEWOPDJdr3O2J7tqPasyXvbA\n7z7CoXNcxxa4EEbQV4pmyNe0MmvYZNeNreYW34ohRZyROjNjmDmCNhWCkF44houSvHO0ctpOwI1u\nek7N3yUDzRL835Dc3mrCgrY4Y9RUaQjuS1wvlfcbLSRVzwmU1UzLS6mxSmjj+hcJXW9VrdnvKpHD\nTuZLOLjc+a9RVw4pgnLLbDRI9lx8jmvdMj2mkeIzCyRaKvAQ4XBBHgvViWNJvax7RkV5Z7dhDh45\nFBmix1g2Ou0+KyUgiJvAAJJ2AC5KjegUUtLBO4Olia8gWuVM9r47axj2X2aTSFR41PLFVNbHI9o0\nNgNt5VMs5Md/q0l2s20NKxwc2FrXA3BG0FbvSan71Uf1nc1y2H1E0uIU0b5pHMdK1rgXHMEhfXuo\nML+6M4nmsfLh8X6X64Zxc95e97nuItpOcXG3vWUcL5LlrchtJyA96sschpaDFnRU9OB9Ex1i4luZ\ndu9yrZJXyW03E22DcPct8Mu03Gdmqz0YY/acZXdjchxXjp3ubotsxndbl/5USK2kN/Da9tMzUz3D\nL3a4C9vArbnxWnYwmJ2tfuDdnvKpUWN4JbteZ2QuSS5xu5xJJ8SiItpNelBERSCIiAiIgIiICIiA\niIgkjldGC3JzDtadhWWqbLnATffGdvu7VCgyNwoBFNrWy5T3vukG339qxkidGAcnMOxw2FNiNERS\nCIiCWiwqevqHTtcNRBomRpeRcZk5eQVlPTUlZBJX4ZHq6eFujKHEgkbXWHll71q4PSYnUVD3UT7Q\nNLNazTtpDP8AJW+N0VZDHp0cQgoYmF08TXAB42nLflkuHkt71vjPSvlw+kxOVlbh8YbRQfXscSC6\n2Zy35KrrZ6I18jqAPFNoN9V243OxWz6p1To1GFQOhooz9PAC1pktmch4KpnqKarxB81NAIWiNo0A\n0AA+t2fNOK3vEZfjADWAE+zuHb5raZ9NDq/tsF2+I3j8+K1S0tN2e9vas45LOD2GxaeBXaxEU07Q\nbSsFmP3dh3hQoCxka58bmMNnOFgb71ksZA5zCIzZ5yafFL+EW0GH02FxGkxNmsq6g/Qua4uAvkM9\n2asMB9HY4RPFiUbJpAWlp0ybA35LV1klDo0+MRGprZj/AJeQEO0NwzOzPNeR+kB9GpnwY4Zpp5QH\nsdGQ/wBXMeG+687ddOlziuFYfT4PVObTABkbnjRcQQQNy4+Vls4Tpt73d8D4qwxX06w+uoZqSCOo\naZYy3ScwbTu2/iq+F5h9i1thB2HzXRwbu2WbBrQ3xJ2kr1TOjbI0vh3ZuZvHl2hQrpjMREUgr70Y\nw+lrRUmphEhaWhtyctqoVu4X6T0WAvmjq45nGXRcNWAe3xWPN/K2H66irwPDWUkrm0rA4NJBuVN1\nDhn3RnErnan/AOoOFS08kbYaq7mkC7RzUv8AiLhH7Kq+Ac1x7rbTS9IKWGjxd0VOwMj1TXaIOV7l\nVymr8Zp8bxF9TStkaxsbWeuLG+fNQrt4v4jHP9FjoAewS3y2cFki1VYXc32m38Wq7weKNlIJzbTk\nvmdwvkFTrew6uZBHqJx9He7XWvbwKw5pbj6Xws2tX6E4MRaHsIs6+xc2fRSqxitkMUsYig9Q3ddx\nzNldzYpTRxnUkSP3NaMveVuehpJ6a5xu5zmknxN1zSXVrXc254eiuIYURVR09M2GnIke9z9OQtGZ\ntcWBt2cVcf4jYR93rPgb/wBy6HG/1JXfyH/JfDlRLuK7GafHMSkq6VkjGCNjLSAA3BcdxPaolUej\n/wBXN5hW67uH+Iwz/REW3DSskia4l1ynLy48U3ko1EW90KPvOToUfecsP97xG40UW90KPvOToUfe\ncn+94jcaKLe6FH3nKKpp2xMBaSbm2atj/q+PPKYw21kRF1JEREBERAREQEREBERAWccroydE5Ha0\n5grBEE+rZNnF6r/2ZO3yP5KFzS02cCD2FeKWaolqAwSu0tAaLTbO3YoEbQ57gxjXOcdjWi5PuWc0\nE8FtfBLEDsL2FoPFdb6LUUcOGtqbAyz3Jd2C+QVxPDHUQuilYHscLEEbVzZc9l9RpOP0530M9is/\niZ8irXF6hrcPqmhunoxOL94aLb+S4yrD6OaWkp5pGaEjtJ7XEEgGzRl4fNQGSUxuYZpix19JpkdY\n323zUXiud7RPaY+lxDGKodMwx2qo4v0lhGiZLZnIZbMlVV1TR1WJSSUMWqiEbAW6Ibnd3YoYwYo3\nRxvexjvaa15AKxjiZHfQaBfarYcNxy2jLOWaZrFzbm7cndqyRdLNJTSNOlDJ6of27juKwe0scWuF\niDYhYuaHDP3HsWvjFU5mH6V7TD1C7vDt81TK9ZtMm/T2Stpo3aLpmg+alvrovoXi7smuB2FciPWJ\nJPmVZYJM/pWobcteNl7LCc1t1Wlw07wu6taIMXd0mrm+oltpaG4ZnMZ9i4z0tgroMQhGISmSR0QL\nSXEkC5Vy+kmle2Sd5e9vsl019HyzWNRQGqeH1DWzOAsDJIHG3vKr4L9T3jjo/rGeYXYLAYRCDcU8\nPxDmtjo0n7vxjmtuLDpvdUyu0bXFrg5pII2EKb1ajsZL2bA7kVj0eT934xzTo8nY34xzWvpRGQWk\ngggjaCo5JWRN0pHBo8VviN0oDJtEEZNk0xl555hcfjEr3V8kbiCI3aIsbhZ58nWLY47XsVXTzO0Y\n5Wk9ipse/TW/wD5lV4OrIIJDhn5Lp6OlbiNLHO+BsrrWJ2lZd7yzqvrrduVRda/CoGe3SBvm2yw6\nBSfd2cFHgv08kaPo/wDVzeYVuo4YIoARFG1l9tlIujDHrjpnld3YiIroERN4AuSdgAuSo/AWTJJI\nyTHLJHfboPLb8F45ro7axj2X2aTSLq89GMPpa7pRqoRIWFobcnLaqZ54zHf6tJdqV800jCx9RO5r\nhYgyuII4rU6BSfd2cF3tXgWGspZHNpGAhuRueam6gwv7mzieax8uH+K/S/XARQRQAiKNrL7bKRWH\npBSw0eMuhp2COPUsdog7yXclXrfCy47jPKaorjDWtfqmvF22N/xVOt+CojZC0F1iFy/6zG3Can/a\ntWraSNwYQX2fYD92+8oKVh9Voc43bfZvBKr2VTXODGPcXOyDWgklSSyOprGYSRX2FzSLrzet/wAT\nX/xt9FjPsmSx0rEjZbtWXRYvpA3TcRpNHmCM/wAVWmsYb3lOe3anS4/2h/FT0y/xp/4spaVkUD3C\n5Nsr7rEc1UV31TfNSmsjO2UnzutermZIwBjrm604cMvLjep/21ERF7SRERAREQEREBERAREQEREB\nERBf+j+ORUcPRKwlsYJLJLXAvuKta30joIISYJRUSfZazt8TuXFosMuCW7Xmd0ye90kj5JDd73Fz\nj4k3WKItpNelBERSCIiAoK2nFVTujOR2g+KnRRZuaJ6ctLQ1THaBiJ8htVphOHvgdrphZ1rAK1RY\n48Ml2vc7YIiLdQREQEREBUuJ4dIZjPA3SDto7FdIqZ4TOaqZdOXgw+pmksYy0byV0cUDYoWRj7Is\nCNqlRVw4pgnLK0ZPLDlpuA7Wm3FS9KlIzfpj94B3zUSw0CM2ZeG5aaVbOvv7UMR/6bfJNZCdsFv4\nXkfO612vBNjk7sKyQTf5c75W8Hck1cR9mcD+NpHyuoUQTdHv7MsTv+q3zVnhUDIabXP0dY8m5vsF\n9iplvYfXR08eoqB9He7XWvbwKx5pbj6Xws2tX6E4MRaHsI9a+xaOE4vFgUtVFNDLKHOGiWOZsFxn\ndwUs2KU0cf0J1j9zWjL3lcNj5JrQ5xu5zbk+JJXNJdWtdzb6FUemFLLA+NtJUAuFs3xf96l/2zo/\nulT8UX/evkiKiX0DEqsYxib6qCJzGCJjLSPZe4LjuJ7VB0aXsb8beaovR76ubzCt13cX8Rhn+puj\nS9jfjbzQUkziA1rSTuDxzRsGiA6Y6tp2D7R8gjp7Athbq2naftHzKv7VdT6N0LKahEzmjXyk6R22\nANrK0mijnidFKwPY4WIO9czgONRUcPRaslsYJLJAL2vuKs6v0hoYIiYZRPJb1Ws2e87lw5Y5dvbe\nWacpUw9Hq5oL3Eby0HwByUS9e90kj5Hm73uLnHxK8XdjuSbYX9ERFYEREBERAREQEREBERAREQER\nEBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERB45ocLELH1mbbub27ws0UDwEEXBuF6sS\nzO7Ton5oH52cLH8CgyRF60Fzg1rS5x2NaLk+5SPFFLSwTO0pYmvIFrlbEsM0FtfDJFfYXsIuqHGp\n5Yqtojke0aGwG28rPLPGY7/VpLtZdX0n3dnBOr6T7uzgqCCSuqZNCGSV7tuTjkO1bXSW0ft1Dqqb\nutedW3zP2vcsfNh8X6X6vqWiYxrjExsTPtO2BTayOL6kaTu+4fIKpwmsnrBM+eQusQGjc0eAVgt8\nL2m2dmq9c4ucXOJJO0leIiugREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERARE\nQEREBERAREQEREBERAREQEREBERAXhAIsRcL1EGFnM2es3s3rr/Rmmihw1tUQBLPclx3C+QXJq8w\nLGYaWDolZlGCSx9rgX3FYc0tx9L4WbdHJq6oOgcxssRFn32eXmuEm9Ep8SxGo1cjHR0x0NDTs52Z\ntc7l1lV6QUFPCejyCeS3qsYMvedy1/RB7pHV8khu972ucfE3XLq6ta7m3LVnodjnR3NaKSGnbnq4\n5Dn55ZnzUH+HuN9tL/UPJfTa39Dl/hU6ql8rgwiqwWolpazQ1ha140HXFjcfkthW3pV+v3f8vH83\nKpXdw/xGGf6IiLVUREQEREBERAREQEREBFz/AF5VdyHgeadeVXch4HmsfNgv0roEXP8AXlV3IuB5\nrzryq7kXA808+B0roUXPdeVXci4HmnXlV3IuB5p58DpXQoue68qu5FwPNOvKruRcDzTz4HSuhRc9\n15VdyLgeadeVXci4HmnnwOldCi57ryq7kXA8068qu5FwPNPPgdK6FFz3XlV3IuB5p15VdyLgeaef\nA6V0KLnuvKruRcDzTryq7kXA808+B0roUXPdeVXci4HmnXlV3IuB5p58DpXQoue68qu5FwPNOvKr\nuRcDzTz4HSuhRc/15VdyLgeadeVXch4HmnnwOldAi5/ryq7kPA8068qu5DwPNPPgdK6BFz/XlV3I\neB5p15VdyHgeaefA6V0CLn+vKruQ8DzTryq7kPA808+B0roEXP8AXlV3IeB5p15VdyHgeaefA6V0\nCLn+vKruQ8DzTryq7kPA808+B0roEXP9eVXch4HmnXlV3IeB5p58DpXQIuf68qu5DwPNOvKruQ8D\nzTz4HSugRc/15VdyHgeadeVXch4HmnnwOldAi5/ryq7kPA8068qu5DwPNPPgdK6BFz/XlV3IeB5p\n15VdyHgeaefA6V0CyY+SMkxyyR326Dy2/Bc715VdyHgeadeVXch4HmovNhf06ZOjM87gQamoIO4z\nO5r3pFR96qP6zua5vryq7kPA8068qu5DwPNR5OP4dcnQuLnvL3uc9xFtJ7i42968XP8AXlV3IeB5\np15VdyHgeamc2EOldAi5/ryq7kPA8068qu5DwPNT58DpXQIuf68qu5DwPNOvKruQ8DzTz4HSugRc\n/wBeVXch4HmnXlV3IeB5p58DpXQIuf68qu5DwPNOvKruQ8DzTz4HSugRc/15VdyHgeadeVXch4Hm\nnnwOldAi5/ryq7kPA8068qu5DwPNPPgdKrERFxNhERAREQEREBERAREQEREBERAREQEREBERAREQ\nEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBER\nAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQER\nEBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAR\nEQEREBERAREQEREBERAREQEREBERAREQf//Z\n",
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"https://www.youtube.com/embed/ILsA4nyG7I0\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.YouTubeVideo at 0x7f4d3ff03110>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "YouTubeVideo('ILsA4nyG7I0', width=800, height=600)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def generate_example(description):\n",
    "    variant = random.choice([1, -1])\n",
    "    if description == 's':  # solid\n",
    "        return (np.array([[ 1.0,  1.0], [ 1.0,  1.0]]) if variant == 1 else\n",
    "                np.array([[-1.0, -1.0], [-1.0, -1.0]]))\n",
    "    elif description == 'v':  # vertical\n",
    "        return (np.array([[ 1.0, -1.0], [ 1.0, -1.0]]) if variant == 1 else\n",
    "                np.array([[-1.0,  1.0], [-1.0,  1.0]]))\n",
    "    elif description == 'd':  # diagonal\n",
    "        return (np.array([[ 1.0, -1.0], [-1.0,  1.0]]) if variant == 1 else\n",
    "                np.array([[-1.0,  1.0], [ 1.0, -1.0]]))\n",
    "    elif description == 'h':  # horizontal\n",
    "        return (np.array([[ 1.0,  1.0], [-1.0, -1.0]]) if variant == 1 else\n",
    "                np.array([[-1.0, -1.0], [ 1.0,  1.0]]))\n",
    "    else:\n",
    "        return np.array([[random.uniform(-1, 1), random.uniform(-1, 1)],\n",
    "                         [random.uniform(-1, 1), random.uniform(-1, 1)]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "num_classes = 4\n",
    "\n",
    "trainset_size = 4000\n",
    "testset_size = 1000\n",
    "\n",
    "y4_train = np.array([random.choice(['s', 'v', 'd', 'h']) for i in range(trainset_size)])\n",
    "x4_train = np.array([generate_example(desc) for desc in y4_train])\n",
    "\n",
    "y4_test = np.array([random.choice(['s', 'v', 'd', 'h']) for i in range(testset_size)])\n",
    "x4_test = np.array([generate_example(desc) for desc in y4_test])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA60AAACQCAYAAADjqY0xAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAADlVJREFUeJzt3V+opPddx/HP192mhUi1bZY2bLJN\nxKQhioI7if9AQkVIi2wEe5F6YSuVBSEIXhkQLPRKvRHEYgk1JPWirXihqyiltUi9aE3OSmPTlk3X\nYpvdRtptpJKq3ezy8+LMs3O6Obs7J+eZmd+Zeb3gYc/MPJnnmZN3fs9+d2ZPqrUWAAAA6NEPrPoE\nAAAA4FoMrQAAAHTL0AoAAEC3DK0AAAB0y9AKAABAtwytAAAAdGtfQ2tVvbGqPllVX5n++oZr7He5\nqj4/3U7t55j0SQsMtECiA2a0wEALJDrg1an9/H9aq+qPkrzYWvuDqno0yRtaa7+7y34vtdZ+cB/n\nSee0wEALJDpgRgsMtECiA16d/Q6tZ5I80Fp7oapuTfJPrbW37bKf6NacFhhogUQHzGiBgRZIdMCr\ns9+/0/rm1toL06//M8mbr7Hf66pqq6o+V1W/ss9j0ictMNACiQ6Y0QIDLZDogFfh8I12qKpPJXnL\nLg/93s4brbVWVdd62/atrbXzVfUjST5dVV9orf37Lsc6meTk9ObxG50bfamqb7XWjmgBLezN8eMH\n82U999xzefnll19x/9GjR3Po0KFMJpN2+vTpC621IzrYbItaE26++ebj99xzzwLPnL260brg+kCS\nTP/duz7M6aD+PmEew+8Tbrhja+1Vb0nOJLl1+vWtSc7M8c88keRdc+zXbAdue0YLNi3sfVtHd999\nd/vGN77RkmzpwJYFrQnHjx9feMuM5+67715YCx00btv75vow57bOkmy1OebO/X48+FSS90y/fk+S\nv7l6h6p6Q1W9dvr1LUl+PsmX9nlc+vSm6a9aQAsb7sSJE3nyySeHmzrAmkBOnDiRaIHvpwPmM89k\ne60t2wvPPyb5SpJPJXnj9P5Jkg9Pv/65JF9I8sz01/fN+dwr/1MN2563/9aCTQt739bRhQsX2tvf\n/vaW5P90YMuC1gTvtB4sFy5cWFgLHTRu2/vm+jDnts4y5zut+/rpwYt0nc+306/TrbXJ2E+qhQNJ\nC3vQ6zo8hqoavYV17WDNLWRNmEwmbWtra+ynZYEWsSZMn9e6cPC4PszJ7xP2/9ODAQAAYGEMrQAA\nAHTL0AoAAEC3DK0AAAB0y9AKAABAtwytAAAAdMvQCgAAQLcMrQAAAHTL0AoAAEC3DK0AAAB0y9AK\nAABAtwytAAAAdMvQCgAAQLcMrQAAAHTL0AoAAEC3DK0AAAB0y9AKAABAtwytAAAAdGuUobWqHqyq\nM1V1tqoe3eXx11bVx6eP/0tV3THGcemPFhhoganX64DEmsCMFphyfWBu+x5aq+pQkg8meUeSe5O8\nu6ruvWq39yX5r9bajyb54yR/uN/j0i0tMNDChrt8+XKSHIsO2GZNYKAFEtcH9mCMd1rvT3K2tfbV\n1trFJB9L8tBV+zyU5Mnp13+V5BerqkY4Nn25OVpgmxbIU089lSTf0wGxJjCjBQauD8xtjKH1aJLn\nd9w+N71v131aa5eSfCfJm0Y4Nn25KVpgmxbI+fPnk+Tijrt0sLmsCQy0wMD1gbkdXvUJ7FRVJ5Oc\nXPV5sHpaYKAFEh0ws7OFY8eOrfhsWCXrAokONsUY77SeT3L7jtu3Te/bdZ+qOpzkh5J8++onaq09\n1lqbtNYmI5wXy3cxWmCbFsjRo0eT7XdVBjrYXAtZE44cObKg02WBXB8YuD4wtzGG1qeT3FVVd1bV\nTUkeTnLqqn1OJXnP9Ot3Jfl0a62NcGz68t1ogW1aIPfdd1+SvE4HxJrAjBYYuD4wt31/PLi1dqmq\nHknyiSSHkjzeWvtiVX0gyVZr7VSSP0/yF1V1NsmL2Q6T9aQFBlrYcIcPH06Sr0cHbLMmMNACiesD\ne1C9/oFFVfV5YlzP6UV8NEMLB5IW9qDXdXgMVTV6C+vawZpbyJowmUza1tbW2E/LAi1iTZg+r3Xh\n4HF9mJPfJ4zz8WAAAABYCEMrAAAA3TK0AgAA0C1DKwAAAN0ytAIAANAtQysAAADdMrQCAADQLUMr\nAAAA3TK0AgAA0C1DKwAAAN0ytAIAANAtQysAAADdMrQCAADQLUMrAAAA3TK0AgAA0C1DKwAAAN0y\ntAIAANAtQysAAADdMrQCAADQrVGG1qp6sKrOVNXZqnp0l8ffW1XfqqrPT7ffHOO49EcLDLTA1Ot1\nQGJN4AprAgMtMLfD+32CqjqU5INJfinJuSRPV9Wp1tqXrtr14621R/Z7PLqnBQZa2HCXL19OkmNJ\n7o0OsCZsPGsCV9ECcxvjndb7k5xtrX21tXYxyceSPDTC83Lw3BwtsE0L5KmnnkqS7+mAWBOINYFX\n0AJz2/c7rUmOJnl+x+1zSX56l/1+tap+IclzSX6ntfb81TtU1ckkJ0c4J1bjpmiBbQtp4dixY/na\n1762gNNdrapa9Sks0sUdX1sTNtfCrg9r/t/POhplTUisC2vA9YG5LesHMf1tkjtaaz+R5JNJntxt\np9baY621SWttsqTzYvm0wGDPLRw5cmSpJ8hSWBMYaIFkzg4SLWwAawJXjDG0nk9y+47bt03vu6K1\n9u3W2vemNz+c5PgIx6U/F6MFtmmBwU07vtbB5rImMLAmMNACcxtjaH06yV1VdWdV3ZTk4SSndu5Q\nVbfuuHkiyZdHOC79+W60wDYtMHidDog1gRlrAgMtMLd9/53W1tqlqnokySeSHEryeGvti1X1gSRb\nrbVTSX67qk4kuZTkxSTv3e9x6ZYWGGiBJPl6dMA2awKJNYEZLTC3aq2t+hx2VVV9nhjXc3oRf59A\nCwfSQlqYTCZta2tr7KdduTX/QTKjt2BNOJBcHxhogYHrw5x6ndfGUFVzdbCsH8QEAAAAe2ZoBQAA\noFuGVgAAALplaAUAAKBbhlYAAAC6ZWgFAACgW4ZWAAAAumVoBQAAoFuGVgAAALplaAUAAKBbhlYA\nAAC6ZWgFAACgW4ZWAAAAumVoBQAAoFuGVgAAALplaAUAAKBbhlYAAAC6ZWgFAACgW6MMrVX1eFV9\ns6qevcbjVVV/UlVnq+rfquqnxjgu3blDB0xpgYEWSHTAjBYYaIG5jfVO6xNJHrzO4+9Ictd0O5nk\nz0Y6Ln25EB2wTQsMtECiA2a0wEALzG2UobW19pkkL15nl4eSfKRt+1ySH66qW8c4Nl15KTpgmxYY\naIFEB8xogYEWmNuy/k7r0STP77h9bnofm0UHDLTAQAskOmBGCwy0wBWHV30CO1XVyWy//c+G0wKD\nnS0cO3ZsxWfDqlgTGGiBgRZIdLAplvVO6/kkt++4fdv0vu/TWnustTZprU2WdF4s11wdJFrYAK+q\nhSNHjizl5Fgq1wcS1wdmtMDA9YErljW0nkry69OfAvYzSb7TWnthScemHzpgoAUGWiDRATNaYKAF\nrhjl48FV9dEkDyS5parOJXl/ktckSWvtQ0n+Psk7k5xN8j9JfmOM49KdO5N8NjpAC8xogUQHzGiB\ngRaYW7XWVn0Ou6qqPk+M6zm9iI9maOFAWkgLk8mkbW1tjf20K1dVqz6FRRq9BWvCgeT6wEALDFwf\n5tTrvDaGqpqrg2V9PBgAAAD2zNAKAABAtwytAAAAdMvQCgAAQLcMrQAAAHTL0AoAAEC3DK0AAAB0\ny9AKAABAtwytAAAAdMvQCgAAQLcMrQAAAHTL0AoAAEC3DK0AAAB0y9AKAABAtwytAAAAdMvQCgAA\nQLcMrQAAAHTL0AoAAEC3Rhlaq+rxqvpmVT17jccfqKrvVNXnp9vvj3FcunOHDpjSAgMtkOiAGS0w\n+EkdMK/DIz3PE0n+NMlHrrPPP7fWfnmk49GnC0l+LTpAC8xogUQHzGiBwVdu8LgOuGKUd1pba59J\n8uIYz8WB9lJ0wDYtMNACiQ6Y0QKDS6s+AQ6OZf6d1p+tqmeq6h+q6seWeFz6ogMGWmCgBRIdMKMF\nEh2ww1gfD76Rf03y1tbaS1X1ziR/neSuq3eqqpNJTk5vvpTkzJLOL0luyfZHVtbNMl/XW2/w+Fwd\nJCttYV07SNakharSwv4s+3WN0oLrw0KsxZoQ14cxaGFv1rWFZb+uO5O8cI3HDkIHyZK+Z1W16ENc\nrac1IUlSrbVRjlZVdyT5u9baj8+x738kmbTWuvkPvqq2WmuTVZ/H2Jb9unTQLy3szbq2sIrXpYU+\nWRP2Zl07SLSwV+vagg72TgvLs5SPB1fVW2r6RwRVdf/0uN9exrHphw4YaIGBFkh0wIwWSHTAK43y\n8eCq+miSB5LcUlXnkrw/yWuSpLX2oSTvSvJbVXUpyf8mebiN9RYv3dABAy0w0AKJDpjRAokO2LvR\nPh580FXVydbaY6s+j7Gt6+talHX+fq3za1uEdf1+revrWqR1/Z6t6+talHX+fq3za1uEdf1+revr\nWqR1/Z71+LoMrQAAAHRrmf/LGwAAANiTjR9aq+rBqjpTVWer6tFVn89YqurxqvpmVT276nM5KLRA\nogNmtMBACyQ6YEYLy7fRQ2tVHUrywSTvSHJvkndX1b2rPavRPJHkwVWfxEGhBRIdMKMFBlog0QEz\nWliNjR5ak9yf5Gxr7auttYtJPpbkoRWf0yhaa59J8uKqz+MA0QKJDpjRAgMtkOiAGS2swKYPrUeT\nPL/j9rnpfWweLZDogBktMNACiQ6Y0cIKbPrQCgAAQMc2fWg9n+T2Hbdvm97H5tECiQ6Y0QIDLZDo\ngBktrMCmD61PJ7mrqu6sqpuSPJzk1IrPidXQAokOmNECAy2Q6IAZLazARg+trbVLSR5J8okkX07y\nl621L672rMZRVR9N8tkkb6uqc1X1vlWfU8+0QKIDZrTAQAskOmBGC6tRrbVVnwMAAADsaqPfaQUA\nAKBvhlYAAAC6ZWgFAACgW4ZWAAAAumVoBQAAoFuGVgAAALplaAUAAKBbhlYAAAC69f+2tfgV5TrN\nDQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f4d3ffc2ed0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      s          s          d          s          h          s          v\n"
     ]
    }
   ],
   "source": [
    "draw_examples(x4_train[:7], captions=y4_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "x4_train = x4_train.reshape(trainset_size, 4)\n",
    "x4_test = x4_test.reshape(testset_size, 4)\n",
    "x4_train = x4_train.astype('float32')\n",
    "x4_test = x4_test.astype('float32')\n",
    "\n",
    "y4_train = np.array([{'s': 0, 'v': 1, 'd': 2, 'h': 3}[desc] for desc in y4_train])\n",
    "y4_test = np.array([{'s': 0, 'v': 1, 'd': 2, 'h': 3}[desc] for desc in y4_test])\n",
    "\n",
    "y4_train = keras.utils.to_categorical(y4_train, num_classes)\n",
    "y4_test = keras.utils.to_categorical(y4_test, num_classes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_16 (Dense)             (None, 4)                 20        \n",
      "_________________________________________________________________\n",
      "dense_17 (Dense)             (None, 4)                 20        \n",
      "_________________________________________________________________\n",
      "dense_18 (Dense)             (None, 8)                 40        \n",
      "_________________________________________________________________\n",
      "dense_19 (Dense)             (None, 4)                 36        \n",
      "=================================================================\n",
      "Total params: 116\n",
      "Trainable params: 116\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model4 = Sequential()\n",
    "model4.add(Dense(4, activation='tanh', input_shape=(4,)))\n",
    "model4.add(Dense(4, activation='tanh'))\n",
    "model4.add(Dense(8, activation='relu'))\n",
    "model4.add(Dense(num_classes, activation='softmax'))\n",
    "model4.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "model4.layers[0].set_weights(\n",
    "    [np.array([[ 1.0,  0.0,  1.0,  0.0],\n",
    "               [ 0.0,  1.0,  0.0,  1.0],\n",
    "               [ 1.0,  0.0, -1.0,  0.0],\n",
    "               [ 0.0,  1.0,  0.0, -1.0]],\n",
    "              dtype=np.float32), np.array([0., 0., 0., 0.], dtype=np.float32)])\n",
    "model4.layers[1].set_weights(\n",
    "    [np.array([[ 1.0, -1.0,  0.0,  0.0],\n",
    "               [ 1.0,  1.0,  0.0,  0.0],\n",
    "               [ 0.0,  0.0,  1.0, -1.0],\n",
    "               [ 0.0,  0.0, -1.0, -1.0]],\n",
    "              dtype=np.float32), np.array([0., 0., 0., 0.], dtype=np.float32)])\n",
    "model4.layers[2].set_weights(\n",
    "    [np.array([[ 1.0, -1.0,  0.0,  0.0,  0.0,  0.0,  0.0,  0.0],\n",
    "               [ 0.0,  0.0,  1.0, -1.0,  0.0,  0.0,  0.0,  0.0],\n",
    "               [ 0.0,  0.0,  0.0,  0.0,  1.0, -1.0,  0.0,  0.0],\n",
    "               [ 0.0,  0.0,  0.0,  0.0,  0.0,  0.0,  1.0, -1.0]],\n",
    "              dtype=np.float32), np.array([0., 0., 0., 0., 0., 0., 0., 0.], dtype=np.float32)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "model4.layers[3].set_weights(\n",
    "    [np.array([[ 1.0,  0.0,  0.0,  0.0],\n",
    "               [ 1.0,  0.0,  0.0,  0.0],\n",
    "               [ 0.0,  1.0,  0.0,  0.0],\n",
    "               [ 0.0,  1.0,  0.0,  0.0],\n",
    "               [ 0.0,  0.0,  1.0,  0.0],\n",
    "               [ 0.0,  0.0,  1.0,  0.0],\n",
    "               [ 0.0,  0.0,  0.0,  1.0],\n",
    "               [ 0.0,  0.0,  0.0,  1.0]],\n",
    "              dtype=np.float32), np.array([0., 0., 0., 0.], dtype=np.float32)])\n",
    "\n",
    "model4.compile(loss='categorical_crossentropy',\n",
    "               optimizer=Adagrad(),\n",
    "               metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[array([[ 1.,  0.,  1.,  0.],\n",
      "       [ 0.,  1.,  0.,  1.],\n",
      "       [ 1.,  0., -1.,  0.],\n",
      "       [ 0.,  1.,  0., -1.]], dtype=float32), array([ 0.,  0.,  0.,  0.], dtype=float32)]\n",
      "[array([[ 1., -1.,  0.,  0.],\n",
      "       [ 1.,  1.,  0.,  0.],\n",
      "       [ 0.,  0.,  1., -1.],\n",
      "       [ 0.,  0., -1., -1.]], dtype=float32), array([ 0.,  0.,  0.,  0.], dtype=float32)]\n",
      "[array([[ 1., -1.,  0.,  0.,  0.,  0.,  0.,  0.],\n",
      "       [ 0.,  0.,  1., -1.,  0.,  0.,  0.,  0.],\n",
      "       [ 0.,  0.,  0.,  0.,  1., -1.,  0.,  0.],\n",
      "       [ 0.,  0.,  0.,  0.,  0.,  0.,  1., -1.]], dtype=float32), array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.], dtype=float32)]\n",
      "[array([[ 1.,  0.,  0.,  0.],\n",
      "       [ 1.,  0.,  0.,  0.],\n",
      "       [ 0.,  1.,  0.,  0.],\n",
      "       [ 0.,  1.,  0.,  0.],\n",
      "       [ 0.,  0.,  1.,  0.],\n",
      "       [ 0.,  0.,  1.,  0.],\n",
      "       [ 0.,  0.,  0.,  1.],\n",
      "       [ 0.,  0.,  0.,  1.]], dtype=float32), array([ 0.,  0.,  0.,  0.], dtype=float32)]\n"
     ]
    }
   ],
   "source": [
    "for layer in model4.layers:\n",
    "    print(layer.get_weights())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.17831734,  0.17831734,  0.17831734,  0.46504799]], dtype=float32)"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model4.predict([np.array([[1.0, 1.0], [-1.0, -1.0]]).reshape(1, 4)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test loss: 0.765614629269\n",
      "Test accuracy: 1.0\n"
     ]
    }
   ],
   "source": [
    "score = model4.evaluate(x4_test, y4_test, verbose=0)\n",
    "\n",
    "print('Test loss: {}'.format(score[0]))\n",
    "print('Test accuracy: {}'.format(score[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_20 (Dense)             (None, 4)                 20        \n",
      "_________________________________________________________________\n",
      "dense_21 (Dense)             (None, 4)                 20        \n",
      "_________________________________________________________________\n",
      "dense_22 (Dense)             (None, 8)                 40        \n",
      "_________________________________________________________________\n",
      "dense_23 (Dense)             (None, 4)                 36        \n",
      "=================================================================\n",
      "Total params: 116\n",
      "Trainable params: 116\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model5 = Sequential()\n",
    "model5.add(Dense(4, activation='tanh', input_shape=(4,)))\n",
    "model5.add(Dense(4, activation='tanh'))\n",
    "model5.add(Dense(8, activation='relu'))\n",
    "model5.add(Dense(num_classes, activation='softmax'))\n",
    "model5.compile(loss='categorical_crossentropy',\n",
    "               optimizer=RMSprop(),\n",
    "               metrics=['accuracy'])\n",
    "model5.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 4000 samples, validate on 1000 samples\n",
      "Epoch 1/8\n",
      "4000/4000 [==============================] - 0s - loss: 1.1352 - acc: 0.5507 - val_loss: 1.0160 - val_acc: 0.7330\n",
      "Epoch 2/8\n",
      "4000/4000 [==============================] - 0s - loss: 0.8918 - acc: 0.8722 - val_loss: 0.8094 - val_acc: 0.8580\n",
      "Epoch 3/8\n",
      "4000/4000 [==============================] - 0s - loss: 0.6966 - acc: 0.8810 - val_loss: 0.6283 - val_acc: 0.8580\n",
      "Epoch 4/8\n",
      "4000/4000 [==============================] - 0s - loss: 0.5284 - acc: 0.8810 - val_loss: 0.4697 - val_acc: 0.8580\n",
      "Epoch 5/8\n",
      "4000/4000 [==============================] - 0s - loss: 0.3797 - acc: 0.9022 - val_loss: 0.3312 - val_acc: 1.0000\n",
      "Epoch 6/8\n",
      "4000/4000 [==============================] - 0s - loss: 0.2555 - acc: 1.0000 - val_loss: 0.2166 - val_acc: 1.0000\n",
      "Epoch 7/8\n",
      "4000/4000 [==============================] - 0s - loss: 0.1612 - acc: 1.0000 - val_loss: 0.1318 - val_acc: 1.0000\n",
      "Epoch 8/8\n",
      "4000/4000 [==============================] - 0s - loss: 0.0939 - acc: 1.0000 - val_loss: 0.0732 - val_acc: 1.0000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f4d34067510>"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model5.fit(x4_train, y4_train, epochs=8, validation_data=(x4_test, y4_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.00708295,  0.00192736,  0.02899081,  0.96199888]], dtype=float32)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model5.predict([np.array([[1.0, 1.0], [-1.0, -1.0]]).reshape(1, 4)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test loss: 0.0731911802292\n",
      "Test accuracy: 1.0\n"
     ]
    }
   ],
   "source": [
    "score = model5.evaluate(x4_test, y4_test, verbose=0)\n",
    "\n",
    "print('Test loss: {}'.format(score[0]))\n",
    "print('Test accuracy: {}'.format(score[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "import contextlib\n",
    "\n",
    "@contextlib.contextmanager\n",
    "def printoptions(*args, **kwargs):\n",
    "    original = np.get_printoptions()\n",
    "    np.set_printoptions(*args, **kwargs)\n",
    "    try:\n",
    "        yield\n",
    "    finally: \n",
    "        np.set_printoptions(**original)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[array([[-0.2, -0.5,  0.8,  1. ],\n",
      "       [-0.9,  0.1, -0.8,  0.2],\n",
      "       [-0.2,  0.4,  0.1, -0.4],\n",
      "       [-0.8,  0.8,  1. ,  0.3]], dtype=float32), array([ 0. , -0. ,  0.1, -0.1], dtype=float32)]\n",
      "[array([[-0.4,  0.9, -1.3,  1.7],\n",
      "       [-0.4, -0.7,  0.3, -0.3],\n",
      "       [ 0.8, -0.9, -1.1, -0.2],\n",
      "       [ 1.3,  0.5,  0.4, -0.2]], dtype=float32), array([-0. , -0. ,  0.2,  0. ], dtype=float32)]\n",
      "[array([[-1.6,  0.3,  0.3, -0.3, -1.1,  1.2,  0.7, -1. ],\n",
      "       [ 0.4,  1.3, -0.9,  0.8, -0.4, -0.7, -1.2, -1. ],\n",
      "       [ 0.6,  1. ,  0.9, -1. , -1.1, -0.2, -0.4, -0.3],\n",
      "       [ 1.1,  0.1, -0.9,  1.3, -0.3, -0.2,  0.2, -0.4]], dtype=float32), array([-0. ,  0.2, -0.1,  0. , -0.1, -0. , -0.1,  0.1], dtype=float32)]\n",
      "[array([[ 0.6, -1.5,  1.3, -1.4],\n",
      "       [-0.4, -1.6, -0.3,  1.2],\n",
      "       [ 1.2,  1.1, -0.3, -1.5],\n",
      "       [ 0.6,  1.4, -1.5, -1.2],\n",
      "       [ 0.2, -1.3, -0.9,  0.8],\n",
      "       [ 0.6, -1.5,  0.8, -1. ],\n",
      "       [ 0.4, -1.3,  0.4,  0.3],\n",
      "       [-1.3,  0.5, -0.9,  0.8]], dtype=float32), array([-0.8,  0.7,  0.4,  0.1], dtype=float32)]\n"
     ]
    }
   ],
   "source": [
    "with printoptions(precision=1, suppress=True):\n",
    "    for layer in model5.layers:\n",
    "        print(layer.get_weights())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 12.3. Odmiany metody gradientu prostego\n",
    "\n",
    "* Batch gradient descent\n",
    "* Stochastic gradient descent\n",
    "* Mini-batch gradient descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### _Batch gradient descent_\n",
    "\n",
    "* Klasyczna wersja metody gradientu prostego\n",
    "* Obliczamy gradient funkcji kosztu względem całego zbioru treningowego:\n",
    "  $$ \\theta := \\theta - \\alpha \\cdot \\nabla_\\theta J(\\theta) $$\n",
    "* Dlatego może działać bardzo powoli\n",
    "* Nie można dodawać nowych przykładów na bieżąco w trakcie trenowania modelu (_online learning_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### _Stochastic gradient descent_ (SGD)\n",
    "\n",
    "* Aktualizacja parametrów dla każdego przykładu:\n",
    "  $$ \\theta := \\theta - \\alpha \\cdot \\nabla_\\theta \\, J \\! \\left( \\theta, x^{(i)}, y^{(i)} \\right) $$\n",
    "* Dużo szybszy niż _batch gradient descent_\n",
    "* Można dodawać nowe przykłady na bieżąco w trakcie trenowania (_online learning_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### _Stochastic gradient descent_ (SGD)\n",
    "\n",
    "* Częsta aktualizacja parametrów z dużą wariancją:\n",
    "\n",
    "<img src=\"http://ruder.io/content/images/2016/09/sgd_fluctuation.png\" style=\"margin: auto;\" width=\"50%\" />\n",
    "\n",
    "* Z jednej strony dzięki temu nie utyka w złych minimach lokalnych, ale z drugiej strony może „wyskoczyć” z dobrego minimum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### _Mini-batch gradient descent_\n",
    "\n",
    "* Kompromis między _batch gradient descent_ i SGD\n",
    "  $$ \\theta := \\theta - \\alpha \\cdot \\nabla_\\theta \\, J \\left( \\theta, x^{(i : i+n)}, y^{(i : i_n)} \\right) $$\n",
    "* Stabilniejsza zbieżność dzięki redukcji wariancji aktualizacji parametrów\n",
    "* Szybszy niż klasyczny _batch gradient descent_\n",
    "* Typowa wielkość batcha: między 50 a 256 przykładów"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Wady klasycznej metody gradientu prostego, czyli dlaczego potrzebujemy optymalizacji\n",
    "\n",
    "* Trudno dobrać właściwą szybkość uczenia (_learning rate_)\n",
    "* Jedna ustalona wartość stałej uczenia się dla wszystkich parametrów\n",
    "* Funkcja kosztu dla sieci neuronowych nie jest wypukła, więc uczenie może utknąć w złym minimum lokalnym lub punkcie siodłowym"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 12.4. Algorytmy optymalizacji metody gradientu\n",
    "\n",
    "* Momentum\n",
    "* Nesterov Accelerated Gradient\n",
    "* Adagrad\n",
    "* Adadelta\n",
    "* RMSprop\n",
    "* Adam\n",
    "* Nadam\n",
    "* AMSGrad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Momentum\n",
    "\n",
    "* SGD źle radzi sobie w „wąwozach” funkcji kosztu\n",
    "* Momentum rozwiązuje ten problem przez dodanie współczynnika $\\gamma$, który można trakować jako „pęd” spadającej piłki:\n",
    "  $$ v_t := \\gamma \\, v_{t-1} + \\alpha \\, \\nabla_\\theta J(\\theta) $$\n",
    "  $$ \\theta := \\theta - v_t $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Przyspiesony gradient Nesterova (_Nesterov Accelerated Gradient_, NAG)\n",
    "\n",
    "* Momentum czasami powoduje niekontrolowane rozpędzanie się piłki, przez co staje się „mniej sterowna”\n",
    "* Nesterov do piłki posiadającej pęd dodaje „hamulec”, który spowalnia piłkę przed wzniesieniem:\n",
    "  $$ v_t := \\gamma \\, v_{t-1} + \\alpha \\, \\nabla_\\theta J(\\theta - \\gamma \\, v_{t-1}) $$\n",
    "  $$ \\theta := \\theta - v_t $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Adagrad\n",
    "\n",
    "* “<b>Ada</b>ptive <b>grad</b>ient”\n",
    "* Adagrad dostosowuje współczynnik uczenia (_learning rate_) do parametrów: zmniejsza go dla cech występujących częściej, a zwiększa dla występujących rzadziej\n",
    "* Świetny do trenowania na rzadkich (_sparse_) zbiorach danych\n",
    "* Wada: współczynnik uczenia może czasami gwałtownie maleć"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Adadelta i RMSprop\n",
    "* Warianty algorytmu Adagrad, które radzą sobie z problemem gwałtownych zmian współczynnika uczenia"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Adam\n",
    "\n",
    "* “<b>Ada</b>ptive <b>m</b>oment estimation”\n",
    "* Łączy zalety algorytmów RMSprop i Momentum\n",
    "* Można go porównać do piłki mającej ciężar i opór\n",
    "* Obecnie jeden z najpopularniejszych algorytmów optymalizacji"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Nadam\n",
    "* “<b>N</b>esterov-accelerated <b>ada</b>ptive <b>m</b>oment estimation”\n",
    "* Łączy zalety algorytmów Adam i Nesterov Accelerated Gradient"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### AMSGrad\n",
    "* Wariant algorytmu Adam lepiej dostosowany do zadań takich jak rozpoznawanie obiektów czy tłumaczenie maszynowe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"contours_evaluation_optimizers.gif\" style=\"margin: auto;\" width=\"80%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"saddle_point_evaluation_optimizers.gif\" style=\"margin: auto;\" width=\"80%\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "livereveal": {
   "start_slideshow_at": "selected",
   "theme": "amu"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}