zuma/wyk/4_Sieci_neuronowe.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Uczenie maszynowe\n",
    "# 4. Sieci neuronowe – wprowadzenie"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "# Przydatne importy\n",
    "\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 4.1. Perceptron"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "https://www.youtube.com/watch?v=cNxadbrN_aI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img style=\"margin: auto\" heighth=\"100%\" src=\"http://m.natemat.pl/b94a41cd7322e1b8793e4644e5f82683,641,0,0,0.png\" alt=\"Frank Rosenblatt\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img style=\"margin: auto\" src=\"http://m.natemat.pl/02943a7dc0f638d786b78cd5c9e75742,641,0,0,0.png\" heighth=\"100%\" alt=\"Frank Rosenblatt\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img style=\"margin: auto\" heighth=\"100%\" src=\"https://upload.wikimedia.org/wikipedia/en/5/52/Mark_I_perceptron.jpeg\" alt=\"perceptron\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Pierwszy perceptron liniowy\n",
    "\n",
    "* Frank Rosenblatt, 1957\n",
    "* aparat fotograficzny podłączony do 400 fotokomórek (rozdzielczość obrazu: 20 x 20)\n",
    "* wagi – potencjometry aktualizowane za pomocą silniczków"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Uczenie perceptronu\n",
    "\n",
    "Cykl uczenia perceptronu Rosenblatta:\n",
    "\n",
    "1. Sfotografuj planszę z kolejnym obiektem.\n",
    "1. Zaobserwuj, która lampka zapaliła się na wyjściu.\n",
    "1. Sprawdź, czy to jest właściwa lampka.\n",
    "1. Wyślij sygnał „nagrody” lub „kary”."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Funkcja aktywacji\n",
    "\n",
    "Funkcja bipolarna:\n",
    "\n",
    "$$ g(z) = \\left\\{ \n",
    "\\begin{array}{rl}\n",
    "1  & \\textrm{gdy $z > \\theta_0$} \\\\\n",
    "-1 & \\textrm{wpp.}\n",
    "\\end{array}\n",
    "\\right. $$\n",
    "\n",
    "gdzie $z = \\theta_0x_0 + \\ldots + \\theta_nx_n$,<br/>\n",
    "$\\theta_0$ to próg aktywacji,<br/>\n",
    "$x_0 = 1$. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "def bipolar_plot():\n",
    "    matplotlib.rcParams.update({'font.size': 16})\n",
    "\n",
    "    plt.figure(figsize=(8,5))\n",
    "    x = [-1,-.23,1] \n",
    "    y = [-1, -1, 1]\n",
    "    plt.ylim(-1.2,1.2)\n",
    "    plt.xlim(-1.2,1.2)\n",
    "    plt.plot([-2,2],[1,1], color='black', ls=\"dashed\")\n",
    "    plt.plot([-2,2],[-1,-1], color='black', ls=\"dashed\")\n",
    "    plt.step(x, y, lw=3)\n",
    "    ax = plt.gca()\n",
    "    ax.spines['right'].set_color('none')\n",
    "    ax.spines['top'].set_color('none')\n",
    "    ax.xaxis.set_ticks_position('bottom')\n",
    "    ax.spines['bottom'].set_position(('data',0))\n",
    "    ax.yaxis.set_ticks_position('left')\n",
    "    ax.spines['left'].set_position(('data',0))\n",
    "\n",
    "    plt.annotate(r'$\\theta_0$',\n",
    "                 xy=(-.23,0), xycoords='data',\n",
    "                 xytext=(-50, +50), textcoords='offset points', fontsize=26,\n",
    "                 arrowprops=dict(arrowstyle=\"->\"))\n",
    "\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAcwAAAEeCAYAAAAHLSWiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de1yUZf7/8fcoZzEVT7mmwapYQsUqth5yKaxdUpHWFHTDxNrU1HbxlNpiWlp00KLdVkOtNA+PBH0Y5LElxbJsk3bZDAo1wbTNNHU7iKDA/fvDL/NrHA43h2EYeD0fj3nIXNd9DZ+5He4398x1X1gMwxAAAKhaC2cXAACAKyAwAQAwgcAEAMAEAhMAABMITAAATCAwAQAwwa2afq45AZwgIiJCu3btcnYZQHNlqaiRM0ygEfruu++cXQKAqxCYAACYQGACAGACgQkAgAkEJgAAJhCYAACYQGACAGACgQkAgAkEJgAAJhCYAACYQGACAGACgQkAgAkEJgAAJhCYAACYQGACAGACgQkAgAkEJlCFkydP6pFHHtHAgQPl4+Mji8WigoICU2PLysqUmJgof39/eXl56ZZbbtGWLVscWzAAhyEwgSocPXpUKSkpateunYYMGVKjsQsWLNCiRYs0ffp07dy5UwMGDNCYMWO0Y8cOB1ULwJEshmFU1V9lJ9DUlZWVqUWLK79Xrl69Wg899JDy8/Pl7+9f5bjTp0+rW7dumjdvnp544glr+9ChQ3XmzBl9+umnVY4PDQ1VVlZWnesHUCuWiho5wwSqUB6WNbV7925dunRJsbGxNu2xsbE6dOiQ8vPz66M8AA2IwAQcICcnR56enurZs6dNe1BQkCQpNzfXGWUBqIMq35K9/fbb7Trj4uIUFxen7777TqNHj7Yb8/DDDysmJkYnTpzQ+PHj7fpnzZqlyMhI5eXlafLkyXb9CQkJuvPOO5Wdna34+Hi7/qefflqDBg3Shx9+qMcee8yuPykpSSEhIcrIyNCSJUvs+pOTk9W7d2+9/fbbWrZsmV3/unXr1K1bN23atEkrVqyw69+8ebM6dOigNWvWaM2aNXb9O3bskI+Pj5YvX66UlBS7/szMTEnS0qVLtW3bNps+b29v7dy5U5K0ePFivfvuuzb97du3t04amT9/vg4cOGDTf91112n9+vWSpPj4eGVnZ9v0BwYGauXKlZKkSZMm6fDhwzb9ISEhSkpKknTlTOjkyZM2/QMHDlRiYqIk6d5779XZs2dt+ocOHaoFCxZIku6++25dvHjRpn/EiBGaPXu2JOn222/X1aKjozV16lQVFhZq2LBhdv11fe2FjJ2lf/zXXRculdr1NTbfrI1XlwlJzi4DaBAFzwyX1HiOe5mZmRW+JetWq2cHuKCdX1lUXNb4wxJA48SkHzQb/vO2O7sE0zjDRHNSfobZiHCGCZSrzQ9oTWbJvvHGG5owYYKOHDli8znmmjVrNHHiRB07dkwBAQGVjg/NWKisxncQAZo1Jv0ADhARESEPDw9t2LDBpn39+vUKDg6uMiwBNE6cYQLV2Lx5syTpk08+kSTt3LlTHTt2VMeOHRUWFiZJcnNz04QJE/Tqq69Kkjp16qQZM2YoMTFRrVu3Vt++fbVp0ybt2bNHaWlpznkiAOqEwASqMWbMGJv7U6dOlSSFhYVZZz2XlpaqtNR2QtFTTz0lX19fvfTSSzp16pR69+6tlJQURUZGNkjdAOoXgQlUo5qJcZVu07JlSyUkJCghIcERZQFoYHyGCQCACQQmAAAmEJgAAJhAYAIAYAKBCQCACQQmAAAmEJgAAJhAYAIAYAKBCQCACQQmAAAmEJgAAJhAYAIAYAKBCQCACQQmAAAmEJgAAJhAYAIAYAKBCQCACQQmAAAmEJgAAJhAYAIAYAKBCQCACQQmAAAmEJgAAJhAYAIAYAKBCQCACQQmAAAmEJgAAJhAYAIAYAKBCQCACQQmAAAmEJgAAJhAYAIAYAKBCQCACQQmAAAmEJgAAJhAYAIAYAKBiTr73//+p8WLFyskJEStW7eWn5+fwsPDtWPHDmeXBgD1xs3ZBcC1ZWZm6g9/+IO++eYbm/a9e/cqMzNTK1as0OTJk51UHQDUH84wUWuZmZkaNmyYvvnmG8XGxurgwYM6f/68Pv74Yw0cOFCGYWjmzJk6efKks0sFgDojMFErp0+f1tixY3Xx4kU999xzWrdunUJDQ9W2bVv1799faWlp8vX1VWFhoTZu3OjscgGgzghM1Mrs2bP17bffasSIEZozZ45df8eOHTV48GBJ0r59+xq6PACodwQmauyLL77Qhg0bZLFY9Nxzz1W6XceOHSVJx48fb6jSAMBhCEzUWHJyssrKynTnnXfqxhtvrHS7y5cv2/wLAK6MwESNlJWV6c0335Qk3XfffVVue+7cOUmSt7e3w+tylBMnTmj06NFq06aNrrnmGo0aNUpfffWVqbEWi6XCW3Z2toOrBuAIXFaCGsnOztapU6ckSXFxcYqLi6t2TLdu3RxclWMUFhYqPDxcnp6eWrt2rSwWixISEnTHHXfo008/VatWrap9jLi4OLvLagIDAx1VMgAHIjBRI7WZwNOrVy8HVOJ4q1at0rFjx5SXl6eePXtKkm6++Wb16tVLycnJmjlzZrWP0bVrVw0YMMDRpQJoALwlixr517/+JUn63e9+p4sXL1Z627p1q3VM3759nVVunaSnp2vAgAHWsJSkgIAADR48WGlpaU6sDIAzEJiokcOHD0uSunfvLi8vr0pvH330kXXMb37zG5vH2LRpk/r37y9vb2+1b99e0dHROnbsWIM+DzNycnIUHBxs1x4UFKTc3FxTj7FixQp5enrKx8dH4eHhev/99+u7TAANhMBEjZSv2uPn51fldtu3b5ck3Xjjjerevbu1/ZVXXtHYsWPl7u6uF198UTNnztSePXs0cOBA05NpGsq5c+fUrl07u3Y/Pz+dP3++2vGxsbFavny5MjIytHLlSp09e1bh4eHKzMx0QLUAHI3PMFEjFy9elCR5eXlVus0XX3yhzz77TJI0fvx4a/u5c+f06KOPKiQkRPv27ZO7u7sk6e6771b//v312GOPaf369Q6svuYsFotdm2EYpsauW7fO+vWQIUMUFRWl4OBgJSQkaP/+/Xbbr1y5UitXrpQknTlzppYVA3AUzjBRIx4eHpKkCxcuVLrN8uXLJV25nOTns2jfeust/fjjj/rzn/9sDUvpymec4eHh2rJliwoLCx1TeC20a9fOemnMz50/f77CM8/qtG7dWsOHD9fBgwcr7J80aZKysrKUlZVlXfQBQONBYKJGrr/+eklSXl5ehf35+fnWs6Tp06erS5cu1r6PP/5YkqxL5v3cbbfdpqKiIuuZaWMQFBSknJwcu/bc3Fz16dOnVo9pGEaFZ60AGj8CEzUSFhYmSXrnnXf03//+16bvwoULiomJUXFxsQIDA7Vw4UKb/q+//lqSdN1119k9bnlbY/rLJiNHjtRHH31kMyGpoKBAH3zwgUaOHFnjx/vhhx+0fft2/frXv67PMgE0EAITNTJx4kS5ubmpuLhYI0eOtP5Jr3feeUeDBw/WwYMH5efnp9TUVLsL+8vfbvX09LR73PLVgBrTW7IPPfSQ/P39FRUVpbS0NKWnpysqKkrdunWzWYzg+PHjcnNz05NPPmltW7p0qR566CFt3LhRmZmZWrt2rQYPHqxTp05pyZIlzng6AOqIST+okRtvvFELFy7UggUL9Mknn+jWW2+16f/lL3+pLVu26Oabb7Yb6+PjI0kqLi62Wy6vqKjIZpvGoFWrVtqzZ49mzJih8ePHyzAMDR06VElJSfL19bVuZxiGSktLVVZWZm3r3bu3tm7dqq1bt+r777/XNddco8GDB+vVV1+122cAXAOBiRpLSEhQYGCgkpKSlJOTI8MwFBgYqJiYGE2bNq3S0OvataukK2+7Xr36z4kTJyRV/HatM3Xv3l1btmypcht/f3+7mbORkZGKjIx0ZGkAGhiBiVqJjo5WdHR0jcbceuutSk5O1oEDB+wC84MPPpCXl1eFCwUAQGPAZ5hoMFFRUfL19dVLL72kkpISa/u///1vvfvuuxo1alSjeksWAH6OM0w0mPbt2+uZZ57R9OnTFRYWpvvvv19nz57Viy++qPbt2+upp55ydokAUCkCEw1q2rRp8vPz09KlSxUfHy9vb28NHTpUiYmJ8vf3d3Z5AFApAhMNbty4cRo3bpyzywCAGuEzTAAATCAwAQAwgcAEAMAEAhMAABMITDQKeXl5OnXqlLPLAIBKEZhoFN577z3dd999pv84MwA0NAITjcLEiRN1/vx5rV+/3tmlAECFCEw0Cm5ubkpOTtajjz6qc+fOObscALBDYKLR6N+/v0aPHq25c+c6uxQAsENgolFZsmSJdu7cqf379zu7FACwQWCiUWnTpo1efPFFTZkyRZcuXXJ2OQBgRWCi0Rk9erS6d++uF154wdmlAIAVgYlGx2Kx6O9//7uWLl2q/Px8Z5cDAJIITDRSAQEBmj17tqZNm8a1mQAaBQITjdasWbP01VdfafPmzc4uBQAITDRe7u7ueuWVVzRjxgx9//33zi4HQDNHYKJRu+222xQREaGEhARnlwKgmSMw0eg9++yzSk1N1cGDB51dCoBmjMBEo9e+fXs999xzmjx5skpKSpxdDoBmisCESxg/frzatm2rl19+2dmlAGimCEy4BIvFohUrVmjJkiU6efKks8sB0AwRmHAZvXv31rRp0/SnP/3J2aUAaIYITLiU+fPn67PPPtPbb7/t7FIANDMEJlyKl5eXli9frunTp+unn35ydjkAmhECEy7nzjvv1JAhQ/TEE084uxQAzQiBCZe0bNkyrV27Vv/5z3+cXQqAZoLAhEvq3LmzlixZoilTpqisrMzZ5QBoBghMuKw//vGPatGihVauXOnsUgA0AwQmXFaLFi30yiuvaMGCBTp16pSzywHQxBGYcGk33XSTHnjgAc2cOdPZpQBo4ghMuLzHH39cBw4c0DvvvGPXt3r1amVkZDihKgBNDYEJl9eqVSu9/PLLmjp1qi5evGjTl5+fr48++shJlQFoSghMNAnDhw9XSEiInn76aZv2zp078/kmgHpBYKLJeOmll7RixQp9/vnn1rZrr71W3377rROrAtBUEJhoMrp27arHH39cDz/8sAzDkHQlMDnDBFAfCEy4rLNnz+rw4cM2bdOmTdNPP/2kN954QxJvyQKoPwQmXFZ+fr5uu+02zZo1Sz/++KMkqWXLlkpOTtbcuXN19uxZ3pIFUG8ITLis0NBQffbZZzp79qz69OmjLVu2yDAM9evXTzExMXr00Ud1zTXX6PLlyyosLHR2uQBcHIEJl9apUyetWbNG69ev14IFCzRixAjl5+dr8eLFeuedd7R//3517tyZs0wAdUZgokkICwtTdna2brvtNoWGhurll1/W888/r8mTJ9fpc8wTJ05o9OjRatOmja655hqNGjVKX331lamxRUVFmjNnjrp06SJvb28NHDhQ7733Xq3qAOB8BCaaDA8PD82fP19ZWVn64IMP9OSTT6pNmzb64YcfanWGWVhYqPDwcH3xxRdau3at1q1bpyNHjuiOO+7QhQsXqh3/4IMPatWqVXryySe1bds2denSRb/73e+UnZ1dm6cHwMncnF0AUN8CAgK0bds2bd26VY888oi++eYbHTp0SFLfGj3OqlWrdOzYMeXl5alnz56SpJtvvlm9evVScnJylevX/uc//9HGjRv12muvaeLEiZKunAUHBQXp8ccfV3p6eq2fHwDn4AwTTZLFYtGoUaOUl5en4cOHy8vLq8aPkZ6ergEDBljDUroSxoMHD1ZaWlq1Y93d3RUTE2Ntc3Nz09ixY7V7924VFxfXuB4AzkVgoknz9fXV22+/rTlz5tR4bE5OjoKDg+3ag4KClJubW+3YgIAA+fj42I29dOmSjh49WuN6ADgXgQlU4ty5c2rXrp1du5+fn86fP1/rseX9AFyLpXwJsYpEREQY3333XQOWU70zZ86oY8eOzi6jWWkq+/zQ199bv76pa5tqt//Xv/6lzp07q2vXrjbtX3/9tU6dOqV+/fpVOvbw4cMqKyvTDTfcYNP+ww8/6MiRI+rdu7d8fX1t+s6cOaPyn7fi4mKFhIRUWyPqT1N5nbuSxrrPP/nkk92GYURc3V5lYEqqstMZQkNDlZWV5ewympWmss/95223fl3wzPBqt+/cubPuueceJScn27RPnTpVqampOnPmTKVjY2JilJ2drby8PJv2lJQUxcTE6LPPPlNQUFCl41u1amVqJi7qT1N5nbuSRrzPLRU18pYsUImgoCDl5OTYtefm5qpPnz7Vjs3Pz7dbYSg3N1ceHh42E4kAuAYCE6jEyJEj9dFHH+nYsWPWtoKCAn3wwQcaOXJktWMvX76s1NRUa1tJSYk2bdqk3/72t/L09HRY3QAcw+UCc9KkSc4uodlprvv8oYcekr+/v6KiopSWlqb09HRFRUWpW7dumjx5snW748ePy83NTU8++aS1LSQkRDExMYqPj9fq1av17rvvauzYscrPz9cTTzxR7ffu0KGDQ54TKtdcX+fO5Gr7nMBEtZrrPm/VqpX27NmjwMBAjR8/Xvfdd58CAgK0Z88emwk7hmGotLRUZWVlNuNff/11TZw4UQkJCRo+fLhOnDihXbt2qW/f6hdQaIwTIZq65vo6dyZX2+cuN+kHqK2aTvpxpkY8GQJoDlxz0s8LL7ygyMhIdenSRRaLRYsWLarR+P3792vQoEHy9vbWtddeq5kzZ+rixYuOKbaJKCsrU2Jiovz9/eXl5aVbbrlFW7ZsMTU2Li5OFovF7hYfH+/gql0Di7k3vLrs84peyxaLhfWAq3Dy5Ek98sgjGjhwoHx8fGSxWFRQUGBqbF2OPQ2h0QfmqlWrdPr0ad1zzz01Hvvpp5/qrrvuUqdOnbRt2zYtWbJEr7/+uuLi4uq/0CZkwYIFWrRokaZPn66dO3dqwIABGjNmjHbs2GFqfMeOHXXgwAGb24wZMxxcdePHYu4Nr677XLryS+DVr+fAwEAHV+66jh49qpSUFLVr105Dhgyp0di6HnsczjCMqm5OV1paahiGYVy+fNmQZCxcuND02Hvuucfo2bOncenSJWvb2rVrDUnGJ598Ut+lNgnffvut4eHhYTz++OM27eHh4cZNN91U7fgJEyYYXbt2dVR5dXL93G3WmzMkJSUZLVq0MI4cOWJtO3bsmNGyZUtj2bJlNtv269fP5n52drYhyXjttdesbZcvXzYCAwONyMhIxxbuwmqyzysiyfjLX/7iyBKbnPJjtmEYxqpVqwxJRn5+frXj6nrsqWcVZmKjP8Ns0aJ2JV6+fFm7du1SdHS03N3dre3R0dHy8PCodvHs5mr37t26dOmSYmNjbdpjY2N16NAh5efnO6ky18di7g2vLvsctVPbY7YrHHsafWDW1pdffqmioiK7xbO9vLzUo0ePahfPbq5ycnLk6elpd2F9+ao0Zvbb6dOn1aFDB7m5uSkwMFDPPvusSktLHVKvK2Ex94ZXl31ebsWKFfL09JSPj4/Cw8P1/vvv13eZUP0cexytyf49zPLFrStbAJvFryt27tw5tW3bVhaL7SQxs4uGh4SEqF+/fgoKClJRUZG2bt2q+fPn68iRI1q9erXD6nYFLObe8Oqyz6UrZzcjRozQL37xCx0/flzPP/+8wsPD9Y9//EO33367Aypuvup67GkIDRqYGRkZuuuuu6rdLiwsTJmZmXX6Xsb/XS5z9c7/eV9zUNN9bhhGnfbZ1bNhhw0bJl9fXyUlJWnu3Lnq1auXucKbqNru27r+vzRnddlv69ats349ZMgQRUVFKTg4WAkJCdq/f3+91QjXeI03aGAOGjRIn3/+ebXbXf22U21U9VvJ+fPnq1z4uimp6T4v/8376hdv+W/j5fu1JsaNG6ekpCRlZWU168Bs165dpa/His6Cfs7Pz6/CSyHq8v/SHNRln1ekdevWGj58uF599dX6KA8/44hjT31r0MD08fGx+3NHjtKjRw95enraLZ5dVFSkY8eOacyYMQ1Sh7PVdJ8HBQWpuLhYX375pc1nCeWfH1S36HhFqjrbb07qupj71q1bVVhYaPMLJYu5V60u+7wylZ0JoW4cceypb0120o+Hh4ciIiKUkpKikpISa/vmzZtVXFxc7eLZzVVERIQ8PDy0YcMGm/b169crODhYAQEBNX7MjRs3ymKxqH///vVVpktiMfeGV5d9XpEffvhB27dv169//ev6LBNyzLGn3lV2vYnRSK7DPHjwoJGammps2rTJkGSMGTPGSE1NNVJTU40LFy5Yt3vggQeMli1b2oz997//bXh5eRm///3vjYyMDGP16tVGu3btjNGjRzf003Apc+fONTw9PY1ly5YZe/fuNaZMmWJYLBYjPT3dZrvw8HCjR48e1vsFBQXGkCFDjL///e/G7t27jfT0dGPixImGxWIxpkyZ0tBPw46zr8P86aefjB49ehjBwcHGW2+9ZaSlpRk333yzERAQYPz444/W7QoKCgxJxhNPPGEzPiYmxmjbtq2xatUqIyMjw7j33nsNT09PrimuQk32ecuWLW32+fPPP2/88Y9/NDZs2GDs3bvXWLNmjREcHGy4u7sb7733njOejssoP0ZPmTLFkGQsX77cSE1NNTIzM63btGzZ0njggQdsxpk99jSACjOx0QfmhAkTDF1Z09bu9vOLYcu3u9q+ffuMAQMGGJ6enkanTp2MP//5zzZBC3slJSXG4sWLje7duxseHh7GTTfdZKSmptptFxYWZlx//fXW+2fPnjWioqKM7t27G56enoaXl5fxq1/9yvjb3/5mczGzszg7MA3DMI4fP26MGjXKaN26teHr62tERUXZXdSdn59f4SIdhYWFxowZM4zOnTsbnp6exq233mrs3bu3wWp3VbXd5+np6cagQYOM9u3bG25uboafn58RGRlp/POf/2zYJ+CCKjtmh4WF2WwzYcIEm3Fmjz0NoMJMZPF1NBssvg7AJNdcfB0AgMaAwAQAwAQCEwAAEwhMAABMIDABADCBwAQAwAQCEwAAEwhMAABMIDABADCBwAQAwAQCEwAAEwhMAABMIDABADCBwAQAwAQCEwAAEwhMAABMIDABADCBwAQAwAQCEwAAEwhMAABMIDABADCBwAQAwAQCEwAAEwhMAABMIDABADCBwAQAwAQCEwAAEwhMAABMIDABADCBwAQAwAQCEwAAEwhMoBJlZWVKTEyUv7+/vLy8dMstt2jLli2mxsbFxclisdjd4uPjHVw1AEdxc3YBQGO1YMECLV26VE899ZT69eunN998U2PGjNG2bds0bNiwasd37NhR6enpNm1dunRxVLkAHIzABCpw+vRpLV26VPPmzdPs2bMlSXfccYeOHj2qefPmmQpMDw8PDRgwwNGlAmggvCULVGD37t26dOmSYmNjbdpjY2N16NAh5efnO6kyAM5CYAIVyMnJkaenp3r27GnTHhQUJEnKzc2t9jFOnz6tDh06yM3NTYGBgXr22WdVWlrqkHoBOB5vyQIVOHfunNq2bSuLxWLT7ufnZ+2vSkhIiPr166egoCAVFRVp69atmj9/vo4cOaLVq1c7rG4AjkNgolnIyMiwuX91EJYLCwtTZmamDMOocBvDMEx9v6tnww4bNky+vr5KSkrS3Llz1atXL7sxK1eu1MqVKyVJZ86cMfV9ADQcAhPNwqBBg6SMvdb7n3/+eYXb+fj4SLpyJnn+/Hm74Dx//ry1v6bGjRunpKQkZWVlVRiYkyZN0qRJkyRJoaGhNX58AI5FYKJZKA/CcjfccEOV2wcFBam4uFhffvmlzeeY5Z9d9unTp8Y1lJ+dVnZ2C6BxY9IPUIGIiAh5eHhow4YNNu3r169XcHCwAgICavyYGzdulMViUf/+/eurTAANiDNMoAKdOnXSjBkzlJiYqNatW6tv377atGmT9uzZo7S0NJtthw4dquPHj+vo0aOSpOPHj2v8+PEaO3asevbsqeLiYm3dulVr1qzR5MmT1aNHD2c8JQB1RGAClXjqqafk6+url156SadOnVLv3r2VkpKiyMhIm+1KS0tVUlJivd+6dWv5+fnp2Wef1bfffiuLxaIbb7xRf/3rXzV16tSGfhoA6omlmll/5qYEAi7Af95269cFzwx3YiXVCw0NVVZWlrPLAJqrCica8BkmAAAmEJgAAJhAYAIAYAKBCQCACQQmAAAmEJgAAJhAYAIAYAKBCQCACQQmAAAmEJgAAJhAYAIAYAKBCQCACQQmAAAmEJgAAJhAYAIAYAKBCQCACQQmAAAmEJgAAJhAYAIAYAKBCQCACQQmAAAmEJgAAJhAYAIAYAKBCQCACQQmAAAmEJgAAJhAYAIAYAKBCQCACQQmAAAmEJgAAJhAYAIAYAKBCQCACQQmAAAmEJgAAJhAYAIAYAKBCQCACQQmAAAmEJgAAJhAYAIAYAKBCQCACQQmAAAmEJgAAJhAYAIAYAKBCVTihRdeUGRkpLp06SKLxaJFixbVaPz+/fs1aNAgeXt769prr9XMmTN18eJFxxQLwOEITKASq1at0unTp3XPPffUeOynn36qu+66S506ddK2bdu0ZMkSvf7664qLi6v/QgE0CDdnFwA0Vjk5OWrRooVKSkr0yiuv1GjswoULdd111yk1NVXu7u6SJA8PD02YMEFz585V3759HVEyAAfiDBOoRIsWtfvxuHz5snbt2qXo6GhrWEpSdHS0PDw8lJaWVl8lAmhABCZQz7788ksVFRUpODjYpt3Ly0s9evRQbm6ukyoDUBdVviV7++2327XFxcUpLi5O3333nUaPHm3X//DDDysmJkYnTpzQ+PHj7fpnzZqlyMhI5eXlafLkyXb9CQkJuvPOO5Wdna34+HibvoIBc6p7PoApFb22K3vtGYYhSTpx4oQk6cMPP9Rjjz1mNz4pKUkhISHavXu3JCkxMVHJycnW/uTkZPn5+enw4cMVfv9169apW7du2rRpk/Ly8uy22bx5szp06KA1a9ZozZo1duN37NghHx8fLV++XPMRzZkAAALqSURBVCkpKXb9mZmZkqSlS5dq27ZtNn3e3t7auXOnJGnx4sV69913bfrbt2+vLVu2SJLmz5+vAwcO2PRfd911Wr9+vSQpPj5e2dnZNv2BgYFauXKlJGnSpEk6fPiwTX9ISIiSkpIkSbGxsTp58qRN/8CBA5WYmChJuvfee3X27Fmb/qFDh2rBggWSpLvvvttuctWIESM0e/ZsSRX/30dHR2vq1KkqLCzUsGHD7PqdedyTpKefflqDBg2q9rWXkZGhJUuW2PUnJyerd+/eevvtt7Vs2TK7/p+/9lasWGHX39xee+X1Xo3PMNHslBUXat++fXbt+/btU1hYmPWHp7bKA7amfRs2bNDmzZt17tw5Xb58uU41AKh/lqp+gCVV2dnQ/Odtd3YJcHHebhbFhvjp3uC2Ffb7+Pioe/fuNm0lJSVyd3fXwoULTV1a8vnnn6tPnz7auHGjxo0bZ9PXp08fBQUFKTU1tcrHCA0NVVZWVrXfC4BDWCpqdKkzzIJnhju7BKBaPXr0kKenp3Jycmzai4qKdOzYMY0ZM8ZJlQGoCyb9APXMw8NDERERSklJUUlJibV98+bNKi4u1siRI51YHYDacqkzTKAhZWVlqaCgQGVlZZKk3Nxcbd68WZI0bNgw+fj4SJIefPBBrV271iYcFy1apIEDByo6OlrTpk1TQUGB5syZo9GjR6tfv34N/2QA1BmBCVTi5Zdf1tq1a633U1NTrZ895ufny9/fX5JUWlqq0tJSm7Hls2Xnzp2r4cOHq02bNrr//vv19NNPN1j9AOqXS036AZoLJv0ATlXhpB8+wwQAwAQCEwAAEwhMAABMIDABADCBwAQAwAQCEwAAEwhMAABMIDABADCBwAQAwAQCEwAAEwhMAABMIDABADCBwAQAwAQCEwAAEwhMAABMIDABADChuj8gDcAJLBbLLsMwIpxdB4D/j8AEAMAE3pIFAMAEAhMAABMITAAATCAwAQAwgcAEAMCE/weovYTBIgUU4wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 576x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "bipolar_plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Perceptron – schemat\n",
    "\n",
    "<img src=\"perceptron.png\" width=\"60%\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Perceptron – zasada działania\n",
    "\n",
    "1. Ustal wartości początkowe $\\theta$ (wektor 0 lub liczby losowe blisko 0).\n",
    "1. Dla każdego przykładu $(x^{(i)}, y^{(i)})$, dla $i=1,\\ldots,m$\n",
    "    * Oblicz wartość wyjścia $o^{(i)} = g(\\theta^{T}x^{(i)}) = g(\\sum_{j=0}^{n} \\theta_jx_j^{(i)})$\n",
    "    * Wykonaj aktualizację wag (tzw. *perceptron rule*):\n",
    "    $$ \\theta := \\theta + \\Delta \\theta $$\n",
    "    $$ \\Delta \\theta = \\alpha(y^{(i)}-o^{(i)})x^{(i)} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "$$\\theta_j := \\theta_j + \\Delta \\theta_j $$\n",
    "\n",
    "Jeżeli przykład został sklasyfikowany **poprawnie**:\n",
    "\n",
    "* $y^{(i)}=1$ oraz $o^{(i)}=1$ : $$\\Delta\\theta_j = \\alpha(1 - 1)x_j^{(i)} = 0$$\n",
    "* $y^{(i)}=-1$ oraz $o^{(i)}=-1$ : $$\\Delta\\theta_j = \\alpha(-1 - -1)x_j^{(i)} = 0$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Czyli: jeżeli trafiłeś, to nic nie zmieniaj."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "$$\\theta_j := \\theta_j + \\Delta \\theta_j $$\n",
    "\n",
    "Jeżeli przykład został sklasyfikowany **niepoprawnie**:\n",
    "\n",
    "* $y^{(i)}=1$ oraz $o^{(i)}=-1$ : $$\\Delta\\theta_j = \\alpha(1 - -1)x_j^{(i)} = 2 \\alpha x_j^{(i)}$$\n",
    "* $y^{(i)}=-1$ oraz $o^{(i)}=1$ : $$\\Delta\\theta_j = \\alpha(-1 - 1)x_j^{(i)} = -2 \\alpha x_j^{(i)}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Czyli: przesuń wagi w odpowiednią stronę."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Perceptron – zalety\n",
    "\n",
    "* intuicyjny i prosty\n",
    "* łatwy w implementacji\n",
    "* jeżeli dane można liniowo oddzielić, algorytm jest zbieżny w skończonym czasie"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Perceptron – wady\n",
    "\n",
    "* jeżeli danych nie można oddzielić liniowo, algorytm nie jest zbieżny"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "def plot_perceptron():\n",
    "    plt.figure(figsize=(12,3))\n",
    "\n",
    "    plt.subplot(131)\n",
    "    plt.ylim(-0.2,1.2)\n",
    "    plt.xlim(-0.2,1.2)\n",
    "\n",
    "    plt.title('AND')\n",
    "    plt.plot([1,0,0], [0,1,0], 'ro', markersize=10)\n",
    "    plt.plot([1], [1], 'go', markersize=10)\n",
    "\n",
    "    ax = plt.gca()\n",
    "    ax.spines['right'].set_color('none')\n",
    "    ax.spines['top'].set_color('none')\n",
    "    ax.xaxis.set_ticks_position('none')\n",
    "    ax.spines['bottom'].set_position(('data',0))\n",
    "    ax.yaxis.set_ticks_position('none')\n",
    "    ax.spines['left'].set_position(('data',0))\n",
    "\n",
    "    plt.xticks(np.arange(0, 2, 1.0))\n",
    "    plt.yticks(np.arange(0, 2, 1.0))\n",
    "\n",
    "\n",
    "    plt.subplot(132)\n",
    "    plt.ylim(-0.2,1.2)\n",
    "    plt.xlim(-0.2,1.2)\n",
    "\n",
    "    plt.plot([1,0,1], [0,1,1], 'go', markersize=10)\n",
    "    plt.plot([0], [0], 'ro', markersize=10)\n",
    "\n",
    "    ax = plt.gca()\n",
    "    ax.spines['right'].set_color('none')\n",
    "    ax.spines['top'].set_color('none')\n",
    "    ax.xaxis.set_ticks_position('none')\n",
    "    ax.spines['bottom'].set_position(('data',0))\n",
    "    ax.yaxis.set_ticks_position('none')\n",
    "    ax.spines['left'].set_position(('data',0))\n",
    "\n",
    "    plt.title('OR')\n",
    "    plt.xticks(np.arange(0, 2, 1.0))\n",
    "    plt.yticks(np.arange(0, 2, 1.0))\n",
    "\n",
    "\n",
    "    plt.subplot(133)\n",
    "    plt.ylim(-0.2,1.2)\n",
    "    plt.xlim(-0.2,1.2)\n",
    "\n",
    "    plt.title('XOR')\n",
    "    plt.plot([1,0], [0,1], 'go', markersize=10)\n",
    "    plt.plot([0,1], [0,1], 'ro', markersize=10)\n",
    "\n",
    "    ax = plt.gca()\n",
    "    ax.spines['right'].set_color('none')\n",
    "    ax.spines['top'].set_color('none')\n",
    "    ax.xaxis.set_ticks_position('none')\n",
    "    ax.spines['bottom'].set_position(('data',0))\n",
    "    ax.yaxis.set_ticks_position('none')\n",
    "    ax.spines['left'].set_position(('data',0))\n",
    "\n",
    "    plt.xticks(np.arange(0, 2, 1.0))\n",
    "    plt.yticks(np.arange(0, 2, 1.0))\n",
    "\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqwAAADGCAYAAAAXMtIlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAWVUlEQVR4nO3df5DUd33H8dd7cRO5XJYcaUQn1YEh0Qop2IYqmBACiS1YQ1J6RB1Bo2PUOUwnV5z6MzVJUepUClYnJiZq6FG1sM4kqYBOmhFsMFQ5M57lgt4dXEhik4YcJnCXHyu8+8f3e7C3t7e3y+339nN3z8fMdzb3/X73u5/95vPm+9rvfr+fNXcXAAAAEKpUrRsAAAAAlEJgBQAAQNAIrAAAAAgagRUAAABBI7ACAAAgaARWAAAABI3ACgAAgKARWEeJmW03Mzez/ymxjsfTz4ZYfmW8fFPB/HvznutmljOzHjP7VbzsXWY2qdrvCZhozOwdZpY1s6fM7GUzO2JmPzazj5jZq4qs311QmyfM7Fkz22lmy2rxHoDxwMw+GdfU14ZY/iYz6zOzg2Z2TsGyt5lZi5k9bmYvmdlRM/upma01s8lDbO/hIrXcY2a7zOy9SbxHDGT8cEDyzOx1kp5Q9AHBJL3V3X9eZL38/xmN7v79guVXSvqxpK+4+8158++V9AFJd0l6On6djKQ3S1ooabKkn0l6t7t3V+t9ARNFHEbvkvQhSccl/UDSIUnnS1om6fWS9kl6l7s/k/e8bkmvlfSP8ayzFdXlNYrq9APu/q+j8y6A8SM+CbNH0lslLXH3XQXLHpb0tiLLvijp05JelrRT0q8l1Uv6c0kXS+qQ9E537yx4vYclXSZpvaRXJKUlXSTpOklnSfqsu38xgbeKfu7OlPAk6VOSXNKX48evD7GeS/qtpBclPSZpUsHyK+N1NhXMvzee/5Yi2zxfUku8/ICk+lrvDyamsTZJ2hDX0COSXluw7GxJX89b/qq8Zd2Sfldke9fH6x+u9XtjYhqrk6IPfy9JOijpnLz5n4zr66sF6zfnHQtnFixLSbo1Xt4h6dyC5Q/Hy+oL5l8u6YSkXkmTa71PxvPEJQGj4wZJz0v6nKTfSHqPmb16iHX/T9LXJP1R/LwRcffnJL1f0oOS3iTp4yPdJjCRmNkbJd0s6TlJy9396fzl7v6ypCZJP5E0X9G3HcPZpuhM7evN7ILqthiYGNz9MUl/L2mGpC9JkpnNknSbpC5FJ4sUzz9f0j8oCrjXuHtXwbZOuvutkr6r6Mzp35bZhocldUqqU3TcRkIIrAkzs8sUBcWsu78kaYuk8yStKPG09YoC7udLBNuyefQxsP+riutHuj1ggvmAon8rv+HuzxZboaDGPljmdi1+zI2secCEtkHSf0tqMrOrJW1W9BX9h9y9N2+96yWdI2mbu3eU2N4X4scPnUFbqOUEEViT13/waokftyj6WmHIg5q790j6J0XXxa2pUjt+qqiY5ha7OQTAkN4ePz40zHo/kfR7SX9Wxk2O71F08HzM3X83wvYBE5a7n1D0beTLiq4tnyfpX9z9JwWrllXH7r5f0jOS3hDff1KSmS1SdO3rEUWXEiAhBJcExXcmXi/psKKDmdz9kJn9VNISM3uDux8e4umbFH19/2kzu9vdXxhJW9z9FTPrkTRN0lRFlx4AGN5r48cnS63k7i+a2XOKaux8na6xV5vZrfF/ny1plqKbrl5U9T6QAhOWux8wsy2SPqwobH6myGpl1XHeOtMkvU7S/xYs+4yZvaIoP10s6a8knZT08fjyICSEwJqsRknnSvpa/JVhvxZFdxveIOn2Yk90914zW6foetZPKLpOZ6Rs+FUAjEB/jeXX+9mSPl+w3ouSlrn77lFpFTCOmdkMSf1DS01TdDb1P0eyyfix2DBKny74+4Sk97r7thG8HsrAJQHJ6v/af0vB/K2KhsW4wcxKhchvKLr7sdnMXjOShpjZ2YrOrJ6Q1DOSbQETTP9NVn9YaqX4evOpimo7v8aed3dzd5M0RdGB9aSkrWZWcpsASouPod9UdInNzYpuqrrbzOoLVi2rjmMXFjwn37lxLddLeqeimzG/bWZzKm07KkNgTYiZzZR0Rfzn/vwBhxUdzM5SdGfjlUNtw91zis7M1CsaYWAk3q7ojPov3f33I9wWMJE8Ej9eNcx6VyiqsZ/H19UN4u4vuPv3JH1E0msUDYcF4Mw1SVos6R53/4qiY+Z0nR77uF9ZdRyPMjBN0ZBzhZcDnOLuve6+U9JKRWH528OcgMIIEViTc4OirxV+rOjTX+F0f7zecHcUf0dSm6SPKirCisVF1P81xr+fyTaACWyzoq8Gb4yHxhmkoMa+PdwG3f07kvZKepeZLaxWQ4GJJL4U4EuKfphnbTx7g6IfymmKb4jqt1XRpTgr4xNKQ+m//vVb5bQhvrnr+5L+VNHNlEgIgTUBZpZSNBTOCUnvc/cPF06KbsZ6TtJfm1lmqG25+0lJn1V0Rrbis6xmNlXRAfcdin7R446K3xAwgbn7ryV9VdIfSLrfzKblLzezs+LlVyoaXqfcX666LX68tSoNBSaQ+EPitxSd3byx/8bk+NuNDyq6NOebZlYXzz+i6OzrqyX9Rxx287eXMrPPSXqfonFV/7mC5tym6EPtLfHxHwngpqtkXK1oSKrtQ32lEN+1/2+S/kbSuyXdPdTG3P0H8c/CXT7M637MzJ5WdGY3o2gQ40Ua+NOsxyt9MwD0CUXjJ79fUoeZFf406xsktUq6Nr6UZ1ju/kMz+5miEUMujwcgB1CeNYo+JN7j7j/KX+Du7WZ2u6IxVdfp9I8AfFnRB8+/k9RuZjsU/ZhP/0+zvlFRWF3m7sfKbYi7/8rM7lM0YsD1kr43gveFIdjAm9dRDWb2XUVfDax092yJ9d4i6VFJe919QXx96y/d/S1F1r1c0n/Ff37F3W/OW3avBv66zglJxxQNzdEqKStpR3y2FsAZMrO/UHR5znxFB77jii7Z+a6kbxWGVTPrlnSeu583xPb+UtHYkQ+5+9UJNh0YN+Kzo79SdD/IJcWGfYzHG98r6U8kXe7uj+QtW6Bo2MiFiq5XfVHRz7VmJd3h7n1FtvewotF9zi124ifveN4u6Y853lYfgRUAAABB41oLAAAABI3ACgAAgKARWAEAABA0AisAAACCRmAFAABA0IYLrD6a09KlS0f19ZiYzmAK3ajuD2qWKfApdKO6P6hXpjEwDSmoM6xHjhypdRMAVICaBcYO6hVjWVCBFQAAAChEYAUAAEDQCKwAAAAIGoEVAAAAQSOwAgAAIGgEVgAAAASNwAoAAICgEVgBAAAQNAIrAAAAgkZgBQAAQNAIrAAAAAgagRUAAABBI7ACAAAgaARWAAAABI3ACgAAgKBVPbA++eSTuummm7RgwQLV1dXJzNTd3V3tlwFQBdQrMHZQr5jIqh5YOzs7tXXrVjU0NGjhwoXDP6GrS2pqkjIZqbU1emxqiuYDRXT1dKlpe5My6zNK3ZZSZn1GTdub1NVDn6lUxfWqgfu/9bet7H+URL1WD/WKUZGfy1KpYHKZuXup5SUXFnPy5EmlUlEOvueee3TjjTfq0KFDmj59+uCVd+6UGhulXE7K5TRP0j5JSqejKZuVli2rtAkYx3Z27FTjtkblTuSUO5k7NT+dSis9Ka3syqyWXZxon7EkN14FFdVsRfWqIvv/LkkfHdX9jzGEeh0W9YqwFOSyU0Yvlw1Zs1U/w9pfTMPq6op2Sl/fwJ0iRX/39UXLOdOKWFdPlxq3Naov1zfg4CdJuZM59eX61LitkTMHFSi7XsX+R2XoL9VHvSJRgeey2t10tWHD4B1SKJeTNm4cnfYgeBse2aDcidJ9Jncip4176TNJYP+jEvSX2mL/o2KB57LaBdYtW8rbMS0to9MeBG9L25ZBZwoK5U7m1NJGn0kC+x+VoL/UFvsfFQs8l9UusB4/Xt31MO4df6W8vlDueqgM+x+VoL/UFvsfFQs8l9UusNbXV3c9jHv1Z5XXF8pdD5Vh/6MS9JfaYv+jYoHnstoF1lWrojvOSkmnpdWrR6c9CN6qOauUTpXuM+lUWqvn0GeSwP5HJegvtcX+R8UCz2W1C6xr15a3Y5qbR6c9CN7aBWuVnjTMP8CT0mqeT59JAvsflaC/1Bb7HxULPJclEliz2ayy2axaW1slSTt37lQ2m9Xu3btPrzRzZjSeV13d4B2UTkfzs9loPUDSzKkzlV2ZVV26btCZg3Qqrbp0nbIrs5o5lT5TibLqVex/VIb+kgzqFYkJPJdV/YcDJMms+LivixYt0q5duwbO7OqKhkhoadG8F17QvkwmOt3c3ExYRVFdPV3auHejWtpadPyV46o/q16r56xW8/zm0fjHd1wNRC5VWK8auP9f+OoLytyUGc39jzGGei2JekV48nKZjh+PrlkdvVw2ZM0mEljP1Lx587Rv377RfEmgUuPuADgS1CwCR73moV4xBozeL10BAAAA1URgBQAAQNAIrAAAAAgagRUAAABBI7ACAAAgaARWAAAABI3ACgAAgKARWAEAABA0AisAAACCRmAFAABA0AisAAAACBqBFQAAAEEjsAIAACBoBFYAAAAEjcAKAACAoBFYAQAAEDQCKwAAAIJGYAUAAEDQCKwAAAAIGoEVAAAAQSOwAgAAIGgEVgAAAASNwAoAAICgEVgBAAAQNAIrAAAAgkZgBQAAQNAIrAAAAAgagRUAAABBI7ACAAAgaARWAAAABI3ACgAAgKARWAEAABA0AisAAACCRmAFAABA0AisAAAACBqBFQAAAEEjsAIAACBoBFYAAAAEjcAKAACAoBFYAQAAEDQCKwAAAIJGYAUAAEDQCKwAAAAIGoEVAAAAQSOwAgAAIGgEVgAAAASNwAoAAICgEVgBAAAQNAIrAAAAgkZgBQAAQNAIrAAAAAgagRUAAABBI7ACAAAgaARWAAAABI3ACgAAgKARWAEAABA0AisAAACCRmAFAABA0AisAAAACBqBFQAAAEEjsAIAACBoBFYAAAAEjcAKAACAoBFYAQAAEDQCKwAAAIJGYAUAAEDQCKwAAAAIGoEVAAAAQSOwAgAAIGgEVgAAAASNwAoAAICgEVgBAAAQNAIrAAAAgkZgBQAAQNAIrAAAAAgagRUAAABBI7ACAAAgaARWAAAABI3ACgAAgKARWAEAABA0AisAAACCRmAFAABA0AisAAAACBqBFQAAAEEjsAIAACBoBFYAAAAEjcAKAACAoBFYAQAAEDQCKwAAAIJGYAUAAEDQCKwAAAAIGoEVAAAAQSOwAgAAIGgEVgAAAASNwAoAAICgEVgBAAAQNAIrAAAAgkZgBQAAQNAIrAAAAAgagRUAAABBI7ACAAAgaARWAAAABI3ACgAAgKARWAEAABC0qgfWJ554Qo2NjZoyZYoymYxWrFihw4cPV/tlAFQJNQuMHdQrJqqqBta+vj4tWbJEBw4c0ObNm9XS0qKOjg4tXrxYvb29xZ/U1SU1NUmZjNTaGj02NUXzgWLy+0wqRZ8ZAWoWSevq6VLT9iZl1meUui2lzPqMmrY3qauH/lIp6hWjIdiadfdSU0U2bdrkqVTKOzo6Ts07ePCgT5o0yTds2DD4CTt2uNfVuafT7pJfKrlL0d91ddFyIF9Bn/HR7zPD1Uytp4pQs0jSjt/s8Lov1Hn69rTrVp2a0renve4Ldb7jN9RrJahXJC3kmjV3L5lnKwm/V111lV566SXt2bNnwPxFixZJknbv3n16ZleXNGeO1Nd3atY8Sfvyn1hXJ7W1STNnVtIMjFdF+swgyfcZS2rDVULNIghdPV2ac+cc9eWGrte6dJ3aPtammVOp13JQr0hS6DVb1UsC9u/fr0suuWTQ/NmzZ6u9vX3gzA0bpFyu9AZzOWnjxiq2EGMafabqqFkkZcMjG5Q7Ubq/5E7ktHEv/aVc1CuSFHrNVjWw9vT0qKGhYdD8qVOn6ujRowNnbtlSXjG1tFSxhRjT6DNVR80iKVvatih3cpiD38mcWtroL+WiXpGk0Gu25CUBS5cu9SNHjpS9sV/84heaNm2aLrzwwgHzn3rqKT399NO69NJLT89sbR30/MckvbnYhvOfh4mrSJ8ZUkJ9prW19UfuvjSRjVcBNYtQtP62zHo16dLXUa/loF6RpNBrtqrXsE6bNk3XXXed7rrrrgHzm5qatG3bNj377LOnZ2Yy0rFjA9YbdH1N/3rPP19JMzBeFekzQ66XXJ8ZV9fEUbNISmZ9RsdeGb5eM2dn9PynqNdyUK9IUug1W9VLAmbPnq39+/cPmt/e3q5Zs2YNnLlqlZROl95gOi2tXl3FFmJMo89UHTWLpKyas0rpVOn+kk6ltXoO/aVc1CuSFHrNVjWwLl++XHv37tXBgwdPzevu7taePXu0fPnygSuvXVteMTU3V7OJGMvoM1VHzSIpaxesVXrSMAe/SWk1z6e/lIt6RZJCr9mqXhLQ29uruXPnavLkyVq3bp3MTLfccouOHTumtrY21dfXD3zCzp1SY2N04Xcud/rrinQ6mrJZadmyCt8SxrWCPnPK6PWZcfUVIzWLJO3s2KnGbY3KncgNuJkjnUorPSmt7Mqsll1MvZaLekXSgq7ZUoO0nsmIr48//rivWLHCzz33XK+vr/drr73WDx06NPQTOjvd16xxz2SiQY0zmejvzs4zeXlMBHl9xlOp0e4ztR5onJrFmNL5XKev2b7GM+sznrot5Zn1GV+zfY13Pke9nskbol6RtFBrtqpnWEdq3rx52rdv0CXhQEjG1RmbkaJmETjqNQ/1ijFgdG66OhNPPPGEGhsbNWXKFD366KNasWKFDh8+XOtmIWBPPvmkbrrpJi1YsEB1dXUyM3V3d9e6WRMGNYtKUK+1Rb2iEiHXa00Da19fn5YsWaIDBw5o8+bNmjFjhjo6OrR48WL19vbWsmkIWGdnp7Zu3aqGhgYtXLiw1s2ZUKhZVIp6rR3qFZUKuV5rGljvvvtuHTx4UPfdd5+uu+46nXfeeXrggQf0+OOPDxpnDuh3xRVX6JlnntGOHTu0cuXKWjdnQqFmUSnqtXaoV1Qq5HqtaWB94IEHNH/+fF100UWn5s2YMUOXXXaZ7r///hq2DCFLpWp+JcuERc2iUtRr7VCvqFTI9VrTlu3fv1+XXHLJoPmzZ89We3t7DVoEoBRqFhg7qFeMJzUNrD09PWpoaBg0f+rUqTp69GgNWgSgFGoWGDuoV4wnNT/3azZ4BINhhtoCUEPULDB2UK8YL2oaWBsaGtTT0zNo/tGjR4t+KgRQW9QsMHZQrxhPahpYZ8+erf379w+a397erlmzZtWgRQBKoWaBsYN6xXhS08C6fPly7d27VwcPHjw1r7u7W3v27NHy5ctr2DIAxVCzwNhBvWI8qelPs/b29mru3LmaPHmy1q1bp7Vr1+qcc87RsWPH1NbWpvr6+iRfHmNYNpuVJD300EO68847dccdd+iCCy7QBRdcoEWLFiX50hP6px6pWZwJ6nVI1CuCU8N6lUrUbE0DqyQdPnxYzc3NevDBB9Xb26trrrlGmzZt0vTp05N+aYxhxW4kkKRFixZp165dib50khuvAmoWwaFeh0S9Ijg1rFcp5MCab968edq3b99oviRQqQl/AMxHzSJw1Gse6hVjwJA1W/NhrQAAAIBSCKwAAAAIGoEVAAAAQRvuGtZRZWY/dPeltW4HgPJQs8DYQb1iLAsqsAIAAACFuCQAAAAAQSOwAgAAIGgEVgAAAASNwAoAAICgEVgBAAAQtP8HLYi+hk3C0xEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x216 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_perceptron()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Funkcje aktywacji\n",
    "\n",
    "Zamiast funkcji bipolarnej możemy zastosować funkcję sigmoidalną jako funkcję aktywacji."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "def plot_activation_functions():\n",
    "    plt.figure(figsize=(16,7))\n",
    "    plt.subplot(121)\n",
    "    x = [-2,-.23,2] \n",
    "    y = [-1, -1, 1]\n",
    "    plt.ylim(-1.2,1.2)\n",
    "    plt.xlim(-2.2,2.2)\n",
    "    plt.plot([-2,2],[1,1], color='black', ls=\"dashed\")\n",
    "    plt.plot([-2,2],[-1,-1], color='black', ls=\"dashed\")\n",
    "    plt.step(x, y, lw=3)\n",
    "    ax = plt.gca()\n",
    "    ax.spines['right'].set_color('none')\n",
    "    ax.spines['top'].set_color('none')\n",
    "    ax.xaxis.set_ticks_position('bottom')\n",
    "    ax.spines['bottom'].set_position(('data',0))\n",
    "    ax.yaxis.set_ticks_position('left')\n",
    "    ax.spines['left'].set_position(('data',0))\n",
    "\n",
    "    plt.annotate(r'$\\theta_0$',\n",
    "                 xy=(-.23,0), xycoords='data',\n",
    "                 xytext=(-50, +50), textcoords='offset points', fontsize=26,\n",
    "                 arrowprops=dict(arrowstyle=\"->\"))\n",
    "\n",
    "    plt.subplot(122)\n",
    "    x2 = np.linspace(-2,2,100)\n",
    "    y2 = np.tanh(x2+ 0.23)\n",
    "    plt.ylim(-1.2,1.2)\n",
    "    plt.xlim(-2.2,2.2)\n",
    "    plt.plot([-2,2],[1,1], color='black', ls=\"dashed\")\n",
    "    plt.plot([-2,2],[-1,-1], color='black', ls=\"dashed\")\n",
    "    plt.plot(x2, y2, lw=3)\n",
    "    ax = plt.gca()\n",
    "    ax.spines['right'].set_color('none')\n",
    "    ax.spines['top'].set_color('none')\n",
    "    ax.xaxis.set_ticks_position('bottom')\n",
    "    ax.spines['bottom'].set_position(('data',0))\n",
    "    ax.yaxis.set_ticks_position('left')\n",
    "    ax.spines['left'].set_position(('data',0))\n",
    "\n",
    "    plt.annotate(r'$\\theta_0$',\n",
    "                 xy=(-.23,0), xycoords='data',\n",
    "                 xytext=(-50, +50), textcoords='offset points', fontsize=26,\n",
    "                 arrowprops=dict(arrowstyle=\"->\"))\n",
    "\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4sAAAGKCAYAAACl0NTvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd3hUZfrG8XvSOxAIEHrokEgNCCKigIIi4KJiWVxAWXApKygqKEVWFHVRwZ+A4NqWoiAugqAioOCKsvSW0CGhk0ACCenJnN8fWQazQwlkMicz8/1cVy7zvmfOzJM45Mmdc857LIZhCAAAAACA3/MyuwAAAAAAQNlDWAQAAAAA2CEsAgAAAADsEBYBAAAAAHYIiwAAAAAAOz7X2c5SqUAxdO/eXd9//73ZZQCuwGJ2AW6A3gwUA70ZKLar9maOLAIOcPbsWbNLAAAAv0NvBkqOsAgAAAAAsENYBAAAAADYISwCAAAAAOwQFgEAAAAAdgiLAAAAAAA7hEUAAAAAgB3CIgAAAADADmERAAAAAGCHsAgAAAAAsENYBAAAAADYISwCAAAAAOwQFgEAAAAAdgiLAAAAAAA7hEUAAAAAgB3CIgAAAADADmERAAAAAGCHsAgAAAAAsENYBAAAAADYISwCAAAAAOwQFgEAAAAAdgiLAAAAAAA7hEUAAAAAgB3CIgAAAADADmERAAAAAGCHsAgAAAAAsENYhNs4fvy4RowYofbt2ysoKEgWi0UJCQnF2tdqtWrKlCmqU6eOAgIC1Lx5c3311VelWzAAAG6O3gy4NsIi3MbBgwe1aNEiVahQQR07dryhfcePH69XXnlFw4cP13fffad27drp4Ycf1rfffltK1QIA4P7ozYBrsxiGca3t19wIlCVWq1VeXoV///jHP/6hP//5zzpy5Ijq1Klzzf2SkpJUs2ZNjRkzRpMmTbLNd+nSRcnJydq5c+d1Xzs2NlabN28uUf2Ah7CYXYAboDfDZdCbAZdw1d7MkUW4jUvN6EatXLlSubm56tevX5H5fv36adeuXTpy5IgjygMAwOPQmwHX5rQji3feeafdXN++fTV06FBlZmbqvvvus9s+YMAADRgwQGfPntVDDz1kt/0vf/mLHnnkER07dkxPPPGE3fbnnntOPXv21L59+zRkyBC77ePGjVPXrl21fft2jRw50m7766+/rttuu02//vqrXnrpJbvt06ZNU4sWLbR69WpNnjzZbvvs2bPVqFEjffPNN3r77bftts+dO1c1a9bUwoULNWvWLLvtixcvVqVKlfTpp5/q008/tdv+7bffKigoSDNnztSiRYvstq9du1aSNHXqVC1fvrzItsDAQH333XeSpFdffVVr1qwpsr1ixYq26wLGjh2r3377rcj2GjVqaN68eZKkkSNHavv27UW2N2zYUHPmzJEkDR48WPv37y+yvUWLFpo2bZqkwh/8x48fL7K9ffv2mjJliiTpwQcf1Llz54ps79Kli8aPHy9Juvfee5WVlVVke0REhBYvXqwjR45owIABtvkLkbE6X6ODDG8/OdKpz0Yqsv80hz4nUBYlvNGjpE/BkcWSozfTm92qN1/y+/de48aNdfz4cd1xxx227QMGDFDTpk116623KiYmRhUrViyy//++97Zs2aLWrVvbtvPeWyvJM997999/v0aPHi3JvX7uXfp/6gAcWQQklUpQBAAAjpWfny8fHx+7+fDwcNv2K5kzZ4569OihLVu2KC8vr1RrBDwB1yzCLV3tuog6Y1aUyutxZBGegiOLZQK9GS7pRq5Z/POf/6zly5fr1KlTReYPHDighg0b6p///OcVj+D8HtcsAsV21d5s/ycbwENc+qX3xRdf1PTp05WVlSWL5fK/lY0bN+rWW2/V8uXL1aPHtX9Bjl09UZtL/ks0AABQ4RHE1NRUGYZRpDenpqbatgPuKCe/QGcv5io5PUdn03N09mKOzmXk6uzFHKVk5ColI1fnLuYqNTNXF3PytXPiPUX+jTgaYREeLzo6Wjk5OTp06JDq169vm4+Pj5ckNW3a1KzSAADwSPRmuBvDMHQ+M08nL2Tp9IVsnbyQrdMXsnQmLUdn0rJ1Ji1bSek5Op95Y6dPZ+YWKNi/9CIdYREer3v37vLz89P8+fM1ceJE2/y8efMUExOjqKgoE6sDAMDz0Jvhii5k5ikxJUPHUrJ0LDVTx1IydTw1SyfOZ+nk+Sxl5hY4/DVTMnIJi0BxLV68WJK0ZcsWSdJ3332niIgIRUREqFOnTlfcp3Llyho1apSmTJmi0NBQtWrVSgsXLtSPP/6opUuXOq12AADcUXF6s4+Pj/r376+PPvpIEr0ZZVdmbr4OJ2fo8NkMHU6+qCNnM5RwLlOJ5zJu+KjglXh7WVQx2E+Vw/xVKaTwo2KInyoF+ys82E/hIX6qGOxX+Hmwn4L8SjfOERbhVh5++OEi46FDh0qSOnXqdM3lhV977TWFhIRo+vTpOn36tBo1aqRFixapZ8+epVkuAABurzi9uaCgQAUFRY+60JthpqzcAu0/k659p9O1/0y6DiRd1MGkizpxPuv6O19FsJ+3IssHKrJcgCLLBahqucLPq4T5q3JogKqEBSg82E/eXmVnLThWQ4VH+f1qqA5Y1dGGFdeAYis7HdB10ZuBYqA3o7iS03O0++QFxZ9MK/w4laaEcxm6dkyy5+/jpdoVg1QrPEg1KgSpZniQalQIVI0KgapePlDlAn1LdTGaEmA1VAAAAACeLT07TzuOXdCO4+e18/h57Tx+QacuZBd7f28vi2pXDFLdSiGqFxGsqEqFH7UrBqtyqL+8ytBRQUcgLAIAAABwO4Zh6GhKpjYeSdHWo6namnhe+5PSi3XE0Msi1akYrEZVQ9WoaqgaVglV/cohqlMxWH4+XqVffBlBWAQAAADg8gzD0JGzGfrt8Dn9duicNiWk6ExaznX3C/D1UtPIMEVXK6foamFqWi1MDauEKsDX2wlVl22ERQAAAAAu6ezFHP1y4Kx+PpCsXw+e0+m0a59S6mWRGlcNU8ta5dW8Rnk1q1lO9SNC5OPtOUcLbwRhEQAAAIBLKLAa2nH8vH7ck6S1+5O0+0TaNR8f6u+j1nUqqE2dcLWqVUHNapQr1fsSuhu+UwAAAADKrKzcAv18IFk/xJ3R2n1JOpeRe9XHhvr76Na6FdW+XkW1qxuuxlXDytStKFwNYREAAABAmXIxJ19r9pzR97tPa+2+ZGXlFVzxcd5eFrWqVV53NIhQx4YRiqkWximlDkRYBAAAAGC67LwCrd2XpGU7TmrNniTl5Fuv+LhKIf7q3DhCnRtXVof6lRQa4OvkSj0HYREAAACAKQzD0KaEVP1r63Gt2HlK6Tn5V3xc/coh6hZdRfc0rapbqpdzu/sZllWERQAAAABOdfpCthZtPqbFW47raErmFR/TuGqoetwSqXtvqar6lUOdXCEkwiIAAAAAJyiwGlq7L0mfbzyqH/cmyWrYP6ZOxSD1al5NPZtXU4MqBESzERYBAAAAlJrzmblauOmY/vlbok6cz7LbHhbgo/ubV9ODrWqoVa3yslg4xbSsICwCAAAAcLiDSen66JcjWrLthLLz7Berua1eRT3atpbuaVpFAb7eJlSI6yEsAgAAAHCISwvWzPn5kFbvSbLbXiHIV31ja+rRtrUUVSnYhApxIwiLAAAAAErEMAz9tC9J7/94UFuPnrfb3jQyTAM61FGv5tU4iuhCCIsAAAAAborVamjVnjP6vx8PaPeJNLvtXZtU0eA76qpNnQpci+iCCIsAAAAAbohhGFq7L1l/X7lP8aeKhkQ/by/1aVVdgzrWVf3KISZVCEcgLAIAAAAoto1HUvT3lXu1KSG1yLy/j5f+eGttDelUV1XCAkyqDo5EWAQAAABwXQeTLmrKt3u0Zm/RhWsCfL30p/Z1NKhjlCqHEhLdCWERAAAAwFWdu5ij6WsOaP5/jqrAatjmfb0terxtLQ3rXJ+Q6KYIiwAAAADs5BdYNXdDot5ZtV/p2fm2eYtF+kOL6hp1d0PVDA8ysUKUNsIiAAAAgCI2JaRo/Ne7tfd0epH5dnXDNa5HU8VUL2dSZXAmwiIAAAAASVJKRq5eW7FHX209XmS+TsUgvdyjqbo2qcwtMDwIYREAAADwcIZhaOn2k/rb8nilZOTa5gN9vTW8c30N6hglfx9vEyuEGQiLAAAAgAc7npqpl5fs1rr9yUXm742pqnH3N1X18oEmVQazERYBAAAAD2QYhhZuOqZXl8crI7fANl+tXIAm/yFGnRtXMbE6lAWERQAAAMDDnEnL1pivduqnfZePJlosUv/2dTS6WyOF+BMTQFgEAAAAPMqKnaf00pJdupCVZ5urWylYf3+4uVrXrmBiZShrCIsAAACAB8jMzdffvonXF5uOFZl/skOUnu/WSIF+LGCDogiLAAAAgJuLP5mmEZ9v1aHkDNtc9fKBmvpwc7WvV9HEylCWERYBAAAAN2UYhhZsPKpJy+KVW2C1zd/fLFKv97lFYQG+JlaHso6wCAAAALihzNx8vbxkt5ZsO2GbC/Lz1iu9ovVw6xqyWCwmVgdXQFgEAAAA3Myh5IsaOm+r9p1Jt801iQzTjMdbqm5EiImVwZUQFgEAAAA38uPeM3rm8+1Kz8m3zT3cuoZefSBGAb4sYoPiIywCAAAAbsAwDH2w7rDeWrlXhlE45+fjpVd7R+uRNrXMLQ4uibAIAAAAuLjsvAK9+NVOLd1+0jZXvXygZj/RWjHVy5lYGVwZYREAAABwYcnpORr0z83acey8ba5tnXDN7NdKlUL8TawMro6wCAAAALiog0npGvDJJh1PzbLNPda2lib1ipafj5eJlcEdEBYBAAAAF/TrobN6eu4WpWUXLmTjZZEm9oxW/9vqmFsY3AZhEQAAAHAxS7ef0OgvdyivoHAlmyA/b/3fYy3VpUkVkyuDOyEsAgAAAC7kk/VHNOmbeNu4cqi/Ph7QhoVs4HCERQAAAMAFGIaht3/Yr/d/Omiba1glRJ8ObKtq5QNNrAzuirAIAAAAlHEFVkPjl+7Wgv8ctc21rl1BH/dvo3JBviZWBndGWAQAAADKsPwCq577ckeReyh2blxZMx5vpUA/bxMrg7sjLAIAAABlVG6+Vc98sU3f7T5tm+vTsrrefKiZfL25NQZKF2ERAAAAKIOy8wo0bP5WrdmbZJvr166W/tYrRl5eFhMrg6cgLAIAAABlTHZegQbP3aKf9yfb5p66PUrjejSRxUJQhHMQFgEAAIAyJCe/QEP+JygOu6ueRt/TiKAIpyIsAgAAAGVEbr5VQ+dt1brfBcWRXRtoZNeGJlYFT8VVsQAAAEAZkFdg1fAFRa9R/Gvn+gRFmIawCAAAAJiswGpo1MLt+iH+jG1u6J31NOpugiLMQ1gEAAAATGQYhsZ9vUvLd56yzQ2+o66e78Y1ijAXYREAAAAwiWEYeuO7vfp84zHbXP/2tTX23sYERZiOsAgAAACYZObaQ5r982HbuE/L6prYM5qgiDKBsAgAAACY4PONR/X3lfts465NqujNh5rJy4ugiLKBsAgAAAA42Zo9Z/Tykl22cfu6FfX+4y3l682v5yg7eDcCAAAATrTtaKqGLdgqq1E4jqkepg/7xyrA19vcwoD/QVgEAAAAnOTI2Qw99dlmZedZJUk1KgTq4wFtFOLvY3JlgD3CIgAAAOAE5y7mqP/HG5WSkStJqhDkq8+ebKvKoQEmVwZcGWERAAAAKGXZeQUaPHeLjqZkSpICfL300YA2qhcRYnJlwNURFgEAAIBSZBiGXvxqp7YkpkqSLBbpvUdbqlWtCiZXBlwbYREAAAAoRe+tOail20/axi/f10T3RFc1sSKgeAiLAAAAQClZtuOk3l293zZ+rG0tPXV7lIkVAcVHWAQAAABKwc7j5/X8lzts49vrV9LfekfLYrGYWBVQfIRFAAAAwMGS0rM1ZO4W5eQX3iKjbkSwZvyxlXy9+fUbroN3KwAAAOBAuflWDZ23VacuZEuSwgJ89FH/NioX6GtyZcCNISzCrRw7dkwPPfSQypUrp7CwMPXp00dHjx4t1r4Wi+WKH9u3by/lqgEAcF+e1psNw9DEZbu1+b8rn3pZpP97vJWiKgWbXBlw43zMLgBwlMzMTHXu3Fn+/v767LPPZLFYNG7cON11113auXOngoOv/0N6wIABGjJkSJG5hg0bllbJAAC4NU/szfP/c1SfbzxmG4+5t7E6NYwwsSLg5hEW4TY+/PBDHT58WPv27VP9+vUlSc2aNVODBg00e/ZsPfvss9d9jurVq6tdu3alXSoAAB7B03rz1qOpmvRNnG38QItq+nPHuiZWBJQMp6HCbSxbtkzt2rWzNSNJioqKUocOHbR06VITKwMAwDN5Um8+ezFHw+ZvVV6BIUmKrhamNx5sxsqncGmERbiNuLg4xcTE2M1HR0crPj6+WM8xa9Ys+fv7KygoSJ07d9a///1vR5cJAIDH8JTenF9g1V8/32Zb0KZcoK8+6NdaAb7eJlcGlAxhEW4jJSVFFSpUsJsPDw9Xamrqdffv16+fZs6cqdWrV2vOnDk6d+6cOnfurLVr117x8XPmzFFsbKxiY2OVnJxc0vIBAHA7ntKb3161X78eOidJslikaY+2UM3wIKe9PlBauGYRbuVKp3oYhlGsfefOnWv7vGPHjurdu7diYmI0btw4/fLLL3aPHzx4sAYPHixJio2NvcmKAQBwb+7em1fFn9GstYds42e6NNBdjSo75bWB0saRRbiNChUqKCUlxW4+NTX1in/VvJ7Q0FD16NFDmzZtckR5AAB4HHfvzcdTM/Xcosu38birUYT+2rmBiRUBjkVYhNuIjo5WXFyc3Xx8fLyaNm16U89pGAYXpgMAcJPcuTfn5ls1fME2pWXnS5Kqlw/Uu4+0kJeX+bUBjkJYhNvo1auXNmzYoMOHD9vmEhIStH79evXq1euGny8tLU0rVqzQrbfe6sgyAQDwGO7cm/++cq+2HzsvSfLxsui9x1qqfJCfyVUBjmW5zjnjxTuhHCgDMjIy1Lx5cwUGBmry5MmyWCwaP3680tPTtXPnToWEhKjOmBW2xz8ZtEUTJkyQJE2dOlX79u3TXXfdpWrVqikxMdE2t2bNGnXs2PGarx0bG6vNmzeX6tcHuAn+5F5y9Ga4jOL0ZklKTExUvXr1NGHCBJfozavjz2jQPy8/90v3NdbgO+qVymsBTnDV3swCN3AbwcHB+vHHHzVq1Cg98cQTMgxDXbp00bRp02zN6PesVqvt80aNGmnJkiVasmSJLly4oLCwMHXo0EEfffSR2rZt68wvAwAAt1Hc3mwYhgoKClyiN588n6XRi3fYxp0bV9ag2+uaVg9QmjiyCI/y+yOLCW/0cNjzcmQRKDaOLJYcvRkohtLozQVWQ49/uEH/OVK4aE9kuQB9+9eOqhDM6adwaVftzVyzCAAAABTDrLUHbUHRyyJNf7QlQRFujbAIAAAAXMeWxFS9u/qAbTyicwO1jQo3sSKg9BEW4RDnz5/Xq6++qhYtWig0NFTh4eHq3Lmzvv32W7NLAwDAI9GbHSctO0/PfLFNBdbCs8Bja1fQiM71Ta4KKH0scIMSW7t2rR5//HGdOnWqyPxPP/2ktWvXatasWRoyZIhJ1QEA4HnozY414evdOp6aJUkKDfDRtEdbyMebYy5wf7zLUSJr167Vfffdp1OnTqlfv37atGmTUlNTtXHjRrVv316GYejZZ5/V8ePHzS4VAACPQG92rGU7Turr7Sdt4zf6NFONCkEmVgQ4D2ERNy0pKUmPPvqosrKy9NZbb2nu3LmKjY1V+fLl1aZNGy1dulQhISHKzMzUggULzC4XAAC3R292rFMXsjRuyS7b+OHWNdSjWaSJFQHORVjETRs9erTOnDmj+++/X88//7zd9oiICHXo0EGStG7dOmeXBwCAx6E3O47Vamj0lzuUlp0vSaoZHqiJvaJNrgpwLsIibsrevXs1f/58WSwWvfXWW1d9XEREhCQpMTHRWaUBAOCR6M2O9emvCVp/8JwkyWKR3unbQiH+LPcBz0JYxE2ZPXu2rFarunbtqiZNmlz1cXl5eUX+CwAASge92XEOnEnXG9/vtY2f7lRPbepwmwx4HsIibpjVatUXX3whSfrjH/94zcempBTeuDYwMLDU6wIAwFPRmx0nr8CqUYu2KzffKklqGhmmUV0bmlwVYA7CIm7Y9u3bdfr0aUnSgAEDZLFYrvqxatUqSVLNmjXNLBkAALdGb3acmT8d0u4TaZIkPx8vTXu0hfx8+JUZnol3Pm7YzVwQ36BBg1KoBAAASPRmR9l94oL+78cDtvFzdzdUwyqhJlYEmIuwiBu2detWSVK3bt2UlZV11Y8lS5bY9mnVqpVZ5QIA4PbozSWXk1+g0V/uUL7VkCS1rl1BgzrWNbkqwFyERdyw/fv3S5Jq1aqlgICAq35s2LDBts8dd9xR5DkWLlyoNm3aKDAwUBUrVlTfvn11+PBhp34dAAC4C3pzyb235oD2nk6XJAX4emnqw83l7WUxuSrAXIRF3LDjx49LksLDr70q2IoVKyRJTZo0Ua1atWzzH3zwgR599FH5+vrq3Xff1bPPPqsff/xR7du319GjR0uvcAAA3BS9uWR2HDuvWWsP2cYvdm+sqErBJlYElA3cLAY3LCsrS5IUEBBw1cfs3btXu3fvliQ98cQTtvmUlBS98MILatGihdatWydfX19J0r333qs2bdropZde0rx580qxegAA3A+9+ebl5Bfo+cU79N+zT9Wubrj6t69jak1AWcGRRdwwPz8/SVJGRsZVHzNz5kxJhctyDxgwwDb/9ddfKz09Xc8884ytGUmF10107txZX331lTIzM0uncAAA3BS9+ebN+OmQ9p+5KEkK9PXWWw82lxennwKSCIu4CbVr15Yk7du374rbjxw5ojlz5kiShg8frsjISNu2jRs3SpI6dOhgt9/tt9+u7Oxs2189AQBA8dCbb078yTTN/OmgbfxC90aqVTHIxIqAsoWwiBvWqVMnSdIPP/ygkydPFtmWkZGhRx55RDk5OWrYsKEmTpxYZPuJEyckSTVq1LB73ktzl667AAAAxUNvvnF5BVY9v/jy6qextStw+inwPwiLuGEDBw6Uj4+PcnJy1KtXL23atEmpqan64Ycf1KFDB23atEnh4eH68ssvFRxc9OLwS6ex+Pv72z1vYGBgkccAAIDioTffuDk/H1bcyTRJkr+Pl956qBmnnwL/g7CIG9akSRPbXyW3bNmitm3bKjw8XN26ddOOHTtUt25drVmzRs2aNbPbNyio8NSOnJwcu23Z2dlFHgMAAIqH3nxjDiZd1PTVB2zjZ+9uqLoRISZWBJRNhEXclHHjxmnhwoVq3769wsLCFBoaqtatW+utt97Srl271KJFiyvuV716dUlXPp3l2LFjkq58GgwAALg2enPxWK2GXvrXLuUWWCVJzWqU01O3R5lcFVA2cesM3LS+ffuqb9++N7RP27ZtNXv2bP32229q0KBBkW3r169XQECAYmJiHFkmAAAeg958fZ9vOqqNCSmSJB8vi958sJl8vDl+AlwJ/zLgVL1791ZISIimT5+u/Px82/y2bdu0Zs0a9enTx+1OdQEAoCzzpN58+kK23vh2r208pFNdNYkMM7EioGzjyCKcqmLFinrjjTc0fPhwderUSX/605907tw5vfvuu6pYsaJee+01s0sEAMCjeFJvnrB0t9JzCgNx3UrBGtG5wXX2ADwbYRFON2zYMIWHh2vq1KkaOXKkAgMD1aVLF02ZMkV16tQxuzwAADyOJ/Tm73ef0g/xZ2zj1/vcogBfbxMrAso+wiJM8dhjj+mxxx4zuwwAAPBf7tyb07LzNGFpnG38WNuaale3ookVAa6BaxYBAADg1qau3Kek9MJbg0SE+mvMvU1MrghwDYRFAAAAuK1tR1M1d0OibTyxZ1OVC/Q1sSLAdRAWAQAA4JbyCqwa+69dMozC8Z2NItTjlkhziwJcCGERAAAAbumT9Ue093S6JCnA10uv9o6RxWIxuSrAdRAWUWbs379fr776qtllAACA/9q7d69ef/11s8u4Kbn5Vr276oBtPLJrQ9UMd4/7RQLOQlhEmREZGak5c+Zo7dq1ZpcCAABU2JtnzJihX375xexSbohhGDp5IUtZeQWSpMZVQ/XU7VEmVwW4HsIiyozQ0FBNnz5dTz/9tHJycswuBwAAj1euXDm9++67evrpp5Wbm2t2OcX2Q/wZpWfnS5IsFum1P9wiX29+7QVuFP9qUKb84Q9/UIMGDfTWW2+ZXQoAAJD08MMPq2bNmnrnnXfMLqVYMnLyNWnZ5XsqPtqmllrXrmBiRYDrIiyiTLFYLHr//fc1bdo0HTx40OxyAADweBaLRTNmzNDUqVN15MgRs8u5rvfWHNDJC9mSpIrBfnqxeyOTKwJcF2ERZU7t2rU1ZswYDR06VMalta4BAIBp6tatq+eee07Dhg0r07153+l0ffTL5UA79r4mKh/kZ2JFgGsjLKJMGjlypE6fPq0vvvjC7FIAAICk5557TomJiVq8eLHZpVyR1Wpo3Ne7lG8tDLPBfj56sFV1k6sCXBthEWWSr6+vZs+ereeee07nz583uxwAADyen5+fZs+erVGjRiktLc3scuz8a9sJbUpIlST5eFlUrXwg91QESoiwiDKrffv26tWrl8aOHWt2KQAAQNLtt9+u7t27a9y4cWaXUsSFzDxN+XaPbTyoY10F+PJrLlBS/CtCmTZlyhQtXbpUGzZsMLsUAAAg6c0339SiRYu0efNms0uxeXvVPp3LKLy1R7VyAfprl/omVwS4B8IiyrQKFSpo6tSpGjJkiPLy8swuBwAAj1exYkW99dZbGjJkiPLz880uR7tPXNC8DYm28fj7myrIz8fEigD3QVhEmffYY4+pcuXKmj59utmlAAAASU888YTKlSunGTNmmFpH4aI2u/XfNW10R8MIdY+pampNgDshLKLMs1gsmjVrlt544w0lJiZefwcAAFCqLvXmV199VcePHzetji+3HNP2Y4UL4fl5e2lSr2gWtQEciLAIl1C/fn0988AjNG8AACAASURBVMwzGjFiRJm+vxMAAJ6iUaNGGjZsmJ555hlTXv98Zq7e+G6vbTz4jrqKqhRsSi2AuyIswmW88MIL2r9/v77++muzSwEAAJLGjh2rXbt2afny5U5/7bd/2K/UzML1DKqXD9Swu1jUBnA0wiJchr+/vz744AP99a9/VXp6utnlAADg8QICAjRz5kwNHz5cGRkZTnvd3ScuaP5/ii5qE+jn7bTXBzwFYREu5c4771SXLl00YcIEs0sBAACSunbtqttvv12TJk1yyusZhqGJy+Jsi9p0bFBJ3aKrOOW1AU9DWITLmTp1qhYsWKCtW7eaXQoAAJD09ttv69NPP9XOnTtL/bX+tfWEtiSmSpJ8vS16hUVtgFJDWITLqVSpkqZMmaIhQ4aooKDA7HIAAPB4VapU0eTJkzVkyBBZrdZSe5207DxN+d2iNk/dXlf1IkJK7fUAT0dYhEsaOHCgAgMDNWvWLLNLAQAAkgYNGiSLxaIPP/yw1F5j+uoDOnsxR5JUNSxAIzqzqA1QmgiLcEkWi0UffPCBJk2apJMnT5pdDgAAHs/Ly0sffPCBxo0bpzNnzjj8+Q+cSddnvybYxi/1aKJgfx+Hvw6AywiLcFlNmzbV4MGDNWrUKLNLAQAAkpo1a6aBAwfq2WefdejzGoahV76JU/5/V7W5NSpcPZtFOvQ1ANgjLMKljRs3Tps3b9b3339vdikAAEDSxIkTtX79eq1atcphz7ky7rTWHzwnSfKyiEVtACchLMKlBQYGaubMmRo6dKgyMzPNLgcAAI8XHBys999/X0OHDlV2dnaJny8rt0CvLt9jGz/RrraaRIaV+HkBXB9hES6vW7duatu2rSZPnmx2KQAAQNL999+vZs2a6fXXXy/xc32w7pBOnM+SJIUH++nZuxuV+DkBFA9hEW7h3Xff1Ycffqi4uDi7bfn5+bpw4YIJVQEA4LmmT5+umTNnau/evXbb8vPzlZaWdt3nOJaSqQ/WHbKNn+/WSOWCfB1aJ4CrIyzCLURGRmrSpEl6+umn7e7v9NNPP2ngwIEmVQYAgGeqUaOGxo8fr6efflqGYRTZtmrVKg0aNOi6z/H6t3uUk1/Y12Oqh6lvbM1SqRXAlREW4TaGDBmi3NxcffLJJ0Xmw8LCdOzYMZOqAgDAcw0fPlzp6en65z//WWS+OL3514Nn9d3u07bxpF7R8vZiURvAmQiLcBve3t6aPXu2XnrpJSUnJ9vmq1atWir3ewIAANd2qTe/+OKLOnfunG3+er05v8CqV765fGlJn5bV1bp2eKnWCsAeYRFupUWLFurXr59Gjx5tm6tSpYrOnDljdwoMAAAofbGxserbt69eeOEF21yVKlV0+vTpq/bmeRsStf/MRUlSkJ+3Xry3sVNqBVAUYRFuZ9KkSVq7dq1++uknSVJAQIACAwOVmppqcmUAAHimyZMna+XKlfr3v/8tSQoJCZGXl5cuXrxo99hzF3P0zqr9tvGIzg1UJSzAabUCuIywCJeWkZGh4cOHKzEx0TYXEhKi9957T08//bRycnIkcSoqAADOkp6erhEjRujo0aO2ubCwME2bNs22voBU2JtPnz5tt//bq/YrLTtfklSnYpCevL2OU+oGYI+wCJcWFBSkyMhItW7dWm+++aby8vIkSb1791aTJk305ptvSrp8ugsAAChdISEhioiIUKtWrTR16lRbb37wwQcVFRWlqVOnSrpyb9594oI+33g5ZI6/v6n8fbydVzyAIgiLcGkWi0Uvv/yyNm7cqHXr1qlly5a2U1z+7//+T++9954OHDhw1b9eAgAAx7JYLJowYYI2bNigVatWqVWrVlq/fr0sFotmzJihd955R4cOHbI768cwDE36Jk6XLmPs1DBCnRtXNumrACARFuEm6tatqxUrVmjSpEl6/PHH9eSTTyowMFAvvfSS/vKXv9gWuQEAAM5Rv359ff/99xo3bpz69u2rQYMGKTQ0VM8//7yGDRtmd2Txm52ntCmhcH0BHy+LJvRsKouFW2UAZiIswm1YLBa1bdtWrVq10vz581W5cmXNnTtXJ0+e1OnTp697ZDE7O1vPP/+8IiMjFRgYqPbt2+vnn392UvUAALgfi8Wi2267TbGxsZo7d64iIiL0xRdf6MiRIzp79qztD7mZufl6fcUe234DO9RRvYgQejNgMh+zCwAcJTMzU507d5a/v78WLlyoI0eO6OWXX1Z+fr4SExPVu3dvqdbV93/qqae0YsUK/f3vf1fdunU1Y8YMdevWTb/99ptatGjhvC8EAAA38b+9+fDhwxo3bpwKCgp0/PhxBQYGSpJmrT2k02nZkqRKIX4a0aWBJHozYDbCItzGhx9+qMOHD2vfvn2qX7++JKlXr15q0KCBrFarfvvtN6nWH6+4744dO7RgwQJ9/PHHGjhwoCSpU6dOio6O1oQJE7Rs2TKnfR0AALiLq/Xmhg0bqqCgQL/++quOpWRq9s+Hbfu80K2xwgJ86c1AGcBpqHAby5YtU7t27WzNSJLq1aunjh07qlWrVnrttdeuua+vr68eeeQR25yPj48effRRrVy50nYLDgAAUHxX6s3169dXx44d1bJlS02ePFmvrdij3HyrJKlZjXJ6qHUN2770ZsBchEW4jbi4OMXExNjNR0dH68iRI3r88cevuW9UVJSCgoLs9s3NzdXBgwcdXi8AAO7uWr05ISFBNVt30fdxl9cUmNgzWl5eFtu+9GbAXIRFuI2UlBRVqFDBbj48PFypqak3ve+l7QAA4MZcszefv6BJ38Tb5v7Qsrpa165QrH0vbQdQuizGpZvZXEH37t2Ns2fPOuSFkpOTFRER4ZDnwvV54vd769atqlKliqpXr15k/sSJEzp9+rRat26tXScu2OZvqV7O9vn+/ftltVrVuHHjIvumpaXpwIEDatSokUJCQopsS05O1qV/Hzk5OVxo70Se+P42m6O+51u2bFlpGEZ3B5TksejNrssTv9/X6s1JaVnyCSv8fnhZLGpYJVS+3pdvlUFvdi2e+P42mzN68zXDoqRrbrwRsbGx2rx5s6OeDtfhid/vKlWq6IEHHtDs2bOLzA8dOlRffvmlkpOTVWfMCtt8whs9bJ8/8sgj2r59u/bt21dk30WLFumRRx7R7t27FR0dfdXXDg4OVkZGhoO+ElyPJ76/zebA7zk3TSs5erOL8sTv99V686Bhz2iV322y+BeGvee7NdKwu+oXeQy92bV44vvbbM7ozZyGCrcRHR2tuLg4u/n4+Hg1bdr0uvseOXJEmZmZdvv6+fkVuTAfAAAUz9V6838yImxBsVZ4kJ66PeqK+9KbAXMRFuE2evXqpQ0bNujw4cvLbyckJGj9+vXq1avXdffNy8vTl19+aZvLz8/XwoULdc8998jf37/U6gYAwF1dqTev2bJXaZWb2cbjejRRgK/3FfelNwPm8n7llVeutf2aG29U69atHfl0uA5P+37fcsst+uKLL7R48WJVr15d+/fv1+DBgxUQEKCPPvpIfn5+mrb6gO3xab9+oU6dOkmSqlatqr1792rGjBmqVKmSUlNTNWbMGG3cuFHz5s1TZGTkNV97zpw5GjVqVKl+fSjK097fZYGDvueTHPEkHu4VRz4Z/5acy9O+3//bm/ft26/hC7bKCK4oSerYoJJG39NIR48eVaVKlSSJ3uzCPO39XRaUdm922jWLgDMcPXpUo0aN0qpVq2QYhrp06aJp06apTp06klTkmsUBAZv0+z+WZGVl6eWXX9aCBQt0/vx5NW/eXG+++abuvPPO674u5+kDxcY1iyVHb4ZL+X1v9q3bVqHdCwOct5dF3z/TUQ2qhCohIUFRUVGaOHEivRlwvqv2ZsIiPMrVFrgpKRoSUGyExZKjN8MlZeUWqOs763TifJYk6ckOUZrQ89prCpQEvRkotrKzwM3+/fv1zDPPqFmzZgoJCVFkZKR69eqlHTt2OLsUj/HOO++oZ8+eioyMlMVi0XVOPUYxHTt2TA899JDKlSunbdu2qU+fPjp69KjZZbmt48ePa8SIEWrfvr2CgoJksViUkJBgdllua/HixXrwwQdVu3ZtBQYGqlGjRho7dqzS09PNLg2lgN7sfJ7Ym2f/fMgWFCsG++mZrg0c/hr0ZueiNzuXGb3Z6WHxhx9+0E8//aT+/fvrm2++0cyZM5WcnKxbb71VW7ZscXY5HuHDDz9UUlKSHnjgAbNLcRuZmZnq3Lmz9u7dq88++0xRUVE6cOCA7rrrLpbpLiUHDx7UokWLVKFCBXXs2NHsctze1KlT5e3trddff13ff/+9/vKXv2jWrFm6++67ZbVazS4PDkZvdj5P683HUzM1a+0h23h0t0YqF+jr0NegNzsfvdm5TOnNhmFc68PhkpOTDavVWmTu/PnzRvny5Y0nnniiNF7S4xUUFBiGYRh5eXmGJGPixInmFmSi2i8ut32UxLRp0wwvLy/jwIEDhmEYRuvWrY3Dhw8b3t7exttvv+2IUvE/Lr2PDcMwPvzwQ0OSceTIEfMKcnNJSUl2c5999pkhyVizZk1Jnvp6fYeP6384HL3Z+TytNw+dt8XWf3u897ORX2C9/k43iN7sfPRm5zKjNzv9yGKlSpVksRQ9LbZcuXJq2LChTpw44exyPIKXF3dIcbRly5apXbt2Re7xFBUVpQ4dOmjp0qUmVua+eB87V0REhN1cmzZtJImf1W6I3ux8nvQz7bdD57Ri1ynb+JWe0fL2cvzly/Rm5/Ok93FZYEZvLhP/h1NSUrR79241adLE7FKAYomLi1NMTIzdfHR0tOLj402oCCh969atkyR+VnsIejMcIb/AqleWxdnGvVtUU2yd8FJ5LXozPFFp9+YyERZHjBghwzA0cuRIs0sBiiUlJUUVKlSwmw8PD1dqaqoJFQGl68SJE5owYYK6du2q2NhYs8uBE9Cb4Qjz/3NU+84ULr4R5OetsfeW3h8f6M3wNM7ozSUOi6tXr5bFYrnux9XuhzNlyhQtWLBA77//fpHTBnBlJf1+w3H+95QtqfAaYMDdXLx4Ub1795aPj48++eQTs8tBMdCbnYvefGUpGbl6+4d9tvGwu+qrarmAUn1NejM8hbN6s09Jn+C2227Tnj17rvu4oKAgu7kPPvhAL730kiZPnqwnn3yypKV4hJJ8v+E4FSpUUEpKit18amrqFf+qCbiq7Oxs9erVS4cPH9a6detUo0YNs0tCMdCbnYvefGVTf9intOx8SVLtikEa1DGqVF+P3gxP4czeXOKwGBQUpMaNG9/wfnPnztXQoUP13HPP6eWXXy5pGR7jZr/fcKzo6GjFxcXZzcfHx6tp09K7wTDgTHl5eXrwwQe1ceNGrV69WrfccovZJaGY6M3ORW+2t/vEBX2+8fL9DSfc31T+Pt6l+pr0ZngCZ/dmU65ZXLJkiQYOHKhBgwZp6tSpZpQAlEivXr20YcMGHT582DaXkJCg9evXq1evXiZWBjiG1WrVH//4R61Zs0ZLly5Vu3btzC4JpYzeDEcxDEOvLIvTpbM/72wUoc6NK5f669Kb4e7M6M0lPrJ4o37++Wc99thjatasmQYMGKANGzbYtvn7+6tly5bOLsntbd68WQkJCbabdcbHx2vx4sWSpPvuu8/jTotxhD//+c96//331bt3b02ePFnnz59X7969VbNmTQ0ZMsTs8tzWpfftpZuEf/fdd4qIiFBERIQ6depkZmluZ9iwYfryyy/18ssvKzg4uMjP6ho1anA6qpuhNzufO/fmr7ef0ObEwgVlfL0tmnB/0yteS+ho9GZz0Judx5TefK2bMJbkzo5XM3HiREPSFT9q165dGi/p8fr373/V77mn3Tj10g2Ba7+4vMTPlZiYaPTp08cIDQ01vLy8jN69e3vc99PZrvY+7tSpk9mluZ3atWtf9ftdwpuHm31De3f4cDh6s/O5a29Oy8o1YievsvXa17+Nd+rr05udj97sPGb0Zotx7RWiWD4KbqXOmBW2zxPe6OGw542NjdXmzZsd9nyAGyv9wwvuj96MMuv1b/dozs+Fp4FWCfPXmufuVIi/009kk0RvBm7AVXtzmbjPIgAAAFzbwaR0ffzLEdv4pfuamBYUATgGYREAAAAlYhiGXlkWr3xr4YHvtnXC1at5NZOrAlBShEUAAACUyMq40/rl4FlJkpdFmtQ72imL2gAoXYRFAAAA3LSs3AK9unyPbfxEu9pqEhlmYkUAHIWwCAAAgJs246eDOnE+S5IUHuynZ+9uZHJFAByFsAgAAICbcuRshm31U0kac29jlQvyNbEiAI5EWAQAAMANMwxDE5fFKbfAKklqWau8HmpVCjcFB2AawiIAAABu2Mq4M/p5f7IkyWKRXu0dIy8vFrUB3AlhEQAAADekcFGbeNu43621FVO9nIkVASgNhEUAAADckPd/OlBkUZvR97CoDeCOCIsAAAAotkPJF4ssavNi90YsagO4KcIiAAAAisUwDE1Yult5BYYkqVWt8nq4dU2TqwJQWgiLAAAAKJZvdp7S+oPnJEleFmnyA7ewqA3gxgiLAAAAuK607Lwii9oMuC1KTauFmVgRgNJGWAQAAMB1vbtqv5LTcyRJlUP9NeruBiZXBKC0ERYBAABwTbtPXNBnvybYxuPvb6rQABa1AdwdYREAAABXVWA19PKSXbIWrmmjDvUr6v5mkeYWBcApCIsAAAC4qvn/SdSO4xckSX7eXnq1d4wsFha1ATwBYREAAABXdCYtW3//fp9tPPSueqobEWJiRQCcibAIAACAK3p1ebzSc/IlSVGVgvV0p3omVwTAmQiLAAAAsLNuf7KW7zxlG7/2QIwCfL1NrAiAsxEWAQAAUERWboHGf73bNv5Dy+q6rX4lEysCYAbCIgAAAIqYtma/jqZkSpLCAnz0co8mJlcEwAyERQAAANjEnbygf/z7iG38co8mqhTib2JFAMxCWAQAAICkwnsqjv3XLhX896aKt0aFq29sTZOrAmAWwiIAAAAkSZ/+mqCdl+6p6OOlKX1u4Z6KgAcjLAIAAEDHUzP19g+X76k44q763FMR8HCERQAAAA9nGIbGfb1bmbkFkqSGVUI0hHsqAh6PsAgAAODhvt5+Qmv3JUuSLBZpSp9b5OfDr4mAp+OnAAAAgAc7ezFHk76Jt437t6+j1rXDTawIQFlBWAQAAPBgE5fF6XxmniSpevlAPd+tkckVASgrCIsAAAAeamXcaa3Yeco2ntLnFgX7+5hYEYCyhLAIAADggS5k5Wn817tt44da19AdDSNMrAhAWUNYBAAA8ECvLo9XUnqOJCki1F/jezQ1uSIAZQ1hEQAAwMP8uPeMFm85bhu/2jta5YJ8TawIQFlEWAQAAPAgFzLzNPZfu2zjns2rqXtMpIkVASirCIsAAAAe5G/L43UmrfD000ohfprUK9rkigCUVYRFAAAAD/Hj3jP6auvl008nP3CLwoP9TKwIQFlGWAQAAPAA5zNzi5x+2qt5NXWPqWpiRQDKOsIiAACAB5iwNO53p5/6c/opgOsiLAIAALi5b3ac1LIdJ23jKX1uUQVOPwVwHYRFAAAAN3YmLVvjvt5tG/eNraG7m1YxsSIAroKwCAAA4KYMw9ALi3fqQlaeJKl6+UCNv7+pyVUBcBWERQAAADc1/z9HtW5/siTJYpHe7ttcoQG+JlcFwFUQFgEAANzQwaSLmrwi3jZ+qkOU2tWtaGJFAFwNYREAAMDN5OZbNXLhNmXnWSVJDSqHaHS3RiZXBcDVEBYBAADczDur9mv3iTRJkp+3l6Y/2lIBvt4mVwXA1RAWAQAA3Mivh85q9s+HbOMXujdS02phJlYEwFURFgEAANzE+cxcPbtwhwyjcNyxQSU92SHK3KIAuCzCIgAAgBswDEMvfrVTp9OyJUkVgnz19sPN5eVlMbkyAK6KsAgAAOAG5m5I1Mq4M7bxmw82U+WwABMrAuDqCIsAAAAuLu7kBU1evsc27t++tu6JrmpiRQDcAWERAADAhWXk5GvEgm3KLSi8TUbTyDCNva+JyVUBcAeERQAAABc2/uvdOnw2Q5IU5Oet9x/nNhkAHIOwCLdhtVo1ZcoU1alTRwEBAWrevLm++uqrYu07YMAAWSwWu4+RI0eWctUAANy8RZuO6V/bTtjGr/0hRnUjQkysqCh6M+DafMwuAHCU8ePHa+rUqXrttdfUunVrffHFF3r44Ye1fPly3XfffdfdPyIiQsuWLSsyFxkZWVrlAgBQInEnL2j80t228UOta+gPLWuYWJE9ejPg2giLcAtJSUmaOnWqxowZo9GjR0uS7rrrLh08eFBjxowpVkPy8/NTu3btSrtUAABKLC07T0Pnb1VOfuF1io2qhOpvvaNNrqooejPg+jgNFW5h5cqVys3NVb9+/YrM9+vXT7t27dKRI0dMqgwAAMcyDEOjF+1Q4rlMSVKwn7dm9mulIL+ydQyA3gy4PsIi3EJcXJz8/f1Vv379IvPR0YV/ZY2Pj7/ucyQlJalSpUry8fFRw4YN9eabb6qgoKBU6gUA4Gb9499H9EP87+6n+FAz1StD1yleQm8GXF/Z+hMUcJNSUlJUvnx5WSyWIvPh4eG27dfSokULtW7dWtHR0crOztaSJUs0duxYHThwQP/4xz+uuM+cOXM0Z84cSVJycrIDvgoAAK7t10Nn9cb3e23jAbfV0f3NqplY0dXRmwHXR1hEmbR69Wrdfffd131cp06dtHbtWhmGYdeMpMJTdYrjf1dWu++++xQSEqJp06bpxRdfVIMGDez2GTx4sAYPHixJio2NLdbrAABws46nZmr4gm0qsBb2tpa1yuslJ95Pkd4MeB7CIsqk2267TXv27Lnu44KCgiQV/pUyNTXVrjGlpqbatt+oxx57TNOmTdPmzZuv2JAAAHCW7LwCPT1vi1IyciVJlUL8NeuPreXn47wriujNgOchLKJMCgoKUuPGjYv9+OjoaOXk5OjQoUNFro24dD1E06ZNb7iGS3/5vNJfRQEAcBbDMPTSv3Zp94k0SZKPl0Wz+rVS1XIBTq2D3gx4Hha4gVvo3r27/Pz8NH/+/CLz8+bNU0xMjKKiom74ORcsWCCLxaI2bdo4qkwAAG7YJ+sT9K9tJ2zjib2i1abOjR+VczZ6M+D6OLIIt1C5cmWNGjVKU6ZMUWhoqFq1aqWFCxfqxx9/1NKlS6+5b2Jiop544gk9+uijql+/vnJycrRkyRJ9+umnGjJkiOrVq+ekrwIAgKLW7kvS5BWXVw3tG1tD/W6tZWJFxXcjvblLly5KTEzUwYMHJdGbgbKCsAi38dprrykkJETTp0/X6dOn1ahRIy1atEg9e/a85n6hoaEKDw/Xm2++qTNnzshisahJkyZ67733NHToUCdVDwBAUQeT0jViwTb9dz0btahZXn/rHeNSp2AWtzcXFBQoPz/fNqY3A2WD5TorUhVvuSrARdQZs8L2ecIbPRz2vLGxsdq8ebPDng9wY67zW27ZRW/2AKkZuXpg5nolnsuUJEWWC9DS4R1UOdS51ym6MnozUGxX7c1cswgAAFCG5OZb9Zf5W2xBMdDXWx/+KZagCMDpCIsAAABlhGEYemnJLm04fPmG9e8+0lwx1cuZWBUAT0VYBAAAKCOmrzmgxVuO28aj72mo7jGRJlYEwJMRFgEAAMqAxVuOa9rqA7bxQ61raNhd9a+xBwCULsIiAACAyX45cFZjvtppG3dsUElT+tziUiufAnA/hEUAAAATxZ28oL/M26L8/94jo3HVUM38Yyv5evNrGgBz8VMIAADAJInnMtT/401Kzym8x2DVsAB9MrCNQgN8Ta4MAAiLAAAApkhKz9YTH23U2Ys5kqTQAB99MrCNIssFmlwZABQiLAIAADhZWnae+n+8SUdTCu+l6O/jpY/6t1GTyDCTKwOAywiLAAAATpSZm69Bn27WnlNpkiRvL4tmPN5KbaPCTa4MAIoiLAIAADhJdl6BBv9zizYmpNjm3uhzi7o2rWJiVQBwZYRFAAAAJ8jNt2ro/K365eBZ29y4Hk30cGxNE6sCgKsjLAIAAJSy/AKrnvlim37cm2Sbe+7uhhrUsa6JVQHAtREWAQAASlF+gVWjFu3Qd7tP2+aG3VVPI7o0MLEqALg+H7MLAAAAcFd5BVaN/GK7Vuw6ZZt7skOURt/TyMSqAKB4CIsAAAClIK/Aqr9+vq3IEcU/ta+t8fc3kcViMbEyACgewiIAAICD5eZbNeLzrVoZd8Y2N+C2OprYsylBEYDLICwCAAA4UGZuvobM3aJ/H7i86ulTt0dpXA+OKAJwLYRFAAAAB7mQlacnP92kLYmptrnBd9TV2HsbExQBuBzCIgAAgAMkp+foTx9v1J5Taba5kV0b6JkuDQiKAFwSYREAAKCEEs9lqP/HG5VwLtM2N+H+pnry9igTqwKAkiEsAgAAlMCOY+f15KebdC4jV5LkZZHeeqi5Hmpdw+TKAKBkCIsAAAA3ac2eMxq+YJuy8gokSf4+Xpr+aEt1j6lqcmUAUHKERQAAgJsw/z+JGv/1blmNwnH5IF/940+xiq0Tbm5hAOAghEUAAIAbkF9g1Wvf7tEn6xNsczUqBOrTgW1Vv3KIeYUBgIMRFgEAAIopLTtPIxZs07r9yba5mOph+nhAG1UODTCxMgBwPMIiAABAMSSczdCgf27WwaSLtrnu0VX1ziPNFeTHr1QA3A8/2QAAAK5jzZ4zGrlwu9Kz821zw++qr2fvbigvL+6hCMA9ERYBAACuwmo1NG3NAb235oBtzs/HS39/qJl6t6huYmUAUPoIiwAAAFeQmpGrUYu2a+2+y9cnVi8fqFn9WqlZjfImVgYAzkFYBAAA+B+bElL+v727D6qqzuM4/rnI40VFEFAUCEUBn0YWzUm0pJRSkHJTaXfGihlnNW2mMdO1Wk1mZK3ZSddm3TTbHXVqZ0fBKZNVbHXXhronKQAACypJREFUWM0pwVDB5weSBx8BEbtCXO7dP1pZr8dSE7nce9+vf8Tf9xzvd4Yz53c/nnN+R6/8/Rudq29sHRvVr7v+9OskhQT6OrEzAGg/hEUAAID/sdnsWvXFKS3/53G13HiBoqSZY/pq/pPx8u7k5cTuAKB9ERYBAAAknau/rnm5B7TnZE3rWDezj5ZNHaqxA3o4sTMAcA7CIgAA8HifHajWwk8O6epNq50+HBOs9371C/XqFuDEzgDAeQiLAADAY9VbmvXWZ6XaXFLdOmYySbNTYvXquDhuOwXg0QiLAADAI31edl4LPy3VxYam1rHI4AAtz0zUiD4hTuwMADoGwiIAAPAoNdealL3lsLYcqHYYnzIsUoszBqqLv4+TOgOAjoWwCAAAPILdbtem/VVauvWIar/7vnU8tLOfciYN1vjBPZ3YHQB0PIRFAADg9k5caNDvPi3V12dqHcYnJ0Vq0cQB6mbm3YkAcCvCIgAAcFvXmqxa+a+T+st/Tst603sTewX56/fPDtHj8eFO7A4AOjbCIgAAcDs2m12b9lfqD9uP6dJNC9h4e5k0fXQfvTK2vwL9+BoEAD+FsyQAAHAr+8prtST/sA5W1juMPxwTrJxJQxTfs4uTOgMA10JYBAAAbuHY+Qb9oeCodh696DDeo6ufXp+QoGeG9paXl8lJ3QGA6yEsAgAAl1ZRa9F7O09o0/5K2f//WKL8vL0047G+emlMLLecAsDPwJkTAAC4pMo6i/7875PKLap0WLzGZJJ+mdhbr6bGKSrE7MQOAcC1ERYBAIBLOVtj0erCU8otqlBzi92h9nh8mH47PkEDIro6qTsAcB+ERQAA4BIOV1/V6i9OKf9gtWyOGVEj+oRobmqcHunb3TnNAYAbIiwCAIAOy263q/DEZf119xkVHr9kqD8cE6xXx8VpZGx3mUwsXgMAbYmwCAAAOhzL91Z98k2V1u4p18mL1wz1R/uHataYWEIiADxAhEUAANBhnLjQoL99dVab9leqodHqUDOZpLTBEXppTKyGRAY5qUMA8ByERQAA4FSW763adui8NhRV6OsztYZ6Zz9vZQ6PUlZyjKK7s7opALQXwiIAAGh3drtdxd/WKa+4UvkHz+lak9WwTZ/QQE175CFlDo9UF38fJ3QJAJ6NsAgAANrN8QsN2lxSpc0l1aqsu26od/Iy6cmBPTTtkYeUzPOIAOBUhEUAAPBAnbjQoK2Hzmtb6TkdPd9w2236hgVq6rAoPZvUWz26+rdzhwCA2yEsAgCANmWz2XWg8op2HLmgz8su6MRtVjOVpK7+3kobEqGpwyOVFB3MVUQA6GAIiwAA4L41NDZrz8nL2nXsknYevahLDU233c7P20tjB4TrmcTeSokPk593p3buFABwtwiLAADgnrXY7CqtqteeU5dVePySisrrZLXZb7utv4+XnkgI14TBEXoiIVyBfnz9AABXwNkaAADckc1m17ELDfrqdI32nq7R3lM1utpoXMH0hpBAXz2REK7UgT30aP9QmX35ygEAroYzNwAAMGhsblFpVb2Kvq1TUXmdir6t1RVL80/uM6hXV6XEhyklPlxJ0cHq5MUziADgygiLAAB4OJvNrvKa73Sg8ooOVNSrpOKKyqrr1dxy+9tKbwjv4qfR/UKV3C9Uj8WFKrwLq5gCgDshLAIA4EFsNrtOX76m0qqrOlRVr9Kqeh2uvqqGph+/pfSG0M6+GtEnRCNiQjSqX6j6hXdmBVMAcGOERQAAPEijtUVP/rFQP7IWjYO+YYEaFh2sYQ8Fa3hMiGLDAgmHAOBBCIsAAHgQs6+3YsM6G959GGz20dCobhoa2U2J0T/8GRLo66QuAQAdAWERAAAP82j/MEWFmDW4V1cN7h2kwb2DFBHkz1VDAIADwiIAAB7mrYyBzm4BAOACvJzdAAAAAACg4yEswm0sX75cGRkZioiIkMlkUnZ29j3tv3v3biUnJysgIEA9e/bU3Llzdf369QfTLAAAHoC5GXBthEW4jQ8//FAXL17UpEmT7nnfgwcPKjU1VeHh4crPz1dOTo7Wrl2rrKystm8UAAAPwdwMuDaeWYTbKCsrk5eXl6xWq1avXn1P+y5evFiRkZHKzc2Vj4+PJMnX11cvvviiFixYoKSkpAfRMgAAbo25GXBtXFmE2/Dy+nmHc3NzswoKCpSZmdk6GUlSZmamfH19tXnz5rZqEQAAj8LcDLg2k93+k2/lvYtX9t5ZzOv/aIt/BmhT5e+ky2KxKCUlRfv27dOAAQMUHh4uScrKylJWVpbi4uJUV1enQYMGOew7a9YsPffcc6qoqNDzzz+v4uJiDRs2rLX+2muvKSMjQ8eOHdPMmTMNn71w4UKNGzdOJSUlmjNnjqG+dOlSJScn68svv9Sbb75pqK9YsUKJiYnasWOHcnJyDPUPPvhA8fHx2rJli5YtW2aof/TRR4qKitKGDRu0atUqQz0vL0+hoaFat26d1q1bZ6hv3bpVZrNZ77//vjZu3Gio79q1S5L07rvvKj8/36EWEBCgbdu2SZKWLFminTt3OtS7d++uTZs2SZLeeOMN7d2716EeGRmpjz/+WJI0Z84clZSUONTj4uK0Zs0aSdKMGTN0/Phxh3piYqJWrFghSZo2bZoqKysd6iNHjtTbb78tSZo8ebJqamoc6mPHjtWiRYskSRMmTDA8OzNx4kTNmzdPkpSSkqJbZWZmavbs2bJYLEpLSzPUbxx7ly9f1pQpUwz1W4+9Wz2oY+/G77QN8G6G+9cmc7PkXsfoDZwfO+b50W63q7CwUIsXL1Z2dvYdjz3mZo69mzE3O29u5soiPJKPydb6s9VqlSR5exvvyg4ODm6t32rNmjVKT09XcXGxmpubH0yjAAB4IOZmoGPgyiI8TqBvJ80ZF6ffPNZXkrRnzx6NHj1aBQUFeuqppxy2HTVqlPz9/Q3/w3ar4cOHq6io6IH1DLgRrizevza7sgjcix07dig1NfWO240ZM8ZwxcNqtcrHx6f1yuKdMDcD7epH5+Z2WeCm/J309vgYuBGLxaKzZ8/ecTuz2azo6GiHsXudkEJCQiRJtbW1htrtbnMBAMATJScn68iRI3fczmw23/dnMTcDHQOroaJDMpvNSkhIaJfPio2NlZ+fn8rKyhzGGxsbdfr0aU2dOrVd+gAAoCNjbgY8D88swuP5+vpq/Pjx2rhxo8MzEHl5eWpqatLTTz/txO4AAPA8zM1Ax8CVRbiNoqIilZeXy2b7YfGaw4cPKy8vT5KUlpbWelvM9OnTtX79eofJJzs7WyNHjlRmZqZefvlllZeXa/78+ZoyZYrDSmoAAODuMTcDro2wCLexcuVKrV+/vvXvubm5ys3NlSSdOXNGMTExkqSWlha1tLQ47JuYmKjt27drwYIFSk9PV1BQkF544QUtXbq03foHAMDdMDcDrq1dVkMF3B0rrgF3jdVQ7x9zM3AXmJuBu8Z7FgEAAAAAd4+wCAAAAAAwICwCAAAAAAwIiwAAAAAAA8IiAAAAAMCAsAgAAAAAMCAsAgAAAAAMCIsAAAAAAAPCIgAAAADAgLAIAAAAADAgLAIAAAAADAiLAAAAAAADwiIAAAAAwICwCAAAAAAwICwCAAAAAAwIiwAAAAAAA8IiAAAAAMCAsAgAAAAAMCAsAgAAAAAMCIsAAAAAAAPCIgAAAADAgLAIAAAAADAgLAIAAAAADAiLAAAAAAADk91ud3YPgMszmUwFdrt9vLP7AAAAP2BuBu4fYREAAAAAYMBtqAAAAAAAA8IiAAAAAMCAsAgAAAAAMCAsAgAAAAAMCIsAAAAAAIP/Ao9YXFZFum1OAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1152x504 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_activation_functions()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Perceptron a regresja liniowa"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"reglin.png\" width=\"70%\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Uczenie regresji liniowej:\n",
    "* Model: $$h_{\\theta}(x) = \\sum_{i=0}^n \\theta_ix_i$$\n",
    "* Funkcja kosztu (błąd średniokwadratowy): $$J(\\theta) = \\frac{1}{m} \\sum_{i=1}^{m} (h_{\\theta}(x^{(i)}) - y^{(i)})^2$$\n",
    "* Po obliczeniu $\\nabla J(\\theta)$ - zwykły SGD."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Perceptron a dwuklasowa regresja logistyczna"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"reglog.png\" width=\"60%\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Uczenie dwuklasowej regresji logistycznej:\n",
    "* Model: $h_{\\theta}(x) = \\sigma(\\sum_{i=0}^n \\theta_ix_i) = P(1|x,\\theta)$\n",
    "* Funkcja kosztu (entropia krzyżowa): $$\\begin{eqnarray} J(\\theta) &=& -\\frac{1}{m} \\sum_{i=1}^{m} \\big( y^{(i)}\\log P(1|x^{(i)},\\theta) \\\\ && + (1-y^{(i)})\\log(1-P(1|x^{(i)},\\theta)) \\big) \\end{eqnarray}$$\n",
    "* Po obliczeniu $\\nabla J(\\theta)$ - zwykły SGD."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Perceptron a wieloklasowa regresja logistyczna"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"multireglog.png\" width=\"40%\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Wieloklasowa regresja logistyczna\n",
    "* Model (dla $c$ klasyfikatorów binarnych): \n",
    "$$\\begin{eqnarray}\n",
    "h_{(\\theta^{(1)},\\dots,\\theta^{(c)})}(x) &=& \\mathrm{softmax}(\\sum_{i=0}^n \\theta_{i}^{(1)}x_i, \\ldots, \\sum_{i=0}^n \\theta_i^{(c)}x_i) \\\\ \n",
    "&=& \\left[ P(k|x,\\theta^{(1)},\\dots,\\theta^{(c)}) \\right]_{k=1,\\dots,c} \n",
    "\\end{eqnarray}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Funkcja kosztu (**przymując model regresji binarnej**): $$\\begin{eqnarray} J(\\theta^{(k)}) &=& -\\frac{1}{m} \\sum_{i=1}^{m} \\big( y^{(i)}\\log P(k|x^{(i)},\\theta^{(k)}) \\\\ && + (1-y^{(i)})\\log P(\\neg k|x^{(i)},\\theta^{(k)}) \\big) \\end{eqnarray}$$\n",
    "* Po obliczeniu $\\nabla J(\\theta)$, **c-krotne** uruchomienie SGD, zastosowanie $\\mathrm{softmax}(X)$ do niezależnie uzyskanych klasyfikatorów binarnych."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Przyjmijmy: \n",
    "$$ \\Theta = (\\theta^{(1)},\\dots,\\theta^{(c)}) $$\n",
    "\n",
    "$$h_{\\Theta}(x) = \\left[ P(k|x,\\Theta) \\right]_{k=1,\\dots,c}$$\n",
    "\n",
    "$$\\delta(x,y) = \\left\\{\\begin{array}{cl} 1 & \\textrm{gdy } x=y \\\\ 0 & \\textrm{wpp.}\\end{array}\\right.$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Wieloklasowa funkcja kosztu $J(\\Theta)$ (kategorialna entropia krzyżowa):\n",
    "$$ J(\\Theta) = -\\frac{1}{m}\\sum_{i=1}^{m}\\sum_{k=1}^{c} \\delta({y^{(i)},k}) \\log P(k|x^{(i)},\\Theta) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Gradient $\\nabla J(\\Theta)$:\n",
    "$$ \\dfrac{\\partial J(\\Theta)}{\\partial \\Theta_{j,k}} = -\\frac{1}{m}\\sum_{i = 1}^{m} (\\delta({y^{(i)},k}) - P(k|x^{(i)}, \\Theta)) x^{(i)}_j \n",
    "$$\n",
    "\n",
    "* Liczymy wszystkie wagi jednym uruchomieniem SGD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Podsumowanie\n",
    "\n",
    "* W przypadku jednowarstowej sieci neuronowej wystarczy znać gradient funkcji kosztu.\n",
    "* Wtedy liczymy tak samo jak w przypadku regresji liniowej, logistycznej, wieloklasowej logistycznej itp. (wymienione modele to szczególne przypadki jednowarstwowych sieci neuronowych).\n",
    "* Regresja liniowa i binarna regresja logistyczna to jeden neuron.\n",
    "* Wieloklasowa regresja logistyczna to tyle neuronów, ile klas."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Funkcja aktywacji i funkcja kosztu są **dobierane do problemu**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 4.2. Funkcje aktywacji"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "* Każda funkcja aktywacji ma swoje zalety i wady.\n",
    "* Różne rodzaje funkcji aktywacji nadają się do różnych zastosowań."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/pawel/.local/lib/python2.7/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n",
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import math\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import random\n",
    "\n",
    "import keras\n",
    "from keras.datasets import mnist\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, SimpleRNN, LSTM\n",
    "from keras.optimizers import Adagrad, Adam, RMSprop, SGD\n",
    "\n",
    "from IPython.display import YouTubeVideo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "def plot(fun):\n",
    "    x = np.arange(-3.0, 3.0, 0.01)\n",
    "    y = [fun(x_i) for x_i in x]\n",
    "    fig = plt.figure(figsize=(14, 7))\n",
    "    ax = fig.add_subplot(111)\n",
    "    fig.subplots_adjust(left=0.1, right=0.9, bottom=0.1, top=0.9)\n",
    "    ax.set_xlim(-3.0, 3.0)\n",
    "    ax.set_ylim(-1.5, 1.5)\n",
    "    ax.grid()\n",
    "    ax.plot(x, y)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Funkcja logistyczna"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$$ g(x) = \\frac{1}{1 + e^{-x}} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Przyjmuje wartości z przedziału $(0, 1)$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Funkcja logistyczna – wykres"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1cAAAG2CAYAAACTRXz+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XmsXudh3/nfw7vvGy/3VRu1Wou1\n2HAWOk4cx0jjpq1bezpN0mXUdhq0M0Axk04GCaaDAl2AWVNMa6RB00GRNJhOWs/UbdI0ZdJ2YFmW\nLVnWZslcxEUSl7vxbrzbmT/el1ekxCvJvke8l+TnA7x43/O+5/I8NB6R/Pqc87ylqqoAAACwPls2\negAAAAA3A3EFAABQA3EFAABQA3EFAABQA3EFAABQA3EFAABQg1riqpTya6WUs6WUb6/x+eFSymQp\n5dnm45fqOC4AAMBm0VrTr/OPk/xKkn/yHvv8h6qqfrKm4wEAAGwqtZy5qqrqD5OM1fFrAQAA3Iiu\n5z1XHy+lPFdK+dellPuu43EBAAA+dHVdFvh+vpFkf1VV06WUzyb5F0nuvNaOpZQnkzyZJJ2dnR/d\nt2/fdRoiN4qVlZVs2WItFq5mXrAWc4NrMS9Yi7nBtXznO985X1XV6PvtV6qqquWApZQDSf7fqqru\n/wD7Hk/yaFVV599rv0OHDlWvvPJKLePj5nHkyJEcPnx4o4fBJmNesBZzg2sxL1iLucG1lFKeqarq\n0ffb77pkeSllRymlNF8/3jzuhetxbAAAgOuhlssCSym/keRwkq2llFNJfjlJW5JUVfUPkvyJJH+5\nlLKUZC7JF6q6TpkBAABsArXEVVVVX3yfz38ljaXaAQAAbkru1gMAAKiBuAIAAKiBuAIAAKiBuAIA\nAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiB\nuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIA\nAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiB\nuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIA\nAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiB\nuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKhBLXFVSvm1UsrZUsq31/i8\nlFL+t1LKa6WUb5VSHqnjuAAAAJtFXWeu/nGSz7zH5z+R5M7m48kk/0dNxwUAANgUaomrqqr+MMnY\ne+zyuST/pGr4apLBUsrOOo4NAACwGVyve652Jzl5xfap5nsAAAA3hdaNHsA7lVKeTOPSwYyOjubI\nkSMbOyA2nenpafOCdzEvWIu5wbWYF6zF3GA9rldcnU6y94rtPc333qWqqi8l+VKSHDp0qDp8+PCH\nPjhuLEeOHIl5wTuZF6zF3OBazAvWYm6wHtfrssAvJ/mZ5qqBH0syWVXVG9fp2AAAAB+6Ws5clVJ+\nI8nhJFtLKaeS/HKStiSpquofJPlKks8meS3JbJI/W8dxAQAANota4qqqqi++z+dVkr9Sx7EAAAA2\no023oAUAAMCHpaqqXFpaycylpcwuLGdmYSkzl5Yyc2k5swuN55mFt7d//L4dH/jXFlcAAMCmd2lp\nOdPzS5m+tJSLzefp+aXMLFy9/fbni1e9N7uwvPq8vFJ9oGOWkuwd7v7AYxRXAADAh+bymaKpucVM\nzi1man4xU3NLmZpvbs8t5uIVETQ9v3T1dvP1wvLK+x6rZUtJb0drejta09fZeB7sbs+e4e70tLek\nu73xXndHS3raW9Pd3tLcbl39vKejJT0drelpb01n25aUUvKnPuDvVVwBAADvaXG5EUdT80vvGUlT\n80tXvG5+Prf4vmHU0bplNYZ6m8+7Brve9d7qo7M1fVe+39mavo621RjaKOIKAABuESsrVS7OL2Vi\nbiHjs4sZn13IZPN5fHYxE7MLmWhuT1zx+cVLS+/567ZuKRnoakv/5Udna3YPdTXe62xLf1dr+jvb\n3t6ns3X1dV9nazpaW67T/wIfLnEFAAA3oKqqMjW/lAvTlzI2s5ALMwu5ML2QsZlLzTBqxNL47EIm\n5hYz0dxe63ajUpL+zrYMdbdloLs9I73tuWNbbwa72zLY1Z7B7mtFUuP1Rp8x2izEFQAAbAKXY2ls\nZiEXpi/lwszCNV5fjqhLGZ9dyOLytUupq60lQ91tGexuz1BPW3YOdGWwuy1D3Y1IGuxuf/vz5vNA\nV1tatgik9RBXAADwIamqKhOzizk3fSnnLjYeZy/Or76+MLOQ882zTWMza8dSb0drRnrbM9zTnt2D\nnfnI7oEM97ZnpKe9+X7H6uuh7vZ0tt0cl9ndaMQVAAB8j+YXl5uh1Iym6Us5NzV/VURdfv9awdTZ\ntiWjfR3Z2tuR3YOdeWB3f0Z6G4E03NO++los3VjEFQAANC2tVDk9MZc3J+fz1tT828/N142IunTN\nBR5KSUZ6OjLa15FtfR25c3tfRvs6Mtr79nujzUdvR6t7lG5C4goAgJteVVW5eGkpb02+HUpvR9Ol\nvDU1nzcm53Nh+lKq3/39q362vWVLtg90ZEd/Z+7Z0Z8fuvPtSLocT9v6OzLc3Z7Wli0b9DtkMxBX\nAADc8OYXl3NmYi5nJuZzZmIupyfmGtuTc3ljohFRswvL7/q5we627OjvzPb+zty7sz/z42/mYw/e\nvfrejoHODHW3OcvEByKuAADY1FZWqlyYWXg7mK6Mp2ZMXZhZuOpnSkm29XVk50BX7t7Zlx8+NJod\nzVi6/Ly9v/Nd9zIdOTKWw4/vu56/PW4i4goAgA21vFLlran5nBybzcnxuZwcm70qpM5MzmdhaeWq\nn+lub8nuwa7sGuzK/bsHsnuwM7ua27sHu7K9vzPtrS7R4/oSVwAAfKguL0d+cnw2J8fm8vrYbPP1\nbE6Nz+X0+FwWlt+Op1KS7X2d2T3UlQf2DObH7+9shNTA2/HU32VBCDYfcQUAwLrNLy7n5NhsI5yu\nOAP1ejOgpt+xut5Qd1v2Dnfn3p39+fR927NvuDt7h7qzd7g7uwY709Fq6XFuPOIKAIAPZG5hOSfG\nZnL8/GyOX5jJiQszOXZ+JicuzOaNyfmr9u1s27IaS08cHM7e4cbrxntd6ets26DfBXx4xBUAAKtm\nF5Zy4sJsjp+fyfELs1cF1JtTVwfUcE97Dox05+O3jeTA1p7sH+nOnmY8jfZ2uGyPW464AgC4xSwt\nr+T1sdkcPTeT756bztFzMzl2YSbHz8/k7MVLV+27tbc9+0d68ok7tubASHf2b+3JwZGe7BvpzkCX\ns09wJXEFAHCTmpxdzGvnpnP03HS+e26m+Tyd18dms7hcre63tbc9B7f25IfuGs2Bke4c2NqTAyON\nM1Eu34MPTlwBANzAlpZXcmp8bvUM1JXPV373U1tLyf6RntyxrTefvm9Hbh/tzW2jPbl9a28GugUU\n1EFcAQDcAJaWV3L8wmxefetivvPWdF49ezGvvjWdY+dnrlrGfKSnPbeN9uTH7t3eiKfR3tw22pu9\nQ11pbfG9T/BhElcAAJvI5Yh67Wwjor7zViOijp6fvupSvr3DXblrW18O3z2a20d7m4+eDHa3b+Do\n4dYmrgAANsDS8kpOjL19Juo7b13Ma2cbl/RdeSbqckR98u5tuXNbb+7a3pfbt/Wku90/42Cz8V8l\nAMCH7Pz0pbz8xsW8/OZUXmo+v3p2OgtL746oHz40mru29YkouAH5rxUAoCbzi8t57ex0Xn7zYl5+\nY6rx/OZUzk+/vbDEtr6O3L2zP5+4Y2sObe/Lndt7c8e2XhEFNwH/FQMAfI+qqsqZyfkrAqoRU0fP\nz2R5pXFfVEfrlhza0ZdPHtqWu3f2554dfTm0oy8jvR0bPHrgwyKuAADew9LySl47N50XTk/lhTNT\neeHMZF56YypT80ur++wZ6srdO/rzmft35O4d/bl7Z18OjPSkZUvZwJED15u4AgBoml9czktvXI6o\nqbx4ZjIvv3kxl5r3RnW2bcndO/rzkw/uyj3Ns1F37ehLvy/aBSKuAIBb1OTsYl54YzIvnnn7jNRr\nZ6fTvKov/Z2tuW/XQP7Mx/bn/t0DuW9Xfw5u7fFdUcCaxBUAcNM7d/FSnj89sXpp3zNHZ3Pu3/zu\n6uc7+jtz367+fOa+Hbl3VyOk9gx1pRSX9QEfnLgCAG4qE7MLef70ZL51ajLfOjWR509N5szk/Orn\nB7f25ODAlvzZH74j9zVDaqtFJoAaiCsA4IY1fWkp3z7diKhvnZrM86cnc+LC7OrnB7f25LGDw3lg\n90A+smcw9+7qT29Ha44cOZLDh+/YwJEDNyNxBQDcEOYWlvPiG40zUs+fmsxzpyZy9PxMquY9UrsH\nu/Lg3oF84bF9eXDPQO7bPZCBLgtNANePuAIANp2l5ZW8/ObFPHtyYjWkXj07vfodUtv7O/LA7sF8\n7qHd+ciegTywe8D3RwEbTlwBABvuzcn5fPP18Tx7ciLffH0iz5+ezNzicpJkqLstH9kzmE/fuz0P\n7BnMR/YMZHt/5waPGODdxBUAcF3NLizl+VOTqyH17MmJvDnVWHCivWVL7tvdny88vjcP7xvKw3sH\nrdoH3DDEFQDwoVlZqXL0/MxVZ6Veeevi6uV9+0e688Rtw3l472Ae2jeUe3b2paO1ZYNHDfD9EVcA\nQG3GZhby7MnxPPv6RL55snFW6uL8UpKkr6M1D+0bzH95z+15eN9gHtwz6D4p4KYirgCA78vKSpXv\nnpvO10+M55kT4/nGifEcPT+TJNlSkkM7+vNHHtyVh/YO5pF9g7lta2+2bHF5H3DzElcAwAcyu7CU\nZ09O5BuXY+r1iUzOLSZJhnva88i+oXz+0b15eN9gHtg9kJ4O/8wAbi3+1AMArunMxFyeaYbUMyfG\n8+IbU6v3St25rTeffWBHHtk3lI/uH8rBrT0WnQBueeIKAMjS8kpeeuNivn5ibPUSvzOTjRX8utpa\n8tDewfzlH749H90/lEf2DWWg25fzAryTuAKAW9Dk3GK+8fp4njneOCv17MmJ1e+V2jXQmUf2D+W/\n2D+UR/cP5+6dfWlr2bLBIwbY/MQVANwC3pycz9eOj+XpY2N5+vhYXnnrYqoqadlScu/O/vypx/bm\no/sbl/jtGuza6OEC3JDEFQDcZKqqyrHzM3n6+Fi+dmw8Tx8fy+tjs0mS7vaWfHT/UD77wM48emAo\nD+0dTHe7fw4A1MGfpgBwg1teqfLSG1P5WvOs1NPHx3J+eiFJYxW/xw4M5Wc+vj+PHxzOvTv70+oS\nP4APhbgCgBvM/OJynjs50Tgzdbyx+MT0pcYX9e4Z6soP3Tmaxw4O57EDw7l91Cp+ANeLuAKATW5q\nfjHPHB9fvWfqW6cms7C8kiS5a3tvPvfQrjx+cDiPHxzOzgH3SwFsFHEFAJvM2am3F5/42vHxvPzm\nVKoqad1Scv/ugfzcJw7ksQPDeXT/UIZ62jd6uAA0iSsA2GBvTM7lqaNjeerYhXz16FiOnZ9J0vh+\nqUf2D+avferOPH5gOA/ts/gEwGbmT2gAuM5OT8zlq9+9kKeOXchTx8Zy4kJjJb++ztY8fmA4X3x8\nbx4/OJL7dvX7fimAG4i4AoAPUVVVOTU+l68ebZyVeurYhZwan0uSDHS15fGDw/mZjx/IEweHc8/O\n/rRssfgEwI1KXAFAjaqqyutjs/nq0QvNS/3GcnqiEVND3W154uBI/vwPHMwTB0dy946+bBFTADcN\ncQUA63D5C3ufOja2GlRvTs0nSUZ62vPEbcP5iz98W544OJI7t/WKKYCbmLgCgO9BVVX57rnp5iV+\nY3nq6IWcvXgpSTLa15EnDg7nidtG8vHbhnP7aK/vmAK4hYgrAHgPVVXl1bPTV1zmdyHnpxeSJNv7\nO/Lx20fyxMGRPHHbcG7b6gt7AW5l4goArrCyUuWVty7mqaONlfyeOjaWsZlGTO0a6MwP3jmaj902\nnCcOjmT/SLeYAmCVuALglrayUuWlN6cal/kdvZCvHR/LxOxikmTPUFc+eWhbnrhtOB+/bSR7hrrE\nFABrqiWuSimfSfK/JmlJ8qtVVf3td3z+c0n+XpLTzbd+paqqX63j2ADwvVheqfLimanmF/ZeyNeO\njWVqfilJsm+4O5++d/vqZX57hro3eLQA3EjWHVellJYkfz/JjyU5leTpUsqXq6p68R27/rOqqn5+\nvccDgO/F8kqV505ONO6ZOjaWp4+N5eKlRkwd3NqTzz6wMx+7rRFTOwe6Nni0ANzI6jhz9XiS16qq\nOpokpZTfTPK5JO+MKwD40C0tr+T505OrS6N/9bXZzP/uf0qS3Dbakz/y0K48cXA4H7ttJNv7Ozd4\ntADcTOqIq91JTl6xfSrJE9fY74+XUn4oyXeS/NdVVZ28xj4ppTyZ5MkkGR0dzZEjR2oYIjeT6elp\n84J3MS9uXUsrVY5PreTlseW8MraSV8eXM7/c+GxXT8ljo1Ue2N6Zu4a3ZLAjSS4kExfy0jdezUsb\nOXA2lD8zWIu5wXpcrwUt/p8kv1FV1aVSyl9M8utJfuRaO1ZV9aUkX0qSQ4cOVYcPH75OQ+RGceTI\nkZgXvJN5cetYXF7Jt05NNs5KHb2QZ06MZ3ahUVN3be/N5x8bycduG8njB4cz2tdhbnBN5gVrMTdY\njzri6nSSvVds78nbC1ckSaqqunDF5q8m+bs1HBeAW8DC0kq+dWpi9TK/rx8fz9xiI6YObe/L5z+6\nZzWmRno7Nni0ANzK6oirp5PcWUo5mEZUfSHJf3blDqWUnVVVvdHc/KnElRgAXNulpeXGmanvXshX\njzXOTM0vriRJ7t7Rlz/12N587LbhPH5wJMM97Rs8WgB427rjqqqqpVLKzyf5nTSWYv+1qqpeKKX8\nzSRfr6rqy0n+ainlp5IsJRlL8nPrPS4AN4dLS8t59vW3z0w9c2I8l5ZWUkpy947+fPHxfY2l0Q8O\nZ0hMAbCJ1XLPVVVVX0nylXe890tXvP4bSf5GHccC4MY2v7icZ5tLo3/16IV88/WJ1Zi6Z0d//vQT\n+5tnpoYz2C2mALhxXK8FLQC4Rc0vLucbr4/nqaONM1PfPDmRhWZM3berP//5x/Y37pk6MJyB7raN\nHi4AfN/EFQC1mltYzjdfH2+emRrLsycnsrC8ki0luW/XQH72442YevTAcAa6xBQANw9xBcC6TF9a\nyjMnxvO1YxfytWONmFpcrrKlJA/sHsjPfeJAPnbbcB49MJz+TjEFwM1LXAHwPRmfWcjTx8fytWNj\n+drxsbxwZirLK1VatpTcv3sgf+4HDuZjB0fy6IGh9IkpAG4h4gqA9/TW1HyeOjaWp481guqVty4m\nSdpbt+ThvYP5K4dvz+MHR/LwvsH0dPhrBYBbl78FAVhVVVVOjs3lqeYlfl87PpYTF2aTJD3tLfno\ngeH81EO78vjB4Xxkz0A6Wls2eMQAsHmIK4Bb2MpKldfOTV91ZurNqfkkyVB3Wx47MJw/87H9eeLg\nSO7Z2ZfWli0bPGIA2LzEFcAtZGl5JS+9cXH1zNTTx8cyPruYJNne35EnDo7ksYPDeeLgcO4Y7c2W\nLWWDRwwANw5xBXATu/yFvY3V/MbyzInxTF9aSpLsH+nOj96zPY8fHM4TB0eyd7grpYgpAPh+iSuA\nm8j56Uv5+vHxPHNiLE8fH88LZyazuFwlSe7a3puffnh3Hjs4nMcPDGfHQOcGjxYAbi7iCuAGVVVV\njp6fydePj+Xrx8fz9RPjOXZ+JkljJb8H9wzkL/zgbXnswFAe2TeUwe72DR4xANzcxBXADeLS0nK+\nfXqqEVMnxvPMifGMzSwkaSw+8dH9w/nCY3vz6IGh3L/bSn4AcL2JK4BNanJ2Mc+83ri875nj43n2\n1EQWllaSJAe39uRH7t6Wxw4M5aP7h3P7aI/7pQBgg4krgE3g8iV+3zgxnm+8PpFnTozlO29NJ0la\nt5Tct3sgP/Ox/Xm0GVOjfR0bPGIA4J3EFcAGmL60lOdOTjRjajzfPDmRieaS6H0drXlk/1B+6sFd\n+ej+4Ty0dzBd7S7xA4DNTlwBfMjeeVbqm6+P55W3LqZqLOKXO7b15tP3bs8j+4byyP4h3y8FADco\ncQVQs4vzi3nu5GS++fo1zkp1tuahvYP58ft25JH9Q3lo72AGuto2eMQAQB3EFcA6rKxUOXZh7bNS\nd27rzY/fuyOP7B/MI/uGcruzUgBw0xJXAN+Ds1Pzee7UZJ47OZHnTk3kuZMTmZpfStI4K/XwvqF8\n5v4deXifs1IAcKsRVwBruDi/mOdPT+a5k42Y+tapiZyZnE+StGwpuXtHX37ywV15cM+As1IAgLgC\nSJKFpZW8/OZU84xUI6ZeOze9ennf/pHuPHpgOA/uHcxDewdy784BK/gBAFcRV8At5/J9Ut86NZHn\nTk7m2ZMTefHMVBaWG1/QO9LTngf3DuYnP7IrD+4dyIN7BjPU077BowYANjtxBdzUVlYay6B/+/Rk\nnm8+XjwzlelLjfukutpa8sCegfzcJw7kwT2DeXDvQHYPdqUUl/cBAN8bcQXcNJZXqhw9N70aUS+c\nnsoLZyYzs7CcJGlv3ZJ7dvbnjz68Kw/sHsiDewdzx2hvWlu2bPDIAYCbgbgCbkhLyyv57rmZPH96\nMt8+PZn/9OJcTv/+72S2GVKdbVty787+/PGP7sn9uwfywO6B3LGtN21CCgD4kIgrYNO7tLScV9+a\nzktvTK1e3vfiG1OZX2zcI9XV1pI9PcmffHTvakjdPtrjjBQAcF2JK2BTGZtZyEtvTOWlN6by4pmp\nvPjGVF47O52llcayfT3tLblv10C++Pi+PNAMqdtGe/Mf/vAPcvjwfRs8egDgViaugA2xslLlxNhs\nXjzTDKlmTL05Nb+6z/b+jtyzsz8/cve23LurP/fs7M+BkZ60+C4pAGATElfAh252YSmvvHkxL15x\nRurlNy+u3h/VsqXkjtHefPz2kdyzsy/37hzIPTv7MtLbscEjBwD44MQVUJvF5ZUcOz+Tl9+8mO+8\nebHx/NbFvD42u7pPX0dr7tnVnz/56N7cu7M/9+7qzx3betPZ5gt5AYAbm7gCvmcrK1VOT8zllTcv\n5pW3LuaVZkR999x0Fpcb90a1bCk5uLUnD+wZyJ/46J4c2tGXe3f2Z8+Q75ACAG5O4gpYU1VVuTCz\ncNVZqJffvJhX37q4+t1RSbJ7sCuHdvTl8KFtuXtHX+7a3pfbt/Wko9XZKADg1iGugKysVDkzOZfX\nzk5f/Tg3nYnZxdX9hrrbcmhHXz7/6N7ctb0vh3b05a7tvenrbNvA0QMAbA7iCm4hi8srOXFhNq+d\nnc53zzUC6tWzF/PdszOZW3z7TNRQd1vu3NaXn7h/Z+7Y1ptD2/ty147ejPZ2uKQPAGAN4gpuQtOX\nlnL8/MxqQF1+HL8ws3pPVJLsHOjMHdt684XHh3PHtt7cMdqbO7b1WqUPAOD7IK7gBrWwtJKT47M5\ndm4mx87P5Oj5mRw9N51j52dy9uKl1f22lGT/SE9uH+3Np+7Znju29ebObb25fVtvejv8EQAAUBf/\nsoJNbGWlyptT86vx1AipRkCdHJ/L8srbZ6GGe9pzcGtPfuiu0Rzc2pPbtvbk4GhPDoz0WOYcAOA6\nEFewwZZXqpyZmMvrY7M5cWE2J8ZmcnJsNkfPzeT4hZnML66s7tvV1pKDW3ty3+6B/JEHd+Xg1p7V\nx2B3+wb+LgAAEFdwHcwtLDfjaeaKiJrNybHZnBqfveo+qLaWkr1D3TmwtSefuGNr4yzUaE9u29qb\n7f0WlAAA2KzEFdSgqqqcn17IyfHZvH7h7TNQrzcj6twV90AlSV9na/aPdOfenf35zP07sn+4O/tG\nurNvuDs7B7rSskVAAQDcaMQVfAArK1XOT1/KyfG5nBqfzemJuZwabzxON7evvHwvaazEt3e4O4fv\nGs3+ke7sG+nJ/uHu7B/pzkBXmzNQAAA3GXEFadz3dPbifDOYZnN6/Ip4mpjL6fG5LCxfHU/DPe3Z\nPdiVu7b35ZOHtmXPUFf2NuNpz1C3RSQAAG4x4oqbXlVVGZ9dzBuTc3ljYr7xPDnffMzlzMR8zkzM\nZemKlfeSZGtvR3YPdeXeXf359H3bs2ewK3uGurN7qCu7B7vSYxlzAACu4F+H3NCqqsrE7GLOTM7l\nzcn5nJmcz5urEfV2SF1auvqsU+uWku39ndk12JmH9g7mJz+yM7uHGvG0pxlPzjwBAPC9EFdsWnML\nyzl7cT5vTV266vlbr87nS69+dTWe3nmvU8uWkh39ndkx0Jn7dw/k0/ftyI5mSO0Y6Mqugc6M9HZY\nNAIAgFqJK667K6Ppran5nL14KWebz29d8XxxfuldP9vesiV9bVX2b1vOvbv686m7t2XnYFd2DnRm\n50Bndg12ZatwAgBgA4granFpaTkXphdyfvpS89F8fXEhF2Yu5dwV4bRWNG3r78j2/s7cua03P3DH\n1oz2Nba3XfE82N2WP/iDP8jhw5/YgN8lAACsTVxxTVVVZXZheTWWzjUj6fzFtwPqckydm752MCVJ\nT3tLRno7MtrXkUM7+vKDd45mW39HtvV1ZvsVz5YmBwDgRieubhGLyyuZmF3M+OxCxmYWMj6zkLHZ\nhYxNN54b24uN55lGSL3zXqbLBrrasrW3PVt7O3LPrv78YE/j9da+jmzt7chIb3tGexuvu9otCgEA\nwK1BXN2ALi0tZ3JuMVNzi5mYXWzE0uxCxmbeHU+XY2lqjTNLSdLb0ZqhnrYMd7dnpLc9d27rzXBP\n+zVjabinPe2tW67j7xYAAG4M4mqDLK9UmZpbzOTcYiaaz5Nzi5mcXWi8N/v2exNXhNTk3GLmFpfX\n/HU727ZkuLs9Qz3tGe5pz96h7gz3tGeouz3DPW2N96/4fLC7LR2tzi4BAMB6iavv08LSSi7OL+bi\n/FLzsZip5vOV712cX8rFS1drvUgqAAAO+0lEQVTEUzOS1rpH6bLu9pYMdLWtPvaPdOcje9oy2N2e\nga629He1ZbD52XBP+2o0uQwPAAA2xi0XV0vLK5lZWM7MpaXGo/n67Ti6MpDeHUqXA+qdX0p7LV1t\nLenrbE1fZ2sGutqyra8zd27rWw2mwe6rnxuPRjy59A4AAG4smz6uFpZWmhG0lJlLy5m+tJTZhUYY\nTV9azuzCUqYvh9Kl5av2bezT2J5t/uwHiaKkceaoEUZtjTjqbs+e4e70X36vo/Wqzy8/9zefeztb\n09YikAAA4FaxqePqxNRK7vrv//UH2ndLSXo6WtPb0Zqejtb0tLekp6M1e7q709vReN14vzU9V2z3\ndrSku7316jDqaE2rMAIAAL4Hmzqu+ttL/vqn77oiiq4Io/ZGBHV3tKS3ozUdrVt8TxIAALBhNnVc\nDXWW/PyP3LnRwwAAAHhfrn0DAACoQS1xVUr5TCnllVLKa6WUX7jG5x2llH/W/PypUsqBOo4LAACw\nWaw7rkopLUn+fpKfSHJvki+WUu59x25/Psl4VVV3JPmfk/yd9R4XAABgM6njzNXjSV6rqupoVVUL\nSX4zyefesc/nkvx68/X/leRTxeoTAADATaSOBS12Jzl5xfapJE+stU9VVUullMkkI0nOv/MXK6U8\nmeTJJBkdHc2RI0dqGCI3k+npafOCdzEvWIu5wbWYF6zF3GA9Nt1qgVVVfSnJl5Lk0KFD1eHDhzd2\nQGw6R44ciXnBO5kXrMXc4FrMC9ZibrAedVwWeDrJ3iu29zTfu+Y+pZTWJANJLtRwbAAAgE2hjrh6\nOsmdpZSDpZT2JF9I8uV37PPlJD/bfP0nkvx+VVVVDccGAADYFNZ9WWDzHqqfT/I7SVqS/FpVVS+U\nUv5mkq9XVfXlJP8oyf9ZSnktyVgaAQYAAHDTqOWeq6qqvpLkK+9475eueD2f5PN1HAsAAGAzquVL\nhAEAAG514goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goA\nAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG\n4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goA\nAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG\n4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goA\nAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG4goAAKAG\n4goAAKAG64qrUspwKeXfllJebT4PrbHfcinl2ebjy+s5JgAAwGa03jNXv5Dk31VVdWeSf9fcvpa5\nqqoeaj5+ap3HBAAA2HTWG1efS/Lrzde/nuSPrvPXAwAAuCGVqqq+/x8uZaKqqsHm65Jk/PL2O/Zb\nSvJskqUkf7uqqn/xHr/mk0meTJLR0dGP/tZv/db3PT5uTtPT0+nt7d3oYbDJmBesxdzgWswL1mJu\ncC2f/OQnn6mq6tH32+9946qU8ntJdlzjo19M8utXxlQpZbyqqnfdd1VK2V1V1elSym1Jfj/Jp6qq\n+u77De7QoUPVK6+88n67cYs5cuRIDh8+vNHDYJMxL1iLucG1mBesxdzgWkopHyiuWt9vh6qqfvQ9\nDvJWKWVnVVVvlFJ2Jjm7xq9xuvl8tJRyJMnDSd43rgAAAG4U673n6stJfrb5+meT/Mt37lBKGSql\ndDRfb03yiSQvrvO4AAAAm8p64+pvJ/mxUsqrSX60uZ1SyqOllF9t7nNPkq+XUp5L8u/TuOdKXAEA\nADeV970s8L1UVXUhyaeu8f7Xk/yF5uv/L8kD6zkOAADAZrfeM1cAAABEXAEAANRCXAEAANRAXAEA\nANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRA\nXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEA\nANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRA\nXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEA\nANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRA\nXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRAXAEAANRgXXFVSvl8KeWFUspKKeXR\n99jvM6WUV0opr5VSfmE9xwQAANiM1nvm6ttJ/liSP1xrh1JKS5K/n+Qnktyb5IullHvXeVwAAIBN\npXU9P1xV1UtJUkp5r90eT/JaVVVHm/v+ZpLPJXlxPccGAADYTK7HPVe7k5y8YvtU8z0AAICbxvue\nuSql/F6SHdf46BerqvqXdQ+olPJkkieTZHR0NEeOHKn7ENzgpqenzQvexbxgLeYG12JesBZzg/V4\n37iqqupH13mM00n2XrG9p/neWsf7UpIvJcmhQ4eqw4cPr/Pw3GyOHDkS84J3Mi9Yi7nBtZgXrMXc\nYD2ux2WBTye5s5RysJTSnuQLSb58HY4LAABw3ax3KfafLqWcSvLxJP+qlPI7zfd3lVK+kiRVVS0l\n+fkkv5PkpSS/VVXVC+sbNgAAwOay3tUCfzvJb1/j/TNJPnvF9leSfGU9xwIAANjMrsdlgQAAADc9\ncQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUA\nAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFAD\ncQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUA\nAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFAD\ncQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUA\nAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFADcQUAAFCD\ndcVVKeXzpZQXSikrpZRH32O/46WU50spz5ZSvr6eYwIAAGxGrev8+W8n+WNJ/uEH2PeTVVWdX+fx\nAAAANqV1xVVVVS8lSSmlntEAAADcoK7XPVdVkt8tpTxTSnnyOh0TAADgunnfM1ellN9LsuMaH/1i\nVVX/8gMe5weqqjpdStmW5N+WUl6uquoP1zjek0kuB9ilUsq3P+AxuHVsTeISU97JvGAt5gbXYl6w\nFnODazn0QXZ637iqqupH1zuSqqpON5/PllJ+O8njSa4ZV1VVfSnJl5KklPL1qqrWXCiDW5N5wbWY\nF6zF3OBazAvWYm5wLR90Ub4P/bLAUkpPKaXv8uskn05jIQwAAICbxnqXYv/pUsqpJB9P8q9KKb/T\nfH9XKeUrzd22J/mPpZTnknwtyb+qqurfrOe4AAAAm816Vwv87SS/fY33zyT5bPP10SQPfp+H+NL3\nPzpuYuYF12JesBZzg2sxL1iLucG1fKB5Uaqq+rAHAgAAcNO7XkuxAwAA3NQ2dVyVUv7HUsq3SinP\nllJ+t5Sya6PHxOZQSvl7pZSXm/Pjt0spgxs9JjZeKeXzpZQXSikrpRQrPd3iSimfKaW8Ukp5rZTy\nCxs9HjaHUsqvlVLO+qoXrlRK2VtK+fellBebf4/8tY0eE5tDKaWzlPK1UspzzbnxP7zn/pv5ssBS\nSn9VVVPN1381yb1VVf2lDR4Wm0Ap5dNJfr+qqqVSyt9Jkqqq/tsNHhYbrJRyT5KVJP8wyV+vquoD\nLZvKzaeU0pLkO0l+LMmpJE8n+WJVVS9u6MDYcKWUH0oyneSfVFV1/0aPh82hlLIzyc6qqr7RXOX6\nmSR/1J8ZlFJKkp6qqqZLKW1J/mOSv1ZV1Vevtf+mPnN1OayaepJs3hLkuqqq6nerqlpqbn41yZ6N\nHA+bQ1VVL1VV9cpGj4NN4fEkr1VVdbSqqoUkv5nkcxs8JjaBqqr+MMnYRo+DzaWqqjeqqvpG8/XF\nJC8l2b2xo2IzqBqmm5ttzceaTbKp4ypJSil/q5RyMsmfTvJLGz0eNqU/l+Rfb/QggE1ld5KTV2yf\nin8oAR9AKeVAkoeTPLWxI2GzKKW0lFKeTXI2yb+tqmrNubHhcVVK+b1Syrev8fhcklRV9YtVVe1N\n8k+T/PzGjpbr6f3mRnOfX0yylMb84BbwQeYFAHw/Sim9Sf55kv/qHVdQcQurqmq5qqqH0rhS6vFS\nypqXFK/re67qUFXVj37AXf9pkq8k+eUPcThsIu83N0opP5fkJ5N8qtrMNw9Sq+/hzwxubaeT7L1i\ne0/zPYBrat5P88+T/NOqqv7vjR4Pm09VVROllH+f5DNJrrkozoafuXovpZQ7r9j8XJKXN2osbC6l\nlM8k+W+S/FRVVbMbPR5g03k6yZ2llIOllPYkX0jy5Q0eE7BJNRct+EdJXqqq6n/a6PGweZRSRi+v\nSl1K6UpjoaQ1m2Szrxb4z5McSmP1rxNJ/lJVVf6fR1JKeS1JR5ILzbe+aiVJSik/neR/TzKaZCLJ\ns1VV/fjGjoqNUkr5bJL/JUlLkl+rqupvbfCQ2ARKKb+R5HCSrUneSvLLVVX9ow0dFBuulPIDSf5D\nkufT+Hdnkvx3VVV9ZeNGxWZQSvlIkl9P4++SLUl+q6qqv7nm/ps5rgAAAG4Um/qyQAAAgBuFuAIA\nAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKiBuAIAAKjB/w8cV2m05IFPywAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fdda9490fd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot(lambda x: 1 / (1 + math.exp(-x)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Tangens hiperboliczny"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$$ g(x) = \\tanh x = \\frac{e^{x} - e^{-x}}{e^{x} + e^{-x}} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Przyjmuje wartości z przedziału $(-1, 1)$.\n",
    "* Powstaje z funkcji logistycznej przez przeskalowanie i przesunięcie."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Tangens hiperboliczny – wykres"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1cAAAG2CAYAAACTRXz+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3Xd0HOW9h/Hvu7vq1bKKZVtyt9xx\nw6ZjUw0pQCgBQjc4cOEGktz0nnADaeSShCQQejVOaCZ0AqLjinuVu+Si3tuW9/6htRFGsmVrpFlJ\nz+ecPTuzO9r5kTNH8pPZnTXWWgEAAAAAOsfj9gAAAAAA0BsQVwAAAADgAOIKAAAAABxAXAEAAACA\nA4grAAAAAHAAcQUAAAAADnAkrowxDxpjio0xa9p5fpYxpsoYsyJ8+6kT+wUAAACASOFz6HUelvQX\nSY8eYpv3rLVfdGh/AAAAABBRHDlzZa19V1K5E68FAAAAAD1Rd37m6nhjzEpjzCvGmPHduF8AAAAA\n6HJOvS3wcJZLGmKtrTXGnCvpeUmj2trQGDNP0jxJio2NnZabm9tNI6KnCIVC8ni4Fgs+i+MC7eHY\nQFs4LtAejg20ZdOmTaXW2ozDbWestY7s0BgzVNK/rbUTOrDtdknTrbWlh9ouLy/Pbty40ZH50Hvk\n5+dr1qxZbo+BCMNxgfZwbKAtHBdoD8cG2mKMWWatnX647boly40xA4wxJrw8I7zfsu7YNwAAAAB0\nB0feFmiMeUrSLEnpxphCST+TFCVJ1tq/S7pI0k3GmICkBkmXWqdOmQEAAABABHAkrqy1lx3m+b+o\n5VLtAAAAANAr8Wk9AAAAAHAAcQUAAAAADiCuAAAAAMABxBUAAAAAOIC4AgAAAAAHEFcAAAAA4ADi\nCgAAAAAcQFwBAAAAgAOIKwAAAABwAHEFAAAAAA4grgAAAADAAcQVAAAAADiAuAIAAAAABxBXAAAA\nAOAA4goAAAAAHEBcAQAAAIADiCsAAAAAcABxBQAAAAAOIK4AAAAAwAHEFQAAAAA4gLgCAAAAAAcQ\nVwAAAADgAOIKAAAAABxAXAEAAACAA4grAAAAAHAAcQUAAAAADiCuAAAAAMABxBUAAAAAOIC4AgAA\nAAAHEFcAAAAA4ADiCgAAAAAcQFwBAAAAgAOIKwAAAABwAHEFAAAAAA4grgAAAADAAcQVAAAAADiA\nuAIAAAAABxBXAAAAAOAA4goAAAAAHEBcAQAAAIADiCsAAAAAcABxBQAAAAAOIK4AAAAAwAHEFQAA\nAAA4gLgCAAAAAAcQVwAAAADgAOIKAAAAABxAXAEAAACAA4grAAAAAHAAcQUAAAAADiCuAAAAAMAB\nxBUAAAAAOIC4AgAAAAAHEFcAAAAA4ADiCgAAAAAcQFwBAAAAgAOIKwAAAABwAHEFAAAAAA5wJK6M\nMQ8aY4qNMWvaed4YY/5kjCkwxqwyxkx1Yr8AAAAAECmcOnP1sKQ5h3j+HEmjwrd5kv7m0H4BAAAA\nICI4ElfW2ncllR9ik/MkPWpbfCwp1RiT7cS+AQAAACAS+LppP4Mk7Wq1Xhh+bE837R8AAABAL2St\nVchKwZBVyFoFQ1ZBaxUKhZfD68GQVSikT5ftp8+HPvPYZ19rdFZSh2fprrjqMGPMPLW8dVAZGRnK\nz893dyBEnNraWo4LfA7HBdrDsYG2cFygPX3x2LDWKmilYEjyh6SAtQqE1OrWsu4Ph4m/jecCISlg\nJX8oHCe25bHQ/tcOv37IhuMmvL7/uQNRc2Bd4SjSZ7YNHfSzofDzXWnuhOgOb9tdcVUkKafV+uDw\nY59jrb1P0n2SlJeXZ2fNmtXlw6Fnyc/PF8cFDsZxgfZwbKAtHBdoTyQcG6GQVYM/qPrmoOqbA6pr\nCqoxEFSjP6gmf0iN/v3r4WV/SA3+oJr8wQPr+7c/sE0g9Jnn/cGQmgMhNQVblq3DgeLzGPm8Rj6P\nJ3zfsuz1GEV5Tfi+Zd3n9SjGYz77Mwf9vNdjFOXxyOs1ivIYeT2eA6/jC697PZLHY+Q1LY97wvcH\nbsa0PO/Rp8/tf2z/zxx4TAceG5qeoJ929L/b2f8Z27VQ0i3GmPmSZkqqstbylkAAAAD0eNZa1TcH\nVdMYUE2jX9WNAVU3+lXTGFB9U0B1zcFP78OxVN/c/uP1zcGjmiPa51Gsz6PYKK9io7yKi/IqNsqj\nmCivUuKiFJsUo9gor2J8HsVEeRTl9Sja51G0t+UWtX+51X3UgXujaJ9HMZ95rO3to7xGxhiH/1fu\nGRyJK2PMU5JmSUo3xhRK+pmkKEmy1v5d0suSzpVUIKle0rVO7BcAAABwgj8YUkV9s4pqQlq0tUwV\n9X5VNTSruuHzwVTT6G95vGn/ekDB0OFP/UR7PYqP8Soh2qf4aK/iY3xKiPYqNT5aCTFexUe3rO9/\nfP99XDiWYsOx9Jlln/dAMHk8fTNoIokjcWWtvewwz1tJNzuxLwAAAOBQgiGr8rpmldQ0qbS2SRX1\nzaqoa1ZFvV+V9S33FfXNqmx1X9sU+PQFPvj4M69njJQY41NybJSSYlvus1NiNTo2UclxLY8lxX56\nn9zqPiHGp4Ron+KivYr2OfUtSIhUEXdBCwAAAOBgoZBVWatg2n9rWW9utdyk8rpmtXciKTnWp34J\n0UqNj1b/xGiNzExUanyU+sVHq198lHZvL9CJ0ye3PJYQ3RJI0T7OCqFDiCsAAAC4yh8MqbimSXur\nGrW3qlF7qhpa7qsbta+qUXuqGrWvulGBNoopxudRRlKM0hNjNLhfvKbkpiojMUbp4cfSE2OUltAS\nTilxUfJ5D332KL9pu04ald5V/6no5YgrAAAAdKlGf1CFFfXaVdGgwvJ6FVY0aFdFvYoqGrSnqlEl\ntU2fu1pdbJRH2SlxGpAcq5nD0jQgJVZZybHKSIo5EFPpidFKjPH12YsnIPIQVwAAAOgUa61Kapq0\ntbRO20vrtLNVQBVWNKikpukz20f7PBrcL06DUuM0ZkCyBqTEHrhlp8QqOzlOyXFEE3oe4goAAAAd\nUtXg17bSOm0rrdW2kjptLa3TtnBQ1bW6fLjXYzQwNVY5/eI1Oy9DOf3ilZMWr5y0OA3uF6+MxBg+\nw4ReibgCAADAZ5TWNmnT3hpt2lejTcW12ryvRltL6lRW13xgG4+RBveL17D0BB07NE3DMxI0tH+C\nhqUnKDsl9rCfbQJ6I+IKAACgj6pu9GvDnnBEHbjVqrxVRKXERWl0VqLOGp+lYekJGpaeqGHpLWei\nYnxeF6cHIg9xBQAA0AeU1DRp7e4qrd1dfeB+R1n9gecTY3walZWos8ZlaVRWkvKykjQ6K1EZSTF8\n9gnoIOIKAACglymuadSKnZVaXfRpTO2r/vSiErlp8Ro/MFkXTxus8QNTNHpAkgamxBJRQCcRVwAA\nAD1YUyCodbur9cnOSn2yq1Kf7KxQYUWDpJbPRY3MTNSJI9I1bmCyxg9M0biByUqJi3J5aqB3Iq4A\nAAB6kH3VjVq8rVzLd1bok52VWre7Ws3BkCRpYEqspuT20zUnDNWU3FSNH5ii2Cg+FwV0F+IKAAAg\nQllrVVjRoEXbyrV4W5kWbSs/8Dmp2CiPJg1O1bUnDdWUnH6akpuqrORYlycG+jbiCgAAIILsKq/X\nBwWlWrStXIu2lml3VaMkKTU+SscOTdOVxw3RjGFpGpudrCgudw5EFOIKAADARdWNfn1YUKb3C0r0\n/uZSbQ+fmUpPjNbMYf114/A0zRiWptGZSXzxLhDhiCsAAIBuFAiGtLKwUu9uKtV7m0u0srBKwZBV\nfLRXxw/vr6tPGKqTR6VrREYiV+8DehjiCgAAoItVN/r1zsYSvbWhWG9vLFZlvV/GSJMGp+qmU0fo\n5FHpmpLbT9E+3uYH9GTEFQAAQBfYUVanN9cX6z/r92nxtnIFQlb94qN0Wl6mThubqZNGpis1Ptrt\nMQE4iLgCAABwgLVW6/ZU6+XVe/Tqmr3aUlInqeV7puaePExnjM3S1Nx+8vK5KaDXIq4AAACOkrVW\na3e3BNXLq/doe1m9PEaaOay/vjZziE4fm6kh/RPcHhNANyGuAAAAjoC1VmuKqvXS6j16Zc0e7Sir\nl9djdPzw/pp3ygidPT5L/RNj3B4TgAuIKwAAgA4orKjX858U6dnlRdpaWievx+iEEf1146kjdPb4\nAUpL4PNTQF9HXAEAALSjutGvV1bv0bPLi7RoW7kkacawNN1wynDNGT9A/QgqAK0QVwAAAK0EgiG9\nt7lUzywv1Bvr9qkpENLw9AR9+8zROn/KIOWkxbs9IoAIRVwBAABI2lVerwVLd2nB0l3aV92k1Pgo\nffXYHF0wZZAm56Tyhb4ADou4AgAAfZY/GNJ/1hfrqcU79e7mEknSrNEZ+sWXc3XamEy+1BfAESGu\nAABAn1PWENLvXtugBUsLVVLTpAHJsfrv00bpkumDNbgfb/sDcHSIKwAA0CdYa7V4W7ke+Wi7Xl3T\nIGmLZudl6rIZuZqVlyGfl7NUADqHuAIAAL1aoz+ohSt266EPt2v9nmqlxEVpztAo/fCSkzhLBcBR\nxBUAAOiVSmqa9MiH2/XEoh2qqPcrLytJd3xlos6fPEiLPnyPsALgOOIKAAD0KttK63Tfu1v1zPJC\n+YMhnTE2S9eeOFTHD+/PFf8AdCniCgAA9Aord1Xq7+9s0atr9yrK69GFUwdr3inDNSw9we3RAPQR\nxBUAAOixrLV6v6BU97xdoI+3lisp1qebTh2ha04cqsykWLfHA9DHEFcAAKDHsdbq3c2luvvNTVq+\ns1IDkmP1o3PH6rKZuUqM4Z83ANzBbx8AANBjWGv1zqYS/d+bm7ViV6UGpsTqfy+YoIumDVaMz+v2\neAD6OOIKAABEPGut8sNRtXJXpQalxunXF0zURdMGK9rH91MBiAzEFQAAiGiLt5XrN69u0LIdFRrc\nL053fGWiLpxKVAGIPMQVAACISOv3VOt3r23UWxuKlZUcw5kqABGPuAIAABFlV3m97npjk55fUaSk\nGJ++f84YXX38UMVF85kqAJGNuAIAABGhvK5Zf/rPZj2xaIe8HqMbTx2hG08ZoZT4KLdHA4AOIa4A\nAICrmgMhPfbxDt395ibVNQd1yfQc3XbGKGUl8z1VAHoW4goAALjCWqu3NhTrf19ar62ldTpldIZ+\n8oWxGpWV5PZoAHBUiCsAANDtNu6t0e0vrdN7m0s1PCNBD11zrGblZcgY4/ZoAHDUiCsAANBtqur9\n+v3rG/XEoh1Kio3Sz740TlccN0RRXq4ACKDnI64AAECXs9bqmeVFuuPl9aqob9aVxw3RbWeMVr+E\naLdHAwDHEFcAAKBLbdxbo588v0aLt5drSm6qHp07Q+MHprg9FgA4jrgCAABdorYpoLvf3KQHP9iu\npFiffnPhRF08LUceD5+rAtA7EVcAAMBxr67Zo58tXKt91U269NgcfXfOGKXxFkAAvRxxBQAAHFNc\n3aifvrBWr67dq7HZyfrbFdM0Nbef22MBQLcgrgAAQKdZa/XPpYW6/aV1agyE9N05ebrh5OFcBRBA\nn0JcAQCATtlZVq8fPrda7xeUasbQNN1x4USNyEh0eywA6HbEFQAAOCrBkNVDH2zTH17fJK/H6Pbz\nJ+jyGblcsAJAn0VcAQCAI7ajrE7fXrBSS3dU6LQxmbr9/AkamBrn9lgA4CriCgAAdJi1Vk8s2qlf\nv7xeXo/RXZccowumDJIxnK0CAOIKAAB0yN6qRn3vmVV6Z1OJThqZrt9eNImzVQDQCnEFAAAOyVqr\nhSt36yfPr1FzMKRfnjdeV8wcwmerAOAgxBUAAGhXVb1fP3x+tV5atUdTc1P1h0sma1h6gttjAUBE\nIq4AAECbFm8r123zP1FxTZO+c3aebjx1hLycrQKAdhFXAADgMwLBkP70VoH+8tZm5abF69n/OkGT\nBqe6PRYARDziCgAAHFBYUa/b5q/Q0h0VunDqYP3ivPFKjOGfCwDQER4nXsQYM8cYs9EYU2CM+X4b\nz19jjCkxxqwI3653Yr8AAMA5/161W+fc/Z427K3R3ZdO1h8uOYawAoAj0OnfmMYYr6R7JJ0pqVDS\nEmPMQmvtuoM2fdpae0tn9wcAAJzV6A/q5wvXav6SXZqck6o/XTpFuf3j3R4LAHocJ/7vqBmSCqy1\nWyXJGDNf0nmSDo4rAAAQYbaV1um/nliu9XuqddOsEfrWmaMV5XXkjS0A0OcYa23nXsCYiyTNsdZe\nH16/UtLM1mepjDHXSLpDUomkTZK+aa3d1c7rzZM0T5IyMjKmLViwoFPzofepra1VYmKi22MgwnBc\noD0cG+1bsjegB1Y3yeuRvj4pRpMy+s5bADku0B6ODbRl9uzZy6y10w+3XXf9Fn1R0lPW2iZjzNcl\nPSLptLY2tNbeJ+k+ScrLy7OzZs3qphHRU+Tn54vjAgfjuEB7ODY+rzkQ0q9fXq+HV2zX5JxU3fO1\nqRqUGuf2WN2K4wLt4dhAZzgRV0WSclqtDw4/doC1tqzV6v2SfuvAfgEAwBEqqmzQzU8s14pdlbr2\nxKH6wTljFe3jbYAA4AQn4mqJpFHGmGFqiapLJV3eegNjTLa1dk949cuS1juwXwAAcATe2VSiW+d/\nokDQ6q9fm6pzJ2a7PRIA9CqdjitrbcAYc4uk1yR5JT1orV1rjPmlpKXW2oWSvmGM+bKkgKRySdd0\ndr8AAKBjrLX6a/4W/f71jcrLStLfrpimYekJbo8FAL2OI5+5sta+LOnlgx77aavlH0j6gRP7AgAA\nHVfbFNB3/rlSr6zZqy8dM1C/uXCi4qP7zoUrAKA78dsVAIBealtpneY9ulRbSmr14y+M1dyThskY\n4/ZYANBrEVcAAPRCb23Yp1vnr5DPY/TY3Jk6cWS62yMBQK9HXAEA0IuEQlb3vF2gu97cpHHZyfr7\nFdOUkxbv9lgA0CcQVwAA9BINzUH9z79W6qVVe3TBlEG64ysTFRvldXssAOgziCsAAHqBfdWNuuHR\npVpdVKUfnjtGN5w8nM9XAUA3I64AAOjhVhdW6fpHl6i2MaB/XDldZ4zLcnskAOiTiCsAAHqwl1fv\n0bcWrFD/hBj966YTNDY72e2RAKDPIq4AAOiBrLX681sFuuuNTZo2pJ/uvXKa0hNj3B4LAPo04goA\ngB6m0R/Ud/61Si+u3K2vTBmkX3PhCgCICMQVAAA9SElNk254dKlWFlbqe3PG6MZTuXAFAEQK4goA\ngB6ioLhW1zy0WGW1zfr7FdN09vgBbo8EAGiFuAIAoAdYtLVM8x5bpiiv0dNfP06TBqe6PRIA4CDE\nFQAAEW7hyt36nwUrlZMWp4evnaGctHi3RwIAtIG4AgAgQllrde+7W3XnKxs0Y1ia7rtymlLjo90e\nCwDQDuIKAIAIFAiG9PMX1+rxj3fqS8cM1O8vnqQYH1cEBIBIRlwBABBh6psD+u8nP9F/NhTrxlNH\n6Ltn58nj4YqAABDpiCsAACJISU2T5j6yRGuKqnT7+RN0xXFD3B4JANBBxBUAABFiZ1m9rnxwkYqr\nm/SPq6br9LFZbo8EADgCxBUAABFg7e4qXfPQEvmDIT15w0xNye3n9kgAgCNEXAEA4LKPtpRp3qNL\nlRTr01M3HK+RmUlujwQAOArEFQAALnp1zR59Y/4KDUmL16NzZyg7Jc7tkQAAR4m4AgDAJU8u2qkf\nP79aU3L76YGrp/MdVgDQwxFXAAB0M2ut/vxWge56Y5NOG5Opey6fqrhovsMKAHo64goAgG4UDFn9\n4sW1evSjHbpw6mDdeeFERXk9bo8FAHAAcQUAQDfxB0P69oKVWrhyt75+ynB9/5wxMoYvBwaA3oK4\nAgCgGzT6g7r5ieX6z4ZifW/OGN00a4TbIwEAHEZcAQDQxWqbArrhkaX6eFuZfnX+BF153BC3RwIA\ndAHiCgCALlRZ36yrH1qiNUVV+uMlk3X+lEFujwQA6CLEFQAAXaS4plFX3r9Y28rq9PcrpunMcVlu\njwQA6ELEFQAAXaCwol5X3L9IxTVNeuiaY3XiyHS3RwIAdDHiCgAAh20pqdUV9y9SXVNAj82dqWlD\n+rk9EgCgGxBXAAA4aO3uKl31wGIZI82fd7zGDUx2eyQAQDchrgAAcMiyHeW65qElSorx6fHrZ2p4\nRqLbIwEAuhFxBQCAAxZtLdO1Dy9RVnKsHr9+pgalxrk9EgCgmxFXAAB00kdbynTdw0s0MDVWT91w\nnDKTY90eCQDgAuIKAIBO+LCgVNc9skQ5/eL15A3HKSMpxu2RAAAuIa4AADhKHxSUau4jS5Sb1hJW\n6YmEFQD0ZR63BwAAoCd6d1OJrnt4iYb2T9BThBUAQJy5AgDgiOVvLNa8x5ZpREainrh+ptISot0e\nCQAQAThzBQDAEXh7Q7HmPbpMozIT9SRhBQBohTNXAAB00H/W79NNjy/X6AGJenzuTKXGE1YAgE9x\n5goAgA54Y90+3fj4Mo3JTtITc48jrAAAn8OZKwAADuO1tXt1y5PLNW5gih69boZS4qLcHgkAEIE4\ncwUAwCG8snqPbn5iuSYMStFjcwkrAED7iCsAANrx0qo9uuWpT3RMTqoevW6GkmMJKwBA+3hbIAAA\nbXhx5W7d9vQKTclJ1cPXzVBiDH8yAQCHxpkrAAAO8sKKIt06/xNNy+1HWAEAOoy/FgAAtPLcJ4X6\n9oKVOnZomh685lglEFYAgA7iLwYAAGHPLCvU//xrpY4b1l8PXDNd8dH8mQQAdBx/NQAAkLRg6S59\n75lVOnFEuv5x1XTFRXvdHgkA0MMQVwCAPu/pJTv1/WdX66SRLWEVG0VYAQCOHBe0AAD0aU8t3qnv\nPbNaJ4/KIKwAAJ1CXAEA+qzHP96hHzy7WrPzMnTfldMIKwBAp/C2QABAn/ToR9v10xfW6vQxmfrr\nFVMV4yOsAACdQ1wBAPqchz/Ypp+/uE5njM3SPV+bQlgBABxBXAEA+pQH3t+mX/17nc4en6U/XzZV\n0T7eIQ8AcAZxBQDoM+5/b6tuf2m9zpkwQH+6bIqivIQVAMA5xBUAoE+4950tuuOVDfrCxGz936WT\nCSsAgOMc+ctijJljjNlojCkwxny/jedjjDFPh59fZIwZ6sR+AQDoiL/mF+iOVzboi5OydTdhBQDo\nIp3+62KM8Uq6R9I5ksZJuswYM+6gzeZKqrDWjpT0R0m/6ex+AQDoiIVbmvXbVzfqvMkD9X9fnSwf\nYQUA6CJO/IWZIanAWrvVWtssab6k8w7a5jxJj4SX/yXpdGOMcWDfAAC06+43N+vZzX5dMGWQ7rqE\nsAIAdC0nPnM1SNKuVuuFkma2t421NmCMqZLUX1LpwS9mjJknaZ4kZWRkKD8/34ER0ZvU1tZyXOBz\nOC5wsOc2N+uFLX7NyLT6UmaF3nv3HbdHQgThdwbaw7GBzoi4C1pYa++TdJ8k5eXl2VmzZrk7ECJO\nfn6+OC5wMI4L7Get1R/f2KQXthToommDdW56uU6bPdvtsRBh+J2B9nBsoDOceH9EkaScVuuDw4+1\nuY0xxicpRVKZA/sGAOAAa61+//pG/emtAn11eo5+e+EkeXgXOgCgmzgRV0skjTLGDDPGREu6VNLC\ng7ZZKOnq8PJFkt6y1loH9g0AgKSWsPrNqxt1z9tbdNmMXN3xlYnyeAgrAED36fTbAsOfobpF0muS\nvJIetNauNcb8UtJSa+1CSQ9IeswYUyCpXC0BBgCAI6y1uuOVDbrv3a264rhc/fLLEwgrAEC3c+Qz\nV9balyW9fNBjP2213CjpYif2BQBAa9Za/e9L63X/+9t01fFD9IsvjxcXpAUAuCHiLmgBAEBHWWv1\ny3+v00MfbNc1JwzVz740jrACALiGuAIA9EjWWv3ixXV6+MPtuu7EYfrJF8cSVgAAVxFXAIAeJxSy\n+tnCtXrs4x264eRh+uG5hBUAwH3EFQCgRwmFrH78who9uWinvn7qcH1/zhjCCgAQEYgrAECPEQpZ\n/ej51Xpq8S7dNGuEvnt2HmEFAIgYxBUAoEcIhax+8OxqPb10l26ZPVLfPms0YQUAiCjEFQAg4gVD\nVt/510o9u7xI3zhtpL55JmEFAIg8xBUAIKIFgiF9+58r9cKK3frmGaN16xmj3B4JAIA2EVcAgIjl\nD4Z029Mr9NKqPfrO2Xm6efZIt0cCAKBdxBUAICI1B0K6df4nemXNXv3w3DGad8oIt0cCAOCQiCsA\nQMRpCgR1y5Of6I11+/STL47T3JOGuT0SAACHRVwBACJKoz+o/3piud7aUKxfnjdeVx0/1O2RAADo\nEOIKABAxGv1BzXtsmd7dVKJfXzBRl8/MdXskAAA6jLgCAESEhuagbnh0qT7YUqrfXjhJlxyb4/ZI\nAAAcEeIKAOC6uqaA5j6yRIu3lev3Fx2jC6cNdnskAACOGHEFAHBVbVNA1z60WMt2VOiPX52s8yYP\ncnskAACOCnEFAHBNdaNf1zy4WCsLq/Sny6boi5MGuj0SAABHjbgCALiiqt6vqx5arLVFVfrLZVN0\nzsRst0cCAKBTiCsAQLcrq23SlQ8sVkFxrf76tak6a/wAt0cCAKDTiCsAQLfaW9Wor93/sYoqG/SP\nq6fr1NEZbo8EAIAjiCsAQLfZVV6vr92/SGW1TXrk2hmaOby/2yMBAOAY4goA0C22ltTqa/cvUl1T\nQE/ccJwm56S6PRIAAI4irgAAXW7D3mpdcf9iWWs1f97xGjcw2e2RAABwHHEFAOhSqworddWDixXj\n8+iJ64/XyMxEt0cCAKBLEFcAgC6zZHu5rn1oiVLjo/Tk9ccpt3+82yMBANBliCsAQJd4f3Opbnh0\nqbJTYvXEDTOVnRLn9kgAAHQp4goA4Lg31u3TzU8s1/CMBD02d6YykmLcHgkAgC5HXAEAHPXMskJ9\n95lVmjAwWY9cN0Op8dFujwQAQLcgrgAAjrn/va26/aX1OnFkf9175XQlxvBnBgDQd/BXDwDQadZa\n/eH1TfrL2wU6Z8IA/d+lkxXj87o9FgAA3Yq4AgB0SjBk9ZMX1ujJRTt12Ywc3X7+RHk9xu2xAADo\ndsQVAOCoNQdC+uaCFXpp1R7dNGuEvnt2nowhrAAAfRNxBQA4KnVNAd34+DK9t7lUPzp3rG44Zbjb\nIwEA4CriCgBwxCrqmnXtw0tUfiR5AAAZp0lEQVS0qrBSv71oki6ZnuP2SAAAuI64AgAckb1Vjbry\ngUXaUV6vv10xTWePH+D2SAAARATiCgDQYQXFtbr6wcWqavDr4WuP1Qkj0t0eCQCAiEFcAQA6ZOn2\ncl3/6FL5PB49dcNxmjg4xe2RAACIKMQVAOCwXl2zR9+Yv0KDU+P0yHUzlJMW7/ZIAABEHOIKAHBI\nD3+wTb/49zpNyUnV/Vcfq7SEaLdHAgAgIhFXAIA2hUJWv3l1g+59d6vOGpeluy+dorhor9tjAQAQ\nsYgrAMDnNAWC+u6/VumFFbt15XFD9PMvj5fXw5cDAwBwKMQVAOAzqhv9+vqjy/TR1jJ9d06ebjp1\nhIwhrAAAOBziCgBwQFFlg+Y+vEQFxbW665Jj9JWpg90eCQCAHoO4AgBIklbuqtTcR5aqyR/UQ9ce\nq5NHZbg9EgAAPQpxBQDQy6v36FsLVig9MUZP3TBTo7KS3B4JAIAeh7gCgD7MWqu/5m/R717bqKm5\nqbrvqulKT4xxeywAAHok4goA+qjmQEg/eHa1nlleqC8fM1C/vWiSYqO41DoAAEeLuAKAPqiirllf\nf3yZFm8r121njNKtp4/iioAAAHQScQUAfczWklpd9/AS7a5s1N2XTtZ5kwe5PRIAAL0CcQUAfcj7\nm0t185PL5fMYPTVvpqYNSXN7JAAAeg3iCgD6AGutHnh/m3798nqNykzSP66artz+8W6PBQBAr0Jc\nAUAv1+gP6ofPrdazy4t09vgs3XXJZCXE8OsfAACn8dcVAHqxvVWN+vpjS7WysErfPGO0/vu0kfJ4\nuHAFAABdgbgCgF5q2Y4K3fj4MtU3BXTvldN09vgBbo8EAECvRlwBQC+0YMku/fj5NRqQEqvH585U\n3oAkt0cCAKDXI64AoBdp9Af184VrNX/JLp00Ml1/uXyKUuOj3R4LAIA+gbgCgF5iV3m9bnpimdYU\nVevm2SP0rTPz5OXzVQAAdBviCgB6gbc3Fuu2+SsUslb/uGq6zhyX5fZIAAD0OZ2KK2NMmqSnJQ2V\ntF3SJdbaija2C0paHV7daa39cmf2CwBoEQxZ3f2fzfrzW5s1ZkCy/n7FVA3pn+D2WAAA9EmdPXP1\nfUn/sdbeaYz5fnj9e21s12CtndzJfQEAWimva9at8z/Re5tLddG0wbr9/AmKjfK6PRYAAH1WZ+Pq\nPEmzwsuPSMpX23EFAHDQ4m3l+sZTn6i8rll3fGWiLj02R8bw+SoAANxkrLVH/8PGVFprU8PLRlLF\n/vWDtgtIWiEpIOlOa+3zh3jNeZLmSVJGRsa0BQsWHPV86J1qa2uVmJjo9hiIMH3luAhZqxe3+PV8\ngV+Z8UY3HROjoSmcrTqUvnJs4MhwXKA9HBtoy+zZs5dZa6cfbrvDnrkyxrwpqa1vnvxR6xVrrTXG\ntFdqQ6y1RcaY4ZLeMsasttZuaWtDa+19ku6TpLy8PDtr1qzDjYg+Jj8/XxwXOFhfOC6Kqxt129Mr\n9OGWep03eaD+94KJSozhukSH0xeODRw5jgu0h2MDnXHYv8rW2jPae84Ys88Yk22t3WOMyZZU3M5r\nFIXvtxpj8iVNkdRmXAEAPu+dTSX61tMrVN8c1G8vmqSLpw3mbYAAAEQYTyd/fqGkq8PLV0t64eAN\njDH9jDEx4eV0SSdKWtfJ/QJAn+APhnTnKxt09YOLlZ4Yo4W3nKhLpvP5KgAAIlFn309yp6QFxpi5\nknZIukSSjDHTJd1orb1e0lhJ9xpjQmqJuTuttcQVABxGQXGtvvn0Cq0uqtJlM3L1sy+N42qAAABE\nsE7FlbW2TNLpbTy+VNL14eUPJU3szH4AoC+x1uqxj3fo1y+vV1yUV3+/YqrmTMh2eywAAHAYfBIa\nACJIcXWjvvOvVXpnU4lOHZ2h3100SZnJsW6PBQAAOoC4AoAI8eqavfrBs6tU3xzUL88bryuPG8Jn\nqwAA6EGIKwBwWXWjX796cZ3+uaxQEwel6I9fnayRmXzHCgAAPQ1xBQAuentDsX7w7GoV1zTqltkj\n9Y3TRyna19kLuQIAADcQVwDggqp6v37573V6ZnmhRmcl6t4rT9QxOalujwUAADqBuAKAbvbGun36\n0XOrVVbXrP8+baRuOW2kYnxcYh0AgJ6OuAKAblJR16xfvLhWz6/YrTEDkvTgNcdqwqAUt8cCAAAO\nIa4AoItZa/X8iiLd/u/1qmrw69bTR+nm2SP5bBUAAL0McQUAXWhrSa1+8sIafVBQpsk5qXrsgoka\nNzDZ7bEAAEAXIK4AoAs0BYL6e/5W3ZNfoBifR786f4Iun5Err4fvrQIAoLcirgDAYR9tKdOPnl+t\nrSV1+uKkbP30i+OUmRzr9lgAAKCLEVcA4JB91Y2685UNeu6TIuWkxenha4/VrLxMt8cCAADdhLgC\ngE5qCgT14Pvb9ee3NisQtLp59gjdMnuU4qK5vDoAAH0JcQUAnfDWhn365YvrtL2sXmeOy9KPvzBW\nQ/onuD0WAABwAXEFAEdha0mtfvXvdXp7Y4mGZyToketm6NTRGW6PBQAAXERcAcARKK9r1p/f2qzH\nP96hGJ9XP/7CWF11/FC+swoAABBXANARjf6gHvxgm/729hbVNQf01WNz9M0zRysziasAAgCAFsQV\nABxCMGT17PJC3fXGJu2patQZYzP1vTljNCorye3RAABAhCGuAKAN1lq9s6lEd76yQRv21uiYwSn6\n41cn67jh/d0eDQAARCjiCgAO8uGWUt31+iYt3VGh3LR4/eXyKfrCxGwZY9weDQAARDDiCgDClm4v\n1x9e36SPtpZpQHKsfnX+BH11eg4XqwAAAB1CXAHo81bsqtRdb2zSu5tKlJ4Yo59+cZwun5mr2Ci+\nBBgAAHQccQWgz1q6vVz3vF2gtzeWqF98lH5wzhhddfxQxUUTVQAA4MgRVwD6FGut3t1cqnveLtDi\nbeVKS4jWd87O09UnDFViDL8SAQDA0eNfEgD6hFDI6rW1e3VPfoHWFFUrOyVWP/vSOF16bC5nqgAA\ngCOIKwC9WqM/qOc+KdL9723VlpI6DUtP0G8vnKTzpwziQhUAAMBRxBWAXqm4plGPf7RDjy/aqfK6\nZo3LTtZfLp+icyZky+vhkuoAAMB5xBWAXmX9nmo98P42LVyxW/5QSKePydLck4bpuOFpfE8VAADo\nUsQVgB7PHwxpyd6A7vvHx/pwS5niory6dEaOrj1xmIalJ7g9HgAA6COIKwA91p6qBj21eJfmL96p\n4pomDUr16HtzxujyGblKiY9yezwAANDHEFcAepRQyOr9glI9/vEO/WdDsULW6tTRGbosoVrfuGg2\nn6cCAACuIa4A9AhFlQ16Zlmh/rWsUDvL65WWEK0bTh6ur83MVU5avPLz8wkrAADgKuIKQMRq9Af1\n2tq9+ufSQn2wpVTWSieM6K9vnzVacyYMUIyP76cCAACRg7gCEFGstVq+s0LPLi/SwpW7VdMY0OB+\ncbr19FG6cOpg5aTFuz0iAABAm4grAK6z1mr9nhotXLlbL67craLKBsVGeXTOhGxdPG2wjhveXx7e\n8gcAACIccQXANdtL67Rw5W4tXLlbBcW18nqMTh6Vrm+fNVpnjstSUixX/AMAAD0HcQWgW20vrdNr\na/fq5dV7tLKwSpI0Y1iabj9/gs6dmK20hGiXJwQAADg6xBWALmWt1Zqiar2+bq9eW7tXm/bVSpIm\nDkrRj84dqy8ek63slDiXpwQAAOg84gqA4wLBkBZvL9fra/fpjXX7VFTZII9pOUP1sy+N01njB2hQ\nKkEFAAB6F+IKgCOKqxuVv6lE72ws0XubS1TdGFCMz6OTR2XotjNG6fSxWbzlDwAA9GrEFYCjEgiG\ntHxnpfI3Fit/Y4nW7amWJGUmxWjOhAGanZepU0ZnKCGGXzMAAKBv4F89ADrEWqstJbX6cEuZPiwo\n04dbSlXdGJDXYzQtt5++c3aeZudlamx2kozhsukAAKDvIa4AtMlaq13lDfpwS6k+2lqmD7eUqaSm\nSZI0KDVOcyYM0Ky8TJ04Ml0pcVwyHQAAgLgCIEkKhVrOTC3dUaGl2yv08dYyFVU2SJIykmJ0/PD+\nOmFEf50wIl05aXGcnQIAADgIcQX0UY3+oFYXVWnp9got3V6uZTsrVFnvlySlJUTr2KH99PVTh+uE\nEf01IiORmAIAADgM4groA0Ihq62ldVpVWKlVhVVaWViptUXVag6GJEnDMxJ01rgsTR+apulD+mlY\negIxBQAAcISIK6CXsdaqqLLhQESt2lWlNUVVqmkKSJLio72aMDBF15w4VNOH9NO0If3UPzHG5akB\nAAB6PuIK6MGaAyEVFNdqw95qrd9TrfV7arR+T7XK6polSVFeo7HZyTpvykBNGpyqYwanamRmorwe\nzkoBAAA4jbgCegBrrYprmrRpX81nImpLSa38QStJivZ5NDorUaeNydSkwSmaNDhVY7KTFOPzujw9\nAABA30BcAREkEAxpV0WDCoprD9y2lNRqS3Htgbf1SVJWcozGZidr9phMjRmQpHHZyRqWniCf1+Pi\n9AAAAH0bcQV0M2utSmubtbO8TttL67WjrE4FJbXaUlynbaV1By4yIUmZSTEakZGo86cM0sjMRI3K\nTNSY7GSlJUS7+F8AAACAthBXQBcIBEPaXdmoHeV12lFWr53lLRG1f7m+OXhgW4+RctPiNTIzUbPy\nMjQiM1EjMxM1IiORL+cFAADoQYgr4CjUNPq1u7JRuysbVFTZoN0Hbo0qqmzQ3upGBUP2wPbRPo9y\n+sVpaP8EHT+iv4akxWtI/wTl9o/X4H5xfC4KAACgFyCugFZCIauyumbtq25USU2TimsaVVzdpL3V\njdpT9WlM1TQGPvNzPo/RgJRYDUyN04xhaRqYGqvctHjlpiVoSP94DUiOlYcr9AEAAPRqxBV6PWut\nqhsCKqtrUnlds8rqmlVW26zimkbtq25SSU2jimuatK+6UaW1zZ8547RfanyUBqbEaXC/eM0clqaB\nqXEHboNS45SRFMPlzQEAAPo44go9irVWjYGWL8mtqverqsGvivqWYCqvbVZ5XVPLcvhWVtesirpm\nBdoIJknqnxCtzORYZSbFKC8rSZnJMcoKr2ck7b+PUWwUb9sDAADAoRFX6FbWWjUFQqptCqi2MdBy\n3xRQTWNAlfXNqmrwq7qhJZoqw/dVDf4DIVXV4G8JpTffavP1k2N96p8Yo7SEaOWkxWtyTqrSEqKV\nlhCt/onR6p8Qc2A5PTFGUVy6HAAAAA4hrnBIwZBVfXNADf6gGptDqvcH1NAcbLn5g6pvDqq+uSWO\n9gdT3UHr+wNq/3p7Z5H2M0ZKivEpNT5aKXFRSomL0sDUuAPLZXt2aur4MQfWU+NbYqlffLSifcQS\nAAAA3NGpuDLGXCzp55LGSpphrV3aznZzJN0tySvpfmvtnZ3Zb19krVUgZNUcCKkpEFJTIHhguTm8\n3uQPqSkYUpM/pOZgSE3+YKvnP/2ZBv9n46gxfN/Qejm8TevvXOqI+GivEmN8LbfYlvuchHglhdcT\nws8lxfqUEN3y2P7nUuKilBoXrcRY3yE/v5Sfv1ezZuR29n9SAAAAwFGdPXO1RtJXJN3b3gbGGK+k\neySdKalQ0hJjzEJr7bpO7rtN1lqFbMsZl1A4SIIhq1CoZTlkW9YP3Oynz7X+mdBB2wSCVv5gSP6g\nVSAUvg+GPveYPxhq2TYU+uzPBEMtcRQMtSwHrfyhz79GINiyTXOrKNofT4c54dMhUV6j2Civ4qO9\niovyKi7ap7goj+KjfeoXH624aK/io7yKiw7fovZv13IfH+1VbKvluChvSzCFY4mLOgAAAKCv6lRc\nWWvXS5Ixh/wH9QxJBdbareFt50s6T9Jh46qoNqTT/pDfEjrWKhgM34ekYCgUjiEpEAopFFL4OQcK\npJM8RvJ5PYryGEX5PPJ5PIryGvm8RlFej6I8Hvm8Rj6vR9FeI5/Ho9io8HPhx2O8HsVEeRTt9Sgm\nyqsY3/5lj2J8XkX7PIrxHbzsCS97W/2sRzHeT9e5HDgAAADQNbrjM1eDJO1qtV4oaWZHfjDKYzQ2\nO1leY+TzGHk8Rl5j5PWG7z2f3jzGyOuRvB5P+LnwskfyhH/e2/o1PAfdTMtzvna28Xn2x09LGO0P\npNZBtH8bzt4AAAAAfc9h48oY86akAW089SNr7QtOD2SMmSdpniRlZGTo4oHVnXvB/R8ZCh75j4Uk\n+Tu3d3SB2tpa5efnuz0GIgzHBdrDsYG2cFygPRwb6IzDxpW19oxO7qNIUk6r9cHhx9rb332S7pOk\nvLw8O2vWrE7uHr1Nfn6+OC5wMI4LtIdjA23huEB7ODbQGd1x3eolkkYZY4YZY6IlXSppYTfsFwAA\nAAC6TafiyhhzgTGmUNLxkl4yxrwWfnygMeZlSbLWBiTdIuk1SeslLbDWru3c2AAAAAAQWTp7tcDn\nJD3XxuO7JZ3bav1lSS93Zl8AAAAAEMm6422BAAAAANDrEVcAAAAA4ADiCgAAAAAcQFwBAAAAgAOI\nKwAAAABwAHEFAAAAAA4grgAAAADAAcQVAAAAADiAuAIAAAAABxBXAAAAAOAA4goAAAAAHEBcAQAA\nAIADiCsAAAAAcABxBQAAAAAOIK4AAAAAwAHEFQAAAAA4gLgCAAAAAAcQVwAAAADgAOIKAAAAABxA\nXAEAAACAA4grAAAAAHAAcQUAAAAADiCuAAAAAMABxBUAAAAAOIC4AgAAAAAHEFcAAAAA4ADiCgAA\nAAAcQFwBAAAAgAOIKwAAAABwAHEFAAAAAA4grgAAAADAAcQVAAAAADiAuAIAAAAABxBXAAAAAOAA\n4goAAAAAHEBcAQAAAIADiCsAAAAAcABxBQAAAAAOIK4AAAAAwAHEFQAAAAA4gLgCAAAAAAcQVwAA\nAADgAOIKAAAAABxAXAEAAACAA4grAAAAAHAAcQUAAAAADiCuAAAAAMABxBUAAAAAOIC4AgAAAAAH\nEFcAAAAA4ADiCgAAAAAcQFwBAAAAgAOIKwAAAABwAHEFAAAAAA4grgAAAADAAcQVAAAAADiAuAIA\nAAAABxBXAAAAAOCATsWVMeZiY8xaY0zIGDP9ENttN8asNsasMMYs7cw+AQAAACAS+Tr582skfUXS\nvR3Ydra1trST+wMAAACAiNSpuLLWrpckY4wz0wAAAABAD9Vdn7mykl43xiwzxszrpn0CAAAAQLc5\n7JkrY8ybkga08dSPrLUvdHA/J1lri4wxmZLeMMZssNa+287+5knaH2BNxpg1HdwH+o50SbzFFAfj\nuEB7ODbQFo4LtIdjA23J68hGh40ra+0ZnZ3EWlsUvi82xjwnaYakNuPKWnufpPskyRiz1Frb7oUy\n0DdxXKAtHBdoD8cG2sJxgfZwbKAtHb0oX5e/LdAYk2CMSdq/LOkstVwIAwAAAAB6jc5eiv0CY0yh\npOMlvWSMeS38+EBjzMvhzbIkvW+MWSlpsaSXrLWvdma/AAAAABBpOnu1wOckPdfG47slnRte3irp\nmKPcxX1HPx16MY4LtIXjAu3h2EBbOC7QHo4NtKVDx4Wx1nb1IAAAAADQ63XXpdgBAAAAoFeL6Lgy\nxvzKGLPKGLPCGPO6MWag2zMhMhhjfmeM2RA+Pp4zxqS6PRPcZ4y52Biz1hgTMsZwpac+zhgzxxiz\n0RhTYIz5vtvzIDIYYx40xhTzVS9ozRiTY4x52xizLvx35Fa3Z0JkMMbEGmMWG2NWho+NXxxy+0h+\nW6AxJtlaWx1e/oakcdbaG10eCxHAGHOWpLestQFjzG8kyVr7PZfHgsuMMWMlhSTdK+l/rLUdumwq\neh9jjFfSJklnSiqUtETSZdbada4OBtcZY06RVCvpUWvtBLfnQWQwxmRLyrbWLg9f5XqZpPP5nQFj\njJGUYK2tNcZESXpf0q3W2o/b2j6iz1ztD6uwBEmRW4LoVtba1621gfDqx5IGuzkPIoO1dr21dqPb\ncyAizJBUYK3daq1tljRf0nkuz4QIYK19V1K523Mgslhr91hrl4eXayStlzTI3akQCez/t3fHoDqF\ncRzHvz83pIxM3GKQTSwmg6LcJDebsshkMJgMblHqrlLmqww3pa7BcA2UwUIWRTFYdBkodQeZ6G94\nX3XTfd2Lw3N0v586w3N6ht/w9L7nf85z/mfg03C4fniMrEl6XVwBJJlOsgCcAi61zqNeOgPcax1C\nUq9sAxaWjN/ihZKkVUiyA9gHPGmbRH2RZCzJM+ADcL+qRq6N5sVVkgdJXixzTAJU1VRVjQOzwLm2\nafUvrbQ2hnOmgC8M1ofWgNWsC0mSfkeSzcAccP6HHVRaw6rqa1XtZbBTan+SkVuK/+g7V12oqsOr\nnDoLzAOX/2Ic9chKayPJaeAYcKj6/PKgOvULvxla294B40vG24fnJGlZw/dp5oDZqrrTOo/6p6oW\nkzwEJoBlm+I0f3L1M0l2LRlOAq9aZVG/JJkALgDHq+pz6zySeucpsCvJziQbgJPA3caZJPXUsGnB\nDPCyqq62zqP+SLL1e1fqJJsYNEoaWZP0vVvgHLCbQfevN8DZqvLOo0jyGtgIfByeemwnSSU5AVwH\ntgKLwLOqOtI2lVpJchS4BowBN6pqunEk9UCSW8BBYAvwHrhcVTNNQ6m5JAeAR8BzBtedABerar5d\nKvVBkj3ATQb/JeuA21V1ZeT8PhdXkiRJkvS/6PW2QEmSJEn6X1hcSZIkSVIHLK4kSZIkqQMWV5Ik\nSZLUAYsrSZIkSeqAxZUkSZIkdcDiSpIkSZI6YHElSZIkSR34Bikt02UAw3ESAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fdda93e9590>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot(lambda x: math.tanh(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### ReLU (*Rectifier Linear Unit*)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$$ g(x) = \\max(0, x) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### ReLU – zalety\n",
    "* Mniej podatna na problem zanikającego gradientu (*vanishing gradient*) niż funkcje sigmoidalne, dzięki czemu SGD jest szybciej zbieżna.\n",
    "* Prostsze obliczanie gradientu.\n",
    "* Dzięki zerowaniu ujemnych wartości, wygasza neurony, „rozrzedzając” sieć (*sparsity*), co przyspiesza obliczenia."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#### ReLU – wady\n",
    "* Dla dużych wartości gradient może „eksplodować”.\n",
    "* „Wygaszanie” neuronów."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### ReLU – wykres"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1cAAAG2CAYAAACTRXz+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAGJhJREFUeJzt3X+sZOd91/HPN/6RRQk0AS+NYzvE\npdalpkBLLNcRVbkmTnFMFTelkWwhkQDVtgirBQmBi0UCgUitkABBItpVY9VJo6QWqZul2eIkda7d\nCNzaiZzWXmfdxQR2t4lN7CT0KmnM1g9/7Gx7vZ659+7Oc2fOzLxe0tXOj7PzPH98vfbb58zZaq0F\nAACA6bxk3hsAAABYBuIKAACgA3EFAADQgbgCAADoQFwBAAB0IK4AAAA66BJXVXVnVT1dVY9OeH+9\nqr5WVY+Mft7RY10AAIChuLDT5/x8kvckef82x/x6a+0HOq0HAAAwKF3OXLXWHkjybI/PAgAAWESz\n/M7V66vqc1X1q1X152e4LgAAwJ7rdVngTj6b5M+01jar6qYkv5zkqnEHVtWBJAeSZN++fa97zWte\nM6Mtsiief/75vOQl7sXCC5kLJjEbjGMukme+0fJ7/6/lWy6uvHJfzXs7g2E2GOeJJ574cmtt/07H\nVWuty4JV9dokv9Ja+85dHPuFJNe01r683XFra2vt6NGjXfbH8tjY2Mj6+vq8t8HAmAsmMRuMs8pz\n0VrLP//oo/mFB/93fvSvfltuv/HPpUpcnbHKs8FkVfWZ1to1Ox03kyyvqlfV6J/aqrp2tO4zs1gb\nAIDThBXsrS6XBVbVh5KsJ7mkqk4keWeSi5KktfYzSX44yd+vqlNJvpHkltbrlBkAADsSVrD3usRV\na+3WHd5/T07fqh0AgBkTVjAbvq0HALDEhBXMjrgCAFhSwgpmS1wBACwhYQWzJ64AAJaMsIL5EFcA\nAEtEWMH8iCsAgCUhrGC+xBUAwBIQVjB/4goAYMEJKxgGcQUAsMCEFQyHuAIAWFDCCoZFXAEALCBh\nBcMjrgAAFoywgmESVwAAC0RYwXCJKwCABSGsYNjEFQDAAhBWMHziCgBg4IQVLAZxBQAwYMIKFoe4\nAgAYKGEFi0VcAQAMkLCCxSOuAAAGRljBYhJXAAADIqxgcYkrAICBEFaw2MQVAMAACCtYfOIKAGDO\nhBUsB3EFADBHwgqWh7gCAJgTYQXLRVwBAMyBsILlI64AAGZMWMFyElcAADMkrGB5iSsAgBkRVrDc\nxBUAwAwIK1h+4goAYI8JK1gN4goAYA8JK1gd4goAYI8IK1gt4goAYA8IK1g94goAoDNhBatJXAEA\ndCSsYHWJKwCAToQVrDZxBQDQgbACxBUAwJSEFZCIKwCAqQgr4AxxBQBwnoQVsJW4AgA4D8IKOJu4\nAgA4R8IKGEdcAQCcA2EFTCKuAAB2SVgB2xFXAAC7IKyAnYgrAIAdCCtgN8QVAMA2hBWwW+IKAGAC\nYQWcC3EFADCGsALOlbgCADiLsALOh7gCANhCWAHnS1wBAIwIK2Aa4goAIMIKmJ64AgBWnrACehBX\nAMBKE1ZAL+IKAFhZwgroqUtcVdWdVfV0VT064f2qqv9QVceq6req6i/3WBcA4HwJK6C3Xmeufj7J\njdu8/6YkV41+DiT5T53WBQA4Z8IK2AsX9viQ1toDVfXabQ65Ocn7W2styYNV9YqqurS19sUe6wMA\n7FZrLR848lzuOy6sgL5m9Z2ry5Ic3/L8xOg1AICZOXPG6r7jp4QV0F2XM1c9VdWBnL50MPv378/G\nxsZ8N8TgbG5umgtexFwwidngjD86Y3UqN1zWct2+L+X++5+a97YYGH9mMI1ZxdXJJFdseX756LUX\naa0dTHIwSdbW1tr6+vqeb47FsrGxEXPB2cwFk5gNkq1nrE5fCnjdvi/l+uuvn/e2GCB/ZjCNWV0W\neCjJ3x7dNfC6JF/zfSsAYBbcvAKYlS5nrqrqQ0nWk1xSVSeSvDPJRUnSWvuZJIeT3JTkWJKvJ/k7\nPdYFANiOsAJmqdfdAm/d4f2W5B/0WAsAYDeEFTBrs7osEABgZoQVMA/iCgBYKsIKmBdxBQAsDWEF\nzJO4AgCWgrAC5k1cAQALT1gBQyCuAICFJqyAoRBXAMDCElbAkIgrAGAhCStgaMQVALBwhBUwROIK\nAFgowgoYKnEFACwMYQUMmbgCABaCsAKGTlwBAIMnrIBFIK4AgEETVsCiEFcAwGAJK2CRiCsAYJCE\nFbBoxBUAMDjCClhE4goAGBRhBSwqcQUADIawAhaZuAIABkFYAYtOXAEAcyesgGUgrgCAuRJWwLIQ\nVwDA3AgrYJmIKwBgLoQVsGzEFQAwc8IKWEbiCgCYKWEFLCtxBQDMjLAClpm4AgBmQlgBy05cAQB7\nTlgBq0BcAQB7SlgBq0JcAQB7RlgBq0RcAQB7QlgBq0ZcAQDdCStgFYkrAKArYQWsKnEFAHQjrIBV\nJq4AgC6EFbDqxBUAMDVhBSCuAIApCSuA08QVAHDehBXAHxFXAMB5EVYALySuAIBzJqwAXkxcAQDn\nRFgBjCeuAIBdE1YAk4krAGBXhBXA9sQVALAjYQWwM3EFAGxLWAHsjrgCACYSVgC7J64AgLGEFcC5\nEVcAwIsIK4BzJ64AgBcQVgDnR1wBAH9IWAGcP3EFACQRVgDTElcAgLAC6EBcAcCKE1YAfYgrAFhh\nwgqgH3EFACtKWAH0Ja4AYAUJK4D+xBUArBhhBbA3xBUArBBhBbB3usRVVd1YVUer6lhV3T7m/bdX\n1f+pqkdGPz/SY10AYPeEFcDeunDaD6iqC5K8N8kbk5xI8lBVHWqtHTnr0F9srd027XoAwLkTVgB7\nr8eZq2uTHGutPdlaey7Jh5Pc3OFzAYAOhBXAbEx95irJZUmOb3l+Isn3jDnub1bV9yV5Isk/aq0d\nH3NMqupAkgNJsn///mxsbHTYIstkc3PTXPAi5oJJVn02Wmv5wJHnct/xU7npyoty3b4v5f77n5r3\ntuZu1eeCycwG0+gRV7vxX5J8qLX2zar60SR3Jflr4w5srR1McjBJ1tbW2vr6+oy2yKLY2NiIueBs\n5oJJVnk2zpyxuu+4M1ZnW+W5YHtmg2n0uCzwZJIrtjy/fPTaH2qtPdNa++bo6c8leV2HdQGACVwK\nCDB7PeLqoSRXVdWVVXVxkluSHNp6QFVduuXpm5M83mFdAGAMYQUwH1NfFthaO1VVtyW5N8kFSe5s\nrT1WVe9K8nBr7VCSH6+qNyc5leTZJG+fdl0A4MWEFcD8dPnOVWvtcJLDZ732ji2PfzLJT/ZYCwAY\nT1gBzFeXv0QYAJgvYQUwf+IKABacsAIYBnEFAAtMWAEMh7gCgAUlrACGRVwBwAISVgDDI64AYMEI\nK4BhElcAsECEFcBwiSsAWBDCCmDYxBUALABhBTB84goABk5YASwGcQUAAyasABaHuAKAgRJWAItF\nXAHAAAkrgMUjrgBgYIQVwGISVwAwIMIKYHGJKwAYCGEFsNjEFQAMgLACWHziCgDmTFgBLAdxBQBz\nJKwAloe4AoA5EVYAy0VcAcAcCCuA5SOuAGDGhBXAchJXADBDwgpgeYkrAJgRYQWw3MQVAMyAsAJY\nfuIKAPaYsAJYDeIKAPaQsAJYHeIKAPaIsAJYLeIKAPaAsAJYPeIKADoTVgCrSVwBQEfCCmB1iSsA\n6ERYAaw2cQUAHQgrAMQVAExJWAGQiCsAmIqwAuAMcQUA50lYAbCVuAKA8yCsADibuAKAcySsABhH\nXAHAORBWAEwirgBgl4QVANsRVwCwC8IKgJ2IKwDYgbACYDfEFQBsQ1gBsFviCgAmEFYAnAtxBQBj\nCCsAzpW4AoCzCCsAzoe4AoAthBUA50tcAcCIsAJgGuIKACKsAJieuAJg5QkrAHoQVwCsNGEFQC/i\nCoCVJawA6ElcAbCShBUAvYkrAFaOsAJgL4grAFaKsAJgr4grAFaGsAJgL4krAFaCsAJgr4krAJae\nsAJgFrrEVVXdWFVHq+pYVd0+5v2XVtUvjt7/jap6bY91AWAnwgqAWZk6rqrqgiTvTfKmJFcnubWq\nrj7rsL+X5CuttW9P8u+S/PS06wLATlpr+cCR54QVADNxYYfPuDbJsdbak0lSVR9OcnOSI1uOuTnJ\nvxg9/s9J3lNV1Vpr233ws7/f8q9/5ch2h7CCjp/4Zj69aS54IXPBOF945uu57/gpYQXATPSIq8uS\nHN/y/ESS75l0TGvtVFV9LcmfSvLlsz+sqg4kOZAkF3/rn80v/Pf/2WGLLJOWljpuLnghc8E4VcmN\nV7Rct+9Luf/+p+a9HQZkc3MzGxsb894GA2Q2mEaPuOqqtXYwycEkWVtba59/99+Y844Ymo2Njayv\nr897GwyMuWASs8E45oJJzAbT6HFDi5NJrtjy/PLRa2OPqaoLk3xLkmc6rA0AADAIPeLqoSRXVdWV\nVXVxkluSHDrrmENJ3jZ6/MNJ7tvp+1YAAACLZOrLAkffobotyb1JLkhyZ2vtsap6V5KHW2uHkrwv\nyQeq6liSZ3M6wAAAAJZGl+9ctdYOJzl81mvv2PL495O8tcdaAAAAQ9TlLxEGAABYdeIKAACgA3EF\nAADQgbgCAADoQFwBAAB0IK4AAAA6EFcAAAAdiCsAAIAOxBUAAEAH4goAAKADcQUAANCBuAIAAOhA\nXAEAAHQgrgAAADoQVwAAAB2IKwAAgA7EFQAAQAfiCgAAoANxBQAA0IG4AgAA6EBcAQAAdCCuAAAA\nOhBXAAAAHYgrAACADsQVAABAB+IKAACgA3EFAADQgbgCAADoQFwBAAB0IK4AAAA6EFcAAAAdiCsA\nAIAOxBUAAEAH4goAAKADcQUAANCBuAIAAOhAXAEAAHQgrgAAADoQVwAAAB2IKwAAgA7EFQAAQAfi\nCgAAoANxBQAA0IG4AgAA6EBcAQAAdCCuAAAAOhBXAAAAHYgrAACADsQVAABAB+IKAACgA3EFAADQ\ngbgCAADoQFwBAAB0IK4AAAA6EFcAAAAdiCsAAIAOxBUAAEAH4goAAKCDqeKqqv5kVX2iqn5n9Osr\nJxz3B1X1yOjn0DRrAgAADNG0Z65uT/JrrbWrkvza6Pk432itfdfo581TrgkAADA408bVzUnuGj2+\nK8kPTvl5AAAAC6laa+f/m6u+2lp7xehxJfnKmednHXcqySNJTiX5qdbaL2/zmQeSHEiS/fv3v+7u\nu+8+7/2xnDY3N/Pyl7983ttgYMwFk5gNxjEXTGI2GOf666//TGvtmp2O2zGuquqTSV415q07kty1\nNaaq6iuttRd976qqLmutnayqb0tyX5I3tNb+x06bW1tba0ePHt3pMFbMxsZG1tfX570NBsZcMInZ\nYBxzwSRmg3GqaldxdeFOB7TWbthmkaeq6tLW2her6tIkT0/4jJOjX5+sqo0k351kx7gCAABYFNN+\n5+pQkreNHr8tyUfPPqCqXllVLx09viTJX0lyZMp1AQAABmXauPqpJG+sqt9JcsPoearqmqr6udEx\n35Hk4ar6XJJP5fR3rsQVAACwVHa8LHA7rbVnkrxhzOsPJ/mR0eP/luQvTLMOAADA0E175goAAICI\nKwAAgC7EFQAAQAfiCgAAoANxBQAA0IG4AgAA6EBcAQAAdCCuAAAAOhBXAAAAHYgrAACADsQVAABA\nB+IKAACgA3EFAADQgbgCAADoQFwBAAB0IK4AAAA6EFcAAAAdiCsAAIAOxBUAAEAH4goAAKADcQUA\nANCBuAIAAOhAXAEAAHQgrgAAADoQVwAAAB2IKwAAgA7EFQAAQAfiCgAAoANxBQAA0IG4AgAA6EBc\nAQAAdCCuAAAAOhBXAAAAHYgrAACADsQVAABAB+IKAACgA3EFAADQgbgCAADoQFwBAAB0IK4AAAA6\nEFcAAAAdiCsAAIAOxBUAAEAH4goAAKADcQUAANCBuAIAAOhAXAEAAHQgrgAAADoQVwAAAB2IKwAA\ngA7EFQAAQAfiCgAAoANxBQAA0IG4AgAA6EBcAQAAdCCuAAAAOhBXAAAAHYgrAACADqaKq6p6a1U9\nVlXPV9U12xx3Y1UdrapjVXX7NGsCAAAM0bRnrh5N8kNJHph0QFVdkOS9Sd6U5Ookt1bV1VOuCwAA\nMCgXTvObW2uPJ0lVbXfYtUmOtdaeHB374SQ3JzkyzdoAAABDMovvXF2W5PiW5ydGrwEAACyNHc9c\nVdUnk7xqzFt3tNY+2ntDVXUgyYEk2b9/fzY2NnovwYLb3Nw0F7yIuWASs8E45oJJzAbT2DGuWms3\nTLnGySRXbHl++ei1SesdTHIwSdbW1tr6+vqUy7NsNjY2Yi44m7lgErPBOOaCScwG05jFZYEPJbmq\nqq6sqouT3JLk0AzWBQAAmJlpb8X+lqo6keT1ST5WVfeOXn91VR1OktbaqSS3Jbk3yeNJ7m6tPTbd\ntgEAAIZl2rsF3pPknjGv/26Sm7Y8P5zk8DRrAQAADNksLgsEAABYeuIKAACgA3EFAADQgbgCAADo\nQFwBAAB0IK4AAAA6EFcAAAAdiCsAAIAOxBUAAEAH4goAAKADcQUAANCBuAIAAOhAXAEAAHQgrgAA\nADoQVwAAAB2IKwAAgA7EFQAAQAfiCgAAoANxBQAA0IG4AgAA6EBcAQAAdCCuAAAAOhBXAAAAHYgr\nAACADsQVAABAB+IKAACgA3EFAADQgbgCAADoQFwBAAB0IK4AAAA6EFcAAAAdiCsAAIAOxBUAAEAH\n4goAAKADcQUAANCBuAIAAOhAXAEAAHQgrgAAADoQVwAAAB2IKwAAgA7EFQAAQAfiCgAAoANxBQAA\n0IG4AgAA6EBcAQAAdCCuAAAAOhBXAAAAHYgrAACADsQVAABAB+IKAACgA3EFAADQgbgCAADoQFwB\nAAB0IK4AAAA6EFcAAAAdiCsAAIAOxBUAAEAH4goAAKCDqeKqqt5aVY9V1fNVdc02x32hqn67qh6p\nqoenWRMAAGCILpzy9z+a5IeS/Owujr2+tfblKdcDAAAYpKniqrX2eJJUVZ/dAAAALKhZfeeqJfl4\nVX2mqg7MaE0AAICZ2fHMVVV9Msmrxrx1R2vto7tc53tbayer6k8n+URVfb619sCE9Q4kORNg36yq\nR3e5BqvjkiQuMeVs5oJJzAbjmAsmMRuMs7abg3aMq9baDdPupLV2cvTr01V1T5Jrk4yNq9bawSQH\nk6SqHm6tTbxRBqvJXDCOuWASs8E45oJJzAbj7PamfHt+WWBVvayq/viZx0m+P6dvhAEAALA0pr0V\n+1uq6kSS1yf5WFXdO3r91VV1eHTYtyb5dFV9LslvJvlYa+2/TrMuAADA0Ex7t8B7ktwz5vXfTXLT\n6PGTSf7SeS5x8Px3xxIzF4xjLpjEbDCOuWASs8E4u5qLaq3t9UYAAACW3qxuxQ4AALDUBh1XVfWv\nquq3quqRqvp4Vb163ntiGKrq31TV50fzcU9VvWLee2L+quqtVfVYVT1fVe70tOKq6saqOlpVx6rq\n9nnvh2Goqjur6ml/1QtbVdUVVfWpqjoy+vfIT8x7TwxDVe2rqt+sqs+NZuNfbnv8kC8LrKo/0Vr7\nv6PHP57k6tbaj815WwxAVX1/kvtaa6eq6qeTpLX2T+e8Leasqr4jyfNJfjbJP26t7eq2qSyfqrog\nyRNJ3pjkRJKHktzaWjsy140xd1X1fUk2k7y/tfad894Pw1BVlya5tLX22dFdrj+T5Af9mUFVVZKX\ntdY2q+qiJJ9O8hOttQfHHT/oM1dnwmrkZUmGW4LMVGvt4621U6OnDya5fJ77YRhaa4+31o7Oex8M\nwrVJjrXWnmytPZfkw0lunvOeGIDW2gNJnp33PhiW1toXW2ufHT3+vSSPJ7lsvrtiCNppm6OnF41+\nJjbJoOMqSarq3VV1PMnfSvKOee+HQfq7SX513psABuWyJMe3PD8R/6EE7EJVvTbJdyf5jfnuhKGo\nqguq6pEkTyf5RGtt4mzMPa6q6pNV9eiYn5uTpLV2R2vtiiQfTHLbfHfLLO00G6Nj7khyKqfngxWw\nm7kAgPNRVS9P8pEk//CsK6hYYa21P2itfVdOXyl1bVVNvKR4qr/nqofW2g27PPSDSQ4neecebocB\n2Wk2qurtSX4gyRvakL88SFfn8GcGq+1kkiu2PL989BrAWKPv03wkyQdba7807/0wPK21r1bVp5Lc\nmGTsTXHmfuZqO1V11ZanNyf5/Lz2wrBU1Y1J/kmSN7fWvj7v/QCD81CSq6rqyqq6OMktSQ7NeU/A\nQI1uWvC+JI+31v7tvPfDcFTV/jN3pa6qP5bTN0qa2CRDv1vgR5Ks5fTdv/5Xkh9rrfk/j6SqjiV5\naZJnRi896E6SVNVbkvzHJPuTfDXJI621vz7fXTEvVXVTkn+f5IIkd7bW3j3nLTEAVfWhJOtJLkny\nVJJ3ttbeN9dNMXdV9b1Jfj3Jb+f0f3cmyT9rrR2e364Ygqr6i0nuyul/l7wkyd2ttXdNPH7IcQUA\nALAoBn1ZIAAAwKIQVwAAAB2IKwAAgA7EFQAAQAfiCgAAoANxBQAA0IG4AgAA6EBcAQAAdPD/AVNA\ngTNkBNHAAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fdda936c6d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot(lambda x: max(0, x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Softplus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$$ g(x) = \\log(1 + e^{x}) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Wygładzona wersja ReLU."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Softplus – wykres"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1cAAAG2CAYAAACTRXz+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xd0XOWd//HPV733YlmWuyz3go0J\nJWCHXoIDgQCppKxDyi7pIWEDu+THhkAKKbtJ2IQEshAgEEIzmBaFboONuy13W5Zly+oa9dE8vz80\nNsZItrCudUfS+3WOjmdGF93vnvOs7HfunWfMOScAAAAAQP9E+T0AAAAAAAwFxBUAAAAAeIC4AgAA\nAAAPEFcAAAAA4AHiCgAAAAA8QFwBAAAAgAc8iSszu9vMqsxsXS/fX2BmDWa2Kvx1kxfnBQAAAIBI\nEePRz/mTpF9Luvcox7zsnLvEo/MBAAAAQETx5MqVc+4lSbVe/CwAAAAAGIwG8j1Xp5rZajN72sym\nDeB5AQAAAOCE8+q2wGNZKWmMcy5gZhdJ+ruk4p4ONLPFkhZLUkJCwtzRo0cP0IgYLEKhkKKi2IsF\n78a6QG9YG+iJX+uiMyQdaAmpIyRlxJsy4m3AZ8DR8TsDPdm8eXO1cy73WMeZc86TE5rZWElPOuem\n9+HYnZLmOeeqj3ZcSUmJKysr82Q+DB2lpaVasGCB32MgwrAu0BvWBnrix7p4em2lvv3wGsVEm35+\n1WwtLMkb0POjb/idgZ6Y2Qrn3LxjHTcgV67MbISk/c45Z2bz1X07Ys1AnBsAAMBPnV0h3fb0Jv3h\nlR2aVZSh//nESSrMSPR7LAAngCdxZWZ/kbRAUo6Z7ZF0s6RYSXLO/VbSFZK+ZGZBSa2SrnZeXTID\nAACIUPsa2vSV+1dqxa46XXvaWH3/oimKi+GWM2Co8iSunHPXHOP7v1b3Vu0AAADDwitbqnX9A2+r\nrbNLv7pmjj48a6TfIwE4wQZqQwsAAIBhIRRy+tWLW3XnC5tVnJei//nEXE3MS/F7LAADgLgCAADw\nSG1zh7724Cq9tPmALptTqFsvm66kOP65BQwX/H87AACAB97eXaev3LdS1YEO/ddlM3TN/CKZsdU6\nMJwQVwAAAP3gnNM9r+3UrUs2akR6gh750mmaMSrd77EA+IC4AgAAOE6B9qC++8gaPbWmUudMydNP\nr5yt9KRYv8cC4BPiCgAA4DiU7WvSl+5boZ3Vzbrhwsla/MHxioriNkBgOCOuAAAA3qe/vlWumx5b\nr5SEGN3/Lx/QB8Zn+z0SgAhAXAEAAPRRc3tQP3hsnf62skKnjs/WL66ZrbzUBL/HAhAhiCsAAIA+\n2LSvUV+5b6W2Vzfra+cU618/VKxobgMEcBjiCgAA4Cicc3rwzXLd/Ph6pSXG6r4vnKLTJuT4PRaA\nCERcAQAA9CLQHtSNj67VY6v26oyJOfr5VbOVmxrv91gAIhRxBQAA0IP1exv0r/e/rZ01zfrmuZP0\n5YUTuQ0QwFERVwAAAIdxzun/lu3WD5/coMykWHYDBNBnxBUAAEBYY1unvve3tXpqTaXOnJSrn39s\nlrJTuA0QQN8QVwAAAJLW7mnQV/+yUnvqWvWdC0p03ZkT+FBgAO8LcQUAAIY155zufX2Xbn1qo7JT\n4vTA4g/o5LFZfo8FYBAirgAAwLBV39Kh7z6yRkvX79eHJufpJ1fOUlZynN9jARikiCsAADAsLdte\no689uErVgXZ9/6LJ+sIZ47kNEEC/EFcAAGBYCXaF9OiWDj2x9A2NzkrSI186TTNHZfg9FoAhgLgC\nAADDRkV9q77+wCot39mpy+cU6paPTFdKPP8cAuANfpsAAIBh4Zl1lfruI2sV7ArpX2bE6carZvs9\nEoAhhrgCAABDWltnl3745Abdt2y3Zo5K1y+vnqOd6970eywAQxBxBQAAhqyyfU3617+s1Ob9AS0+\nc7y+dV6J4mKitNPvwQAMScQVAAAYcpxz+r9lu/X/ntyg1IQY3fO5+TprUq7fYwEY4ogrAAAwpBz+\n2VVnTsrVT6+cpdzUeL/HAjAMEFcAAGDIWL6jVtc/8LaqA+268aIp+vwZ4/jsKgADhrgCAACDXmdX\nSHc+v1m/Kd3GZ1cB8A1xBQAABrXtBwL6+oOrtHpPg66cO0o3XzqNz64C4At+8wAAgEHJOacH3izX\nLU9sUFxMlH7ziZN04YwCv8cCMIwRVwAAYNCpCbTrhr+t1XMb9uv0idn66ZWzNSI9we+xAAxzxBUA\nABhUSsuq9O2H16ihpVP/fvEUfe50Nq0AEBmIKwAAMCi0dXbptqc36U+v7VRJfqru/dx8TSlI83ss\nADiEuAIAABFvw95Gfe3Bt7V5f0CfPX2svnvBZCXERvs9FgC8C3EFAAAiVijk9IdXduiOpWVKT4rV\nPZ+br7Mm5fo9FgD0iLgCAAARqbKhVd/662q9urVG503N120fnams5Di/xwKAXhFXAAAg4jy5Zq9u\nfHSdOoIh3Xb5DF11cpHM2LQCQGQjrgAAQMSob+nQDx5brydW79WsogzdedVsjctJ9nssAOgT4goA\nAESE0rIqfefhNapt7tA3z52kLy2YoJjoKL/HAoA+I64AAICvmtuDunXJRt2/bLcm5afo7mtP1vTC\ndL/HAoD3jbgCAAC+eXNnrb750GqV17Xoi2eO19fPncQW6wAGLeIKAAAMuLbOLv38uc266+XtGpWZ\nqAcXn6r547L8HgsA+oW4AgAAA2r93gZ948HVKtvfpGvmj9aNF09RSjz/JAEw+PGbDAAADIhgV0i/\n/ec23fn8FmUlx+mP156shZPz/B4LADxDXAEAgBNu+4GAvvHQaq0qr9clMwv0w0XTlckHAgMYYogr\nAABwwoRCTve+vlO3PbNJ8THR+uU1c3TprJF+jwUAJwRxBQAATohdNc369sNrtHxHrc6alKvbr5ip\n/LQEv8cCgBOGuAIAAJ4KhZzueX2nbn+mTDFRpts/OlNXzhslM/N7NAA4oYgrAADgmZ3VzfrOw2u0\nfGf31arbPjpDBemJfo8FAAOCuAIAAP0WCjn98bWdumPpJsVGR+mOK2bqirlcrQIwvBBXAACgX3ZU\nN+s7D6/WmzvrtLAkVz+6fKZGpPPeKgDDD3EFAACOS1fI6Y+v7tAdS8sUFxOln1w5Sx89qZCrVQCG\nLeIKAAC8b9sPBPTth9doxa46fWhynv7rshlcrQIw7BFXAACgz7pCTne/skM/ebZM8TFR+tnHZumy\nOVytAgCJuAIAAH20sbJRNzyyRqv3NOjsyXn6r8tn8LlVAHAY4goAABxVe7BLv35xq35Tuk1pibH6\nxdWzdemskVytAoAjEFcAAKBXb+2s1XcfWaNtB5p1+ZxC/fslU5WVHOf3WAAQkYgrAADwHoH2oG5/\nZpP+/MYujUxP1J8+e7IWlOT5PRYARDTiCgAAvMs/NlXpxkfXqrKxTZ85day+dX6JUuL5JwMAHIsn\nvynN7G5Jl0iqcs5N7+H7JukXki6S1CLpWufcSi/ODQAAvFETaNctT27QY6v2amJeih6+7jTNHZPp\n91gAMGh49T9D/UnSryXd28v3L5RUHP46RdJvwn8CAACfOef02Kq9uuXJDWpq69T1ZxfrywsnKD4m\n2u/RAGBQ8SSunHMvmdnYoxyySNK9zjkn6Q0zyzCzAudcpRfnBwAAx6e8tkU3PbZO/yg7oNlFGfrx\nR2eqZESq32MBwKA0UDdQF0oqP+z5nvBrxBUAAD7o7Arp7ld26M7nt8hM+sElU3XtaWMVHcX26gBw\nvCLu3almtljSYknKzc1VaWmpvwMh4gQCAdYF3oN1gd6wNt5ra32X7lnfofKmkObkReuTU+KUHdyl\nl1/a5fdoA4Z1gd6wNtAfAxVXFZKKDns+Kvzaezjn7pJ0lySVlJS4BQsWnPDhMLiUlpaKdYEjsS7Q\nG9bGOxpaO3XH0k26b9lu5acm6Hefmqbzp43weyxfsC7QG9YG+mOg4upxSV81swfUvZFFA++3AgBg\nYDjn9OSaSt3y5AbVBNp17Wlj9c3z2F4dALzm1Vbsf5G0QFKOme2RdLOkWElyzv1W0hJ1b8O+Vd1b\nsX/Wi/MCAICj213Toh88tk7/3HxA0wvTdPdnTtaMUel+jwUAQ5JXuwVec4zvO0lf8eJcAADg2Dq7\nQvrfl7frF89vUUyU6aZLpurTp45RTHSU36MBwJDF/QAAAAwxK3bV6vt/W6ey/U06f1q+/uPSaSpI\nT/R7LAAY8ogrAACGiJpAu378zCY99NYejUxP0P9+ep7OnZrv91gAMGwQVwAADHJdIae/LN+tO5aW\nqbk9qMVnjtf1ZxcrmQ0rAGBA8VsXAIBBbFV5vX7w93VaW9GgD4zP0g8XTVdxfqrfYwHAsERcAQAw\nCNU1d+j2pZv0wJvlyk2J1y+unq1LZ42Umfk9GgAMW8QVAACDSCjk9OBb5frxM5vU1BbU508fp+vP\nKVZqQqzfowHAsEdcAQAwSKzZU68fPLZeq8vrNX9c9y2AJSO4BRAAIgVxBQBAhKtv6dAdS8t0//Ld\nyk6O151Xzdai2dwCCACRhrgCACBCdYWcHnyzXD95tkz1LR269rSx+vq5k5TGLYAAEJGIKwAAItDy\nHbX6j8fXa0Nlo+aPzdJ/XDpNU0em+T0WAOAoiCsAACJIRX2rfrRko55cU6mR6Qn69cfn6OIZBdwC\nCACDAHEFAEAEaO3o0u9e2qbf/nObnJOuP7tY1501QYlx0X6PBgDoI+IKAAAfOef01NpK/WjJJlXU\nt+rimQX6/kVTVJiR6PdoAID3ibgCAMAnG/Y26j+fWK9lO2o1pSBNP/vYLJ0yPtvvsQAAx4m4AgBg\ngNU2d+gnz5bpgeW7lZ4Yq1svm66rTx6t6CjeVwUAgxlxBQDAAGkPdunPr+/SL1/YouaOLn3mtLH6\n2tmTlJ7E1uoAMBQQVwAAnGDOOT29bp9ue3qTdte26KxJufr3i6eoOD/V79EAAB4irgAAOIHe3l2n\nW5/aqLd21akkP1X3fm6+zpyU6/dYAIATgLgCAOAEKK9t0e1Ly/TE6r3KTY3XbZfP0JXzinhfFQAM\nYcQVAAAeamjt1P/8Y6v++OpORUVJ//ahifriWROUHM9fuQAw1PGbHgAAD3R2hXT/st268/nNqm/t\n1OVzRulb509SQTqfVwUAwwVxBQBAPzjn9PzGKv3o6Y3afqBZp47P1o0XT9H0wnS/RwMADDDiCgCA\n47Ryd51ue3qTlu+o1fjcZP3+0/N09pQ8mfG+KgAYjogrAADep61VTbr9mTI9u2G/clLidMuiabpm\n/mjFRkf5PRoAwEfEFQAAfVTZ0Ko7n9uiv64oV1JcjL5x7iR9/oxxbFYBAJBEXAEAcEz1LR36Tek2\n/em1nXJOuva0cfrKwgnKTon3ezQAQAQhrgAA6EVbZ5f++OpO/aZ0q5rag7psdqG+fu4kFWUl+T0a\nACACEVcAABwh2BXSwyv26M7nt2hfY5s+NDlP3z6/RFMK0vweDQAQwYgrAADCQiGnZ9bv00+fLdO2\nA82aMzpDv7h6tk4Zn+33aACAQYC4AgAMe845vbCxSj97brM2VDZqYl6KfvepuTpvaj7bqgMA+oy4\nAgAMW845vbylWj99brNWl9drTHaSfn7VLF06q1DRUUQVAOD9Ia4AAMPSG9tr9LNnN2v5zloVZiTq\nxx+doctPGsVnVQEAjhtxBQAYVlburtPPnt2sV7ZWKy81XrcsmqarTi5SfEy036MBAAY54goAMCys\nq2jQz57brBc3VSk7OU7/fvEUffIDY5QQS1QBALxBXAEAhrTyppCu+/MKPbN+n9ITY/Xt80t07Wlj\nlRzPX4EAAG/xNwsAYEhaV9GgX724RUvXtyolvlPXn12sz39wnNISYv0eDQAwRBFXAIAhZXV5vX71\n4hY9v7FKqQkxunRCrG75xAJlJMX5PRoAYIgjrgAAQ8KKXXX61YtbVFp2QOmJsfrGuZP0mdPG6u1l\nrxJWAIABQVwBAAa15Ttq9csXtuiVrdXKTIrVdy4o0ac+MEap3P4HABhgxBUAYNBxzun17TX65Qtb\n9Mb2WuWkxOn7F03WJ04Zw0YVAADf8DcQAGDQcM7ppS3V+vWLW/Tmzjrlpcbrpkum6pr5o5UYx5bq\nAAB/EVcAgIjXFXJasrZSvyndpg2VjSpIT9Ati6bpY/OK+JwqAEDEIK4AABGrrbNLj6zco7te2q5d\nNS0an5us26+YqY/MLlRcTJTf4wEA8C7EFQAg4jS2deq+N3brD6/sUHWgXbNGpet7n5yr86bmKyrK\n/B4PAIAeEVcAgIhR1dSmP766U//3+i41tQf1weIcfWnBbJ06PltmRBUAILIRVwAA3+2qadZdL23X\nX1fsUbArpAtnFOhLZ03Q9MJ0v0cDAKDPiCsAgG9Wldfr9y9v15K1lYqJitJH547S4jPHa1xOst+j\nAQDwvhFXAIAB1RVyem7Dfv3hle16c2edUuNj9C8fHK/PnzFOeWkJfo8HAMBxI64AAAOiuT2oh1fs\n0d2v7tCumhaNykzUTZdM1cdOLlIKH/wLABgC+NsMAHBCVTa06p7Xdun+ZbvU2BbUSaMz9N0LJuu8\nqfmKiWY7dQDA0EFcAQBOiHUVDfrDKzv0xOq9CjmnC6cX6HNnjNPcMZl+jwYAwAlBXAEAPNMVcvrH\npir9/pXtemN7rZLjovXpU8fqs6ePVVFWkt/jAQBwQhFXAIB+a2jp1ENvleveN3aqvLZVBekJ+v5F\nk3X1/NFKS4j1ezwAAAYEcQUAOG4bKxt17+s79ejbFWrrDGn+2CzdcMEUnTctX7G8nwoAMMwQVwCA\n9yXYFdKzG/brT6/t1PIdtUqIjdJHZhfq06eO1dSRaX6PBwCAb4grAECfVAfa9cDy3bpv2W5VNrRp\nVGaivnfhZF11cpEykuL8Hg8AAN8RVwCAo1pdXq97Xt+pJ1dXqqMrpDMm5uiWRdP1ocl5io4yv8cD\nACBiEFcAgPcItAf1+Kq9un/5Lq2raFRyXLSunl+kT586RhPzUv0eDwCAiERcAQAOWVfRoPuX79Zj\nb1eouaNLk0ek6pZF0/SROYXs+gcAwDF4EldmdoGkX0iKlvR759xtR3z/Wkl3SKoIv/Rr59zvvTg3\nAKB/WjqCemL1Xt2/bLdW72lQfEyULpk5Uh8/ZbROGp0hM279AwCgL/odV2YWLem/JZ0raY+kN83s\ncefchiMOfdA599X+ng8A4I2NlY26f9lu/f3tCjW1B1Wcl6KbPzxVl88ZpfQkrlIBAPB+eXHlar6k\nrc657ZJkZg9IWiTpyLgCAPispSOop9ZU6v7lu/X27nrFxUTp4hkF+vgpozVvTCZXqQAA6AdzzvXv\nB5hdIekC59wXws8/JemUw69ShW8L/JGkA5I2S/q6c668l5+3WNJiScrNzZ370EMP9Ws+DD2BQEAp\nKSl+j4EIw7ronXNOW+tDerkiqOWVQbV1SSOSTQuLYnX6yBilxA3toGJtoCesC/SGtYGeLFy4cIVz\nbt6xjhuoDS2ekPQX51y7mX1R0j2SPtTTgc65uyTdJUklJSVuwYIFAzQiBovS0lKxLnAk1sV77W9s\n099WVuivK8u1/UCbkuKidcnsUbpy7ijNH5c1bK5SsTbQE9YFesPaQH94EVcVkooOez5K72xcIUly\nztUc9vT3km734LwAgCN0BEN6cdN+PfTWHpWWVSnkpJPHZuq6MyfoopkFSolnk1gAAE4UL/6WfVNS\nsZmNU3dUXS3p44cfYGYFzrnK8NNLJW304LwAgLCNlY3661t79PdVFapt7lB+WryuO2uCrpg7SuNz\nub0FAICB0O+4cs4Fzeyrkpaqeyv2u51z683sFklvOecel/RvZnappKCkWknX9ve8ADDcHWhq1xOr\n9+rRtyu0tqJBsdGmc6fm68p5RfrgxBzFREf5PSIAAMOKJ/eHOOeWSFpyxGs3Hfb4e5K+58W5AGA4\na+kI6rkN+/Xo2xV6eUu1ukJO00am6eYPT9Wi2YXKSo7ze0QAAIYtbr4HgAjXFXJ6dWu1/v52hZ5Z\nv08tHV0qzEjUF88cr4/MKdSk/FS/RwQAACKuACAiOee0fm+j/v52hR5fvVdVTe1KTYjRotkj9ZHZ\nhTp5bJaioobHbn8AAAwWxBUARJDy2hY9sWavHl1ZoS1VAcVGmxaW5OmyOYVaODlPCbHRfo8IAAB6\nQVwBgM/21rfqqTWVenLNXq3e0yCpe/v0Wy+brotnFCgjifdRAQAwGBBXAOCDqsY2PbW2Uk+uqdSK\nXXWSpOmFabrhwsm6eEaBirKSfJ4QAAC8X8QVAAyQ6kC7nl63T0+u3qvlO2vlnDR5RKq+dd4kXTJz\npMbmJPs9IgAA6AfiCgBOoNrmDj27fp+eXFOp17ZVK+SkCbnJuv7sYl0ys0AT89jpDwCAoYK4AgCP\nVTa06tn1+/XMun1atqNGISeNzU7SlxdM1CWzClSSnyozdvoDAGCoIa4AwAM7qpv1zLp9emb9Pq0u\nr5ckFeel6CsLJ+r8aSM0bWQaQQUAwBBHXAHAcXDOaWNlk55Zv09L1+1T2f4mSdLMUen69vklOn/a\nCE3MS/F5SgAAMJCIKwDoo66Q08rddXpuQ/ctf7trWxRl0sljs3Tzh6fqvGkjVJiR6PeYAADAJ8QV\nABxFY1unXtp8QC9urNI/yqpU19Kp2GjT6RNz9OUFE3TO1HzlpMT7PSYAAIgAxBUAHGF3TYue37hf\nL2zar2XbaxUMOWUmxWphSZ7OnpKvMyflKDUh1u8xAQBAhCGuAAx7XSGnt3fX6fmNVXph435tqQpI\nkibmpejzHxync6bk66TRmYqOYkMKAADQO+IKwLBUE2jXy1uq9c/NB/TPzQdU29yhmCjT/HFZunr+\naJ0zJU9jsvlQXwAA0HfEFYBhIdgV0uo99Sot646ptRUNck7KSo7TmcU5OntKvs4qyVUat/sBAIDj\nRFwBGLL2NbTppfCVqZe3HFBjW1BRJs0ZnalvnDNJZ5XkavrIdEVxux8AAPAAcQVgyGjr7NKKXXV6\nacsB/bPsgDbt6/7sqfy0eF0wfYTOmpSnMybmKD2Jq1MAAMB7xBWAQasr5LSuokGvbqvWk2+2atvz\nz6o9GFJstGnemCzdcOFknTUpV5NHpMqMq1MAAODEIq4ADBrOOe2obtarW6v16tYavbatWo1tQUnS\nqBTTJ04ZqzOKszV/XLZS4vn1BgAABhb/+gAQ0aoa2/Tathq9srVar26tVmVDmySpMCNRF0wfodMn\n5ui0CTlav+J1LVgw1edpAQDAcEZcAYgoe+tbtWxHjZZtr9WyHbXaUd0sScpIitVpE7L11Yk5On1C\njsZkJ3GrHwAAiCjEFQDfOOe0p65Vb2yv0bIdtVq2o0blta2SpLSEGM0fl6WPzx+tD4zP1rSRaezq\nBwAAIhpxBWDAOOe0s6ZFyw7G1PYa7Q3f5peZFKv547L02dPG6ZTxWZo8Ik3RxBQAABhEiCsAJ0xH\nMKQNlY1asatOK3bV6q2ddapqapck5aTE6ZRx2bpufJZOGZet4rwUrkwBAIBBjbgC4Jm65o7ukNpd\npxW76rS6vF7twZCk7g0oPjA+W6eEY2pCbjLvmQIAAEMKcQXguIRCTturA+GrUnV6a1edth/o3nwi\nJso0rTBdnzhljOaNzdRJozM1Ij3B54kBAABOLOIKQJ/UBNq1Zk+DVpXXa/Weeq0qr1d9S6ek7vdL\nzR2TqSvmjtLc0ZmaVZShhNhonycGAAAYWMQVgPdo7ejSur0NWl1efyimDu7iZyZNykvV+VNHaO7Y\nTM0dk6nxOdziBwAAQFwBw1ywK6QtVQGtPnRFqkGb9zepK+Qkdb9XalZRuj55yhjNKsrQ9MJ0pcTz\nqwMAAOBI/AsJGEY6giFtqWrS+opGrdvboHUVDdpY2aTWzi5JUnpirGYVZejcKXmaVZShmaMylJsa\n7/PUAAAAgwNxBQxRrR1d2rSvUev2Nmp9RYPW7W3Q5n0BdXR1796XEh+jqSPTdPX8Is0alaHZRRka\nk53E7X0AAADHibgChoC65g5t3NeojZVNh0Jqa1VA4Tv7lJkUq+mF6frcGeM0bWSaphema0xWEp8r\nBQAA4CHiChhEOoIhbTsQ0KZ9jdpU2aRN+5q0aV+j9je2HzomPy1e00em64LpBYdCamR6AlekAAAA\nTjDiCohAzjntb2zXxkMR1aiyfU3aWhVQMHw5Ki46ShPzUnT6xBxNGZGmyQWpmjwijfdIAQAA+IS4\nAnzknNOBQLu27g9oS1VAW6sC2ry/+4pUQ2vnoeNGpidockGaPjQ5T5ML0jRlRKrG5iQrNjrKx+kB\nAABwOOIKGADOOVU2tGlLVUBb9ndfgToYU4dHVGp8jCbmp+iiGQWaEr4SVZKfqvSkWB+nBwAAQF8Q\nV4CHOoIhlde1aMeBZm09ENCW/QFtreqOqeaOrkPHZSbFqjg/VRfPLFBxXoqK81JVnJ+ivNR43hsF\nAAAwSBFXwPsUCjnta2zTjupmba9u1o4DzdpRHdCO6maV17Ue+vBdScpLjVdxfoqunFekiXkpmpiX\nouK8FGWn8L4oAACAoYa4AnrgnFNdS6d2VAe0/UCzdlS/+6s9GDp0bEJslMblpGjayHRdMnOkxuUk\na2xOsibmpSg9kdv5AAAAhgviCsNWsCukyoY27a5tefdXTYt21TSrsS146NiYKNPorCSNy0nWGRNz\nNDYnWeNzkjUuN1n5qQl8XhQAAACIKwxtjW2d2l3z7ngqD/9ZUdd6aFtzqTugRmUmqigrSTNHdV+B\nGp+brHE5KRqVmcjOfAAAADgq4gqDlnNODa2d2lPXqor6Vu2tb1VF+HFFfat217aovqXzXf9NZlKs\nRmclaUZhui6ZWaDRWUkqykrS6KwkFaQnKporUAAAADhOxBUiVlfIaX9jW3c01bdqT113QK3d1qZb\nV/5Te+tb37UDn9T9/qeRGYkqzEjUxTMKNCY76VBAFWUlKS2B90ABAADgxCCu4IuukFN1oF2VDW3a\n19Cm/Y1tqgz/efAq1L6Gtnfdtid1X3lKi3EqGZWsM4pzVBgOqcLM7j+zkuPYyhwAAAC+IK7gudaO\nLu1rfCeaDj7e1/DO4wOB9nfW/TevAAARZ0lEQVRtWS51v+cpPy1BIzMSNHdM5ruiqTAjUSMzEpUc\nH6PS0lItWDDPp//rAAAAgJ4RV+gT55wa24I60NSmqqZ2HTj4FWjXgcbuP6sa27WvsU0NrZ3v+e9T\n42OUn56gEWkJOqM4RyPSEpSfnqCCtASNSE9QflqCspPj2HUPAAAAgxZxNcy1B7tUHehQVWPbO7EU\nDqcjI6rjsM92OiguOkq5qfHKTY3X6OwkzR+XpRHhiDoYTSPSE5QSz1IDAADA0Ma/eIeYts4u1TZ3\nqLa5Q9WB9kOPa5o7VBN+XhN+rTbQoab2YI8/Jys5Trkp8cpLi9f4nORDAXXwKy81XrkpCUpLjOE9\nTgAAAICIq4gW7AqpsS2o+pYO1bV0qqG1Q3XNnb3GUk2g/T275x0UE2XKSo5TVnKcslPiVJSZ0f04\nOe6wYEpQbmq8slPi+EwnAAAA4H0irgZAV8ipsbVT9a2dqm/pUH1Lp+rDoVTf2qmGcDy963FLhxrb\ner6qJEmx0RaOo+4YGpOddCiWslPiDz3OCj9PS+AKEwAAAHAiEVd94JxTc0eXGls71djWqaa24KHH\nja1BNbV1qvGw1w5+vzumul9zruefbSalJcQqIylWGUlxykiK09icZGUmxSk9MVaZ4dfTk2KVmRSn\njMRYZaXEKTWeWAIAAAAiyZCPq2BXSM3tXWpq71Rze5cC7UEF2oNqDv8ZaAseiqTuMHrn8eHxFOol\njg5KiI1SWkKs0hJjlZYQo4ykOI3JTlZmUqzSw1GUmRyrjMS4d0Iqsfv4aHbIAwAAAAa9iI6rYEja\nsLdRzR3vhNChKDoUSF3viaXmju7nTW1Btfeww11PUuJjlJYQE46jWI1IS9Ck/NRDr6UmxBwWT+Hn\n4ZBKTYhVXAzvUQIAAACGs4iOqz2BkC765cu9fj8xNlopCTFKiY9Rcny0UuJjNDIjQcnx3a91v37E\n44QYpcRHKzk+RslxMUoNxxFXjwAAAAD0R0THVU6i6befPKnHQEqOiyGIAAAAAESMiI6rlFjTBdML\n/B4DAAAAAI6JNwoBAAAAgAc8iSszu8DMysxsq5nd0MP3483swfD3l5nZWC/OCwAAAACRot9xZWbR\nkv5b0oWSpkq6xsymHnHY5yXVOecmSvq5pB/397wAAAAAEEm8uHI1X9JW59x251yHpAckLTrimEWS\n7gk/fljS2cYn4AIAAAAYQrzY0KJQUvlhz/dIOqW3Y5xzQTNrkJQtqfrIH2ZmiyUtlqTc3FyVlpZ6\nMCKGkkAgwLrAe7Au0BvWBnrCukBvWBvoj4jbLdA5d5ekuySppKTELViwwN+BEHFKS0vFusCRWBfo\nDWsDPWFdoDesDfSHF7cFVkgqOuz5qPBrPR5jZjGS0iXVeHBuAAAAAIgIXsTVm5KKzWycmcVJulrS\n40cc87ikz4QfXyHpReec8+DcAAAAABAR+n1bYPg9VF+VtFRStKS7nXPrzewWSW855x6X9AdJfzaz\nrZJq1R1gAAAAADBkePKeK+fcEklLjnjtpsMet0m60otzAQAAAEAk8uRDhAEAAABguCOuAAAAAMAD\nxBUAAAAAeIC4AgAAAAAPEFcAAAAA4AHiCgAAAAA8QFwBAAAAgAeIKwAAAADwAHEFAAAAAB4grgAA\nAADAA8QVAAAAAHiAuAIAAAAADxBXAAAAAOAB4goAAAAAPEBcAQAAAIAHiCsAAAAA8ABxBQAAAAAe\nIK4AAAAAwAPEFQAAAAB4gLgCAAAAAA8QVwAAAADgAeIKAAAAADxAXAEAAACAB4grAAAAAPAAcQUA\nAAAAHiCuAAAAAMADxBUAAAAAeIC4AgAAAAAPEFcAAAAA4AHiCgAAAAA8QFwBAAAAgAeIKwAAAADw\nAHEFAAAAAB4grgAAAADAA8QVAAAAAHiAuAIAAAAADxBXAAAAAOAB4goAAAAAPEBcAQAAAIAHiCsA\nAAAA8ABxBQAAAAAeIK4AAAAAwAPEFQAAAAB4gLgCAAAAAA8QVwAAAADgAeIKAAAAADxAXAEAAACA\nB4grAAAAAPAAcQUAAAAAHiCuAAAAAMADxBUAAAAAeIC4AgAAAAAPEFcAAAAA4AHiCgAAAAA8QFwB\nAAAAgAeIKwAAAADwAHEFAAAAAB4grgAAAADAA/2KKzPLMrPnzGxL+M/MXo7rMrNV4a/H+3NOAAAA\nAIhE/b1ydYOkF5xzxZJeCD/vSatzbnb469J+nhMAAAAAIk5/42qRpHvCj++R9JF+/jwAAAAAGJTM\nOXf8/7FZvXMuI/zYJNUdfH7EcUFJqyQFJd3mnPv7UX7mYkmLJSk3N3fuQw89dNzzYWgKBAJKSUnx\newxEGNYFesPaQE9YF+gNawM9Wbhw4Qrn3LxjHXfMuDKz5yWN6OFbN0q65/CYMrM659x73ndlZoXO\nuQozGy/pRUlnO+e2HWu4kpISV1ZWdqzDMMyUlpZqwYIFfo+BCMO6QG9YG+gJ6wK9YW2gJ2bWp7iK\nOdYBzrlzjnKS/WZW4JyrNLMCSVW9/IyK8J/bzaxU0hxJx4wrAAAAABgs+vueq8clfSb8+DOSHjvy\nADPLNLP48OMcSadL2tDP8wIAAABAROlvXN0m6Vwz2yLpnPBzmdk8M/t9+Jgpkt4ys9WS/qHu91wR\nVwAAAACGlGPeFng0zrkaSWf38Ppbkr4QfvyapBn9OQ8AAAAARLr+XrkCAAAAAIi4AgAAAABPEFcA\nAAAA4AHiCgAAAAA8QFwBAAAAgAeIKwAAAADwAHEFAAAAAB4grgAAAADAA8QVAAAAAHiAuAIAAAAA\nDxBXAAAAAOAB4goAAAAAPEBcAQAAAIAHiCsAAAAA8ABxBQAAAAAeIK4AAAAAwAPEFQAAAAB4gLgC\nAAAAAA8QVwAAAADgAeIKAAAAADxAXAEAAACAB4grAAAAAPAAcQUAAAAAHiCuAAAAAMADxBUAAAAA\neIC4AgAAAAAPEFcAAAAA4AHiCgAAAAA8QFwBAAAAgAeIKwAAAADwAHEFAAAAAB4grgAAAADAA8QV\nAAAAAHiAuAIAAAAADxBXAAAAAOAB4goAAAAAPEBcAQAAAIAHiCsAAAAA8ABxBQAAAAAeIK4AAAAA\nwAPEFQAAAAB4gLgCAAAAAA8QVwAAAADgAeIKAAAAADxAXAEAAACAB4grAAAAAPAAcQUAAAAAHiCu\nAAAAAMADxBUAAAAAeIC4AgAAAAAPEFcAAAAA4AHiCgAAAAA8QFwBAAAAgAeIKwAAAADwAHEFAAAA\nAB4grgAAAADAA8QVAAAAAHiAuAIAAAAAD/QrrszsSjNbb2YhM5t3lOMuMLMyM9tqZjf055wAAAAA\nEIn6e+VqnaTLJb3U2wFmFi3pvyVdKGmqpGvMbGo/zwsAAAAAESWmP/+xc26jJJnZ0Q6bL2mrc257\n+NgHJC2StKE/5wYAAACASDIQ77kqlFR+2PM94dcAAAAAYMg45pUrM3te0ogevnWjc+4xrwcys8WS\nFktSbm6uSktLvT4FBrlAIMC6wHuwLtAb1gZ6wrpAb1gb6I9jxpVz7px+nqNCUtFhz0eFX+vtfHdJ\nukuSSkpK3IIFC/p5egw1paWlYl3gSKwL9Ia1gZ6wLtAb1gb6YyBuC3xTUrGZjTOzOElXS3p8AM4L\nAAAAAAOmv1uxX2ZmeySdKukpM1safn2kmS2RJOdcUNJXJS2VtFHSQ8659f0bGwAAAAAiS393C3xU\n0qM9vL5X0kWHPV8iaUl/zgUAAAAAkWwgbgsEAAAAgCGPuAIAAAAADxBXAAAAAOAB4goAAAAAPEBc\nAQAAAIAHiCsAAAAA8ABxBQAAAAAeIK4AAAAAwAPEFQAAAAB4gLgCAAAAAA8QVwAAAADgAeIKAAAA\nADxAXAEAAACAB4grAAAAAPAAcQUAAAAAHiCuAAAAAMADxBUAAAAAeIC4AgAAAAAPEFcAAAAA4AHi\nCgAAAAA8QFwBAAAAgAeIKwAAAADwAHEFAAAAAB4grgAAAADAA8QVAAAAAHiAuAIAAAAADxBXAAAA\nAOAB4goAAAAAPEBcAQAAAIAHiCsAAAAA8ABxBQAAAAAeIK4AAAAAwAPEFQAAAAB4gLgCAAAAAA8Q\nVwAAAADgAeIKAAAAADxAXAEAAACAB4grAAAAAPAAcQUAAAAAHiCuAAAAAMADxBUAAAAAeIC4AgAA\nAAAPEFcAAAAA4AHiCgAAAAA8QFwBAAAAgAeIKwAAAADwAHEFAAAAAB4grgAAAADAA8QVAAAAAHiA\nuAIAAAAADxBXAAAAAOAB4goAAAAAPEBcAQAAAIAHiCsAAAAA8ABxBQAAAAAeIK4AAAAAwAPEFQAA\nAAB4gLgCAAAAAA/0K67M7EozW29mITObd5TjdprZWjNbZWZv9eecAAAAABCJYvr536+TdLmk3/Xh\n2IXOuep+ng8AAAAAIlK/4so5t1GSzMybaQAAAABgkBqo91w5Sc+a2QozWzxA5wQAAACAAXPMK1dm\n9rykET1860bn3GN9PM8ZzrkKM8uT9JyZbXLOvdTL+RZLOhhg7Wa2ro/nwPCRI4lbTHEk1gV6w9pA\nT1gX6A1rAz0p6ctBx4wr59w5/Z3EOVcR/rPKzB6VNF9Sj3HlnLtL0l2SZGZvOed63SgDwxPrAj1h\nXaA3rA30hHWB3rA20JO+bsp3wm8LNLNkM0s9+FjSeereCAMAAAAAhoz+bsV+mZntkXSqpKfMbGn4\n9ZFmtiR8WL6kV8xstaTlkp5yzj3Tn/MCAAAAQKTp726Bj0p6tIfX90q6KPx4u6RZx3mKu45/Ogxh\nrAv0hHWB3rA20BPWBXrD2kBP+rQuzDl3ogcBAAAAgCFvoLZiBwAAAIAhLaLjysx+aGZrzGyVmT1r\nZiP9ngmRwczuMLNN4fXxqJll+D0T/GdmV5rZejMLmRk7PQ1zZnaBmZWZ2VYzu8HveRAZzOxuM6vi\no15wODMrMrN/mNmG8N8j1/s9EyKDmSWY2XIzWx1eG/951OMj+bZAM0tzzjWGH/+bpKnOuet8HgsR\nwMzOk/Sicy5oZj+WJOfcd30eCz4zsymSQpJ+J+lbzrk+bZuKocfMoiVtlnSupD2S3pR0jXNug6+D\nwXdmdqakgKR7nXPT/Z4HkcHMCiQVOOdWhne5XiHpI/zOgJmZpGTnXMDMYiW9Iul659wbPR0f0Veu\nDoZVWLKkyC1BDCjn3LPOuWD46RuSRvk5DyKDc26jc67M7zkQEeZL2uqc2+6c65D0gKRFPs+ECOCc\ne0lSrd9zILI45yqdcyvDj5skbZRU6O9UiASuWyD8NDb81WuTRHRcSZKZ3Wpm5ZI+Iekmv+dBRPqc\npKf9HgJARCmUVH7Y8z3iH0oA+sDMxkqaI2mZv5MgUphZtJmtklQl6TnnXK9rw/e4MrPnzWxdD1+L\nJMk5d6NzrkjSfZK+6u+0GEjHWhvhY26UFFT3+sAw0Jd1AQDA8TCzFEmPSPraEXdQYRhzznU552ar\n+06p+WbW6y3F/fqcKy84587p46H3SVoi6eYTOA4iyLHWhpldK+kSSWe7SH7zIDz1Pn5nYHirkFR0\n2PNR4dcAoEfh99M8Iuk+59zf/J4Hkcc5V29m/5B0gaQeN8Xx/crV0ZhZ8WFPF0na5NcsiCxmdoGk\n70i61DnX4vc8ACLOm5KKzWycmcVJulrS4z7PBCBChTct+IOkjc65n/k9DyKHmeUe3JXazBLVvVFS\nr00S6bsFPiKpRN27f+2SdJ1zjv/lETKzrZLiJdWEX3qDnSRhZpdJ+pWkXEn1klY55873dyr4xcwu\nknSnpGhJdzvnbvV5JEQAM/uLpAWSciTtl3Szc+4Pvg4F35nZGZJelrRW3f/ulKTvO+eW+DcVIoGZ\nzZR0j7r/LomS9JBz7pZej4/kuAIAAACAwSKibwsEAAAAgMGCuAIAAAAADxBXAAAAAOAB4goAAAAA\nPEBcAQAAAIAHiCsAAAAA8ABxBQAAAAAeIK4AAAAAwAP/H1Jy90TyLtqpAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fdde8452e10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot(lambda x: math.log(1 + math.exp(x)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Problem zanikającego gradientu (*vanishing gradient problem*)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Sigmoidalne funkcje aktywacji ograniczają wartości na wyjściach neuronów do niewielkich przedziałów ($(-1, 1)$, $(0, 1)$ itp.).\n",
    "* Jeżeli sieć ma wiele warstw, to podczas propagacji wstecznej mnożymy przez siebie wiele małych wartości → obliczony gradient jest mały.\n",
    "* Im więcej warstw, tym silniejszy efekt zanikania."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Sposoby na zanikający gradient\n",
    "\n",
    "* Modyfikacja algorytmu optymalizacji (*RProp*, *RMSProp*)\n",
    "* Użycie innej funckji aktywacji (ReLU, softplus)\n",
    "* Dodanie warstw *dropout*\n",
    "* Nowe architektury (LSTM itp.)\n",
    "* Więcej danych, zwiększenie mocy obliczeniowej"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 4.3. Wielowarstwowe sieci neuronowe\n",
    "\n",
    "czyli _Artificial Neural Networks_ (ANN) lub _Multi-Layer Perceptrons_ (MLP)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"nn1.png\" width=\"70%\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Architektura sieci\n",
    "\n",
    "* Sieć neuronowa jako graf neuronów. \n",
    "* Organizacja sieci przez warstwy.\n",
    "* Najczęściej stosowane są sieci jednokierunkowe i gęste."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* $n$-warstwowa sieć neuronowa ma $n+1$ warstw (nie liczymy wejścia).\n",
    "* Rozmiary sieci określane poprzez liczbę neuronów lub parametrów."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Sieć neuronowa jednokierunkowa (*feedforward*)\n",
    "\n",
    "* Mając daną $n$-warstwową sieć neuronową oraz jej parametry $\\Theta^{(1)}, \\ldots, \\Theta^{(L)} $ oraz $\\beta^{(1)}, \\ldots, \\beta^{(L)} $ liczymy:<br/><br/> \n",
    "$$a^{(l)} = g^{(l)}\\left( a^{(l-1)} \\Theta^{(l)} + \\beta^{(l)} \\right). $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"nn2.png\" width=70%/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Funkcje $g^{(l)}$ to tzw. **funkcje aktywacji**.<br/>\n",
    "Dla $i = 0$ przyjmujemy $a^{(0)} = \\mathrm{x}$ (wektor wierszowy cech) oraz $g^{(0)}(x) = x$ (identyczność)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Parametry $\\Theta$ to wagi na połączeniach miedzy neuronami dwóch warstw.<br/>\n",
    "Rozmiar macierzy $\\Theta^{(l)}$, czyli macierzy wag na połączeniach warstw $a^{(l-1)}$ i $a^{(l)}$, to $\\dim(a^{(l-1)}) \\times \\dim(a^{(l)})$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Parametry $\\beta$ zastępują tutaj dodawanie kolumny z jedynkami do macierzy cech.<br/>Macierz $\\beta^{(l)}$ ma rozmiar równy liczbie neuronów w odpowiedniej warstwie, czyli $1 \\times \\dim(a^{(l)})$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* **Klasyfikacja**: dla ostatniej warstwy $L$ (o rozmiarze równym liczbie klas) przyjmuje się $g^{(L)}(x) = \\mathop{\\mathrm{softmax}}(x)$.\n",
    "* **Regresja**: pojedynczy neuron wyjściowy jak na obrazku. Funkcją aktywacji może wtedy być np. funkcja identycznościowa."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Pozostałe funkcje aktywacji najcześciej mają postać sigmoidy, np. sigmoidalna, tangens hiperboliczny.\n",
    "* Mogą mieć też inny kształt, np. ReLU, leaky ReLU, maxout."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Jak uczyć sieci neuronowe?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* W poznanych do tej pory algorytmach (regresja liniowa, regresja logistyczna) do uczenia używaliśmy funkcji kosztu, jej gradientu oraz algorytmu gradientu prostego (GD/SGD)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Dla sieci neuronowych potrzebowalibyśmy również znaleźć gradient funkcji kosztu."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Sprowadza się to do bardziej ogólnego problemu:<br/>jak obliczyć gradient $\\nabla f(x)$ dla danej funkcji $f$ i wektora wejściowego $x$?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 4.4. Metoda propagacji wstecznej – wprowadzenie"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Pochodna funkcji\n",
    "\n",
    "* **Pochodna** mierzy, jak szybko zmienia się wartość funkcji względem zmiany jej argumentów:\n",
    "\n",
    "$$ \\frac{d f(x)}{d x} = \\lim_{h \\to 0} \\frac{ f(x + h) - f(x) }{ h } $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Pochodna cząstkowa i gradient\n",
    "\n",
    "* **Pochodna cząstkowa** mierzy, jak szybko zmienia się wartość funkcji względem zmiany jej *pojedynczego argumentu*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* **Gradient** to wektor pochodnych cząstkowych:\n",
    "\n",
    "$$ \\nabla f = \\left( \\frac{\\partial f}{\\partial x_1}, \\ldots, \\frac{\\partial f}{\\partial x_n} \\right) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Gradient – przykłady"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$$ f(x_1, x_2) = x_1 + x_2 \\qquad \\to \\qquad \\frac{\\partial f}{\\partial x_1} = 1, \\quad \\frac{\\partial f}{\\partial x_2} = 1, \\quad \\nabla f = (1, 1) $$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$$ f(x_1, x_2) = x_1 \\cdot x_2 \\qquad \\to \\qquad \\frac{\\partial f}{\\partial x_1} = x_2, \\quad \\frac{\\partial f}{\\partial x_2} = x_1, \\quad \\nabla f = (x_2, x_1) $$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "$$ f(x_1, x_2) = \\max(x_1 + x_2) \\hskip{12em} \\\\\n",
    "\\to \\qquad \\frac{\\partial f}{\\partial x_1} = \\mathbb{1}_{x \\geq y}, \\quad \\frac{\\partial f}{\\partial x_2} = \\mathbb{1}_{y \\geq x}, \\quad \\nabla f = (\\mathbb{1}_{x \\geq y}, \\mathbb{1}_{y \\geq x}) $$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Własności pochodnych cząstkowych\n",
    "\n",
    "Jezeli $f(x, y, z) = (x + y) \\, z$ oraz $x + y = q$, to:\n",
    "$$f = q z,\n",
    "\\quad \\frac{\\partial f}{\\partial q} = z,\n",
    "\\quad \\frac{\\partial f}{\\partial z} = q,\n",
    "\\quad \\frac{\\partial q}{\\partial x} = 1,\n",
    "\\quad \\frac{\\partial q}{\\partial y} = 1 $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Reguła łańcuchowa\n",
    "\n",
    "$$ \\frac{\\partial f}{\\partial x} = \\frac{\\partial f}{\\partial q} \\, \\frac{\\partial q}{\\partial x},\n",
    "\\quad \\frac{\\partial f}{\\partial y} = \\frac{\\partial f}{\\partial q} \\, \\frac{\\partial q}{\\partial y} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Propagacja wsteczna – prosty przykład"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Dla ustalonego wejścia\n",
    "x = -2; y = 5; z = -4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3, -12)\n"
     ]
    }
   ],
   "source": [
    "# Krok w przód\n",
    "q = x + y\n",
    "f = q * z\n",
    "print(q, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-4, -4, 3]\n"
     ]
    }
   ],
   "source": [
    "# Propagacja wsteczna dla f = q * z\n",
    "dz = q\n",
    "dq = z\n",
    "# Propagacja wsteczna dla q = x + y\n",
    "dx = 1 * dq  # z reguły łańcuchowej\n",
    "dy = 1 * dq  # z reguły łańcuchowej\n",
    "print([dx, dy, dz])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"exp1.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Właśnie tak wygląda obliczanie pochodnych metodą propagacji wstecznej!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Spróbujmy czegoś bardziej skomplikowanego:<br/>metodą propagacji wstecznej obliczmy pochodną funkcji sigmoidalnej."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Propagacja wsteczna – funkcja sigmoidalna"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Funkcja sigmoidalna:\n",
    "\n",
    "$$f(\\theta,x) = \\frac{1}{1+e^{-(\\theta_0 x_0 + \\theta_1 x_1 + \\theta_2)}}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "$$\n",
    "\\begin{array}{lcl}\n",
    "f(x) = \\frac{1}{x} \\quad & \\rightarrow & \\quad \\frac{df}{dx} = -\\frac{1}{x^2} \\\\\n",
    "f_c(x) = c + x \\quad & \\rightarrow & \\quad \\frac{df}{dx} = 1 \\\\\n",
    "f(x) = e^x \\quad & \\rightarrow & \\quad \\frac{df}{dx} = e^x \\\\\n",
    "f_a(x) = ax \\quad & \\rightarrow & \\quad \\frac{df}{dx} = a \\\\\n",
    "\\end{array}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"exp2.png\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.3932238664829637, -0.5898357997244456]\n",
      "[-0.19661193324148185, -0.3932238664829637, 0.19661193324148185]\n"
     ]
    }
   ],
   "source": [
    "# Losowe wagi i dane\n",
    "w = [2,-3,-3]\n",
    "x = [-1, -2]\n",
    "\n",
    "# Krok w przód\n",
    "dot = w[0]*x[0] + w[1]*x[1] + w[2]\n",
    "f = 1.0 / (1 + math.exp(-dot))  # funkcja sigmoidalna\n",
    "\n",
    "# Krok w tył\n",
    "ddot = (1 - f) * f  # pochodna funkcji sigmoidalnej\n",
    "dx = [w[0] * ddot, w[1] * ddot]\n",
    "dw = [x[0] * ddot, x[1] * ddot, 1.0 * ddot]\n",
    "\n",
    "print(dx)\n",
    "print(dw)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Obliczanie gradientów – podsumowanie"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Gradient $f$ dla $x$ mówi, jak zmieni się całe wyrażenie przy zmianie wartości $x$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Gradienty łączymy, korzystając z **reguły łańcuchowej**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* W kroku \"wstecz\" gradienty informują, które części grafu powinny być zwiększone lub zmniejszone (i z jaką siłą), aby zwiększyć wartość na wyjściu."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* W kontekście implementacji chcemy dzielić funkcję $f$ na części, dla których można łatwo obliczyć gradienty."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 4.5. Uczenie wielowarstwowych sieci neuronowych metodą propagacji wstecznej"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Mając algorytm SGD oraz gradienty wszystkich wag, moglibyśmy trenować każdą sieć."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Niech $\\Theta = (\\Theta^{(1)},\\Theta^{(2)},\\Theta^{(3)},\\beta^{(1)},\\beta^{(2)},\\beta^{(3)})$\n",
    "* Funkcja sieci neuronowej z grafiki:\n",
    "$$\\small h_\\Theta(x) = \\tanh(\\tanh(\\tanh(x\\Theta^{(1)}+\\beta^{(1)})\\Theta^{(2)} + \\beta^{(2)})\\Theta^{(3)} + \\beta^{(3)})$$\n",
    "* Funkcja kosztu dla regresji:\n",
    "$$J(\\Theta) = \\dfrac{1}{2m} \\sum_{i=1}^{m} (h_\\Theta(x^{(i)})- y^{(i)})^2 $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Jak obliczymy gradienty?\n",
    "\n",
    "$$\\nabla_{\\Theta^{(l)}} J(\\Theta) = ? \\quad \\nabla_{\\beta^{(l)}} J(\\Theta) = ?$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### W kierunku propagacji wstecznej\n",
    "\n",
    "* Pewna (niewielka) zmiana wagi $\\Delta z^l_j$ dla $j$-ego neuronu w warstwie $l$ pociąga za sobą (niewielką) zmianę kosztu: \n",
    "\n",
    "$$\\frac{\\partial J(\\Theta)}{\\partial z^{l}_j}  \\Delta z^{l}_j$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Jeżeli $\\frac{\\partial J(\\Theta)}{\\partial z^{l}_j}$ jest duża, $\\Delta z^l_j$ ze znakiem przeciwnym zredukuje koszt.\n",
    "* Jeżeli $\\frac{\\partial J(\\Theta)}{\\partial z^l_j}$ jest bliska zeru, koszt nie będzie mocno poprawiony."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Definiujemy błąd $\\delta^l_j$ neuronu $j$ w warstwie $l$: \n",
    "\n",
    "$$\\delta^l_j := \\dfrac{\\partial J(\\Theta)}{\\partial z^l_j}$$ \n",
    "$$\\delta^l := \\nabla_{z^l} J(\\Theta) \\quad \\textrm{ (zapis wektorowy)} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Podstawowe równania propagacji wstecznej\n",
    "\n",
    "$$\n",
    "\\begin{array}{rcll}\n",
    "\\delta^L & = & \\nabla_{a^L}J(\\Theta) \\odot { \\left( g^{L} \\right) }^{\\prime} \\left( z^L \\right) & (BP1) \\\\[2mm]\n",
    "\\delta^{l} & = & \\left( \\left( \\Theta^{l+1} \\right) \\! ^\\top \\, \\delta^{l+1} \\right) \\odot {{ \\left( g^{l} \\right) }^{\\prime}} \\left( z^{l} \\right) & (BP2)\\\\[2mm]\n",
    "\\nabla_{\\beta^l} J(\\Theta) & = & \\delta^l & (BP3)\\\\[2mm]\n",
    "\\nabla_{\\Theta^l} J(\\Theta) & = & a^{l-1} \\odot \\delta^l & (BP4)\\\\\n",
    "\\end{array}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### (BP1)\n",
    "$$ \\delta^L_j \\; = \\; \\frac{ \\partial J }{ \\partial a^L_j } \\, g' \\!\\! \\left( z^L_j \\right) $$\n",
    "$$ \\delta^L \\; = \\; \\nabla_{a^L}J(\\Theta) \\odot { \\left( g^{L} \\right) }^{\\prime} \\left( z^L \\right) $$\n",
    "Błąd w ostatniej warstwie jest iloczynem szybkości zmiany kosztu względem $j$-tego wyjścia i szybkości zmiany funkcji aktywacji w punkcie $z^L_j$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### (BP2)\n",
    "$$ \\delta^{l} \\; = \\; \\left( \\left( \\Theta^{l+1} \\right) \\! ^\\top \\, \\delta^{l+1} \\right) \\odot {{ \\left( g^{l} \\right) }^{\\prime}} \\left( z^{l} \\right) $$\n",
    "Aby obliczyć błąd w $l$-tej warstwie, należy przemnożyć błąd z następnej ($(l+1)$-szej) warstwy przez transponowany wektor wag, a uzyskaną macierz pomnożyć po współrzędnych przez szybkość zmiany funkcji aktywacji w punkcie $z^l$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### (BP3)\n",
    "$$ \\nabla_{\\beta^l} J(\\Theta) \\; = \\; \\delta^l $$\n",
    "Błąd w $l$-tej warstwie jest równy wartości gradientu funkcji kosztu."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### (BP4)\n",
    "$$ \\nabla_{\\Theta^l} J(\\Theta) \\; = \\; a^{l-1} \\odot \\delta^l $$\n",
    "Gradient funkcji kosztu względem wag $l$-tej warstwy można obliczyć jako iloczyn po współrzędnych $a^{l-1}$ przez $\\delta^l$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Algorytm propagacji wstecznej"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Dla pojedynczego przykładu $(x,y)$:\n",
    "1. **Wejście**: Ustaw aktywacje w warstwie cech $a^{(0)}=x$ \n",
    "2. **Feedforward:** dla $l=1,\\dots,L$ oblicz \n",
    "$z^{(l)} = a^{(l-1)} \\Theta^{(l)} + \\beta^{(l)}$ oraz $a^{(l)}=g^{(l)} \\!\\! \\left( z^{(l)} \\right)$\n",
    "3. **Błąd wyjścia $\\delta^{(L)}$:** oblicz wektor $$\\delta^{(L)}= \\nabla_{a^{(L)}}J(\\Theta) \\odot {g^{\\prime}}^{(L)} \\!\\! \\left( z^{(L)} \\right) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "4. **Propagacja wsteczna błędu:** dla $l = L-1,L-2,\\dots,1$ oblicz $$\\delta^{(l)} = \\delta^{(l+1)}(\\Theta^{(l+1)})^T \\odot {g^{\\prime}}^{(l)} \\!\\! \\left( z^{(l)} \\right) $$\n",
    "5. **Gradienty:** \n",
    "    * $\\dfrac{\\partial}{\\partial \\Theta_{ij}^{(l)}} J(\\Theta) = a_i^{(l-1)}\\delta_j^{(l)} \\textrm{ oraz } \\dfrac{\\partial}{\\partial \\beta_{j}^{(l)}} J(\\Theta) = \\delta_j^{(l)}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "W naszym przykładzie:\n",
    "\n",
    "$$\\small J(\\Theta) = \\frac{1}{2} \\left( a^{(L)} - y \\right) ^2 $$\n",
    "$$\\small  \\dfrac{\\partial}{\\partial a^{(L)}} J(\\Theta) = a^{(L)} - y$$\n",
    "\n",
    "$$\\small \\tanh^{\\prime}(x) = 1 - \\tanh^2(x)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"nn3.png\" width=\"65%\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Algorytm SGD z propagacją wsteczną\n",
    "\n",
    "Pojedyncza iteracja:\n",
    "1. Dla parametrów $\\Theta = (\\Theta^{(1)},\\ldots,\\Theta^{(L)})$ utwórz pomocnicze macierze zerowe $\\Delta = (\\Delta^{(1)},\\ldots,\\Delta^{(L)})$ o takich samych wymiarach (dla uproszczenia opuszczono wagi $\\beta$)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "2. Dla $m$ przykładów we wsadzie (*batch*), $i = 1,\\ldots,m$:\n",
    "    * Wykonaj algortym propagacji wstecznej dla przykładu $(x^{(i)}, y^{(i)})$ i przechowaj gradienty $\\nabla_{\\Theta}J^{(i)}(\\Theta)$ dla tego przykładu;\n",
    "    * $\\Delta := \\Delta + \\dfrac{1}{m}\\nabla_{\\Theta}J^{(i)}(\\Theta)$\n",
    "3. Wykonaj aktualizację wag: $\\Theta := \\Theta - \\alpha \\Delta$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Propagacja wsteczna – podsumowanie\n",
    "\n",
    "* Algorytm pierwszy raz wprowadzony w latach 70. XX w.\n",
    "* W 1986 David Rumelhart, Geoffrey Hinton i Ronald Williams pokazali, że jest znacznie szybszy od wcześniejszych metod.\n",
    "* Obecnie najpopularniejszy algorytm uczenia sieci neuronowych."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 4.6. Przykłady implementacji wielowarstwowych sieci neuronowych"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "### Uwaga!\n",
    "\n",
    "Poniższe przykłady wykorzystują interfejs [Keras](https://keras.io), który jest częścią biblioteki [TensorFlow](https://www.tensorflow.org).\n",
    "\n",
    "Aby uruchomić TensorFlow w środowisku Jupyter, należy wykonać następujące czynności:\n",
    "\n",
    "#### Przed pierwszym uruchomieniem (wystarczy wykonać tylko raz)\n",
    "\n",
    "Instalacja biblioteki TensorFlow w środowisku Anaconda:\n",
    "\n",
    "1. Uruchom *Anaconda Navigator*\n",
    "1. Wybierz kafelek *CMD.exe Prompt*\n",
    "1. Kliknij przycisk *Launch*\n",
    "1. Pojawi się konsola. Wpisz następujące polecenia, każde zatwierdzając wciśnięciem klawisza Enter:\n",
    "```\n",
    "conda create -n tf tensorflow\n",
    "conda activate tf\n",
    "conda install pandas matplotlib\n",
    "jupyter notebook\n",
    "```\n",
    "\n",
    "#### Przed każdym uruchomieniem\n",
    "\n",
    "Jeżeli chcemy korzystać z biblioteki TensorFlow, to środowisko Jupyter Notebook należy uruchomić w następujący sposób:\n",
    "\n",
    "1. Uruchom *Anaconda Navigator*\n",
    "1. Wybierz kafelek *CMD.exe Prompt*\n",
    "1. Kliknij przycisk *Launch*\n",
    "1. Pojawi się konsola. Wpisz następujące polecenia, każde zatwierdzając wciśnięciem klawisza Enter:\n",
    "```\n",
    "conda activate tf\n",
    "jupyter notebook\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Przykład: MNIST\n",
    "\n",
    "_Modified National Institute of Standards and Technology database_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Zbiór cyfr zapisanych pismem odręcznym\n",
    "* 60 000 przykładów uczących, 10 000 przykładów testowych\n",
    "* Rozdzielczość każdego przykładu: 28 × 28 = 784 piksele"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# źródło: https://github.com/keras-team/keras/examples/minst_mlp.py\n",
    "\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras.datasets import mnist\n",
    "from tensorflow.keras.layers import Dense, Dropout\n",
    "\n",
    "# załaduj dane i podziel je na zbiory uczący i testowy\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "\n",
    "def draw_examples(examples, captions=None):\n",
    "    plt.figure(figsize=(16, 4))\n",
    "    m = len(examples)\n",
    "    for i, example in enumerate(examples):\n",
    "        plt.subplot(100 + m * 10 + i + 1)\n",
    "        plt.imshow(example, cmap=plt.get_cmap('gray'))\n",
    "    plt.show()\n",
    "    if captions is not None:\n",
    "        print(6 * ' ' + (10 * ' ').join(str(captions[i]) for i in range(m)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA54AAACOCAYAAABZsdfhAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAcc0lEQVR4nO3deZBU5bnH8ecVAVFERJAQEYYIQYhsAgpeC0gAV2SRgBL2GKHEBVJCgUoMxiCIStUAEkUujCAlWGHVSJCwSFRCgQTvBVkGjMDgBAYF2Qxc9Nw/6O15menpnjnn9Onu76eKmvOb093nnebh9Lycfvo1juMIAAAAAABeuSTVAwAAAAAAZDYmngAAAAAATzHxBAAAAAB4ioknAAAAAMBTTDwBAAAAAJ5i4gkAAAAA8FS5Jp7GmLuMMbuNMXuNMePcGhTSD7WAMGoBItQBoqgFiFAHiKIWspcp6zqexpgKIrJHRLqKSIGIbBaRfo7jfO7e8JAOqAWEUQsQoQ4QRS1AhDpAFLWQ3S4tx31vEZG9juN8ISJijFkoIj1EpMTCMcaUbZaLVDrqOE6tUm5DLWQBx3FMAjdLqhaog7TEOQFh1AJEJKHXB+ogO3BOQFixtVCet9peJyIHY3JB6HvILPsTuA21gDBqIfNxTkAYtYBEUQfZgXMCwoqthfJc8Szuf7cu+h8JY8wwERlWjuMg+KgFhJVaC9RBVuCcgDBqASLUAaKohSxWnolngYhcH5PrishX9o0cx5klIrNEuFSewagFhJVaC9RBVuCcgDBqASLUAaKohSxWnrfabhaRRsaYBsaYSiLyoIiscGdYSDPUAsKoBYhQB4iiFiBCHSCKWshiZb7i6TjOeWPMYyKySkQqiMgcx3F2uDYypA1qAWHUAkSoA0RRCxChDhBFLWS3Mi+nUqaDcak8HX3qOE4btx+UWkg/CX6qbVKog7TEOQFh1AJEhNcHRHBOQFixtVCet9oCAAAAAFAqJp4AAAAAAE8x8QQAAAAAeIqJJwAAAADAU0w8AQAAAACeYuIJAAAAAPAUE08AAAAAgKeYeAIAAAAAPMXEEwAAAADgqUtTPQAgk7Vu3Vrlxx57TOVBgwapPG/ePJWnT5+u8tatW10cHQAAALySm5ur8hNPPBHZ3r59u9rXrVs3lffv3+/dwFKEK54AAAAAAE8x8QQAAAAAeIq32iaoQoUKKl911VUJ39d+e+Xll1+ucuPGjVV+9NFHVX755ZdV7tevn8r/+c9/VJ48eXJk+7nnnkt4nCi/li1bqrx69WqVq1WrprLjOCoPHDhQ5e7du6t8zTXXlHOEyASdO3dWecGCBSp37NhR5d27d3s+Jnhj/PjxKtvn9Esu0f9/3KlTJ5U//PBDT8YFwB1XXnmlylWrVlX53nvvVblWrVoqT506VeWzZ8+6ODokKycnR+UBAwao/MMPP0S2mzRpovbdeOONKvNWWwAAAAAAksTEEwAAAADgKSaeAAAAAABPZU2PZ7169VSuVKmSyrfddpvKt99+u8rVq1dXuXfv3q6NraCgQOVp06ap3KtXL5VPnjyp8meffaYyPT3+uuWWWyLbixcvVvvsXmC7p9P+uzx37pzKdk9nu3btVLaXV7Hvn+k6dOgQ2bafq6VLl/o9HN+0bdtW5c2bN6doJHDbkCFDVB47dqzKsf1BxbHPMQBSL7bvz/433b59e5VvuummpB67Tp06Kscu1wH/FRUVqbxhwwaV7c/uyDZc8QQAAAAAeIqJJwAAAADAU0w8AQAAAACeytgeT3s9xbVr16qczDqcbrN7dOx12k6dOqWyvUZfYWGhyseOHVOZNfvcZa+7evPNN6v81ltvRbbtXovS5OfnqzxlyhSVFy5cqPLHH3+ssl07kyZNSur46S52zcJGjRqpfZnU42mv1digQQOV69evr7IxxvMxwRv23+Vll12WopGgLG699VaVY9fws9fX/dnPfhb3sUaPHq3yV199pbL9WRSxr0UiIps2bYo/WLjGXn9x1KhRKvfv3z+yXaVKFbXPPl8fPHhQZfuzIOy1H/v27avyzJkzVd61a1cJo4YXTp8+rXImrsVZHlzxBAAAAAB4ioknAAAAAMBTTDwBAAAAAJ7K2B7PAwcOqPz111+r7GaPp91Hcfz4cZV//vOfq2yvtTh//nzXxgL3vf766yr369fPtce2+0WrVq2qsr0ma2xPo4hI8+bNXRtLOho0aFBke+PGjSkcibfs3uGHH35YZbu3i56e9NGlSxeVH3/88bi3t/9uu3XrpvLhw4fdGRgS8sADD6icm5urcs2aNSPbdi/f+vXrVa5Vq5bKL730Utxj249n3//BBx+Me38kzv6d8cUXX1TZroMrr7wy4ce2P+vhzjvvVLlixYoq2+eA2BorLsNf1atXV7lFixapGUhAccUTAAAAAOApJp4AAAAAAE8x8QQAAAAAeCpjezy/+eYblceMGaOy3Rfzz3/+U+Vp06bFffxt27ZFtrt27ar22Wv42Gt1jRw5Mu5jI7Vat26t8r333qtyvDUS7Z7Md999V+WXX35ZZXtdNrsO7TVaf/GLXyQ8lmxgr2+ZqWbPnh13v90jhOCy116cO3euyqV9/oDd98cacd669FL9a1KbNm1UfuONN1S2133esGFDZPv5559X+z766COVK1eurPI777yj8h133BF3rFu2bIm7H2XXq1cvlX/zm9+U+bH27dunsv07pL2OZ8OGDct8LPjPPgfUq1cv4fu2bdtWZbufNxPO99nxWxsAAAAAIGVKnXgaY+YYY44YY7bHfK+GMWa1MSY/9PVqb4eJIKAWEEYtQIQ6QBS1gDBqASLUAYqXyBXPPBG5y/reOBFZ4zhOIxFZE8rIfHlCLeCCPKEWQB0gKk+oBVyQJ9QCqAMUo9QeT8dxNhhjcqxv9xCRTqHtN0VkvYiMdXNgblu2bJnKa9euVfnkyZMq2+vuPPTQQyrH9urZPZ22HTt2qDxs2LC4tw+qTKkFW8uWLVVevXq1ytWqVVPZcRyVV65cGdm21/js2LGjyuPHj1fZ7t0rKipS+bPPPlP5hx9+UNnuP7XXBd26dat4IVW1YK9bWrt2bTcfPrBK6/uza9YvmXpO8NLgwYNV/vGPfxz39vZaj/PmzXN7SK7I1FoYMGCAyqX1W9v/FmPXdzxx4kTc+9prQZbW01lQUKDym2++Gff2fsnEWujTp09St//yyy9V3rx5c2R77Fj9Y9s9nbYmTZokdeygyMQ6SIT92R15eXkqT5gwocT72vuOHz+u8owZM8oxsmAoa49nbcdxCkVEQl+vdW9ISDPUAsKoBYhQB4iiFhBGLUCEOsh6nn+qrTFmmIik5yU+uIpagAh1gChqAWHUAkSoA0RRC5mprFc8Dxtj6oiIhL4eKemGjuPMchynjeM4bUq6DdIatYCwhGqBOsh4nBMQRi0gjNcHiHBOyHplveK5QkQGi8jk0Nflro3IJ6X1Wnz77bdx9z/88MOR7UWLFql9dh9ehku7WvjpT3+qsr3Gq91Pd/ToUZULCwtVju2rOXXqlNr3l7/8JW4urypVqqj85JNPqty/f39Xj1cKz2vhnnvuUdn++TOF3bvaoEGDuLc/dOiQl8NJVtqdE7xUs2ZNlX/961+rbL9e2D09f/zjHz0Zl0/SrhbstTaffvpple0e/5kzZ6ps9/GX9rtGrGeeeSbh24qIPPHEEyrbnxEQMGlXC7Fif+cTufizOj744AOV9+7dq/KRIyXOr0qVYZ9lkNZ1UBb2OSVej2c2SGQ5lbdFZKOINDbGFBhjHpILBdPVGJMvIl1DGRmOWkAYtQAR6gBR1ALCqAWIUAcoXiKfatuvhF2dXR4LAo5aQBi1ABHqAFHUAsKoBYhQByheWXs8AQAAAABIiOefapuu7Pdgt27dWuXY9Rm7dOmi9tnv9UdqVa5cWeXYNVhFLu4btNd0HTRokMpbtmxROUh9hvXq1Uv1EDzVuHHjEvfZ6+WmM7tG7R6fPXv2qGzXLFIrJycnsr148eKk7jt9+nSV161b58aQUIJnn31WZbun89y5cyqvWrVKZXtNxu+++67EY1122WUq2+t02udvY4zKdr/v8uUZ3x4XGPbajH726bVv3963Y8F7l1wSveaXZZ8JIyJc8QQAAAAAeIyJJwAAAADAU0w8AQAAAACeosezBKdPn1bZXsNp69atke033nhD7bN7cuyewFdffVVle10wuKtVq1Yq2z2dth49eqj84Ycfuj4muG/z5s2pHkKJqlWrpvJdd92l8oABA1S2e79s9rpg9tqPSK3Yv9/mzZvHve2aNWtUzs3N9WRMuKB69eoqjxgxQmX79dju6ezZs2dSx2vYsGFke8GCBWqf/dkRtj//+c8qT5kyJaljIzhi11y94oorkrpvs2bN4u7/5JNPVN64cWNSjw9/xfZ1ZuPv/1zxBAAAAAB4ioknAAAAAMBTvNU2Qfv27VN5yJAhke25c+eqfQMHDoyb7bdZzJs3T+XCwsKyDhPFmDp1qsr2R9Tbb6UN8ltrYz+GWyQ7P4q7JDVq1CjX/Vu0aKGyXSf2skl169ZVuVKlSpHt/v37q33235u95MKmTZtUPnv2rMqXXqpP1Z9++qkgOOy3X06ePLnE23700UcqDx48WOVvv/3WtXHhYrH/TkVEatasGff2sW+RFBG59tprVR46dKjK3bt3V/mmm26KbFetWlXts99mZ+e33npLZbsFCKlz+eWXq9y0aVOVf//736scr8Un2dd1e2kXuwa///77uPcHUokrngAAAAAATzHxBAAAAAB4ioknAAAAAMBT9HiW0dKlSyPb+fn5ap/dU9i5c2eVX3jhBZXr16+v8sSJE1U+dOhQmceZjbp166Zyy5YtVbb7aFasWOH1kFxj937YP8u2bdt8HI3/7N7I2J//tddeU/uefvrppB7bXvbC7vE8f/68ymfOnFH5888/j2zPmTNH7bOXVLL7iA8fPqxyQUGBylWqVFF5165dgtTJyclRefHixQnf94svvlDZ/ruHt86dO6dyUVGRyrVq1VL5X//6l8rJLn8Q24934sQJta9OnToqHz16VOV33303qWPBPRUrVlTZXpbN/jdv/13ar1WxdWAvd2Ivr2X3j9rsnv/7779fZXtJJrvmgVTiiicAAAAAwFNMPAEAAAAAnmLiCQAAAADwFD2eLti+fbvKffv2Vfm+++5T2V73c/jw4So3atRI5a5du5Z3iFnF7oez1207cuSIyosWLfJ8TImqXLmyyhMmTIh7+7Vr16r81FNPuT2kQBkxYoTK+/fvj2zfdttt5XrsAwcOqLxs2TKVd+7cqfI//vGPch0v1rBhw1S2+8zsvkCk1tixY1VOZj3deGt8wnvHjx9X2V6D9b333lPZXh/YXtN7+fLlKufl5an8zTffRLYXLlyo9tl9gfZ++Mf+PcHuu1yyZEnc+z/33HMq26/NH3/8cWTbrin7trFrvxbHfn2YNGmSyqW9ltnrRMNfseu2lvba0aFDB5VnzJjhyZj8xBVPAAAAAICnmHgCAAAAADzFxBMAAAAA4Cl6PD1g95DMnz9f5dmzZ6tsr8lkv6e7U6dOKq9fv75c48t2dn9DYWFhikZycU/n+PHjVR4zZozK9vqOr7zyisqnTp1ycXTB9+KLL6Z6CK6w1/q1JbNOJNxnrwV8xx13JHxfuwdw9+7dbgwJLtm0aZPKdv9cecW+nnfs2FHts/u76OX2j71Op92jab/22lauXKny9OnTVbZ/D4ytq/fff1/ta9asmcr2uptTpkxR2e4B7dGjh8oLFixQ+W9/+5vK9uvmsWPHpCSZvjZ4KsT+uy9tXWB7jdamTZuqHLt+eLrgiicAAAAAwFNMPAEAAAAAnmLiCQAAAADwFD2eLmjevLnKv/zlL1Vu27atynZPp81+z/aGDRvKMTrYVqxYkbJj271idh/JAw88oLLdH9a7d29PxoVgW7p0aaqHkNU++OADla+++uq4t49d43XIkCFeDAlpInZdabun0+7vYh1P71SoUEHl559/XuXRo0erfPr0aZXHjRunsv13Zfd0tmnTRuXY9RdbtWql9uXn56v8yCOPqLxu3TqVq1WrprK9hnX//v1V7t69u8qrV6+Wkhw8eFDlBg0alHhblM1rr70W2R4+fHhS97XX/B41apQbQ/IVVzwBAAAAAJ5i4gkAAAAA8BQTTwAAAACAp+jxTFDjxo1VfuyxxyLb9jo7P/rRj5J67O+//15le11Juy8E8Rlj4uaePXuqPHLkSM/G8tvf/lbl3/3udypfddVVKtvrbw0aNMibgQFI2DXXXKNyaefkmTNnRrazbW1daKtWrUr1ECAX98bZPZ1nzpxR2e69s/u827Vrp/LQoUNVvvvuu1WO7fX9wx/+oPbNnTtXZbvP0nbixAmV//rXv8bN/fr1U/lXv/pViY9t/84C9+3atSvVQ0gprngCAAAAADxV6sTTGHO9MWadMWanMWaHMWZk6Ps1jDGrjTH5oa/xP+YPaY9agAh1gChqAWHUAkSoA0RRCyhOIlc8z4vIk47jNBGRdiLyqDGmqYiME5E1juM0EpE1oYzMRi1AhDpAFLWAMGoBItQBoqgFXMTY60iVegdjlovIjNCfTo7jFBpj6ojIesdxGpdy3+QO5iO7L9N+T3xsT6eISE5OTpmPtWXLFpUnTpyocirXmSzGp47jtCluR1BroU+fPiq//fbbKts9ta+//rrKc+bMUfnrr79W2e7tGDhwYGS7RYsWal/dunVVPnDggMqx6/2JiOTm5sbdn0qO45jivh/UOkgnixYtUrlv374qDx48WOV58+Z5PqY40u6ckCy758pei7O0Hs+f/OQnke39+/e7Nq4AyvhaKK8777wzsv3++++rffbvX3Xq1FG5qKjIu4G5LOivD/ZnZ9SqVUvls2fPqmz34V1xxRUqN2zYMKnjT5gwIbI9adIktc/+nSTNcU5Iwp49e1S+4YYb4t7+kkv09UK7Dvft2+fOwNxRbC0k1eNpjMkRkVYisklEajuOUygiEvp6rQuDRJqgFiBCHSCKWkAYtQAR6gBR1ALCEv5UW2NMVRFZLCKjHMc5YX9SaJz7DRORYaXeEGmDWoAIdYAoagFh1AJEqANEUQuIldAVT2NMRblQNAscx1kS+vbh0CVyCX09Utx9HceZ5ThOm5IuvSO9UAsQoQ4QRS0gjFqACHWAKGoBtlKveJoL/zXx3yKy03GcqTG7VojIYBGZHPq63JMRuqR27doqN23aVOUZM2aofOONN5b5WJs2bVL5pZdeUnn5cv1Upcs6nZlSCxUqVFB5xIgRKvfu3Vtle82sRo0aJXysTz75ROV169ap/Oyzzyb8WEGRKXUQZHbvl93XERSZUgstW7ZUuUuXLirb5+hz586p/Oqrr6p8+PBh9waXJjKlFtwW2++bDYJaB//+979Vtns8K1eurLL9eQ02u193w4YNKi9btkzlL7/8MrKdYT2dJQpqLQTJjh07VC7tfJEu84V4Enmr7X+JyEAR+V9jzLbQ956WCwXzjjHmIRE5ICJ9ir87Mgi1ABHqAFHUAsKoBYhQB4iiFnCRUieejuN8JCIlvSG7s7vDQZBRCxChDhBFLSCMWoAIdYAoagHFCeb7twAAAAAAGSPhT7UNuho1aqhsr81o9/CUt+8itnfvlVdeUftWrVql8nfffVeuYyE5GzduVHnz5s0qt23bNu797TVd7f5gW+w6nwsXLlT7Ro4cGfe+QCLat2+vcl5eXmoGkqGqV6+usn0OsB06dEjl0aNHuz0kZIi///3vkW27VzsT+rXSRYcOHVTu2bOnyjfffLPKR47oz7ux1/c+duyYynbfN5CIWbNmqXzfffelaCT+4YonAAAAAMBTTDwBAAAAAJ5i4gkAAAAA8FRa9Xjeeuutke0xY8aofbfccovK1113XbmOdebMGZWnTZum8gsvvBDZPn36dLmOBXcVFBSofP/996s8fPhwlcePH5/U4+fm5qr8pz/9KbK9d+/epB4LKM6F5c8ApLvt27dHtvPz89U++7MmbrjhBpWLioq8G1iWOXnypMrz58+PmwE/fP755yrv3LlT5SZNmvg5HF9wxRMAAAAA4CkmngAAAAAAT6XVW2179epV7HYi7MvZ7733nsrnz59X2V4i5fjx40kdD8FRWFio8oQJE+JmwG8rV65UuU+fPikaSXbatWuXyrHLZYmI3H777X4OBxkqtkVHRGT27NkqT5w4UeXHH39cZfv3GADpbf/+/So3a9YsRSPxD1c8AQAAAACeYuIJAAAAAPAUE08AAAAAgKeM4zj+HcwY/w4Gt3zqOE4btx+UWkg/juO4vsYHdZCWOCcgjFpIQrVq1VR+5513VO7SpYvKS5YsUXno0KEqB2kpN14fEMI5AWHF1gJXPAEAAAAAnmLiCQAAAADwFBNPAAAAAICn0modTwAAgHR04sQJlfv27auyvY7nI488orK95jTregJIN1zxBAAAAAB4ioknAAAAAMBTTDwBAAAAAJ5iHU+UhjWZICKs04YIzgkIoxYgIrw+IIJzAsJYxxMAAAAA4D8mngAAAAAATzHxBAAAAAB4yu91PI+KyH4RqRnaDiLGptX36HGDXgtBHZcIdeA3xqZRC8FELfgnqOMSoQ78xtg0aiGYAlMLvn64UOSgxmzxovnYDYzNX0H9mYI6LpFgj62sgvwzMTZ/BflnYmz+CurPFNRxiQR7bGUV5J+JsfkryD8TY0sMb7UFAAAAAHiKiScAAAAAwFOpmnjOStFxE8HY/BXUnymo4xIJ9tjKKsg/E2PzV5B/Jsbmr6D+TEEdl0iwx1ZWQf6ZGJu/gvwzMbYEpKTHEwAAAACQPXirLQAAAADAU75OPI0xdxljdhtj9hpjxvl57BLGM8cYc8QYsz3mezWMMauNMfmhr1enYFzXG2PWGWN2GmN2GGNGBmVsbglSLQS1DkLjoBb8HUsga4E68H0sgayD0DioBX/HQi2kELWQ0LioA3/HEsg6CI0j8LXg28TTGFNBRF4VkbtFpKmI9DPGNPXr+CXIE5G7rO+NE5E1juM0EpE1oey38yLypOM4TUSknYg8GnqugjC2cgtgLeRJMOtAhFrwW54EsxaoA3/lSTDrQIRa8FueUAspQS0kjDrwV54Esw5E0qEWHMfx5Y+ItBeRVTH5KRF5yq/jxxlXjohsj8m7RaROaLuOiOwOwBiXi0jXII4tU2ohHeqAWqAWqAPqgFqgFqgFaoE6oA7StRb8fKvtdSJyMCYXhL4XNLUdxykUEQl9vTaVgzHG5IhIKxHZJAEbWzmkQy0E7rmmFlImUM81dZAygXuuqYWUCdxzTS2kTKCea+ogZQL3XAe1FvyceJpivsdH6sZhjKkqIotFZJTjOCdSPR4XUQtJohYgQh0gilpAGLUAEeoAUUGuBT8nngUicn1MrisiX/l4/EQdNsbUEREJfT2SikEYYyrKhaJZ4DjOkiCNzQXpUAuBea6phZQLxHNNHaRcYJ5raiHlAvNcUwspF4jnmjpIucA810GvBT8nnptFpJExpoExppKIPCgiK3w8fqJWiMjg0PZgufD+aF8ZY4yI/LeI7HQcZ2qQxuaSdKiFQDzX1EIgpPy5pg4CIRDPNbUQCIF4rqmFQEj5c00dBEIgnuu0qAWfm1zvEZE9IrJPRJ5JVWNrzHjeFpFCEfk/ufA/Kg+JyDVy4ROf8kNfa6RgXLfLhbcR/I+IbAv9uScIY8vEWghqHVAL1AJ1QB1QC9QCtUAtUAfUQabUggkNFAAAAAAAT/j5VlsAAAAAQBZi4gkAAAAA8BQTTwAAAACAp5h4AgAAAAA8xcQTAAAAAOApJp4AAAAAAE8x8QQAAAAAeIqJJwAAAADAU/8PSXdIbxrRR2wAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1152x288 with 7 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      5          0          4          1          9          2          1\n"
     ]
    }
   ],
   "source": [
    "draw_examples(x_train[:7], captions=y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "60000 przykładów uczących\n",
      "10000 przykładów testowych\n"
     ]
    }
   ],
   "source": [
    "num_classes = 10\n",
    "\n",
    "x_train = x_train.reshape(60000, 784)  # 784 = 28 * 28\n",
    "x_test = x_test.reshape(10000, 784)\n",
    "x_train = x_train.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "print('{} przykładów uczących'.format(x_train.shape[0]))\n",
    "print('{} przykładów testowych'.format(x_test.shape[0]))\n",
    "\n",
    "# przekonwertuj wektory klas na binarne macierze klas\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_21\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_59 (Dense)             (None, 512)               401920    \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_60 (Dense)             (None, 512)               262656    \n",
      "_________________________________________________________________\n",
      "dropout_3 (Dropout)          (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_61 (Dense)             (None, 10)                5130      \n",
      "=================================================================\n",
      "Total params: 669,706\n",
      "Trainable params: 669,706\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = keras.Sequential()\n",
    "model.add(Dense(512, activation='relu', input_shape=(784,)))\n",
    "model.add(Dropout(0.2))\n",
    "model.add(Dense(512, activation='relu'))\n",
    "model.add(Dropout(0.2))\n",
    "model.add(Dense(num_classes, activation='softmax'))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 784) (60000, 10)\n"
     ]
    }
   ],
   "source": [
    "print(x_train.shape, y_train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "469/469 [==============================] - 11s 23ms/step - loss: 0.2463 - accuracy: 0.9238 - val_loss: 0.1009 - val_accuracy: 0.9690\n",
      "Epoch 2/5\n",
      "469/469 [==============================] - 10s 22ms/step - loss: 0.1042 - accuracy: 0.9681 - val_loss: 0.0910 - val_accuracy: 0.9739\n",
      "Epoch 3/5\n",
      "469/469 [==============================] - 11s 23ms/step - loss: 0.0774 - accuracy: 0.9762 - val_loss: 0.0843 - val_accuracy: 0.9755\n",
      "Epoch 4/5\n",
      "469/469 [==============================] - 11s 24ms/step - loss: 0.0606 - accuracy: 0.9815 - val_loss: 0.0691 - val_accuracy: 0.9818\n",
      "Epoch 5/5\n",
      "469/469 [==============================] - 10s 22ms/step - loss: 0.0504 - accuracy: 0.9848 - val_loss: 0.0886 - val_accuracy: 0.9772\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1ed6e0565b0>"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(loss='categorical_crossentropy', optimizer=keras.optimizers.RMSprop(), metrics=['accuracy'])\n",
    "\n",
    "model.fit(x_train, y_train, batch_size=128, epochs=5, verbose=1,\n",
    "          validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test loss: 0.08859136700630188\n",
      "Test accuracy: 0.9771999716758728\n"
     ]
    }
   ],
   "source": [
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "\n",
    "print('Test loss: {}'.format(score[0]))\n",
    "print('Test accuracy: {}'.format(score[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Warstwa _dropout_ to metoda regularyzacji, służy zapobieganiu nadmiernemu dopasowaniu sieci. Polega na tym, że część węzłów sieci jest usuwana w sposób losowy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_22\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_62 (Dense)             (None, 512)               401920    \n",
      "_________________________________________________________________\n",
      "dense_63 (Dense)             (None, 512)               262656    \n",
      "_________________________________________________________________\n",
      "dense_64 (Dense)             (None, 10)                5130      \n",
      "=================================================================\n",
      "Total params: 669,706\n",
      "Trainable params: 669,706\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/5\n",
      "469/469 [==============================] - 10s 20ms/step - loss: 0.2203 - accuracy: 0.9317 - val_loss: 0.0936 - val_accuracy: 0.9697\n",
      "Epoch 2/5\n",
      "469/469 [==============================] - 10s 21ms/step - loss: 0.0816 - accuracy: 0.9746 - val_loss: 0.0747 - val_accuracy: 0.9779\n",
      "Epoch 3/5\n",
      "469/469 [==============================] - 10s 20ms/step - loss: 0.0544 - accuracy: 0.9827 - val_loss: 0.0674 - val_accuracy: 0.9798\n",
      "Epoch 4/5\n",
      "469/469 [==============================] - 10s 22ms/step - loss: 0.0384 - accuracy: 0.9879 - val_loss: 0.0746 - val_accuracy: 0.9806\n",
      "Epoch 5/5\n",
      "469/469 [==============================] - 10s 22ms/step - loss: 0.0298 - accuracy: 0.9901 - val_loss: 0.0736 - val_accuracy: 0.9801\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1ed7eba8070>"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Bez warstw Dropout\n",
    "\n",
    "num_classes = 10\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "\n",
    "x_train = x_train.reshape(60000, 784)  # 784 = 28 * 28\n",
    "x_test = x_test.reshape(10000, 784)\n",
    "x_train = x_train.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes)\n",
    "\n",
    "model_no_dropout = keras.Sequential()\n",
    "model_no_dropout.add(Dense(512, activation='relu', input_shape=(784,)))\n",
    "model_no_dropout.add(Dense(512, activation='relu'))\n",
    "model_no_dropout.add(Dense(num_classes, activation='softmax'))\n",
    "model_no_dropout.summary()\n",
    "\n",
    "model_no_dropout.compile(loss='categorical_crossentropy',\n",
    "                         optimizer=keras.optimizers.RMSprop(),\n",
    "                         metrics=['accuracy'])\n",
    "\n",
    "model_no_dropout.fit(x_train, y_train,\n",
    "                     batch_size=128,\n",
    "                     epochs=5,\n",
    "                     verbose=1,\n",
    "                     validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test loss (no dropout): 0.07358124107122421\n",
      "Test accuracy (no dropout): 0.9800999760627747\n"
     ]
    }
   ],
   "source": [
    "# Bez warstw Dropout\n",
    "\n",
    "score = model_no_dropout.evaluate(x_test, y_test, verbose=0)\n",
    "\n",
    "print('Test loss (no dropout): {}'.format(score[0]))\n",
    "print('Test accuracy (no dropout): {}'.format(score[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_23\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_65 (Dense)             (None, 2500)              1962500   \n",
      "_________________________________________________________________\n",
      "dense_66 (Dense)             (None, 2000)              5002000   \n",
      "_________________________________________________________________\n",
      "dense_67 (Dense)             (None, 1500)              3001500   \n",
      "_________________________________________________________________\n",
      "dense_68 (Dense)             (None, 1000)              1501000   \n",
      "_________________________________________________________________\n",
      "dense_69 (Dense)             (None, 500)               500500    \n",
      "_________________________________________________________________\n",
      "dense_70 (Dense)             (None, 10)                5010      \n",
      "=================================================================\n",
      "Total params: 11,972,510\n",
      "Trainable params: 11,972,510\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/10\n",
      "469/469 [==============================] - 129s 275ms/step - loss: 0.9587 - accuracy: 0.7005 - val_loss: 0.5066 - val_accuracy: 0.8566\n",
      "Epoch 2/10\n",
      "469/469 [==============================] - 130s 276ms/step - loss: 0.2666 - accuracy: 0.9234 - val_loss: 0.3376 - val_accuracy: 0.9024\n",
      "Epoch 3/10\n",
      "469/469 [==============================] - 130s 277ms/step - loss: 0.1811 - accuracy: 0.9477 - val_loss: 0.1678 - val_accuracy: 0.9520\n",
      "Epoch 4/10\n",
      "469/469 [==============================] - 134s 287ms/step - loss: 0.1402 - accuracy: 0.9588 - val_loss: 0.1553 - val_accuracy: 0.9576\n",
      "Epoch 5/10\n",
      "469/469 [==============================] - 130s 278ms/step - loss: 0.1153 - accuracy: 0.9662 - val_loss: 0.1399 - val_accuracy: 0.9599\n",
      "Epoch 6/10\n",
      "469/469 [==============================] - 130s 277ms/step - loss: 0.0956 - accuracy: 0.9711 - val_loss: 0.1389 - val_accuracy: 0.9612\n",
      "Epoch 7/10\n",
      "469/469 [==============================] - 131s 280ms/step - loss: 0.0803 - accuracy: 0.9761 - val_loss: 0.1008 - val_accuracy: 0.9724\n",
      "Epoch 8/10\n",
      "469/469 [==============================] - 134s 286ms/step - loss: 0.0685 - accuracy: 0.9797 - val_loss: 0.1137 - val_accuracy: 0.9679\n",
      "Epoch 9/10\n",
      "469/469 [==============================] - 130s 278ms/step - loss: 0.0602 - accuracy: 0.9819 - val_loss: 0.1064 - val_accuracy: 0.9700\n",
      "Epoch 10/10\n",
      "469/469 [==============================] - 129s 274ms/step - loss: 0.0520 - accuracy: 0.9843 - val_loss: 0.1095 - val_accuracy: 0.9698\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1ed0e628250>"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Więcej warstw, inna funkcja aktywacji\n",
    "\n",
    "num_classes = 10\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "\n",
    "x_train = x_train.reshape(60000, 784)  # 784 = 28 * 28\n",
    "x_test = x_test.reshape(10000, 784)\n",
    "x_train = x_train.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes)\n",
    "\n",
    "model3 = keras.Sequential()\n",
    "model3.add(Dense(2500, activation='tanh', input_shape=(784,)))\n",
    "model3.add(Dense(2000, activation='tanh'))\n",
    "model3.add(Dense(1500, activation='tanh'))\n",
    "model3.add(Dense(1000, activation='tanh'))\n",
    "model3.add(Dense(500, activation='tanh'))\n",
    "model3.add(Dense(num_classes, activation='softmax'))\n",
    "model3.summary()\n",
    "\n",
    "model3.compile(loss='categorical_crossentropy',\n",
    "               optimizer=keras.optimizers.RMSprop(),\n",
    "               metrics=['accuracy'])\n",
    "\n",
    "model3.fit(x_train, y_train,\n",
    "           batch_size=128,\n",
    "           epochs=10,\n",
    "           verbose=1,\n",
    "           validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test loss: 0.10945799201726913\n",
      "Test accuracy: 0.9697999954223633\n"
     ]
    }
   ],
   "source": [
    "# Więcej warstw, inna funkcja aktywacji\n",
    "\n",
    "score = model3.evaluate(x_test, y_test, verbose=0)\n",
    "\n",
    "print('Test loss: {}'.format(score[0]))\n",
    "print('Test accuracy: {}'.format(score[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Przykład: 4-pikselowy aparat fotograficzny\n",
    "\n",
    "https://www.youtube.com/watch?v=ILsA4nyG7I0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def generate_example(description):\n",
    "    variant = random.choice([1, -1])\n",
    "    if description == 's':  # solid\n",
    "        return (np.array([[ 1.0,  1.0], [ 1.0,  1.0]]) if variant == 1 else\n",
    "                np.array([[-1.0, -1.0], [-1.0, -1.0]]))\n",
    "    elif description == 'v':  # vertical\n",
    "        return (np.array([[ 1.0, -1.0], [ 1.0, -1.0]]) if variant == 1 else\n",
    "                np.array([[-1.0,  1.0], [-1.0,  1.0]]))\n",
    "    elif description == 'd':  # diagonal\n",
    "        return (np.array([[ 1.0, -1.0], [-1.0,  1.0]]) if variant == 1 else\n",
    "                np.array([[-1.0,  1.0], [ 1.0, -1.0]]))\n",
    "    elif description == 'h':  # horizontal\n",
    "        return (np.array([[ 1.0,  1.0], [-1.0, -1.0]]) if variant == 1 else\n",
    "                np.array([[-1.0, -1.0], [ 1.0,  1.0]]))\n",
    "    else:\n",
    "        return np.array([[random.uniform(-1, 1), random.uniform(-1, 1)],\n",
    "                         [random.uniform(-1, 1), random.uniform(-1, 1)]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "num_classes = 4\n",
    "\n",
    "trainset_size = 4000\n",
    "testset_size = 1000\n",
    "\n",
    "y4_train = np.array([random.choice(['s', 'v', 'd', 'h']) for i in range(trainset_size)])\n",
    "x4_train = np.array([generate_example(desc) for desc in y4_train])\n",
    "\n",
    "y4_test = np.array([random.choice(['s', 'v', 'd', 'h']) for i in range(testset_size)])\n",
    "x4_test = np.array([generate_example(desc) for desc in y4_test])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6oAAACQCAYAAAABdZZIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAPGUlEQVR4nO3dX4ild33H8c+3uwYhVKTN+ie7WZOLZSUVWswkVXqTXliSYJNeaIm9UKSwKOSyFwsF7WUveiUVQy5C4o22eGGXNq1NhRALymZXVJK2SRdRs0nARkvaVTFm+fVizrMzpLOzc/Y858xvnvN6wZA5c56c5zkn7/09893zzKRaawEAAIBe/Np+HwAAAABsZ1AFAACgKwZVAAAAumJQBQAAoCsGVQAAALpiUAUAAKArhxf5l6vqN5L8TZJbk/wgyR+31v57h+1+kOR/k1xO8kZrbWOR/dIfLTDQAokO2KIFBlog0QF7t+g7qqeTfL21diLJ12e3r+b3W2u/I7LJ0gIDLZDogC1aYKAFEh2wR4sOqg8keXz2+eNJ/mjBx+Pg0gIDLZDogC1aYKAFEh2wR4sOqu9srb2SJLN/vuMq27Uk/1xV56vq1IL7pE9aYKAFEh2wRQsMtECiA/bomj+jWlX/kuRdO9z153Ps5/daay9X1TuSPFlV/9Fae/oq+zuVZIjxjjn2wT6rqtZaq2tstqcW1qGDO+44uE/rhRdeyK9+9av/9/WjR4/m0KFDqar/aq0ducbDzN3CjTfeeMd73/veMZ4CI7hWBxsbG+38+fOvXqMF54c1sKzzw1TXhPPnz+/3ISzNss4PmeiacFC/V1jl+UEHB9tuHVRr7bofuKqeT3J3a+2Vqnp3kqdaayev8e/8RZJLrbW/2sPjX//BsS9aazV2C1PtYJE/ez07efJkXnjhhe+11n577BY2NjbauXPnRjxaluXkyZN56qmncvPNN59P8odxflh7yzg/THVNqLrWTH+gLeX8MNU1YYrfKyzz/KCDg6eqzl/tZ5AXvfT3TJJPzD7/RJK/22HnN1bVrw+fJ/mDJM8uuF/6poU1dv/99yfJb85uamFN3X///Xn88eFHkHTAFVrA+WHNOT+wV4sOqn+Z5ENV9Z9JPjS7naq6uaqemG3zziT/WlXfTXI2yT+01v5pwf3SKS1w+vTpJHmbFtbb6dOn8+STTybJ+6ID4vzAFc4Pa875gb1a6NLfZZvq2/dTtoefQZrbVDvo+c/eona7jGMRU73Mb8qW1cJU14UpW8b5YaprwsQv/bUmzMH3CnM/5iRfsHXtYNF3VAEAAGBUBlUAAAC6YlAFAACgKwZVAAAAumJQBQAAoCsGVQAAALpiUAUAAKArBlUAAAC6YlAFAACgKwZVAAAAumJQBQAAoCsGVQAAALpiUAUAAKArBlUAAAC6YlAFAACgKwZVAAAAumJQBQAAoCsGVQAAALoyyqBaVfdU1fNVdaGqTu9wf1XV52b3f6+q3j/GfumPFhhogZm36YDEmsAV1gQGWmBXCw+qVXUoyeeT3Jvk9iQfq6rb37TZvUlOzD5OJfnCovulW1pgoIU1d/ny5SQ5Hh2wyZpAYk0gzg/szRjvqN6V5EJr7futtdeTfDnJA2/a5oEkX2ybvpXk7VX17hH2TX+0QJLcGC2svbNnzybJL3XAjDWBxJpAnB/YmzEG1aNJXtx2++Lsa/NuwzRogSS5IVpYey+99FKSvL7tSzpYb9YEEmsCcX5gbw6P8Bi1w9fadWyzuWHVqWy+vc80XFcLOpikhVs4fvz4Eg6LZWltx2Xe+YGBNYHEmrCWxjw/6GC6xnhH9WKSW7bdPpbk5evYJknSWnuktbbRWtsY4dhYvVFa0MGB93qW0MKRI0dGP1CW59ixY8nmu+tXvhTnh3VmTSCxJpBxzw86mK4xBtVnkpyoqtuq6oYkDyY586ZtziT5+Oy3d30gyWuttVdG2Df90QJJ8rNoYe3deeedSfJWHTBjTSCxJhDnB/Zm4Ut/W2tvVNVDSb6W5FCSR1trz1XVp2b3P5zkiST3JbmQ5OdJPrnofumWFhhoYc0dPnw4SX4UHbDJmkBiTSDOD+xNXeUa8S5UVb8Hx45aazv9PMFCptpBz3/2FlVV55dxCc7GxkY7d+7c2A/LEi2rhamuC1O2jPPDVNeEqtFfqp5YE+bge4W5H3OSL9i6djDGpb8AAAAwGoMqAAAAXTGoAgAA0BWDKgAAAF0xqAIAANAVgyoAAABdMagCAADQFYMqAAAAXTGoAgAA0BWDKgAAAF0xqAIAANAVgyoAAABdMagCAADQFYMqAAAAXTGoAgAA0BWDKgAAAF0xqAIAANAVgyoAAABdMagCAADQlVEG1aq6p6qer6oLVXV6h/vvrqrXquo7s4/PjLFf+qMFBlpg5m06ILEmcIU1gYEW2NXhRR+gqg4l+XySDyW5mOSZqjrTWvu3N236jdbahxfdH93TAgMtrLnLly8nyfEkt0cHWBPYZE3A+YE9GeMd1buSXGitfb+19nqSLyd5YITH5WDSAklyY7Sw9s6ePZskv9QBM9YEEmsCcX5gbxZ+RzXJ0SQvbrt9Mcnv7rDdB6vqu0leTvJnrbXndnqwqjqV5NQIx8X+GKWF7R0cP348P/zhD5dxrPuqqvb7EJbphiyhhdntkQ+VJXt92+fOD+vNmrBHrbX9PoSl+MpXvpKPfvSj1gTy0ksvJSOdH3QwXWMMqjudId68wn47yXtaa5eq6r4kX01yYqcHa609kuSRJKmqaa7U6+W6WtjewcbGhg6mYeEWrAmT4PzAwJqwZq4ygFsT1tCYLehgusa49Pdiklu23T6Wzb/1uKK19j+ttUuzz59I8paqummEfdMfLZBs/i2pFkg2310f6GC9WRPW3LFjxxJrAtECezPGoPpMkhNVdVtV3ZDkwSRntm9QVe+q2bU5VXXXbL8/GWHf9EcLJMnPogU2vVUHzFgT1tydd96ZWBOIFtibhS/9ba29UVUPJflakkNJHm2tPVdVn5rd/3CSjyT5dFW9keQXSR5sU/0BDLTAQAskyY+iAzZZE9bc4cOHE2sC0QJ7Uz3/93ad+cHTWhv9t1psbGy0c+fOjf2w+26KvwBkm/OttY2xH9SacCBpgSTLOT9MtYOevzdbVFVZE+aghbkfc5Iv2Lp2MMalvwAAADAagyoAAABdMagCAADQFYMqAAAAXTGoAgAA0BWDKgAAAF0xqAIAANAVgyoAAABdMagCAADQFYMqAAAAXTGoAgAA0BWDKgAAAF0xqAIAANAVgyoAAABdMagCAADQFYMqAAAAXTGoAgAA0BWDKgAAAF0ZZVCtqker6sdV9exV7q+q+lxVXaiq71XV+8fYL/3RATO3aoEZLZDE+YErrAkMtMCuxnpH9bEk9+xy/71JTsw+TiX5wkj7pT86IElejRbYpAUGOiCxJrBFC+xqlEG1tfZ0kp/usskDSb7YNn0rydur6t1j7Jvu6IAkuRQtsEkLDHRAYk1gixbY1ap+RvVokhe33b44+xrrRQcMtMBACyQ6YIsWGGhhzR1e0X5qh6+1HTesOpXNt/eZnuvq4Pjx48s8JvaHNYGBFkh0wBYtMNhTCzqYrlW9o3oxyS3bbh9L8vJOG7bWHmmtbbTWNlZyZKzSdXVw5MiRlRwcK2VNYKAFEh2wRQsM9tSCDqZrVYPqmSQfn/32rg8kea219sqK9k0/dMBACwy0QKIDtmiBgRbW3CiX/lbVl5LcneSmqrqY5LNJ3pIkrbWHkzyR5L4kF5L8PMknx9gvXfpmdEByW7TAJi0w0AGJNYEtWmBX1dqOl/13oar6PTh21Frb6ecJFrKxsdHOnTs39sPuu6rRX6qenF/GJTjWhANJCyRZzvlhqh30/L3ZoqrKmjAHLcz9mJN8wda1g1Vd+gsAAAB7YlAFAACgKwZVAAAAumJQBQAAoCsGVQAAALpiUAUAAKArBlUAAAC6YlAFAACgKwZVAAAAumJQBQAAoCsGVQAAALpiUAUAAKArBlUAAAC6YlAFAACgKwZVAAAAumJQBQAAoCsGVQAAALpiUAUAAKArowyqVfVoVf24qp69yv13V9VrVfWd2cdnxtgv/dEBM7dqgRktkMT5gSusCQy0wK4Oj/Q4jyX56yRf3GWbb7TWPjzS/ujXPdEByatJ/iRaQAtscX4gsSawRQvsapR3VFtrTyf56RiPxYGnA5LkUrTAJi0w0AGJNYEtWmBXq/wZ1Q9W1Xer6h+r6rdWuF/6ogMGWmCgBRIdsEULDLSwxsa69Pdavp3kPa21S1V1X5KvJjmx04ZVdSrJqdnNS0meX8kRJjdl8xKEqVnl83rPNe6/7g6qSgeLm0QLsSaMQQvzmWoLOpjPyl6vqlrFbrbTwny0MI5RWtjHDpIVvV7r2kG11kbZQ1XdmuTvW2vv28O2P0iy0Vrr5sRfVedaaxv7fRxjW/Xz0kG/tDAfLYy6v1ujhe7oYD5T7SDRwry0MOr+bo0WutPL81rJpb9V9a6a/VVAVd012+9PVrFv+qEDBlpgoAUSHbBFCwy0wCiX/lbVl5LcneSmqrqY5LNJ3pIkrbWHk3wkyaer6o0kv0jyYBvrrVy6oQMGWmCgBRIdsEULDLTAtYx26e9BV1WnWmuP7PdxjG2qz2tZpvx6Tfm5LcOUX68pP7dlmOrrNdXntSxTfr2m/NyWYcqv15Sf2zJM9fXq5XkZVAEAAOjKKv/3NAAAAHBNaz+oVtU9VfV8VV2oqtP7fTxjqapHq+rHVfXsfh/LQaEFBlNsQQfzm2IHiRauhxYYTLEFHcxvih0k/bWw1oNqVR1K8vkk9ya5PcnHqur2/T2q0TyW5J79PoiDQgsMJtzCY9HBnk24g0QLc9ECgwm38Fh0sGcT7iDprIW1HlST3JXkQmvt+62115N8OckD+3xMo2itPZ3kp/t9HAeIFhhMsgUdzG2SHSRauA5aYDDJFnQwt0l2kPTXwroPqkeTvLjt9sXZ11g/WmCgBRIdsEULDLRAooOVWfdBtXb4ml+DvJ60wEALJDpgixYYaIFEByuz7oPqxSS3bLt9LMnL+3Qs7C8tMNACiQ7YogUGWiDRwcqs+6D6TJITVXVbVd2Q5MEkZ/b5mNgfWmCgBRIdsEULDLRAooOVWetBtbX2RpKHknwtyb8n+dvW2nP7e1TjqKovJflmkpNVdbGq/nS/j6lnWmAw1RZ0MJ+pdpBoYV5aYDDVFnQwn6l2kPTXQrXmkmoAAAD6sdbvqAIAANAfgyoAAABdMagCAADQFYMqAAAAXTGoAgAA0BWDKgAAAF0xqAIAANAVgyoAAABd+T89XtOTyy31ugAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1152x288 with 7 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      s          d          h          s          d          v          v\n"
     ]
    }
   ],
   "source": [
    "draw_examples(x4_train[:7], captions=y4_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "x4_train = x4_train.reshape(trainset_size, 4)\n",
    "x4_test = x4_test.reshape(testset_size, 4)\n",
    "x4_train = x4_train.astype('float32')\n",
    "x4_test = x4_test.astype('float32')\n",
    "\n",
    "y4_train = np.array([{'s': 0, 'v': 1, 'd': 2, 'h': 3}[desc] for desc in y4_train])\n",
    "y4_test = np.array([{'s': 0, 'v': 1, 'd': 2, 'h': 3}[desc] for desc in y4_test])\n",
    "\n",
    "y4_train = keras.utils.to_categorical(y4_train, num_classes)\n",
    "y4_test = keras.utils.to_categorical(y4_test, num_classes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_24\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_71 (Dense)             (None, 4)                 20        \n",
      "_________________________________________________________________\n",
      "dense_72 (Dense)             (None, 4)                 20        \n",
      "_________________________________________________________________\n",
      "dense_73 (Dense)             (None, 8)                 40        \n",
      "_________________________________________________________________\n",
      "dense_74 (Dense)             (None, 4)                 36        \n",
      "=================================================================\n",
      "Total params: 116\n",
      "Trainable params: 116\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model4 = keras.Sequential()\n",
    "model4.add(Dense(4, activation='tanh', input_shape=(4,)))\n",
    "model4.add(Dense(4, activation='tanh'))\n",
    "model4.add(Dense(8, activation='relu'))\n",
    "model4.add(Dense(num_classes, activation='softmax'))\n",
    "model4.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "model4.layers[0].set_weights(\n",
    "    [np.array([[ 1.0,  0.0,  1.0,  0.0],\n",
    "               [ 0.0,  1.0,  0.0,  1.0],\n",
    "               [ 1.0,  0.0, -1.0,  0.0],\n",
    "               [ 0.0,  1.0,  0.0, -1.0]],\n",
    "              dtype=np.float32), np.array([0., 0., 0., 0.], dtype=np.float32)])\n",
    "model4.layers[1].set_weights(\n",
    "    [np.array([[ 1.0, -1.0,  0.0,  0.0],\n",
    "               [ 1.0,  1.0,  0.0,  0.0],\n",
    "               [ 0.0,  0.0,  1.0, -1.0],\n",
    "               [ 0.0,  0.0, -1.0, -1.0]],\n",
    "              dtype=np.float32), np.array([0., 0., 0., 0.], dtype=np.float32)])\n",
    "model4.layers[2].set_weights(\n",
    "    [np.array([[ 1.0, -1.0,  0.0,  0.0,  0.0,  0.0,  0.0,  0.0],\n",
    "               [ 0.0,  0.0,  1.0, -1.0,  0.0,  0.0,  0.0,  0.0],\n",
    "               [ 0.0,  0.0,  0.0,  0.0,  1.0, -1.0,  0.0,  0.0],\n",
    "               [ 0.0,  0.0,  0.0,  0.0,  0.0,  0.0,  1.0, -1.0]],\n",
    "              dtype=np.float32), np.array([0., 0., 0., 0., 0., 0., 0., 0.], dtype=np.float32)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "model4.layers[3].set_weights(\n",
    "    [np.array([[ 1.0,  0.0,  0.0,  0.0],\n",
    "               [ 1.0,  0.0,  0.0,  0.0],\n",
    "               [ 0.0,  1.0,  0.0,  0.0],\n",
    "               [ 0.0,  1.0,  0.0,  0.0],\n",
    "               [ 0.0,  0.0,  1.0,  0.0],\n",
    "               [ 0.0,  0.0,  1.0,  0.0],\n",
    "               [ 0.0,  0.0,  0.0,  1.0],\n",
    "               [ 0.0,  0.0,  0.0,  1.0]],\n",
    "              dtype=np.float32), np.array([0., 0., 0., 0.], dtype=np.float32)])\n",
    "\n",
    "model4.compile(loss='categorical_crossentropy',\n",
    "               optimizer=keras.optimizers.Adagrad(),\n",
    "               metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[array([[ 1.,  0.,  1.,  0.],\n",
      "       [ 0.,  1.,  0.,  1.],\n",
      "       [ 1.,  0., -1.,  0.],\n",
      "       [ 0.,  1.,  0., -1.]], dtype=float32), array([0., 0., 0., 0.], dtype=float32)]\n",
      "[array([[ 1., -1.,  0.,  0.],\n",
      "       [ 1.,  1.,  0.,  0.],\n",
      "       [ 0.,  0.,  1., -1.],\n",
      "       [ 0.,  0., -1., -1.]], dtype=float32), array([0., 0., 0., 0.], dtype=float32)]\n",
      "[array([[ 1., -1.,  0.,  0.,  0.,  0.,  0.,  0.],\n",
      "       [ 0.,  0.,  1., -1.,  0.,  0.,  0.,  0.],\n",
      "       [ 0.,  0.,  0.,  0.,  1., -1.,  0.,  0.],\n",
      "       [ 0.,  0.,  0.,  0.,  0.,  0.,  1., -1.]], dtype=float32), array([0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32)]\n",
      "[array([[1., 0., 0., 0.],\n",
      "       [1., 0., 0., 0.],\n",
      "       [0., 1., 0., 0.],\n",
      "       [0., 1., 0., 0.],\n",
      "       [0., 0., 1., 0.],\n",
      "       [0., 0., 1., 0.],\n",
      "       [0., 0., 0., 1.],\n",
      "       [0., 0., 0., 1.]], dtype=float32), array([0., 0., 0., 0.], dtype=float32)]\n"
     ]
    }
   ],
   "source": [
    "for layer in model4.layers:\n",
    "    print(layer.get_weights())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.17831734, 0.17831734, 0.17831734, 0.465048  ]], dtype=float32)"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model4.predict([np.array([[1.0, 1.0], [-1.0, -1.0]]).reshape(1, 4)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test loss: 0.7656148672103882\n",
      "Test accuracy: 1.0\n"
     ]
    }
   ],
   "source": [
    "score = model4.evaluate(x4_test, y4_test, verbose=0)\n",
    "\n",
    "print('Test loss: {}'.format(score[0]))\n",
    "print('Test accuracy: {}'.format(score[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_25\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_75 (Dense)             (None, 4)                 20        \n",
      "_________________________________________________________________\n",
      "dense_76 (Dense)             (None, 4)                 20        \n",
      "_________________________________________________________________\n",
      "dense_77 (Dense)             (None, 8)                 40        \n",
      "_________________________________________________________________\n",
      "dense_78 (Dense)             (None, 4)                 36        \n",
      "=================================================================\n",
      "Total params: 116\n",
      "Trainable params: 116\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model5 = keras.Sequential()\n",
    "model5.add(Dense(4, activation='tanh', input_shape=(4,)))\n",
    "model5.add(Dense(4, activation='tanh'))\n",
    "model5.add(Dense(8, activation='relu'))\n",
    "model5.add(Dense(num_classes, activation='softmax'))\n",
    "model5.compile(loss='categorical_crossentropy',\n",
    "               optimizer=keras.optimizers.RMSprop(),\n",
    "               metrics=['accuracy'])\n",
    "model5.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/8\n",
      "125/125 [==============================] - 0s 3ms/step - loss: 1.3126 - accuracy: 0.3840 - val_loss: 1.1926 - val_accuracy: 0.6110\n",
      "Epoch 2/8\n",
      "125/125 [==============================] - 0s 2ms/step - loss: 1.0978 - accuracy: 0.5980 - val_loss: 1.0085 - val_accuracy: 0.6150\n",
      "Epoch 3/8\n",
      "125/125 [==============================] - 0s 2ms/step - loss: 0.9243 - accuracy: 0.7035 - val_loss: 0.8416 - val_accuracy: 0.7380\n",
      "Epoch 4/8\n",
      "125/125 [==============================] - 0s 2ms/step - loss: 0.7522 - accuracy: 0.8740 - val_loss: 0.6738 - val_accuracy: 1.0000\n",
      "Epoch 5/8\n",
      "125/125 [==============================] - 0s 2ms/step - loss: 0.5811 - accuracy: 1.0000 - val_loss: 0.5030 - val_accuracy: 1.0000\n",
      "Epoch 6/8\n",
      "125/125 [==============================] - 0s 2ms/step - loss: 0.4134 - accuracy: 1.0000 - val_loss: 0.3428 - val_accuracy: 1.0000\n",
      "Epoch 7/8\n",
      "125/125 [==============================] - 0s 2ms/step - loss: 0.2713 - accuracy: 1.0000 - val_loss: 0.2161 - val_accuracy: 1.0000\n",
      "Epoch 8/8\n",
      "125/125 [==============================] - 0s 1ms/step - loss: 0.1621 - accuracy: 1.0000 - val_loss: 0.1225 - val_accuracy: 1.0000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1ed00809700>"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model5.fit(x4_train, y4_train, epochs=8, validation_data=(x4_test, y4_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[3.2040708e-02, 1.0065207e-03, 4.9596769e-04, 9.6645677e-01]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model5.predict([np.array([[1.0, 1.0], [-1.0, -1.0]]).reshape(1, 4)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test loss: 0.1224619448184967\n",
      "Test accuracy: 1.0\n"
     ]
    }
   ],
   "source": [
    "score = model5.evaluate(x4_test, y4_test, verbose=0)\n",
    "\n",
    "print('Test loss: {}'.format(score[0]))\n",
    "print('Test accuracy: {}'.format(score[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "import contextlib\n",
    "\n",
    "@contextlib.contextmanager\n",
    "def printoptions(*args, **kwargs):\n",
    "    original = np.get_printoptions()\n",
    "    np.set_printoptions(*args, **kwargs)\n",
    "    try:\n",
    "        yield\n",
    "    finally: \n",
    "        np.set_printoptions(**original)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[array([[ 0.7,  0.2, -0.7,  0.7],\n",
      "       [-0.5,  0.9,  0.6,  0.6],\n",
      "       [ 1.1,  0.2,  0.1,  0.2],\n",
      "       [ 0.7,  0.1,  0.3, -0.7]], dtype=float32), array([ 0. ,  0.1, -0.1, -0.2], dtype=float32)]\n",
      "[array([[ 0.7,  0.5, -1.1, -1.2],\n",
      "       [ 0.7,  0.9, -0.6,  0.3],\n",
      "       [ 0.1,  1.4, -0.6,  0.8],\n",
      "       [ 1.5,  0.1, -0.1,  0.9]], dtype=float32), array([-0.4,  0.2, -0. ,  0.2], dtype=float32)]\n",
      "[array([[-1. ,  1. , -0.7, -0.3,  0.2,  1.3, -0.7,  0.9],\n",
      "       [-0.9,  0.5,  0.8, -1.3, -1.2,  1.3,  0.4, -1. ],\n",
      "       [ 0.9,  0.2,  0.3,  0.4,  1.3, -0.9, -0.1, -0.2],\n",
      "       [-0.4,  0.5,  1.1, -0.6,  1.1,  0.1, -1.5, -1. ]], dtype=float32), array([-0.1,  0.1,  0.1,  0.1,  0.2, -0. ,  0.1,  0.2], dtype=float32)]\n",
      "[array([[ 0.7, -0.5,  0.8, -0.5],\n",
      "       [-0.3, -1.6, -0.2,  0.1],\n",
      "       [-1.5,  0.9,  0.1, -0.5],\n",
      "       [ 0.6,  0.7,  1. , -1.4],\n",
      "       [ 0.7, -1.2, -1.6,  1.2],\n",
      "       [ 1. , -1.2,  0.3, -1.5],\n",
      "       [-0.2,  0. ,  0.6,  1.3],\n",
      "       [-0.8,  0.2, -0.6, -1. ]], dtype=float32), array([-0.6,  0.5, -0.3,  0.4], dtype=float32)]\n"
     ]
    }
   ],
   "source": [
    "with printoptions(precision=1, suppress=True):\n",
    "    for layer in model5.layers:\n",
    "        print(layer.get_weights())"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "livereveal": {
   "start_slideshow_at": "selected",
   "theme": "white"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}