{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Uczenie maszynowe – zastosowania\n",
    "# 13. Konwolucyjne sieci neuronowe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Konwolucyjne sieci neuronowe wykorzystuje się do:\n",
    "\n",
    "* rozpoznawania obrazu\n",
    "* analizy wideo\n",
    "* innych zagadnień o podobnej strukturze"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Innymi słowy, CNN przydają się, gdy mamy bardzo dużo danych wejściowych, w których istotne jest ich sąsiedztwo."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Warstwy konwolucyjne"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Dla uproszczenia przyjmijmy, że mamy dane w postaci jendowymiarowej – np. chcemy stwierdzić, czy na danym nagraniu obecny jest głos człowieka."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Tak wygląda nasze nagranie:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<img style=\"margin: auto\" width=\"80%\" src=\"http://colah.github.io/posts/2014-07-Conv-Nets-Modular/img/Conv-9-xs.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "(ciąg próbek dźwiękowych – możemy traktować je jak jednowymiarowe „piksele”)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Najprostsza metoda – „zwykła” jednowarstwowa sieć neuronowa (każdy z każdym):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<img style=\"margin: auto\" width=\"80%\" src=\"http://colah.github.io/posts/2014-07-Conv-Nets-Modular/img/Conv-9-F.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Wady:\n",
    "\n",
    "* dużo danych wejściowych\n",
    "* nie wykrywa własności „lokalnych” wejścia"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Chcielibyśmy wykrywać pewne lokalne „wzory” w danych wejściowych.\n",
    "\n",
    "W tym celu tworzymy mniejszą sieć neuronową (mniej neuronów wejściowych) i _kopiujemy_ ją tak, żeby każda jej kopia działała na pewnym fragmencie wejścia (fragmenty mogą nachodzić na siebie):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img style=\"margin: auto\" width=\"80%\" src=\"http://colah.github.io/posts/2014-07-Conv-Nets-Modular/img/Conv-9-Conv2.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Każda z sieci A ma 2 neurony wejściowe (mało realistycznie). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img style=\"margin: auto\" width=\"80%\" src=\"http://colah.github.io/posts/2014-07-Conv-Nets-Modular/img/Conv-9-Conv3.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Każda z sieci A ma 3 neurony wejściowe (wciąż mało realistycznie, ale już trochę bardziej). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Warstwę sieci A nazywamy **warstwą konwolucyjną** (konwolucja = splot).\n",
    "\n",
    "Warstw konwolucyjnych może być więcej niż jedna:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<img style=\"margin: auto\" height=\"80%\" src=\"http://colah.github.io/posts/2014-07-Conv-Nets-Modular/img/Conv-9-Conv2Conv2.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "W dwóch wymiarach wygląda to tak:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<img style=\"margin: auto\" height=\"80%\" src=\"http://colah.github.io/posts/2014-07-Conv-Nets-Modular/img/Conv2-9x5-Conv2.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img style=\"margin: auto\" height=\"80%\" src=\"http://colah.github.io/posts/2014-07-Conv-Nets-Modular/img/Conv2-9x5-Conv2Conv2.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Zblizenie na pojedynczą jednostkę A:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<img style=\"margin: auto\" height=\"80%\" src=\"http://colah.github.io/posts/2014-07-Conv-Nets-Modular/img/Conv2-unit.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Tak definiujemy formalnie funckję splotu dla 2 wymiarów:\n",
    "\n",
    "$$\n",
    "\\left[\\begin{array}{ccc}\n",
    "a & b & c\\\\\n",
    "d & e & f\\\\\n",
    "g & h & i\\\\\n",
    "\\end{array}\\right]\n",
    "*\n",
    "\\left[\\begin{array}{ccc}\n",
    "1 & 2 & 3\\\\\n",
    "4 & 5 & 6\\\\\n",
    "7 & 8 & 9\\\\\n",
    "\\end{array}\\right] \n",
    "=\\\\\n",
    "(1 \\cdot a)+(2 \\cdot b)+(3 \\cdot c)+(4 \\cdot d)+(5 \\cdot e)\\\\+(6 \\cdot f)+(7 \\cdot g)+(8 \\cdot h)+(9 \\cdot i)\n",
    "$$\n",
    "\n",
    "Więcej: https://en.wikipedia.org/wiki/Kernel_(image_processing)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "A tak to mniej więcej działa:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<img style=\"margin: auto\" height=\"80%\" src=\"https://devblogs.nvidia.com/wp-content/uploads/2015/11/Convolution_schematic.gif\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Jednostka warstwy konwolucyjnej może się składać z jednej lub kilku warstw neuronów:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<img style=\"margin: auto\" height=\"80%\" src=\"http://colah.github.io/posts/2014-07-Conv-Nets-Modular/img/Conv-A.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img style=\"margin: auto\" height=\"80%\" src=\"http://colah.github.io/posts/2014-07-Conv-Nets-Modular/img/Conv-A-NIN.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Jeden neuron może odpowiadać np. za wykrywanie pionowych krawędzi, drugi poziomych, a jeszcze inny np. krzyżujących się linii."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Przykładowe filtry, których nauczyła się pierwsza warstwa konwolucyjna:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img style=\"margin: auto\" height=\"80%\" src=\"http://colah.github.io/posts/2014-07-Conv-Nets-Modular/img/KSH-filters.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### _Pooling_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Obrazy składają się na ogół z milionów pikseli. Oznacza to, że nawet po zastosowaniu kilku warstw konwolucyjnych mielibyśmy sporo parametrów do wytrenowania.\n",
    "\n",
    "Żeby zredukować liczbę parametrów, a dzięki temu uprościć obliczenia, stosuje się warstwy **_pooling_**.\n",
    "\n",
    "_Pooling_ to rodzaj próbkowania. Najpopularniejszą jego odmianą jest _max-pooling_, czyli wybieranie najwyższej wartości spośród kilku sąsiadujących pikseli."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img style=\"margin: auto\" height=\"80%\" src=\"https://upload.wikimedia.org/wikipedia/commons/e/e9/Max_pooling.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Warstwy _pooling_ i konwolucyjne można przeplatać ze sobą:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<img style=\"margin: auto\" height=\"80%\" src=\"http://colah.github.io/posts/2014-07-Conv-Nets-Modular/img/Conv2-9x5-Conv2Max2Conv2.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "_Pooling_ – idea: nie jest istotne, w którym *dokładnie* miejscu na obrazku dana cecha (krawędź, oko, itp.) się znajduje, wystarczy przybliżona lokalizacja."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Do sieci konwolucujnych możemy dokładać też warstwy ReLU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAUDBAgHCAkICAgGCAkFBwcICAYGBwcICAgHBwgHBwcH\nBwcHChALBwgOCQcIDRUNDhERExMTCAsWGBYSGBASExIBBQUFCAcIDwgIDxIPDw0SEhUSEhISFxUS\nEhISEhUSEhIVEhISFRISFRIVFRUSEhUVFRIVEhIVFRIVFRIVFRUVFf/AABEIAWgB4AMBIgACEQED\nEQH/xAAcAAEAAgMBAQEAAAAAAAAAAAAAAwYCBAUHCAH/xABYEAABAwEDBggICAgMBAcAAAAAAgME\nEgEFEwYUIzNTkhEXIjJDUnLTByQxNEJjc4MIFSFEYnSTw1RhZHGBgpSiFiU1QVGRoaOxs7TEhKTw\n81WFwdHU4eP/xAAbAQEAAwADAQAAAAAAAAAAAAAAAQIDBAUHBv/EADYRAQABAwEGBQMDAgUFAAAA\nAAACAwQSARMVFiIyUhEUQlGRITEzBQY0Q/AjQWJyoSRjcYGT/9oADAMBAAIRAxEAPwD4yAAAAAAA\nAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\nAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\nBlwDgL5xZTdtD33u5HFlN20Pfe7k7Hdd12ODvO079PlQ+AcBfOLKbtoe+93I4spu2h773cjdd12G\n87Xv0+VD4BwF84spu2h773cjiym7aHvvdyN13XYbztO/T5UPgHAXziym7aHvvdyOLKbtoe+93I3X\nddhvO079PlQ+AcBfOLKbtoe+93I4spu2h773cjdd12G87Tv0+VD4BwF84spu2h773cjiym7aHvvd\nyN13XYbztO/T5UPgHAXziym7aHvvdyOLKbtoe+93I3XddhvO079PlQ+AcBfOLKbtoe+93I4spu2h\n773cjdd12G87Xv0+VD4BwF84spu2h773cjiym7aHvvdyN13XYbztO/T5UPgHAXziym7aHvvdyOLK\nbtoe+93I3VddmpvO179PlQ+AcBfOLKbtoe+93I4spu2h773cjdd12G87Tv0+VD4BwF84spu2h773\ncjiym7aHvvdyN13XYbztO/T5UPgHAXziym7aHvvdyOLKbtoe+93I3VddmpvO179PlQ+AcBfOLKbt\noe+93I4spu2h773cjdd12G87Tv0+VD4BwF84spu2h773cjiym7aHvvdyN13XYbztO/T5UPgHAXzi\nym7aHvvdyOLKbtoe+93I3XddhvO079PlQ+AcBfOLKbtoe+93I4spu2h773cjdd12G87Tv0+VD4Bw\nF84spu2h773cjiym7aHvvdyN13XYbztO/T5UPgHAXziym7aHvvdyOLKbtoe+93I3XddhvO079PlQ\n+AcBfOLKbtoe+93I4spu2h773cjdd12G87Tv0+VD4BwF84spu2h773cjiym7aHvvdyN13XYbztO/\nT5UPgHAXziym7aHvvdyOLKbtoe+93I3XddhvO079PlQ+AcBfOLKbtoe+93I4spu2h773cjdd12G8\n7Tv0+VD4BwF84spu2h773cjiym7aHvvdyN13XYbztO/T5UPgHAXziym7aHvvdyOLKbtoe+93I3Xd\ndhvO079PlQ+AcBfOLKbtoe+93I4spu2h773cjdd12G87Tv0+VD4BwF84spu2h773cjiym7aHvvdy\nN13XYbztO/T5UPgHAXziym7aHvvdyOLKbtoe+93I3XddhvO079PlQ+AcBfOLKbtoe+93I4spu2h7\n73cjdd12G87Tv0+VD4BwF84spu2h773cjiym7aHvvdyN13XYbztO/T5esgA9EeXgNliA+uypDElx\nO0bYfIXEKTbSpNKguwAAUAZtoUq2lKVqVs2zAAAAeIAZuIUmxKlJXS50gGAAAAmiRlvuJaaQtxbn\nMbbGbOJbS6pteHIXQhzaBdCAAoAAuAAKACaJGW+qhpC3Fc/DbIQAALgACgAAAAAAAAAAAAAAAAAA\nAAAAAAAAAAAAAAAAAAAAAAAAAAALled6SYt03Vm78lnE+Ma8BeH89NrKSAu8fi/GVhznLrlypK3G\n+ezHziRHzj8owCF/KCVCum60xn8PE+Ma+Qxtjn5EzHH7xqedqcnsS2cd9fTSIciPHOt2c8c/97s8\n4ZYf7HIuK7lTFOosVTm8WXJ/Z2c4LPeV1XZ8WQVqmLbU4uXW+3C0rn9+Z5G3DLjqmLlNLjpbuu8E\nafR4juD83NKdDdk3PBUyhbiYj1442H0fm5apU1lPkl/f1VhDCPS7N9XPGRekVu75i2X3G4lGHFw6\nNDrzgRMn0KipvCVKwWnH3mdGjEccej/g5ZnIy/jq7ZVC8Bxi7qH+i8yKrfv8lQfrt7f7crRqT8YR\nWrU4c037LyZUpyJmTmdNXvyIzi0YbmL84YkGzByZgyn0Q2bzqkrXRiLi+LOfV3zrZGS2mo124q0N\npclXszW56GcM5uc/IzJKc1ekbGYWyiHKarff1f7QXnWnz86mxhy8qpy2cJ1xqqrN1vILZkvAS7Zd\nmdyl5q/NeQzEwMTTY0f/ADis3751J+tPf5xZrteSiLcalKQlLd6OrX9tHNK/jgwoYZuTlnGisS3U\nxn1uJx3q28DDw9NqDTumNGXWuVJWylvo22MR1w2cs4D8edJxWlt5xKeWjE6RnGOzk3BUmAmVEiom\nyXJTrL2IjOc0a+b+L+uL7Twomz/xU+S1zpi3pdrrLucRp+Mtl+jDd1MnOGJEc5M7+RYP128f9uXZ\ntbuPcOcZsl1td44zbCGG2m/UFJnfyLB+u3j/ALc4tOc5z0/vvcqpDCH9/wCls+DqBDkOSs5XSpu7\npa0N4OJ0OvNaJcLCrHZKplMFhdCJbjGlfe2EeOfng6RXKdbTrJF3XiyhvaPYJ0/ip+RdaIqGl53d\nE11b0CjS4MhnX5uXnPWE+pjCGdJyryuFGBnUJ9chptbLLzbjGHJbekeb+Lkz+T0NhSY0q8MOX6bb\nbGJGYe2EiQdPJ9C7pjOuzULZzuVdyGWHNa5m8zOJD+bmllBkxMdnuraaW4xLfdeZl/NlsyHtfnA2\n+vRkvhyZYoZWSS2pkqO6+y21dLbK5M+jRkfxDDW1nTMx5UaO+0iTiRfGWM4838X+cl6vJ5Wd3zDj\npjPSXF3c8y3IRiNPsx2fGM3/AGgqV+zLzTEcbehxorEtbKF4cViM6szp15zXnQhFZkXVERlBhRHa\nnHES0ZpgYbTf8WfhBUG8m4ztjrUWdjSYjDy8PAw2n8384zeQWl2GtN/Lk2oWliXFlrZf6Jz+LSp+\nDbz5P1WX/o5Ip5w8Z5f5E8OhHEuSMhhD8+UuOmX5swwxiOuM7f6ubUTIxbspTCH2VNuQXZrMvonI\nh3JE+U/DguwocOa2xBZiveJMSXWHo5DHmTFOyUS0Mx1RLgvFCGG0MN4bJO3rGwoqffLMZpSUxX1y\nE+mtbGGaAB2ujq6gAAkAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABQK/RABcTPzHV891\n5WHtFmDby02KSla0pc57ba9YYAbPRG0Z5yulKK10t8xuswrVwUgBKaJSpTaXVLS1Xy8PSYZbLtmQ\nYDiJPxjJnqicuNBwH228b8ozkpoM6lvmvCvgzfWpalLVznF1mFYBoozfeWumtxasPmYizNiStrmL\nW3ibNeGQgHizxldZZhX8lIAPFm2tSbakqpUZ5y7ViVvYm0r0pCCBm+8t22pa1qVtHF4hZoiLuRYl\nS71mKY564DcV9txz1GvwCrApUpZrQqYOnft6qmS3ZWrU+utCEdGaT8lbvPWtz2i8QhBNOnCCm0TZ\nyvgSmtdLfMbr1ZC2tSeaAXSmYkra1Ti2/ZrwzCtXCpVS6nOeYAAAAAACAABIAAAAAAAAAAAAAAAA\nAAAAAAAAAAAAAAAAAAAAAAACx5EXOxKU4uWpbbDeCyhaPwuR5ucKXGWw440tNLkddC2y0vyYcOBF\niyGHnnX/AOMHsCVh4eceb9BsP881svaJDjV4MppRe7Fa268TDlx/F5H/AF684NOpPNyZwhg5N23J\nJlJU4yipLfIxHFsN/wCpEC5JUirCa5LC6FuOLYbaxv8AiTs5NIVKZRDeiSXozkrkS2OgekecHTfZ\nQm72GkRfjHMJV4svLYW/rsbxd/xb8nFSvOM16dCEoOHdWScmRnSVN4bl2sV4bi2G9N4ucJ9CkWqS\nrnN8guyJLsp2chUZbLvxJgIiax1zN3o5T4FLTyMVPJbfZrbc9sXoVJ+tScIeh0G8mZykJVYwvSIr\nQitjF/Z9edBu51sXTnWbRnFS33a33FsaNnBj+b6f15NeV2zHb4U60l6lyVjMzuiwdvnBhle807Fr\nZoocvu9lo9j4uU1qTlrBfZwjCSuQYDslakMoW4ptFeG2bV5XDKjN4rrWjroxG1sOYf7MdPwdIWt2\nYllVK3LrloQLthuxYd4KktrZQ+wyyht9GHiS8aOXqV55qU6EMHMvm7XEKdWmM8y02/RhrXiYbwYu\nGUtyhLC6sDG0mjw2S7WraavGc/L5UHEZQ8xt3vFs3+x1xqtoWlq82nWM/kuSojy8N9/TxPGNPHzb\n5uU83Nv5WDh3LcLjcxtiW1yXGHlo5eI05oZJz2LhlLZx0taOitDji2G8QtNyyV48Ri2DmTcdi8Vs\ntuLfxfGGfyk592xlT2GmpUZ5vAivZtezaNFgx/GPGCdvNnsYKzERU62lXNrZLHlBktJzqTm8bRNv\nu0IrYxMH6vryuQNa37RkvN5XbMVf6n1Y2G3Nxs+6Jtn6wWr1NYT+ilCEJQVC7LkkykqcZbqS2ujE\ncWw21jf8SG7nlKeVGSwvFj89vZlvvJ5h+Iwtm71y2m35dbbD7+gdkPZz5vG9RgEL7z8xm8Gkxlx5\nLaLuRmja8R1yJH84+4KeYmv5WCsuXDKxksYVTj6K0IbWw4YXlckqOlLjrWjcXRW2thxr/ljrZLw3\n0JmMYTzcmXC8WQ4jDcXpo+cMRzO6YzsaBMzhDzKZeaIZbfRh4j2N/wDHNJ11IUIOevJW8EWK4Yy9\nHz262MX9nIbyuGVFTW+1SiujErYc0xbG5a/4RSV18r+MUV+5kHJyTebTDlqe5SG5t0rXie2kGelx\nNpsIOY5k3OQ2p1THJbRWvlsYuD9X15g3ckpTOc4WiorxFrYbrZ+rlvvWSuNPfkoupbiq3l53jvuN\nOMyOn+rnMYjKnxkolRnmVQILq415UaJxmP4xgSBTrzJ2sFQOs/k3MaaU6tjkNorXy2MVtn6ucyIi\ntxtClU4i6MRzoz0O7YCY705pMFbaY8G8UZ/LW/p9DI836A3r18GdGhk88iMrdWlCE1OOLoQ22bs+\n5JLCm0Lb5UhdCG21sOf6YjyekutSmHY7eI6wutDdGJiF3uKHGakxpmAuIqfnbKIMtfMewfF348gV\n6+ClGhkqUvJuYxZUtrk10V1sOaYzyryedu5xCV81xDS8StjY5wdCc8tiJLaTdT0RMjBQ8++t/kab\n8pNXLlleM07SvDfhXfQ50Tnicczp1JzmvOEIQcy7bqflWqwEVYfPcrw2mztZN5KvrnsRpDWjc5a+\nWxpGfrAiMrfulTUdK3HGJ2NJYY1uDg+LvnZyTZVGTd7UhK23XLxlvMtua1uJmXfla1efNovRhDKK\noX6hecqSthllWwY1RH8Wv4+a4S8eujA9cabnlLznKc2TfFSMXNcy/wDMfN8f7A2nPBjrDNWYGT0p\n9OIhrR10IccWw3iftJuXLkrJkWS+ThuXYjVuLYb0xtZSw3ZSYa4rTzzGYssowEYmG784YOs+hS87\nYTpH27kiMvNt6R1yXHzbODCdebaFCCpwcnpj9ilIa5La6MRxbDemES5JTqnEIaXVEXQ9WvDw/wBp\nLNLpXAu/gu5ctLDDzK8Nb+jl40jOM4jxjD4yfkOyUPXY841IWzjMRMfFYejs5vj5wNvNOwgpj7Km\nlKQvnNroWYHWygudUV6Slqt5iA9g53RozknOp1M3BqcgAAAACQAAAAAAAAAAAAAAAAAAAAUBxalc\n4zrVwU1ckwBcTMTHUWKSh15KXOe22vWCJJdatqadebxNmvDIQMEbRm28tKq0rWlzaV6UOLUq2pXK\nUYAJTNzHUt4WK9h7OvRENYARm37inpjY9SV+NxXmUYf5Qaz8l12nFWtzD2i8QhA2a+bNx5Sucpag\n3JWhSVJWtKm+Y42swA8FfFm5JWtdanFqVtHFmeeO4eFivYezr0RCBgjaBNnjtNGK9h7OvREICU0S\nS61qlvN+zXhmDby0qrSpaVbSswA2YmfkurVWta1ObRxYlzHXbanXXnFesXiEIGzRtGdauGqpdRhW\nAPBKZyY6pvCxXsPZ16IZ47h4WK9h7OvREIGzRtJhM5MdVYlKnXlJb5mIshASzbWpNtSVUqb6QPyV\nu21LWtxW0cWYACZ+Y67YlK3XnEt8zEXiGDjzikpQpa1Jb5jderMABmw8tq2pDi21bRtYckrUqtS1\nqVtHFmAACtXBT6IAEzEx1qxSWnXm8TZrwzBh5aLakKWlW0bWYAbMTMTHUWqodeTic/DXrAxMdaqo\ndebxOfhrIQMEbRnjK4FJqXS50ZgAEgAAAAAAAAAAAAAAAAAAAAAAAAAKAAC4AAIAAEgAAAAAAAAA\nCAABIAAAACAABIAAAAAAAAAAAAAAAAAAAACAABIAAAAAAAAAAAAAAAAAbMGA+/VhNPOYfLXho1YG\nsDZfgPoaS6tp5LT/ADHFo0azWAAAAACgAAuAAAAAAARuLpsqAkBBjK2axjK2ayma2wknBBjK2SzN\nt6q2mlaRmbCaQEDbylWVJR++K1dT98uYTTggrV1P3xjKTzmwYTTggxleikVq6ixmYTTggrV1Fhby\nk2VKQsZmE04I3Fq4aUpqFa+p++MzCaQEda+p++K19T98GE0gIcbk1U+nQZVq6n74NhNICOtXU/fF\naup++DCaQEGN8tKk0itarKqEb4zNhNOCGtfURvitfURvgwTAgxlJ5yf3xWrhUlKdX9MoYTTgwrX1\nEb4rX1Eb4MGYMK19RG+R43yJpTrAbOacGFa+ojfPzSdRG+MzYJAQ1r6iN8yQuqqpPKbL5mCQEDa1\nqsqoRvkla+ojfBgzBhWvqI3yPGUm1NSEb4zME4I3FqqpSn98xrX1Eb4MEwIa19RG+K19RG+DBMCN\ntdVhIFQtmRt5MRdeuMppt9mbz38XGj/6klieDG/n0Idau91SH0VoXiR/5/IT8U+UH/h7v2jBwal/\nbS5NZuwp2FzGWeDlXs80mHQy+yrO1tPSW9Pi4uw9znBXC68U+UH/AIc79owF+CfKCyyq27nf0OME\nU7+2h6ypYXMvQpQDnJtpUDnuvADSl3pHYVQ6+02uiuhazCpUhDnmvTpzqckG6Dl/whg8rxyNo/WD\n+EMHgT41G0n0zPeVv3tvI3HZUdQHM+PoKrUpzyNpPWD+EN38FWeRqW/pk+ftu88jcdlR0wcxeUEG\nyxPjkbl/TM2L4hurS0iTGUtz0ELJ89bd6fIXPY6BG/5E9tkkI3/IntsnJceHWkOgxdTq2FSakYfv\nznlsu1Ck3Y7yV0uY3oP4Tn9x0JSc8E0YZqmQdJ+oTkHSe4Eyh1kTVo7BOQRNWjsG6xGWvmIWr2aD\nSHQip1oSCX5E9snIJfkT2yKhDrSMeRXbNy6YapTqWEc5w02PIrtnZyTZUuU3SmpTfLw6CnoaT62l\nPjKYdU0rnN/QwznytWr2Z3crPO3OwzrPYnClatXsy/oU9Z0nuCcg6T3BO3Uq2lIgvXATPxltc9C2\n/aIwyEuxaXoe/wDvjdNL0Pf/AHxumMHJugHWyaupMy1yp1beHg6tGIc+8oyWnVISvES30jfSF8/Q\nxw5M2kvWJ9m8Zsc1PYMF6xPs3jNjmp7BT1Lz6HUuK7c8cU3Xh4aK+ZiGk+ilSk7MsGQiFKddUlK6\nW2OXh61vTHFvLXO1JQnlvasQnzs5w5M2k/6PbDHOcEj0e2yYMc5wv62lPoTgmiMrdVQhNStm2JcZ\nbSqHULbVs3CjFCarHQdg2jVY6DsCban0NoAsFxXIxJZxXXVt9fDwMJsZ4M4QzV8gRznCf0iBHOcL\nzaQZxdWn2ZIQRdWn2ZtOIUmmpK04hRnU6258VSVNY6WHsLn10aI5Mr0e2XO4mf4sfWpK+mwXMD1P\njBTJfkT2ymebfDCUUlmsV2DMgs1iuwbTaFKsUpKVqS3zzZjNnAhuyFUMoW4rZtiXGWwuh1C21bNw\n6GSaFLltUpWrr4aOhM8sEJTLUlPNbQzRyMMpnzYL4cmbhMeRXbeJPRI2PIrtvEnol9Px6qS/Jo+o\n/CTc15TclmFXRKkxZ0GDFfatjuYeNwM2Y8e389nD/UeNfBW8LF4LvpV3XtMkykXsihnO12W4EqPb\nZw2fL/TZYfU+Rn8nQfqMT/IsPi74U2R38Gr/AGrxhW2IavZ+y8GkWdBMju2WyP8AGw8rqa/XV6vT\n05NHrHwwvClIuqyNdd2S3o8txdkmS/H1jbXy2sM/i4bbf7LC1fBmuy97LnXeN8TJcl+9kVx2Jbnm\n8Tg+Sz8/y8P6Pxnzl4JLkdy+yqck3kpGH/KExuvWMx82jR2D7rfssSyqxKabEI5KP8Ckfuirp9Hw\njL1i+28Qk0vWL7bxCep0uh5TV/IFJysRXeaW0oXpITSFr9VjWF2KNlvw/GCVpXhqbitYPIY0juMd\nL+4/47vv23/IeseE/wCDzGuO5Jk9N6yJdt2Iak5r8XMN12z7eDX4/wCf+o8JbWrEjUsLxdDRy9Fg\n4Pi/QHtXgdy/nXxejUDKW9c6uyXnaJcG8sxbjLaj+b5wdPw8ZBw5q7sVkrd7LraM7+MV5PsZxheT\nN8fgxs20H+J8L4vvXz/AWnhVS09h0RMbEWxto3qDVcpzZtKmHqaHsHlsYuujeoOu5ckyNLzWQ2tm\nW3miMwcYYbdcexo2gPVvAJ4OLYt5Icyqu1xm7nIMtDtl8xbI0ZEvGj5vZ+fXjxE3gW8DTWU8V28X\np64DsC8LbvwERWJDTmcQo2n17X4QUN+5bLpygmXYit5u6LxZirlOaOvN/F4756Z4Z8sFZPy4MbJa\nfGuuDLu7OpzF1YDjfxj4xb84xfwdmw8jumZJlXsqVLfxJM99l55xxDGnewfGDkW/5oOPd6f4M19I\n3/IntskhG/5E9tk9P0+zyyPWkLZdrNN2OqUrETQ9o9n/AHBUyzXaum7HalU65DOoxfyjzkyrLWvU\nrJB0nuCcg6T3BeatDrImrR2Cx5J1cKqf9dmxXImrR2C25CPRmLXHXXUJVzMNzVOFJ9C0PyuFe3Kf\ndVyFad7VnPl+RPbN2eut1xe0W8s0pfkT2y/oUp9aRjyK7Z1sl2a5LfLw1f5hyWPIrtnWyXqztqnr\nifQ0/qs8sPPHeShOp1fsThStWr2Z2sqFpXKdUhVSdDRqNj+THFlatXsxDoP6p0nuDaic5Pb6+Gav\nSe4N2AhCnW0uqpbr5bnqRAr9bu5UIUlhpKqE6evDbffk+/ziSVws2W95NPpYQ06y4ltHR/8AYKyK\nPSVutpeh7/743TS9D3/3xuiC1dZsjGVYb9KkaRHRuYbqP78rLnlLTkKilL7ik6NvC0mJhttvf9dC\nVl/nK9LlmdPrkrU6YNVesT7N4zY5qewYP6xPYdM2OansGnqXn0LPkQhOKpeLhqbR0mjOLe3nDtSa\ndO9ozr5CcmQ4r0W2OWcWXrHFfTKQ61J/hasj0e2yGOc52xI9HtshjnOdsv6kQ6HQuVFT7aaWVYm3\nx8L/AJY3sr0KS62hVGjYow20P4TZq5PLYRJbckKWltvl4je2NnK+emVJrQvEbw+Q5RhlPWn+k43o\nmqx0BtGqx0Amin0NotkRn+LHOVUlvl6P/vlTLZA0V2KU7yc4xsFxz/b+2FYtepUyBHOcJyBHOcNp\npgzu2mlurml0yo5EJtLSdA5yEYmk+4KZdtPA1XzdDWdy/Z7C222465NDfRvsMN/6YxqQ6F8+pu3E\nyld3yVqUypUfGobcX+EMlSl+RPbLbcyKrsf9LDW9zHMPD0PtypS/Intin6jsSWaxXYLfkQhNLikV\n4/0F/wD4FQs1iuwdy5ZkZhtSlKktv+g42ww5/qRPpZx62FyrxZ7alUN4i6F9G0MrGUtSlNJopb5G\njXiGGS/njXK6QZUIplLSpK04fRuLxCNOtb+k5DHkV23iT0SNjyK7bxIb6/j1Zy/Jo+2rmvJqFc8e\nVIWlpmHdzLzzi/QasZst4T4nvmXN8IuVaG0JXgPv0Mt/gl3R/ltkHp/hdyxj39c0S6Yl4Zm0hlnP\nseK+5jWR7E2Zv4t+OzhIPg8TrgySzh6TLelSpfITIYhP4bcSz5bDzWX6dcffB6ZS/Urfw63mOUEe\nZ4O8q1Kj1qbiLrZxPnd3SPm59y5P39HvW7250VaHGZ0atH4/kt/9T5y+ERe+T+VzEfN5TsaXAc0b\n78J/Dcat8rFv6TDwBZXRsmIsiFLvLPY71tcZtiFIscYd/n4c58v/ANDT9OuPvgVf1G38Ot5pL1i+\n28Qmb66lKVtFmB6TS/G8zq/kChZdoRn3KQtX8XayjmabXl9KFlvVn/JoT4l6/T+oOh/cH8d337b/\nAJCuYLGEvkr18vT4Gs9Qe0/B78LUHJiJJivxZ6nL3fu5bOaIY+bsnjjeKpLiUJQrEXL8UbQ/o/Xn\n0B8GXwb3dfESY/e8aS8/dj93ZthvvxnG2ZDP3J8M+9d+8vBnblLeH8NWZUZmNeuBejMCQx4y2zdG\nb2P24/482/vys+H7w73dlNccmCxHvJvPpUSUy5KRZh4UfNs4Z4bH/wCn/E6eVXhCvC5MoLclYFt3\ntXVdj0O747bzOLJzS8c2zjgk4/y+cHQ+Ej4I7kuW4JT91QXkvxJsRlqxyVOktrakcGcaDh/OB8xy\n0NY7WiXVpdBQxsY50MjENJnt0pWrlxNPR6k1pyH0PNJWpCVaXxtaH9jH9f7k3Mj1qz5vkstpriaB\nvH0mhN7f80HHuPwzfRPgs8Gci/eF1dbMPDdonNrY878nm5y/CP4N7xuNOK9YhyNY4zwymF/LZpuH\n5bLfIYeD/L2bciqmVrcaodogrf8AFsaR8uPgHKyzypn3s5izZDzmkaobsXom/wDhz7qMLzbf9t59\nGdpso97klmu1dEJyMquqQw9Kw9oztysm58ayUs4GKvC2Z2k4ZOthPFpkHSe4JyDpP1BNFDrImrR2\nCcji6tPsyQvAn1hBL8ie2TkEvyJ7YmQ60jHkV2zrZNVIex+XTE5a8PozkseRXbNmJJWwutpa21bR\nsp6Wk+tu5ULqlKXtEMr0nsTiytWr2ZtPvLdUpa1rcU50jhqytWr2Zf0M4dZ0nuCcg6T3BOINK4AC\n7Fpeh7/743TSVq/f/fG6YwbXTpwL7dYaU0lLKkOY2sRiHMAL4MM0C9Yn2bxmxzU9gwf1iew6SROY\nnsFPW5U+hY8j1qYUp9KXtJodBpHWzkXsil91NNOneIWJ62OY6tvE2a8Mw5w9bGc+RHI9HtshjnOd\nsP8Ao9sMc5wepaHQkAFYZeAarHQG0arHRez+5E21HobR0/jh9TObKUjDoo5jBzAMGGYQI5zhOQI5\nzhebamkiatHYMyCLq0+zJxBSfW7l0z0oiutqdZTrtG4h/F8YZ+blcl+RPbJyCX5E9sps16dTnSWa\nxXYMzCzWK7BmXVm3LiewpDa1KQnD6RwkyheS69UlxDmjZRo9V/zJzwNnzZm05cFj8G2Rkq+3sJpL\nyWq3kLl0YjTbxs5ZeDu9botUqRHWppv52xpG/wCoy8EuXsi4XfkUtUVb7y3Yrfp2/wBNhu5e+E68\nr7qRavNoznzVhfy8H5RadV/1nmeTodpPymx5+tRAAdo6j6gAH0PqAAkCh5aIrvFLfOU5FaQjlv8A\nIdxrNOXwo2WDykXklxCasCE0tfL6LGsOh/cf8d9D+2/5D1/wVeDS8MjrwZv6/W4zcK6cZcuU3Kzm\n1DUhnxfxf9NhyfhNeEW7L7cuj4tlPPZpnectobnQsTOM2zfoPGNAQeEL4QMy+7omQXbqhMIvOy2K\nuQxNfxPEeBTH82CeON14sXgTpdDQ5XosLB8X6A+F8H3njoMLqdqU6txdESh/Hf0emPSvgy5cQbmv\nhude0p5MdF3Xi28+tb8ltx7HjZueZQF8pVKV00RMbEWxto3qCBxas1bqQunAeweWxto3qCfA8dH0\n14V8ln/CDJg3lcKmZcS6IXxZMW+v4u/jHzjzeT9YYPDIl0u3dfS4MpXjN2ymYz1D+I0h2P5wXfwU\n+GWXk1HVdyLuhzFTpvxgpx+U+3hux4cXA82Y/J/7CnN36u9r/lXiprB+N5rMp5vE1b0jxjxc3t9N\ndtBhca6bGa7tkb/kT22SQjf8ie2yemT+zyyj1uRlFei2tEylddFbq9i1/QRs3Q06hLudS1VorxsY\nlu+1Ody6+dZh/ZcJwZiLbVrUyl7MsTTUfcHFnU9c3dUqenRDkdW4b1XiYTiluIfXQxJWjnnc6T9Q\n5d9WIU1HwqKVyW8Kg67ms/UL0+xxa2kOtrvS7I8fFt9BBoM3c9IsrkvPJr6FleHQQ5UcOBFoo85b\n55Ohq8bfK5FT+rYNetppT0hDOHh9fdoSZC4L1LDrr6W+W6yvSUNfjO/jJdbQtHNXgrIoF3ojNrq+\nW1fLdeX6dhp5NeaNe25H2wp5wRVwnDPT/JvuyrI7K3bfIhw4MC1Ux1aJi5CF2ctEVCsOzC/Sb2Uu\noTw83PW6/tbTDKuheEhrlSOiw/QKVF7eHhp4/wCdT/hqXlVDeSiKt5dvPXFWrEswv0HdTIsej4qP\nItBzclOBFq0PcmVw1u4npjJz5Yjn9GI7R7IUy4h4/wDmn/y7fSe4OW/IelPLZYXgoZ1r/wD7nR6T\n3BXbgTMoXgripsxna60WYhpUqM6VPTrbU660R0KdzqW2pHp14hNk3eq3tE8laXW0V+0aP1m63nXU\nuzFtOYHMQwjRmUngz9ng8ubOVf1WmcOSWa0pQnHCXO2fQ9/98ad5ZRMRX8F1K/kRzzcs5nv/AL4o\nvhBsqmK7LX+FhNxXnSjyJtreFarhNfYN5MP2aJ1CjaPI7vuSU9ym2l9svVwXXNa1krk9TWChdzn6\nC7/TaNLTxhN216xPs3jQvicpllCWuU6/yGjfXrE+zeOReHJkQlq8mkR73/q2w2qMbfTxx/8AbVua\nEzKqznGckN61tayG2Rmb9uaYzjDHnDdeI2glyhQqQ/RE17HPfb/yCWHIZRAcRYnDUwixLyHNZa7w\nnFc7w8efX/5u64tKrG1J5ri2Q2um1xSjXu1NtkaPZb5UJaPy8UqtbkWWc6j7k5W0dVs/rg0ozK59\nTji1Nx+iZQvDr/FYat8QGYtOAp5t5eqQheJWZXHGlLjtKblNJTRq8HmHTuu7MJVrri8Z1fp9QwhD\nNzqlTSlPq+nsXLOteQpLllLrHIdQbDfRdg0bv+WZLVZzdGj3pvN9F2DSDCrTh4uG3lewlxbbqFpo\nXRWdmDekZ6zRuoUeW3i2q2Q7YlK1aZ037syYmP8AKow09dw4ULqtm7Ov+lW2GeeD1AgRznDnXLdL\nkezSSXV+rs1Z0Uc5w7TaZw53Sa04Q15NfFh839x0ZS7rv5cfgTY7ioc5zb2sQXRvUc6nkaw8+vR1\nS0p4XmneQ78iEYZwbuphg7L9PpwnnCa3w8qIrieG1eGrqLOq/wAqxKk9c84ycdjNWrzlpbircKhd\nnoHo7lNLdPNrZNLevOcedneWkKMuRJZrFdgqmVUvAlWKQ8ttbbPIR0ZZ7NYrsFdyrkKQ5TnDTacH\nmLRifoL3X4WH6f8AmZ3TlYhVliX6U2r6RB3YU1qRYpTa6koPKfkscSq1NSfTRWek5MvR3G15s042\nmvl8KDC0rznPBy/1KxhCGcHSY8iu28SEbHkV23iT0TsJfZ00vyaPpe5PAzcLsFiS8iWlTkVl55ee\nyG+h/OeRZb5R5BXY6piPHvm8FoXQtcSTa3G/aZNp1PhT5duw7ouq5I61tqvO74kqXh/gfBZoP67C\nP4NPgGgXlAbve+G1vpnWcMSDXahvB2z9lnlttPNal7cePW9MpWNt4dDh5L5Z5CTHrGpMK/IKXOnk\nSbXGv641p7rH8C+TjrGOymQ424itK0TX8Nf9FvyWnm3wjfAHdkW6370udpcd67bMR2K2u1bTjX8/\nJt+Wz+b+s5vwKMuHlWzLifWtTWC7Kh4nR/hDA0vbjx69SX6dbeHQ86fRStSU9GswJpesX23iE9Hp\ndDzOr+QI3GUKtqUhClesQSAmpThPrRTqTh0I8FvqM7jB+4KOoz9gwZgy8pR7Gvma3ejwW+ozuMDB\nb6jO4wSAeUo9h5mt3sMFHUZ+wYPzBRw1YbKVbSgkBp5ej2HmKneEcvyJ7bJIRv8AkT22TSbKh1OR\nlFdKpHLZVSuihSNs1sTCPeFrbaWsxfTRyKEcOGd4GGw9blU73kwmrtwXKtLmO9ZhpQutmLXiNoO8\n5rP1CQg6T9QYYJ1rzrTzmheiJkR8JfpoNKPeEiNZQ+y65RzH2UYlf6TpxdWnsE42aulx6JuE/IkT\nLMJtt1hpeuee1n6bTp4KWm0IQmlLa2jaIJfkT2xszW4z5IMFxkvtLaXzV4xw4DDsBa62XJOJzJSN\nI5hfiLGx5FdszGtDNfS8nDk1VyfGdnrRSyqMlv50vhr9jwnZwUtMUITSlCDaIJerV7MaUMOdSVzt\nMYHSe4OfKiPR3VSIycSx7WxvvjodJ+oTl9nmiVxs5uPbfL9vJRClV/T4W2ya54TiLVvPKqef/ct2\nJ0gNn3qzueXCEPBqr1fv/vj9XBZUvFtaaUvrrQfi9X7/AO+NomCa1TWH2AAXcZA/rE9h0hnQUyo6\nUL5PI5C+oTP6xPYdJGNWjsFHJ2mEc4ODdjcmEnCzZDvr0L54TdDkt+yQ+hDSUdCj0/blhBnsGvn5\n9Xggkej22QxznO2Zv+j2zBjnOGjjw15XLtiPxFKVGsS405y81Xo6PYGa5c16yhEfA9eteJR+k7AM\n9g0853waV3QUx0cCVVK5619czR0HYNpw1WOg7BoiNTPnSMxGkW22oaaTavnLQgmALeDja1NdfuEC\nOc4TkCOc4TNemwb5TPJTVyCvQMm1rtscfpZo5jLKCyRdWn2ZPYYbCE+tvpdzo56QasWCw1ZwIaQl\nPYM5Xo9snIJXo9s02eDOFTWc/qWaxXYORlBdr0l6yhEdKHEULlLRpEHXs1iuwTipTzKdedGecHMu\n24mGLE8nEUj01nTbQlPNSAXp04QUqXE59aNjyK7bxJ6JGx5Fdt4kIl9tTX8mjc+FgytN53WtVeG5\nkzd1H/Mn1h8HaQ27kvdGHai3Du9ltVG0Z4LLbP7P7SieHLwYLylyeu9+HYhV4XRCZWyhzp2rWbLH\n4/6eE8A8FHhfvbIpbsB2Mt5ixfLu2XonWHvyc8tq/fV6zS+z7Q8L0lti4r0cdtRQ3dsvn+xtPjj4\nG7C1ZSpcs5rF33gpf57WrLLP7bT98LfhxvbK9tN2R4mbMPrZ8UieMyX3f6PzHv8A8F/wUu5N3fJl\nT0oTOvdHLb2EWz5bI5np9yWv0fPUvWL7ZCTS9YvtvEJ6nS6Hk1X8gACVAAFwAAAABARvoUqzk84k\nAITwQVu9VG+K3eqjfJwUwX2n+hBW71Ub4bQqqpVHMo0ZOBgvt2q3ipsSmlGj+mZ1u9VG+TguZoK3\neqjfDlaqako5/XJwFM0DdaaqUo5/XFa+qj7QnBOBmgrX1UfaGDlarFJpRpPpm0CDNAtCqqk0cygc\nv6BOCcDNBpfoDS/QJwQZoMFVNPIqrrFbvUR9oTgYL7dDW71Eb4rd6iN8mAUzQULUqpVCeQG0OJsS\nnkaMnAwNuj5f0By/oEgBmgocVTVQKFJtUpKkaQnBQzR6TrI+zGk6yPsyQFzNHpOsj7MwwVJsRT0e\n0JwMDNBpfUjS+pJwDNBpfUhtCuVV0hOAZoG0LTZSlSNH9A/dL1mdwmAwM0Ol6zO4flC1U1KR7tBO\nBgZoHEKqqSpHMM9J1kfZkgBmj0nWR9mNJ1kfZkgBmjbRTYSABVb4nhPv1hCGkXi6lDCKEow2DQyh\nywvC8/PXWZHt4sFz7gr5YMkIcZ/FaWllT7mCiG3IcfbxP2Y4M7C2hz4Owhf3MuTNHk9lbOuzzJ1m\nN6xiLB7g7K/CtlDbZTbebv2bHcHDlw2E3e06lC0u508w85XrNCcYQsLaf1wU8/cw9Y5yralAA5zh\nAAKAAC4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgAASAAAAAAAAAAAAAAAAAAAAAAAAAAAd\nCJfElqylDq08ijRnPAGecrw8KrR114frjAAAAAAAKAAC4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\nAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACgAAuAAAAAAAAAAAAAAAAAAAAAAAAAA\nAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA8m4zZuyhbj3fDjNm7KFuPd8dDx\nBa+z6Dhq799HrIPJeM6bsYe493w4zpuxh7j3fF+ILQ4Zu/fR60DyXjOm7GHuPd8OM6bsYe493w4g\ntDhm799HrQPJeM6bsYe493w4zpuxh7j3fDiC0OGbv30etA8l4zpuxh7j3fDjOm7GHuPd8OILQ4Zu\n/fR60DyXjOm7GHuPd8OM6bsYe493w4gtDhm799HrQPJeM6bsYe493w4zpuxh7j3fDiC0OGbv30et\nA8l4zpuxh7j3fDjOm7GHuPd8OILQ4Zu/fR60DyXjOm7GHuPd8OM6bsYe493w4gtDhm799HrQPJeM\n6bsYe493w4zpuxh7j3fDiC0OGbv30etA8l4zpuxh7j3fDjOm7GHuPd8OILQ4Zu/fR60DyXjOm7GH\nuPd8OM6bsYe493w4gtDhm799HrQPJeM6bsYe493w4zpuxh7j3fDiC0OGbv30etA8l4zpuxh7j3fD\njOm7GHuPd8OILQ4Zu/fR60DyXjOm7GHuPd8OM6bsYe493w4gtDhm799HrQPJeM6bsYe493w4zpux\nh7j3fDiC0OGbv30etA8l4zpuxh7j3fDjOm7GHuPd8OILQ4Zu/fR60DyXjOm7GHuPd8OM6bsYe493\nw4gtDhm799HrQPJeM6bsYe493w4zpuxh7j3fDiC0OGbv30etA8l4zpuxh7j3fDjOm7GHuPd8OILQ\n4Zu/fR60DyXjOm7GHuPd8OM6bsYe493w4gtDhm799HrQPJeM6bsYe493w4zpuxh7j3fDiC0OGbv3\n0etA8l4zpuxh7j3fDjOm7GHuPd8OILQ4Zu/fR60DyXjOm7GHuPd8OM6bsYe493w4gtDhm799HrQP\nJeM6bsYe493w4zpuxh7j3fDiC0OGbv30etA8l4zpuxh7j3fDjOm7GHuPd8OILQ4Zu/fR60DyXjOm\n7GHuPd8OM6bsYe493w4gtDhm799HrQPJeM6bsYe493w4zpuxh7j3fDiC0OGbv30etA8l4zpuxh7j\n3fDjOm7GHuPd8OILQ4Zu/fR60DyXjOm7GHuPd8OM6bsYe493w4gtDhm799FCAB8E++AAAAAAAAAA\nAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\nAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\nAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAH\n/9k=\n",
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"https://www.youtube.com/embed/FmpDIaiMIeA\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.YouTubeVideo at 0x7f70ba22e910>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import IPython\n",
    "IPython.display.YouTubeVideo('FmpDIaiMIeA', width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Możliwości konwolucyjnych sieci neuronowych"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<img style=\"margin: auto\" height=\"80%\" src=\"http://colah.github.io/posts/2014-07-Conv-Nets-Modular/img/KSH-results.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Przykład: MNIST"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import math\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import random\n",
    "\n",
    "from IPython.display import YouTubeVideo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "# źródło: https://github.com/keras-team/keras/examples/minst_mlp.py\n",
    "\n",
    "import keras\n",
    "from keras.datasets import mnist\n",
    "\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Flatten\n",
    "from keras.layers import Conv2D, MaxPooling2D\n",
    "\n",
    "# załaduj dane i podziel je na zbiory uczący i testowy\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "def draw_examples(examples, captions=None):\n",
    "    plt.figure(figsize=(16, 4))\n",
    "    m = len(examples)\n",
    "    for i, example in enumerate(examples):\n",
    "        plt.subplot(100 + m * 10 + i + 1)\n",
    "        plt.imshow(example, cmap=plt.get_cmap('gray'))\n",
    "    plt.show()\n",
    "    if captions is not None:\n",
    "        print(6 * ' ' + (10 * ' ').join(str(captions[i]) for i in range(m)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6IAAACPCAYAAADgImbyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAHEtJREFUeJzt3XmQVNXZx/HniIAYREQIISKCggiR\nTUDB1wITwBVZJKKEPUYoUYSUUKASgzEIolLFIlEkMIKUaIVVI0EElKiEAgnmZXXAyJYRUEE2Iy96\n3z/o5T5Hpqd7uvvc2z3fT9UU9ze3u+/pnme659D93GM8zxMAAAAAAFw5J+gBAAAAAADKFiaiAAAA\nAACnmIgCAAAAAJxiIgoAAAAAcIqJKAAAAADAKSaiAAAAAACnmIgCAAAAAJxiIgoAAAAAcCqtiagx\n5hZjzA5jzE5jzOhMDQq5h1pAFLUAEeoAcdQCRKgDxFELiPE8r1RfIlJORHaJyOUiUkFEPhaRxiVc\nx+Mr574OZboWQnCf+MpCHVALZeYr488J1ELOfvH6wFdW6oBayNkvaoGvpGvB87y03hG9VkR2ep73\nqed5p0Rkvoh0TeP2EE67k7gMtZD/kqkDEWqhLOA5AVHUAkSoA8RRC4hK6u/GdCail4jIXl/eF/me\nYowZZIzZYIzZkMaxEG4l1gJ1UGZQCxDh9QFxPCdAhOcExFELiDk32wfwPG+GiMwQETHGeNk+HsKJ\nOkAUtYAoagEi1AHiqAVEUQtlQzrviO4XkUt9uXbkeyh7qAVEUQsQoQ4QRy1AhDpAHLWAmHQmoutF\npIExpp4xpoKI3CMiSzMzLOQYagFR1AJEqAPEUQsQoQ4QRy0gptQfzfU877Qx5kERWS5nzoA1y/O8\nLRkbGXIGtYAoagEi1AHiqAWIUAeIoxbgZyKnRXZzMD7jnYs+8jyvVSZvkDrISRmvAxFqIUdRC4ji\n9QEiPCcgjlpAVFK1kM5HcwEAAAAASBkTUQAAAACAU0xEAQAAAABOMREFAAAAADjFRBQAAAAA4BQT\nUQAAAACAU0xEAQAAAABOMREFAAAAADh1btADAPJVy5YtVX7wwQdV7tevn8pz5sxReerUqSpv3Lgx\ng6MDAABANk2ePFnlhx56KLa9efNmta9z584q7969O3sDCwneEQUAAAAAOMVEFAAAAADgFB/NTVK5\ncuVUvvDCC5O+rv2RzPPPP1/lhg0bqvzAAw+o/Oyzz6rcq1cvlf/73/+qPGHChNj2E088kfQ4kZ7m\nzZurvGLFCpWrVKmisud5Kvft21flLl26qHzxxRenO0TkiQ4dOqg8b948ldu3b6/yjh07sj4mZMeY\nMWNUtp/TzzlH/3/yjTfeqPJ7772XlXEByIwLLrhA5cqVK6t8++23q1yjRg2VJ02apPK3336bwdEh\nVXXr1lW5T58+Kn///fex7UaNGql9V111lcp8NBcAAAAAgAxjIgoAAAAAcIqJKAAAAADAqTLTI1qn\nTh2VK1SooPL111+v8g033KBy1apVVe7Ro0fGxrZv3z6Vp0yZonL37t1VPnbsmMoff/yxyvQEuXPt\ntdfGthcsWKD22X3Edk+o/XM8deqUynZPaJs2bVS2l3Oxr18WtGvXLrZtP16LFi1yPRxnWrdurfL6\n9esDGgkybcCAASqPGjVKZX9/0dnYzzMAgufvG7R/p9u2bavy1VdfndJt16pVS2X/8iBw79ChQyqv\nWbNGZfv8H2Ud74gCAAAAAJxiIgoAAAAAcIqJKAAAAADAqbztEbXXdFy1apXKqawDmml2j4+9Ttzx\n48dVttcILCoqUvnw4cMqs2Zg5thrvl5zzTUqv/LKK7Ftu0+jJIWFhSpPnDhR5fnz56v8wQcfqGzX\nzfjx41M6fj7wr5nYoEEDtS+fekTttSLr1aun8mWXXaayMSbrY0J22D/L8847L6CRIFXXXXedyv71\nA+21fX/2s58lvK0RI0ao/J///Edl+zwW/tciEZF169YlHiwyyl7/cfjw4Sr37t07tl2pUiW1z36+\n3rt3r8r2+STstSd79uyp8vTp01Xevn17ccNGFpw4cULlsrAWaDp4RxQAAAAA4BQTUQAAAACAU0xE\nAQAAAABO5W2P6J49e1T+8ssvVc5kj6jdi3HkyBGVf/7zn6tsr/c4d+7cjI0FmfXiiy+q3KtXr4zd\ntt1vWrlyZZXt9WD9/ZAiIk2bNs3YWHJVv379Yttr164NcCTZZfcf33fffSrb/WH0BOWOjh07qjx0\n6NCEl7d/tp07d1b5wIEDmRkYSnT33XerPHnyZJWrV68e27b7AN99912Va9SoofIzzzyT8Nj27dnX\nv+eeexJeH6mx/2Z8+umnVbZr4YILLkj6tu3zRdx8880qly9fXmX7OcBfZ2fLcKtq1aoqN2vWLKCR\n5AbeEQUAAAAAOMVEFAAAAADgFBNRAAAAAIBTedsj+tVXX6k8cuRIle2+mn/+858qT5kyJeHtb9q0\nKbbdqVMntc9eQ8heL2zYsGEJbxvBadmypcq33367yonWZ7R7Ot944w2Vn332WZXtdeHsGrTXh/3F\nL36R9FjKCnt9zXw1c+bMhPvtHiOEl73+4+zZs1Uu6fwFdu8ga9Rlz7nn6j+RWrVqpfJLL72ksr3u\n9Jo1a2LbTz75pNr3/vvvq1yxYkWVX3/9dZVvuummhGPdsGFDwv1IT/fu3VX+zW9+U+rb2rVrl8r2\n35D2OqL169cv9bHgnv08UKdOnaSv27p1a5XtfuB8fL4vG3/FAQAAAABCg4koAAAAAMCpEieixphZ\nxpiDxpjNvu9VM8asMMYURv69KLvDRBhQC4iiFiBCHSCOWkAUtQAR6gDJSaZHtEBEponIHN/3RovI\nSs/zJhhjRkfyqMwPL3MWL16s8qpVq1Q+duyYyva6P/fee6/K/n4/uyfUtmXLFpUHDRqUeLDhVSB5\nUAt+zZs3V3nFihUqV6lSRWXP81RetmxZbNteY7R9+/YqjxkzRmW77+/QoUMqf/zxxyp///33Ktv9\nq/a6pBs3bpQsKpAAasFeO7VmzZqZvPnQKqlv0K5bhwokz54Tsq1///4q//SnP014eXu9yTlz5pz9\ngsErkDyrhT59+qhcUq+2/XvoX1vy6NGjCa9rr0NZUk/ovn37VH755ZcTXt6xAsmzWrjrrrtSuvxn\nn32m8vr162Pbo0bpu233hNoaNWqU0rFDpEDyrA6SYZ//o6CgQOWxY8cWe11735EjR1SeNm1aOkML\npRLfEfU8b42IfGV9u6uIRJ/1XhaRbhkeF0KIWkAUtQAR6gBx1AKiqAWIUAdITmnPmlvT87yiyPbn\nIlLs2xLGmEEikrNvAaJESdUCdVAmUAsQ4fUBcTwnIIpagAivD7CkvXyL53meMcZLsH+GiMwQEUl0\nOeS+RLVAHZQt1AJEeH1AHM8JiKIWIMLrA84o7UT0gDGmlud5RcaYWiJyMJODcqGkfo2vv/464f77\n7rsvtv3aa6+pfXYvX57LqVq48sorVbbXl7V78b744guVi4qKVPb35Rw/flzt++tf/5owp6tSpUoq\nP/zwwyr37t07o8dLQtZr4bbbblPZfgzyhd37Wq9evYSX379/fzaHk6qcek7IturVq6v861//WmX7\n9cLuCfrjH/+YnYG5kVO1YK/1+eijj6psnyNg+vTpKtvnASjp7wy/xx57LOnLiog89NBDKtvnGAih\nnKoFm/9vPpEfnuvj7bffVnnnzp0qHzxY+rubZ+dCyOk6KA37eSVRj2hZVNrlW5aKSPSMC/1FZElm\nhoMcRC0gilqACHWAOGoBUdQCRKgDWJJZvuVVEVkrIg2NMfuMMfeKyAQR6WSMKRSRjpGMPEctIIpa\ngAh1gDhqAVHUAkSoAySnxI/mep7Xq5hdHTI8FoQctYAoagEi1AHiqAVEUQsQoQ6QnLRPVpSv7M9w\nt2zZUmX/GpEdO3ZU++xeAQSnYsWKKvvXfxX5Yc+hvZ5sv379VN6wYYPKYepRrFOnTtBDyLqGDRsW\nu89erzeX2XVq9wh98sknKtt1i2DVrVs3tr1gwYKUrjt16lSVV69enYkh4Swef/xxle2e0FOnTqm8\nfPlyle31IL/55ptij3XeeeepbK8Taj9/G2NUtnuFlyzhE40u2WtDuuzza9u2rbNjIfvOOSf+YdQy\ndk6ZsyptjygAAAAAAKXCRBQAAAAA4BQTUQAAAACAU/SIFuPEiRMq22tIbdy4Mbb90ksvqX12T4/d\nV/j888+rbK9Nhsxp0aKFynZPqK1r164qv/feexkfE7Jj/fr1QQ+hWFWqVFH5lltuUblPnz4q2/1j\nNntdMnvtSQTL//Nt2rRpwsuuXLlS5cmTJ2dlTBCpWrWqykOGDFHZfi22e0K7deuW0vHq168f2543\nb57aZ593wvaXv/xF5YkTJ6Z0bISLf93XH/3oRyldt0mTJgn3f/jhhyqvXbs2pduHW/6+UP7+5x1R\nAAAAAIBjTEQBAAAAAE7x0dwk7dq1S+UBAwbEtmfPnq329e3bN2G2P5YxZ84clYuKiko7TFgmTZqk\nsn1KfPujt2H+KK7/lN8inPbbVq1atbSu36xZM5XtWrGXaapdu7bKFSpUiG337t1b7bN/dvYyD+vW\nrVP522+/Vfncc/VT9UcffSQID/sjmxMmFL9G+/vvv69y//79Vf76668zNzAo/t9REZHq1asnvLz/\n45QiIj/+8Y9VHjhwoMpdunRR+eqrr45tV65cWe2zP5Jn51deeUVlu10IwTr//PNVbty4scq///3v\nVU7UFpTqa7u9lIxdh999913C6wNhwjuiAAAAAACnmIgCAAAAAJxiIgoAAAAAcIoe0VJatGhRbLuw\nsFDts/sSO3TooPJTTz2l8mWXXabyuHHjVN6/f3+px1nWdO7cWeXmzZurbPfhLF26NOtjyhS7b8S+\nL5s2bXI5nEDYvZX+x+CFF15Q+x599NGUbtteZsPuET19+rTKJ0+eVHnr1q2x7VmzZql99hJOdi/y\ngQMHVN63b5/KlSpVUnn79u2C4NStW1flBQsWJH3dTz/9VGX7Z4/sOXXqlMqHDh1SuUaNGir/+9//\nVjnVpRb8vXxHjx5V+2rVqqXyF198ofIbb7yR0rGQWeXLl1fZXgrO/p23f572a5W/FuzlVezlvOz+\nU5t9zoA777xTZXsJKLvugTDhHVEAAAAAgFNMRAEAAAAATjERBQAAAAA4RY9oBmzevFnlnj17qnzH\nHXeobK87OnjwYJUbNGigcqdOndIdYplh99LZ68YdPHhQ5ddeey3rY0pWxYoVVR47dmzCy69atUrl\nRx55JNNDCp0hQ4aovHv37tj29ddfn9Zt79mzR+XFixervG3bNpX/8Y9/pHU8v0GDBqls96rZfYUI\n1qhRo1ROZU3fRGuMIruOHDmisr3+65tvvqmyvTaxvZ74kiVLVC4oKFD5q6++im3Pnz9f7bN7Cu39\ncMv+W8Hu21y4cGHC6z/xxBMq26/PH3zwQWzbriv7sv71Z8/Gfn0YP368yiW9ltnrVMMt/7qxJb12\ntGvXTuVp06ZlZUxB4h1RAAAAAIBTTEQBAAAAAE4xEQUAAAAAOEWPaBbYfShz585VeebMmSrba0LZ\nnwm/8cYbVX733XfTG2AZZvdGFBUVBTSSH/aEjhkzRuWRI0eqbK8t+dxzz6l8/PjxDI4uNzz99NNB\nDyEj7LWGbamsU4nMs9cjvummm5K+rt1HuGPHjoyMCelbt26dynbvXbr8r+Xt27dX++zeMPrA3bLX\nCbV7PO3XX9uyZctUnjp1qsr234H+2nrrrbfUviZNmqhsr/s5ceJEle0e0q5du6o8b948ld955x2V\n7dfNw4cPS3HKwvrkrvl/90tam9heI7Zx48Yq+9cvz1W8IwoAAAAAcIqJKAAAAADAKSaiAAAAAACn\n6BHNgKZNm6r8y1/+UuXWrVurbPeE2uzPfK9ZsyaN0cFv6dKlgR3b7jOze1Duvvtule3esh49emRn\nYAi9RYsWBT2EMu3tt99W+aKLLkp4ef8aswMGDMjGkJAD/Ota2z2hdm8Y64hmV7ly5VR+8sknVR4x\nYoTKJ06cUHn06NEq2z8vuye0VatWKvvXf2zRooXaV1hYqPL999+v8urVq1WuUqWKyvYa2r1791a5\nS5cuKq9YsUKKs3fvXpXr1atX7GVROi+88EJse/DgwSld115zfPjw4RkZU5B4RxQAAAAA4BQTUQAA\nAACAU0xEAQAAAABO0SOapIYNG6r84IMPxrbtdX5+8pOfpHTb3333ncr22pZ2bwmKZ4xJmLt166by\nsGHDsjaW3/72tyr/7ne/U/nCCy9U2V77q1+/ftkZGICUXHzxxSqX9Jw8ffr02HZZXN8XZyxfvjzo\nISDC7q2ze0JPnjypst27Z/eJt2nTRuWBAweqfOutt6rs7xf+wx/+oPbNnj1bZbtP03b06FGV//a3\nvyXMvXr1UvlXv/pVsbdt/92CzNu+fXvQQwgV3hEFAAAAADhV4kTUGHOpMWa1MWarMWaLMWZY5PvV\njDErjDGFkX8Tn0YQOY9agAh1gDhqAVHUAkSoA8RRC0hGMu+InhaRhz3PaywibUTkAWNMYxEZLSIr\nPc9rICIrIxn5jVqACHWAOGoBUdQCRKgDxFELKFGJPaKe5xWJSFFk+5gxZpuIXCIiXUXkxsjFXhaR\nd0VkVFZG6YDd12l/pt7fEyoiUrdu3VIfa8OGDSqPGzdO5SDXukwkF2rBXpvNzvbPecqUKSrPmjVL\n5S+//FJluy+kb9++se1mzZqpfbVr11Z5z549Ktv9Q/6+sjDLhTrIdXZv85VXXqmyf53KIOVrLdg9\nW+eck1oXy4cffpjJ4eSEfK2FdNx8881BD8G5sNbB448/nnC/vc6ovc732LFjVa5fv35Kx/dff/z4\n8WqffZ6QTHv11VcT5mwJay0EberUqbHtoUOHqn1XXHFFwuva5zXx35aIyK5du9IcnXspvboaY+qK\nSAsRWSciNSNFJiLyuYjUzOjIEGrUAkSoA8RRC4iiFiBCHSCOWkBxkj5rrjGmsogsEJHhnucd9f+P\nved5njHGK+Z6g0Rk0Nn2ITeVphaog/zDcwKiqAVE8foAEZ4TEEctIJGk3hE1xpSXM0U0z/O8hZFv\nHzDG1IrsryUiB892Xc/zZnie18rzvFaZGDCCVdpaoA7yC88JiKIWEMXrA0R4TkActYCSlPiOqDnz\nXxd/FpFtnudN8u1aKiL9RWRC5N8lWRlhhtSsqd/5b9y4scrTpk1T+aqrrir1sdatW6fyM888o/KS\nJfqhypV1QvOhFuw+kCFDhqjco0cPle31uho0aJD0sew+sdWrV6tcUs9KWOVDHYSd3ducao+iK/lS\nC82bN1e5Y8eOKtvP0adOnVL5+eefV/nAgQMZHF1uyJdayKTLL7886CE4F9Y6+Pzzz1WuUaOGyhUr\nVlTZPueD7a233lJ5zZo1Ki9evFjlzz77LLad7Z7QsAhrLYTJli1bVC7pOSNX5gupSOajuf8jIn1F\n5H+NMZsi33tUzhTQ68aYe0Vkt4j0zM4QESLUAkSoA8RRC4iiFiBCHSCOWkCJkjlr7vsiYorZ3SGz\nw0GYUQsQoQ4QRy0gilqACHWAOGoByQjn570AAAAAAHkr6bPmhl21atVUfvHFF1W2e4DS7d3w9/89\n99xzap+9PuQ333yT1rGQvLVr16q8fv16lVu3bp3w+vY6o3Zvsc2/zuj8+fPVPnu9J6C02rZtq3JB\nQUEwA8lTVatWVdl+HrDt379f5REjRmR8TMh9f//732Pbdp93PvZ6hVm7du1U7tatm8rXXHONygcP\n6vPn2GuMHz58WGW7bxxIxowZM1S+4447AhpJcHhHFAAAAADgFBNRAAAAAIBTTEQBAAAAAE7lVI/o\nddddF9seOXKk2nfttdeqfMkll6R1rJMnT6o8ZcoUlZ966qnY9okTJ9I6FjJn3759Kt95550qDx48\nWOUxY8akdPuTJ09W+U9/+lNse+fOnSndFlCcM8uvAchlmzdvjm0XFhaqffZ5Kq644gqVDx06lL2B\nlUHHjh1Tee7cuQkz4MLWrVtV3rZtm8qNGjVyOZxA8I4oAAAAAMApJqIAAAAAAKdy6qO53bt3P+t2\nMuy3v998802VT58+rbK9JMuRI0dSOh7CoaioSOWxY8cmzEAQli1bpvJdd90V0EjKpu3bt6vsX55L\nROSGG25wORzkIX87j4jIzJkzVR43bpzKQ4cOVdn+GwZA7tu9e7fKTZo0CWgkweEdUQAAAACAU0xE\nAQAAAABOMREFAAAAADhlPM9zdzBj3B0MmfKR53mtMnmD1EFOyngdiFALOYpaQBSvD0mqUqWKyq+/\n/rrKHTt2VHnhwoUqDxw4UOWQLRvHcwKiqAVEJVULvCMKAAAAAHCKiSgAAAAAwCkmogAAAAAAp3Jq\nHVEAAIBcc/ToUZV79uypsr2O6P3336+yveY164oCyAe8IwoAAAAAcIqJKAAAAADAKSaiAAAAAACn\n6BEFAABwyO4ZHTp0aMIMAPmId0QBAAAAAE4xEQUAAAAAOMVEFAAAAADglOse0S9EZLeIVI9shxFj\n0y7Lwm1SB+nJlzoQoRbSRS24xdg0Xh/CJ1/qQIRaSBe14FZYxxbUuJKqBeN5XrYH8sODGrPB87xW\nzg+cBMbmTpjvD2NzK8z3ibG5Feb7xNjcCfP9YWxuhfk+MTa3wnyfwjq2sI4rio/mAgAAAACcYiIK\nAAAAAHAqqInojICOmwzG5k6Y7w9jcyvM94mxuRXm+8TY3Anz/WFsboX5PjE2t8J8n8I6trCOS0QC\n6hEFAAAAAJRdfDQXAAAAAOAUE1EAAAAAgFNOJ6LGmFuMMTuMMTuNMaNdHvssY5lljDlojNns+141\nY8wKY0xh5N+LAhrbpcaY1caYrcaYLcaYYWEaXyZQC0mPjVpwO5ZQ1gJ14HwsoayDyDioBbdjoRYC\nRC0kNS7qwO1YQlkHkXHkXC04m4gaY8qJyPMicquINBaRXsaYxq6OfxYFInKL9b3RIrLS87wGIrIy\nkoNwWkQe9jyvsYi0EZEHIo9VWMaXFmohJdSCWwUSzlqgDtwqkHDWgQi14FqBUAuBoBaSRh24VSDh\nrAORXKwFz/OcfIlIWxFZ7suPiMgjro5fzJjqishmX94hIrUi27VEZEeQ4/ONa4mIdArr+KgFaoFa\noA6oA2qBWgj8saMWqAXqgDrIqVpw+dHcS0Rkry/vi3wvTGp6nlcU2f5cRGoGORgREWNMXRFpISLr\nJITjKyVqoRSohcCE6rGmDgITuseaWghM6B5raiEwoXqsqYPAhO6xzpVa4GRFxfDO/LdBoGvbGGMq\ni8gCERnued5R/74wjK+sCMNjTS2EQ9CPNXUQDmF4rKmFcAjDY00thEPQjzV1EA5heKxzqRZcTkT3\ni8ilvlw78r0wOWCMqSUiEvn3YFADMcaUlzNFNM/zvIVhG1+aqIUUUAuBC8VjTR0ELjSPNbUQuNA8\n1tRC4ELxWFMHgQvNY51rteByIrpeRBoYY+oZYyqIyD0istTh8ZOxVET6R7b7y5nPVjtnjDEi8mcR\n2eZ53iTfrlCMLwOohSRRC6EQ+GNNHYRCKB5raiEUQvFYUwuhEPhjTR2EQige65ysBcdNs7eJyCci\nsktEHguyOVZEXhWRIhH5PznzefN7ReRiOXM2qUIReUdEqgU0thvkzNvm/xKRTZGv28IyPmqBWqAW\nqAPqgOcEaoFaoBaCf6ypA+ogl2vBRAYOAAAAAIATnKwIAAAAAOAUE1EAAAAAgFNMRAEAAAAATjER\nBQAAAAA4xUQUAAAAAOAUE1EAAAAAgFNMRAEAAAAATv0/VRGGEPckXi4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f70ba2e9090>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      5          0          4          1          9          2          1\n"
     ]
    }
   ],
   "source": [
    "draw_examples(x_train[:7], captions=y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "batch_size = 128\n",
    "num_classes = 10\n",
    "epochs = 12\n",
    "\n",
    "# input image dimensions\n",
    "img_rows, img_cols = 28, 28"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "if keras.backend.image_data_format() == 'channels_first':\n",
    "    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)\n",
    "    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)\n",
    "    input_shape = (1, img_rows, img_cols)\n",
    "else:\n",
    "    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n",
    "    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n",
    "    input_shape = (img_rows, img_cols, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train shape: (60000, 28, 28, 1)\n",
      "60000 train samples\n",
      "10000 test samples\n"
     ]
    }
   ],
   "source": [
    "x_train = x_train.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "print('x_train shape: {}'.format(x_train.shape))\n",
    "print('{} train samples'.format(x_train.shape[0]))\n",
    "print('{} test samples'.format(x_test.shape[0]))\n",
    "\n",
    "# convert class vectors to binary class matrices\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Conv2D(32, kernel_size=(3, 3),\n",
    "                 activation='relu',\n",
    "                 input_shape=input_shape))\n",
    "model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Dropout(0.25))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(128, activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(num_classes, activation='softmax'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer=keras.optimizers.Adadelta(),\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/12\n",
      "60000/60000 [==============================] - 333s - loss: 0.3256 - acc: 0.9037 - val_loss: 0.0721 - val_acc: 0.9780\n",
      "Epoch 2/12\n",
      "60000/60000 [==============================] - 342s - loss: 0.1088 - acc: 0.9683 - val_loss: 0.0501 - val_acc: 0.9835\n",
      "Epoch 3/12\n",
      "60000/60000 [==============================] - 366s - loss: 0.0837 - acc: 0.9748 - val_loss: 0.0429 - val_acc: 0.9860\n",
      "Epoch 4/12\n",
      "60000/60000 [==============================] - 311s - loss: 0.0694 - acc: 0.9788 - val_loss: 0.0380 - val_acc: 0.9878\n",
      "Epoch 5/12\n",
      "60000/60000 [==============================] - 325s - loss: 0.0626 - acc: 0.9815 - val_loss: 0.0334 - val_acc: 0.9886\n",
      "Epoch 6/12\n",
      "60000/60000 [==============================] - 262s - loss: 0.0552 - acc: 0.9835 - val_loss: 0.0331 - val_acc: 0.9890\n",
      "Epoch 7/12\n",
      "60000/60000 [==============================] - 218s - loss: 0.0494 - acc: 0.9852 - val_loss: 0.0291 - val_acc: 0.9903\n",
      "Epoch 8/12\n",
      "60000/60000 [==============================] - 218s - loss: 0.0461 - acc: 0.9859 - val_loss: 0.0294 - val_acc: 0.9902\n",
      "Epoch 9/12\n",
      "60000/60000 [==============================] - 219s - loss: 0.0423 - acc: 0.9869 - val_loss: 0.0287 - val_acc: 0.9907\n",
      "Epoch 10/12\n",
      "60000/60000 [==============================] - 218s - loss: 0.0418 - acc: 0.9875 - val_loss: 0.0299 - val_acc: 0.9906\n",
      "Epoch 11/12\n",
      "60000/60000 [==============================] - 218s - loss: 0.0388 - acc: 0.9879 - val_loss: 0.0304 - val_acc: 0.9905\n",
      "Epoch 12/12\n",
      "60000/60000 [==============================] - 218s - loss: 0.0366 - acc: 0.9889 - val_loss: 0.0275 - val_acc: 0.9910\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f70b80b1a10>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(x_train, y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,\n",
    "          verbose=1,\n",
    "          validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Test loss:', 0.027530849870144449)\n",
      "('Test accuracy:', 0.99099999999999999)\n"
     ]
    }
   ],
   "source": [
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "livereveal": {
   "start_slideshow_at": "selected",
   "theme": "white"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}