{ "cells": [ { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Uczenie maszynowe – zastosowania\n", "# 10. Sieci neuronowe – propagacja wsteczna" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "slideshow": { "slide_type": "notes" } }, "outputs": [], "source": [ "%matplotlib inline\n", "\n", "import numpy as np\n", "import math" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## 10.1. Metoda propagacji wstecznej – wprowadzenie" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Architektura sieci neuronowych\n", "\n", "* Budowa warstwowa, najczęściej sieci jednokierunkowe i gęste.\n", "* Liczbę i rozmiar warstw dobiera się do każdego problemu.\n", "* Rozmiary sieci określane poprzez liczbę neuronów lub parametrów." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### _Feedforward_\n", "\n", "Mając daną $n$-warstwową sieć neuronową oraz jej parametry $\\Theta^{(1)}, \\ldots, \\Theta^{(L)} $ oraz $\\beta^{(1)}, \\ldots, \\beta^{(L)} $, obliczamy:\n", "\n", "$$a^{(l)} = g^{(l)}\\left( a^{(l-1)} \\Theta^{(l)} + \\beta^{(l)} \\right). $$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "* Funkcje $g^{(l)}$ to **funkcje aktywacji**.
\n", "Dla $i = 0$ przyjmujemy $a^{(0)} = x$ (wektor wierszowy cech) oraz $g^{(0)}(x) = x$ (identyczność)." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "* Parametry $\\Theta$ to wagi na połączeniach miedzy neuronami dwóch warstw.
\n", "Rozmiar macierzy $\\Theta^{(l)}$, czyli macierzy wag na połączeniach warstw $a^{(l-1)}$ i $a^{(l)}$, to $\\dim(a^{(l-1)}) \\times \\dim(a^{(l)})$." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "* Parametry $\\beta$ zastępują tutaj dodawanie kolumny z jedynkami do macierzy cech.
Macierz $\\beta^{(l)}$ ma rozmiar równy liczbie neuronów w odpowiedniej warstwie, czyli $1 \\times \\dim(a^{(l)})$." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "* **Klasyfikacja**: dla ostatniej warstwy $L$ (o rozmiarze równym liczbie klas) przyjmuje się $g^{(L)}(x) = \\mathop{\\mathrm{softmax}}(x)$.\n", "* **Regresja**: pojedynczy neuron wyjściowy; funkcją aktywacji może wtedy być np. funkcja identycznościowa." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "* Pozostałe funkcje aktywacji najcześciej mają postać sigmoidy, np. sigmoidalna, tangens hiperboliczny.
Ale niekoniecznie, np. ReLU, leaky ReLU, maxout." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Jak uczyć sieci neuronowe?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "* W poznanych do tej pory algorytmach (regresja liniowa, regresja logistyczna) do uczenia używaliśmy funkcji kosztu, jej gradientu oraz algorytmu gradientu prostego (GD/SGD)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "* Dla sieci neuronowych potrzebowalibyśmy również znaleźć gradient funkcji kosztu." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "* Co sprowadza się do bardziej ogólnego problemu:
jak obliczyć gradient $\\nabla f(x)$ dla danej funkcji $f$ i wektora wejściowego $x$?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Pochodna funkcji\n", "\n", "* **Pochodna** mierzy, jak szybko zmienia się wartość funkcji względem zmiany jej argumentów:\n", "\n", "$$ \\frac{d f(x)}{d x} = \\lim_{h \\to 0} \\frac{ f(x + h) - f(x) }{ h } $$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Pochodna cząstkowa i gradient\n", "\n", "* **Pochodna cząstkowa** mierzy, jak szybko zmienia się wartość funkcji względem zmiany jej *pojedynczego argumentu*." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "* **Gradient** to wektor pochodnych cząstkowych:\n", "\n", "$$ \\nabla f = \\left( \\frac{\\partial f}{\\partial x_1}, \\ldots, \\frac{\\partial f}{\\partial x_n} \\right) $$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Gradient – przykłady\n", "\n", "$$ f(x_1, x_2) = x_1 + x_2 \\qquad \\to \\qquad \\frac{\\partial f}{\\partial x_1} = 1, \\quad \\frac{\\partial f}{\\partial x_2} = 1, \\quad \\nabla f = (1, 1) $$ " ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "$$ f(x_1, x_2) = x_1 \\cdot x_2 \\qquad \\to \\qquad \\frac{\\partial f}{\\partial x_1} = x_2, \\quad \\frac{\\partial f}{\\partial x_2} = x_1, \\quad \\nabla f = (x_2, x_1) $$ " ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "$$ f(x_1, x_2) = \\max(x_1 + x_2) \\hskip{12em} \\\\\n", "\\to \\qquad \\frac{\\partial f}{\\partial x_1} = \\mathbb{1}_{x \\geq y}, \\quad \\frac{\\partial f}{\\partial x_2} = \\mathbb{1}_{y \\geq x}, \\quad \\nabla f = (\\mathbb{1}_{x \\geq y}, \\mathbb{1}_{y \\geq x}) $$ " ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Własności pochodnych cząstkowych\n", "\n", "Jezeli $f(x, y, z) = (x + y) \\, z$ oraz $x + y = q$, to:\n", "$$f = q z,\n", "\\quad \\frac{\\partial f}{\\partial q} = z,\n", "\\quad \\frac{\\partial f}{\\partial z} = q,\n", "\\quad \\frac{\\partial q}{\\partial x} = 1,\n", "\\quad \\frac{\\partial q}{\\partial y} = 1 $$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Reguła łańcuchowa\n", "\n", "$$ \\frac{\\partial f}{\\partial x} = \\frac{\\partial f}{\\partial q} \\, \\frac{\\partial q}{\\partial x},\n", "\\quad \\frac{\\partial f}{\\partial y} = \\frac{\\partial f}{\\partial q} \\, \\frac{\\partial q}{\\partial y} $$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Propagacja wsteczna – prosty przykład" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "# Dla ustalonego wejścia\n", "x = -2; y = 5; z = -4" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(3, -12)\n" ] } ], "source": [ "# Krok w przód\n", "q = x + y\n", "f = q * z\n", "print(q, f)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[-4, -4, 3]\n" ] } ], "source": [ "# UWAGA: teraz za pomocą zmiennych `dfx`, `dfy`, `dfz` i `dfq`\n", "# oznaczę pochodne cząstkowe ∂f/∂x, ∂f/∂y, ∂f/∂z i ∂f/∂q odpowiednio\n", "\n", "# Propagacja wsteczna dla f = q * z\n", "dfz = q\n", "dfq = z\n", "# Propagacja wsteczna dla q = x + y\n", "dfx = 1 * dfq # z reguły łańcuchowej\n", "dfy = 1 * dfq # z reguły łańcuchowej\n", "print([dfx, dfy, dfz])" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "* Właśnie tak wygląda obliczanie pochodnych metodą propagacji wstecznej!" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "* Spróbujmy czegoś bardziej skomplikowanego:
metodą propagacji wstecznej obliczmy pochodną funkcji sigmoidalnej." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Propagacja wsteczna – funkcja sigmoidalna" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "Funkcja sigmoidalna:\n", "\n", "$$f(\\theta,x) = \\frac{1}{1+e^{-(\\theta_0 x_0 + \\theta_1 x_1 + \\theta_2)}}$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "$$\n", "\\begin{array}{lcl}\n", "f(x) = \\frac{1}{x} \\quad & \\rightarrow & \\quad \\frac{df}{dx} = -\\frac{1}{x^2} \\\\\n", "f_c(x) = c + x \\quad & \\rightarrow & \\quad \\frac{df}{dx} = 1 \\\\\n", "f(x) = e^x \\quad & \\rightarrow & \\quad \\frac{df}{dx} = e^x \\\\\n", "f_a(x) = ax \\quad & \\rightarrow & \\quad \\frac{df}{dx} = a \\\\\n", "\\end{array}\n", "$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[0.3932238664829637, -0.5898357997244456]\n", "[-0.19661193324148185, -0.3932238664829637, 0.19661193324148185]\n" ] } ], "source": [ "# Losowe wagi i dane\n", "w = [2,-3,-3]\n", "x = [-1, -2]\n", "\n", "# Krok w przód\n", "dot = w[0]*x[0] + w[1]*x[1] + w[2]\n", "f = 1.0 / (1 + math.exp(-dot)) # funkcja sigmoidalna\n", "\n", "# Krok w tył\n", "ddot = (1 - f) * f # pochodna funkcji sigmoidalnej\n", "dx = [w[0] * ddot, w[1] * ddot]\n", "dw = [x[0] * ddot, x[1] * ddot, 1.0 * ddot]\n", "\n", "print(dx)\n", "print(dw)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Obliczanie gradientów – podsumowanie\n", "\n", "* Gradient $f$ dla $x$ mówi jak zmieni się całe wyrażenie przy zmianie wartości $x$.\n", "* Gradienty łączymy korzystając z **reguły łańcuchowej**.\n", "* W kroku wstecz gradienty informują, które części grafu powinny być zwiększone lub zmniejszone (i z jaką siłą), aby zwiększyć wartość na wyjściu.\n", "* W kontekście implementacji chcemy dzielić funkcję $f$ na części, dla których można łatwo obliczyć gradienty." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## 10.2. Uczenie wielowarstwowych sieci neuronowych metodą propagacji wstecznej" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Mając algorytm SGD oraz gradienty wszystkich wag, moglibyśmy trenować każdą sieć." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "* Niech:\n", "$$\\Theta = (\\Theta^{(1)},\\Theta^{(2)},\\Theta^{(3)},\\beta^{(1)},\\beta^{(2)},\\beta^{(3)})$$\n", "\n", "* Funkcja sieci neuronowej z grafiki:\n", "\n", "$$\\small h_\\Theta(x) = \\tanh(\\tanh(\\tanh(x\\Theta^{(1)}+\\beta^{(1)})\\Theta^{(2)} + \\beta^{(2)})\\Theta^{(3)} + \\beta^{(3)})$$\n", "* Funkcja kosztu dla regresji:\n", "$$J(\\Theta) = \\dfrac{1}{2m} \\sum_{i=1}^{m} (h_\\Theta(x^{(i)})- y^{(i)})^2 $$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "* Jak obliczymy gradienty?\n", "\n", "$$\\nabla_{\\Theta^{(l)}} J(\\Theta) = ? \\quad \\nabla_{\\beta^{(l)}} J(\\Theta) = ?$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### W kierunku propagacji wstecznej\n", "\n", "* Pewna (niewielka) zmiana wagi $\\Delta z^l_j$ dla $j$-ego neuronu w warstwie $l$ pociąga za sobą (niewielką) zmianę kosztu: \n", "\n", "$$\\frac{\\partial J(\\Theta)}{\\partial z^{l}_j} \\Delta z^{l}_j$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "* Jeżeli $\\frac{\\partial J(\\Theta)}{\\partial z^{l}_j}$ jest duża, $\\Delta z^l_j$ ze znakiem przeciwnym zredukuje koszt.\n", "* Jeżeli $\\frac{\\partial J(\\Theta)}{\\partial z^l_j}$ jest bliska zeru, koszt nie będzie mocno poprawiony." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "* Definiujemy błąd $\\delta^l_j$ neuronu $j$ w warstwie $l$: \n", "\n", "$$\\delta^l_j := \\dfrac{\\partial J(\\Theta)}{\\partial z^l_j}$$ \n", "$$\\delta^l := \\nabla_{z^l} J(\\Theta) \\quad \\textrm{ (zapis wektorowy)} $$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Podstawowe równania propagacji wstecznej\n", "\n", "$$\n", "\\begin{array}{rcll}\n", "\\delta^L & = & \\nabla_{a^L}J(\\Theta) \\odot { \\left( g^{L} \\right) }^{\\prime} \\left( z^L \\right) & (BP1) \\\\[2mm]\n", "\\delta^{l} & = & \\left( \\left( \\Theta^{l+1} \\right) \\! ^\\top \\, \\delta^{l+1} \\right) \\odot {{ \\left( g^{l} \\right) }^{\\prime}} \\left( z^{l} \\right) & (BP2)\\\\[2mm]\n", "\\nabla_{\\beta^l} J(\\Theta) & = & \\delta^l & (BP3)\\\\[2mm]\n", "\\nabla_{\\Theta^l} J(\\Theta) & = & a^{l-1} \\odot \\delta^l & (BP4)\\\\\n", "\\end{array}\n", "$$\n" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### (BP1)\n", "$$ \\delta^L_j \\; = \\; \\frac{ \\partial J }{ \\partial a^L_j } \\, g' \\!\\! \\left( z^L_j \\right) $$\n", "$$ \\delta^L \\; = \\; \\nabla_{a^L}J(\\Theta) \\odot { \\left( g^{L} \\right) }^{\\prime} \\left( z^L \\right) $$\n", "Błąd w ostatniej warstwie jest iloczynem szybkości zmiany kosztu względem $j$-tego wyjścia i szybkości zmiany funkcji aktywacji w punkcie $z^L_j$." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### (BP2)\n", "$$ \\delta^{l} \\; = \\; \\left( \\left( \\Theta^{l+1} \\right) \\! ^\\top \\, \\delta^{l+1} \\right) \\odot {{ \\left( g^{l} \\right) }^{\\prime}} \\left( z^{l} \\right) $$\n", "Aby obliczyć błąd w $l$-tej warstwie, należy przemnożyć błąd z następnej ($(l+1)$-szej) warstwy przez transponowany wektor wag, a uzyskaną macierz pomnożyć po współrzędnych przez szybkość zmiany funkcji aktywacji w punkcie $z^l$." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### (BP3)\n", "$$ \\nabla_{\\beta^l} J(\\Theta) \\; = \\; \\delta^l $$\n", "Błąd w $l$-tej warstwie jest równy wartości gradientu funkcji kosztu." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### (BP4)\n", "$$ \\nabla_{\\Theta^l} J(\\Theta) \\; = \\; a^{l-1} \\odot \\delta^l $$\n", "Gradient funkcji kosztu względem wag $l$-tej warstwy można obliczyć jako iloczyn po współrzędnych $a^{l-1}$ przez $\\delta^l$." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Algorytm propagacji wstecznej" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Dla jednego przykładu $(x,y)$:\n", "\n", "1. **Wejście**: Ustaw aktywacje w warstwie cech $a^{(0)}=x$ \n", "2. **Feedforward:** dla $l=1,\\dots,L$ oblicz \n", "$$z^{(l)} = a^{(l-1)} \\Theta^{(l)} + \\beta^{(l)} \\textrm{ oraz } a^{(l)}=g^{(l)} \\!\\! \\left( z^{(l)} \\right) $$\n", "3. **Błąd wyjścia $\\delta^{(L)}$:** oblicz wektor $$\\delta^{(L)}= \\nabla_{a^{(L)}}J(\\Theta) \\odot {g^{\\prime}}^{(L)} \\!\\! \\left( z^{(L)} \\right) $$\n", "4. **Propagacja wsteczna błędu:** dla $l = L-1,L-2,\\dots,1$ oblicz $$\\delta^{(l)} = \\delta^{(l+1)}(\\Theta^{(l+1)})^T \\odot {g^{\\prime}}^{(l)} \\!\\! \\left( z^{(l)} \\right) $$\n", "5. **Gradienty:** \n", " * $\\dfrac{\\partial}{\\partial \\Theta_{ij}^{(l)}} J(\\Theta) = a_i^{(l-1)}\\delta_j^{(l)} \\textrm{ oraz } \\dfrac{\\partial}{\\partial \\beta_{j}^{(l)}} J(\\Theta) = \\delta_j^{(l)}$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "W naszym przykładzie:\n", "\n", "$$\\small J(\\Theta) = \\frac{1}{2} \\left( a^{(L)} - y \\right) ^2 $$\n", "$$\\small \\dfrac{\\partial}{\\partial a^{(L)}} J(\\Theta) = a^{(L)} - y$$\n", "\n", "$$\\small \\tanh^{\\prime}(x) = 1 - \\tanh^2(x)$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Algorytm SGD z propagacją wsteczną\n", "\n", "Pojedyncza iteracja:\n", "* Dla parametrów $\\Theta = (\\Theta^{(1)},\\ldots,\\Theta^{(L)})$ utwórz pomocnicze macierze zerowe $\\Delta = (\\Delta^{(1)},\\ldots,\\Delta^{(L)})$ o takich samych wymiarach (dla uproszczenia opuszczono wagi $\\beta$).\n", "* Dla $m$ przykładów we wsadzie (_batch_), $i = 1,\\ldots,m$:\n", " * Wykonaj algortym propagacji wstecznej dla przykładu $(x^{(i)}, y^{(i)})$ i przechowaj gradienty $\\nabla_{\\Theta}J^{(i)}(\\Theta)$ dla tego przykładu;\n", " * $\\Delta := \\Delta + \\dfrac{1}{m}\\nabla_{\\Theta}J^{(i)}(\\Theta)$\n", "* Wykonaj aktualizację wag: $\\Theta := \\Theta - \\alpha \\Delta$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Propagacja wsteczna – podsumowanie\n", "\n", "* Algorytm pierwszy raz wprowadzony w latach 70. XX w.\n", "* W 1986 David Rumelhart, Geoffrey Hinton i Ronald Williams pokazali, że jest znacznie szybszy od wcześniejszych metod.\n", "* Obecnie najpopularniejszy algorytm uczenia sieci neuronowych." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## 10.3. Przykłady implementacji wielowarstwowych sieci neuronowych" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "notes" } }, "source": [ "### Uwaga!\n", "\n", "Poniższe przykłady wykorzystują interfejs [Keras](https://keras.io), który jest częścią biblioteki [TensorFlow](https://www.tensorflow.org).\n", "\n", "Aby uruchomić TensorFlow w środowisku Jupyter, należy wykonać następujące czynności:\n", "\n", "#### Przed pierwszym uruchomieniem (wystarczy wykonać tylko raz)\n", "\n", "Instalacja biblioteki TensorFlow w środowisku Anaconda:\n", "\n", "1. Uruchom *Anaconda Navigator*\n", "1. Wybierz kafelek *CMD.exe Prompt*\n", "1. Kliknij przycisk *Launch*\n", "1. Pojawi się konsola. Wpisz następujące polecenia, każde zatwierdzając wciśnięciem klawisza Enter:\n", "```\n", "conda create -n tf tensorflow\n", "conda activate tf\n", "conda install pandas matplotlib\n", "jupyter notebook\n", "```\n", "\n", "#### Przed każdym uruchomieniem\n", "\n", "Jeżeli chcemy korzystać z biblioteki TensorFlow, to środowisko Jupyter Notebook należy uruchomić w następujący sposób:\n", "\n", "1. Uruchom *Anaconda Navigator*\n", "1. Wybierz kafelek *CMD.exe Prompt*\n", "1. Kliknij przycisk *Launch*\n", "1. Pojawi się konsola. Wpisz następujące polecenia, każde zatwierdzając wciśnięciem klawisza Enter:\n", "```\n", "conda activate tf\n", "jupyter notebook\n", "```" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Przykład: MNIST\n", "\n", "_Modified National Institute of Standards and Technology database_" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "* Zbiór cyfr zapisanych pismem odręcznym\n", "* 60 000 przykładów uczących, 10 000 przykładów testowych\n", "* Rozdzielczość każdego przykładu: 28 × 28 = 784 piksele" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "# źródło: https://github.com/keras-team/keras/examples/minst_mlp.py\n", "\n", "from tensorflow import keras\n", "from tensorflow.keras.datasets import mnist\n", "from tensorflow.keras.layers import Dense, Dropout\n", "\n", "# załaduj dane i podziel je na zbiory uczący i testowy\n", "(x_train, y_train), (x_test, y_test) = mnist.load_data()" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "slideshow": { "slide_type": "notes" } }, "outputs": [], "source": [ "from matplotlib import pyplot as plt\n", "\n", "def draw_examples(examples, captions=None):\n", " plt.figure(figsize=(16, 4))\n", " m = len(examples)\n", " for i, example in enumerate(examples):\n", " plt.subplot(100 + m * 10 + i + 1)\n", " plt.imshow(example, cmap=plt.get_cmap('gray'))\n", " plt.show()\n", " if captions is not None:\n", " print(6 * ' ' + (10 * ' ').join(str(captions[i]) for i in range(m)))" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA54AAACOCAYAAABZsdfhAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAcc0lEQVR4nO3deZBU5bnH8ecVAVFERJAQEYYIQYhsAgpeC0gAV2SRgBL2GKHEBVJCgUoMxiCIStUAEkUujCAlWGHVSJCwSFRCgQTvBVkGjMDgBAYF2Qxc9Nw/6O15menpnjnn9Onu76eKmvOb093nnebh9Lycfvo1juMIAAAAAABeuSTVAwAAAAAAZDYmngAAAAAATzHxBAAAAAB4ioknAAAAAMBTTDwBAAAAAJ5i4gkAAAAA8FS5Jp7GmLuMMbuNMXuNMePcGhTSD7WAMGoBItQBoqgFiFAHiKIWspcp6zqexpgKIrJHRLqKSIGIbBaRfo7jfO7e8JAOqAWEUQsQoQ4QRS1AhDpAFLWQ3S4tx31vEZG9juN8ISJijFkoIj1EpMTCMcaUbZaLVDrqOE6tUm5DLWQBx3FMAjdLqhaog7TEOQFh1AJEJKHXB+ogO3BOQFixtVCet9peJyIHY3JB6HvILPsTuA21gDBqIfNxTkAYtYBEUQfZgXMCwoqthfJc8Szuf7cu+h8JY8wwERlWjuMg+KgFhJVaC9RBVuCcgDBqASLUAaKohSxWnolngYhcH5PrishX9o0cx5klIrNEuFSewagFhJVaC9RBVuCcgDBqASLUAaKohSxWnrfabhaRRsaYBsaYSiLyoIiscGdYSDPUAsKoBYhQB4iiFiBCHSCKWshiZb7i6TjOeWPMYyKySkQqiMgcx3F2uDYypA1qAWHUAkSoA0RRCxChDhBFLWS3Mi+nUqaDcak8HX3qOE4btx+UWkg/CX6qbVKog7TEOQFh1AJEhNcHRHBOQFixtVCet9oCAAAAAFAqJp4AAAAAAE8x8QQAAAAAeIqJJwAAAADAU0w8AQAAAACeYuIJAAAAAPAUE08AAAAAgKeYeAIAAAAAPMXEEwAAAADgqUtTPQAgk7Vu3Vrlxx57TOVBgwapPG/ePJWnT5+u8tatW10cHQAAALySm5ur8hNPPBHZ3r59u9rXrVs3lffv3+/dwFKEK54AAAAAAE8x8QQAAAAAeIq32iaoQoUKKl911VUJ39d+e+Xll1+ucuPGjVV+9NFHVX755ZdV7tevn8r/+c9/VJ48eXJk+7nnnkt4nCi/li1bqrx69WqVq1WrprLjOCoPHDhQ5e7du6t8zTXXlHOEyASdO3dWecGCBSp37NhR5d27d3s+Jnhj/PjxKtvn9Esu0f9/3KlTJ5U//PBDT8YFwB1XXnmlylWrVlX53nvvVblWrVoqT506VeWzZ8+6ODokKycnR+UBAwao/MMPP0S2mzRpovbdeOONKvNWWwAAAAAAksTEEwAAAADgKSaeAAAAAABPZU2PZ7169VSuVKmSyrfddpvKt99+u8rVq1dXuXfv3q6NraCgQOVp06ap3KtXL5VPnjyp8meffaYyPT3+uuWWWyLbixcvVvvsXmC7p9P+uzx37pzKdk9nu3btVLaXV7Hvn+k6dOgQ2bafq6VLl/o9HN+0bdtW5c2bN6doJHDbkCFDVB47dqzKsf1BxbHPMQBSL7bvz/433b59e5VvuummpB67Tp06Kscu1wH/FRUVqbxhwwaV7c/uyDZc8QQAAAAAeIqJJwAAAADAU0w8AQAAAACeytgeT3s9xbVr16qczDqcbrN7dOx12k6dOqWyvUZfYWGhyseOHVOZNfvcZa+7evPNN6v81ltvRbbtXovS5OfnqzxlyhSVFy5cqPLHH3+ssl07kyZNSur46S52zcJGjRqpfZnU42mv1digQQOV69evr7IxxvMxwRv23+Vll12WopGgLG699VaVY9fws9fX/dnPfhb3sUaPHq3yV199pbL9WRSxr0UiIps2bYo/WLjGXn9x1KhRKvfv3z+yXaVKFbXPPl8fPHhQZfuzIOy1H/v27avyzJkzVd61a1cJo4YXTp8+rXImrsVZHlzxBAAAAAB4ioknAAAAAMBTTDwBAAAAAJ7K2B7PAwcOqPz111+r7GaPp91Hcfz4cZV//vOfq2yvtTh//nzXxgL3vf766yr369fPtce2+0WrVq2qsr0ma2xPo4hI8+bNXRtLOho0aFBke+PGjSkcibfs3uGHH35YZbu3i56e9NGlSxeVH3/88bi3t/9uu3XrpvLhw4fdGRgS8sADD6icm5urcs2aNSPbdi/f+vXrVa5Vq5bKL730Utxj249n3//BBx+Me38kzv6d8cUXX1TZroMrr7wy4ce2P+vhzjvvVLlixYoq2+eA2BorLsNf1atXV7lFixapGUhAccUTAAAAAOApJp4AAAAAAE8x8QQAAAAAeCpjezy/+eYblceMGaOy3Rfzz3/+U+Vp06bFffxt27ZFtrt27ar22Wv42Gt1jRw5Mu5jI7Vat26t8r333qtyvDUS7Z7Md999V+WXX35ZZXtdNrsO7TVaf/GLXyQ8lmxgr2+ZqWbPnh13v90jhOCy116cO3euyqV9/oDd98cacd669FL9a1KbNm1UfuONN1S2133esGFDZPv5559X+z766COVK1eurPI777yj8h133BF3rFu2bIm7H2XXq1cvlX/zm9+U+bH27dunsv07pL2OZ8OGDct8LPjPPgfUq1cv4fu2bdtWZbufNxPO99nxWxsAAAAAIGVKnXgaY+YYY44YY7bHfK+GMWa1MSY/9PVqb4eJIKAWEEYtQIQ6QBS1gDBqASLUAYqXyBXPPBG5y/reOBFZ4zhOIxFZE8rIfHlCLeCCPKEWQB0gKk+oBVyQJ9QCqAMUo9QeT8dxNhhjcqxv9xCRTqHtN0VkvYiMdXNgblu2bJnKa9euVfnkyZMq2+vuPPTQQyrH9urZPZ22HTt2qDxs2LC4tw+qTKkFW8uWLVVevXq1ytWqVVPZcRyVV65cGdm21/js2LGjyuPHj1fZ7t0rKipS+bPPPlP5hx9+UNnuP7XXBd26dat4IVW1YK9bWrt2bTcfPrBK6/uza9YvmXpO8NLgwYNV/vGPfxz39vZaj/PmzXN7SK7I1FoYMGCAyqX1W9v/FmPXdzxx4kTc+9prQZbW01lQUKDym2++Gff2fsnEWujTp09St//yyy9V3rx5c2R77Fj9Y9s9nbYmTZokdeygyMQ6SIT92R15eXkqT5gwocT72vuOHz+u8owZM8oxsmAoa49nbcdxCkVEQl+vdW9ISDPUAsKoBYhQB4iiFhBGLUCEOsh6nn+qrTFmmIik5yU+uIpagAh1gChqAWHUAkSoA0RRC5mprFc8Dxtj6oiIhL4eKemGjuPMchynjeM4bUq6DdIatYCwhGqBOsh4nBMQRi0gjNcHiHBOyHplveK5QkQGi8jk0Nflro3IJ6X1Wnz77bdx9z/88MOR7UWLFql9dh9ehku7WvjpT3+qsr3Gq91Pd/ToUZULCwtVju2rOXXqlNr3l7/8JW4urypVqqj85JNPqty/f39Xj1cKz2vhnnvuUdn++TOF3bvaoEGDuLc/dOiQl8NJVtqdE7xUs2ZNlX/961+rbL9e2D09f/zjHz0Zl0/SrhbstTaffvpple0e/5kzZ6ps9/GX9rtGrGeeeSbh24qIPPHEEyrbnxEQMGlXC7Fif+cTufizOj744AOV9+7dq/KRIyXOr0qVYZ9lkNZ1UBb2OSVej2c2SGQ5lbdFZKOINDbGFBhjHpILBdPVGJMvIl1DGRmOWkAYtQAR6gBR1ALCqAWIUAcoXiKfatuvhF2dXR4LAo5aQBi1ABHqAFHUAsKoBYhQByheWXs8AQAAAABIiOefapuu7Pdgt27dWuXY9Rm7dOmi9tnv9UdqVa5cWeXYNVhFLu4btNd0HTRokMpbtmxROUh9hvXq1Uv1EDzVuHHjEvfZ6+WmM7tG7R6fPXv2qGzXLFIrJycnsr148eKk7jt9+nSV161b58aQUIJnn31WZbun89y5cyqvWrVKZXtNxu+++67EY1122WUq2+t02udvY4zKdr/v8uUZ3x4XGPbajH726bVv3963Y8F7l1wSveaXZZ8JIyJc8QQAAAAAeIyJJwAAAADAU0w8AQAAAACeosezBKdPn1bZXsNp69atke033nhD7bN7cuyewFdffVVle10wuKtVq1Yq2z2dth49eqj84Ycfuj4muG/z5s2pHkKJqlWrpvJdd92l8oABA1S2e79s9rpg9tqPSK3Yv9/mzZvHve2aNWtUzs3N9WRMuKB69eoqjxgxQmX79dju6ezZs2dSx2vYsGFke8GCBWqf/dkRtj//+c8qT5kyJaljIzhi11y94oorkrpvs2bN4u7/5JNPVN64cWNSjw9/xfZ1ZuPv/1zxBAAAAAB4ioknAAAAAMBTvNU2Qfv27VN5yJAhke25c+eqfQMHDoyb7bdZzJs3T+XCwsKyDhPFmDp1qsr2R9Tbb6UN8ltrYz+GWyQ7P4q7JDVq1CjX/Vu0aKGyXSf2skl169ZVuVKlSpHt/v37q33235u95MKmTZtUPnv2rMqXXqpP1Z9++qkgOOy3X06ePLnE23700UcqDx48WOVvv/3WtXHhYrH/TkVEatasGff2sW+RFBG59tprVR46dKjK3bt3V/mmm26KbFetWlXts99mZ+e33npLZbsFCKlz+eWXq9y0aVOVf//736scr8Un2dd1e2kXuwa///77uPcHUokrngAAAAAATzHxBAAAAAB4ioknAAAAAMBT9HiW0dKlSyPb+fn5ap/dU9i5c2eVX3jhBZXr16+v8sSJE1U+dOhQmceZjbp166Zyy5YtVbb7aFasWOH1kFxj937YP8u2bdt8HI3/7N7I2J//tddeU/uefvrppB7bXvbC7vE8f/68ymfOnFH5888/j2zPmTNH7bOXVLL7iA8fPqxyQUGBylWqVFF5165dgtTJyclRefHixQnf94svvlDZ/ruHt86dO6dyUVGRyrVq1VL5X//6l8rJLn8Q24934sQJta9OnToqHz16VOV33303qWPBPRUrVlTZXpbN/jdv/13ar1WxdWAvd2Ivr2X3j9rsnv/7779fZXtJJrvmgVTiiicAAAAAwFNMPAEAAAAAnmLiCQAAAADwFD2eLti+fbvKffv2Vfm+++5T2V73c/jw4So3atRI5a5du5Z3iFnF7oez1207cuSIyosWLfJ8TImqXLmyyhMmTIh7+7Vr16r81FNPuT2kQBkxYoTK+/fvj2zfdttt5XrsAwcOqLxs2TKVd+7cqfI//vGPch0v1rBhw1S2+8zsvkCk1tixY1VOZj3deGt8wnvHjx9X2V6D9b333lPZXh/YXtN7+fLlKufl5an8zTffRLYXLlyo9tl9gfZ++Mf+PcHuu1yyZEnc+z/33HMq26/NH3/8cWTbrin7trFrvxbHfn2YNGmSyqW9ltnrRMNfseu2lvba0aFDB5VnzJjhyZj8xBVPAAAAAICnmHgCAAAAADzFxBMAAAAA4Cl6PD1g95DMnz9f5dmzZ6tsr8lkv6e7U6dOKq9fv75c48t2dn9DYWFhikZycU/n+PHjVR4zZozK9vqOr7zyisqnTp1ycXTB9+KLL6Z6CK6w1/q1JbNOJNxnrwV8xx13JHxfuwdw9+7dbgwJLtm0aZPKdv9cecW+nnfs2FHts/u76OX2j71Op92jab/22lauXKny9OnTVbZ/D4ytq/fff1/ta9asmcr2uptTpkxR2e4B7dGjh8oLFixQ+W9/+5vK9uvmsWPHpCSZvjZ4KsT+uy9tXWB7jdamTZuqHLt+eLrgiicAAAAAwFNMPAEAAAAAnmLiCQAAAADwFD2eLmjevLnKv/zlL1Vu27atynZPp81+z/aGDRvKMTrYVqxYkbJj271idh/JAw88oLLdH9a7d29PxoVgW7p0aaqHkNU++OADla+++uq4t49d43XIkCFeDAlpInZdabun0+7vYh1P71SoUEHl559/XuXRo0erfPr0aZXHjRunsv13Zfd0tmnTRuXY9RdbtWql9uXn56v8yCOPqLxu3TqVq1WrprK9hnX//v1V7t69u8qrV6+Wkhw8eFDlBg0alHhblM1rr70W2R4+fHhS97XX/B41apQbQ/IVVzwBAAAAAJ5i4gkAAAAA8BQTTwAAAACAp+jxTFDjxo1VfuyxxyLb9jo7P/rRj5J67O+//15le11Juy8E8Rlj4uaePXuqPHLkSM/G8tvf/lbl3/3udypfddVVKtvrbw0aNMibgQFI2DXXXKNyaefkmTNnRrazbW1daKtWrUr1ECAX98bZPZ1nzpxR2e69s/u827Vrp/LQoUNVvvvuu1WO7fX9wx/+oPbNnTtXZbvP0nbixAmV//rXv8bN/fr1U/lXv/pViY9t/84C9+3atSvVQ0gprngCAAAAADxV6sTTGHO9MWadMWanMWaHMWZk6Ps1jDGrjTH5oa/xP+YPaY9agAh1gChqAWHUAkSoA0RRCyhOIlc8z4vIk47jNBGRdiLyqDGmqYiME5E1juM0EpE1oYzMRi1AhDpAFLWAMGoBItQBoqgFXMTY60iVegdjlovIjNCfTo7jFBpj6ojIesdxGpdy3+QO5iO7L9N+T3xsT6eISE5OTpmPtWXLFpUnTpyocirXmSzGp47jtCluR1BroU+fPiq//fbbKts9ta+//rrKc+bMUfnrr79W2e7tGDhwYGS7RYsWal/dunVVPnDggMqx6/2JiOTm5sbdn0qO45jivh/UOkgnixYtUrlv374qDx48WOV58+Z5PqY40u6ckCy758pei7O0Hs+f/OQnke39+/e7Nq4AyvhaKK8777wzsv3++++rffbvX3Xq1FG5qKjIu4G5LOivD/ZnZ9SqVUvls2fPqmz34V1xxRUqN2zYMKnjT5gwIbI9adIktc/+nSTNcU5Iwp49e1S+4YYb4t7+kkv09UK7Dvft2+fOwNxRbC0k1eNpjMkRkVYisklEajuOUygiEvp6rQuDRJqgFiBCHSCKWkAYtQAR6gBR1ALCEv5UW2NMVRFZLCKjHMc5YX9SaJz7DRORYaXeEGmDWoAIdYAoagFh1AJEqANEUQuIldAVT2NMRblQNAscx1kS+vbh0CVyCX09Utx9HceZ5ThOm5IuvSO9UAsQoQ4QRS0gjFqACHWAKGoBtlKveJoL/zXx3yKy03GcqTG7VojIYBGZHPq63JMRuqR27doqN23aVOUZM2aofOONN5b5WJs2bVL5pZdeUnn5cv1Upcs6nZlSCxUqVFB5xIgRKvfu3Vtle82sRo0aJXysTz75ROV169ap/Oyzzyb8WEGRKXUQZHbvl93XERSZUgstW7ZUuUuXLirb5+hz586p/Oqrr6p8+PBh9waXJjKlFtwW2++bDYJaB//+979Vtns8K1eurLL9eQ02u193w4YNKi9btkzlL7/8MrKdYT2dJQpqLQTJjh07VC7tfJEu84V4Enmr7X+JyEAR+V9jzLbQ956WCwXzjjHmIRE5ICJ9ir87Mgi1ABHqAFHUAsKoBYhQB4iiFnCRUieejuN8JCIlvSG7s7vDQZBRCxChDhBFLSCMWoAIdYAoagHFCeb7twAAAAAAGSPhT7UNuho1aqhsr81o9/CUt+8itnfvlVdeUftWrVql8nfffVeuYyE5GzduVHnz5s0qt23bNu797TVd7f5gW+w6nwsXLlT7Ro4cGfe+QCLat2+vcl5eXmoGkqGqV6+usn0OsB06dEjl0aNHuz0kZIi///3vkW27VzsT+rXSRYcOHVTu2bOnyjfffLPKR47oz7ux1/c+duyYynbfN5CIWbNmqXzfffelaCT+4YonAAAAAMBTTDwBAAAAAJ5i4gkAAAAA8FRa9Xjeeuutke0xY8aofbfccovK1113XbmOdebMGZWnTZum8gsvvBDZPn36dLmOBXcVFBSofP/996s8fPhwlcePH5/U4+fm5qr8pz/9KbK9d+/epB4LKM6F5c8ApLvt27dHtvPz89U++7MmbrjhBpWLioq8G1iWOXnypMrz58+PmwE/fP755yrv3LlT5SZNmvg5HF9wxRMAAAAA4CkmngAAAAAAT6XVW2179epV7HYi7MvZ7733nsrnz59X2V4i5fjx40kdD8FRWFio8oQJE+JmwG8rV65UuU+fPikaSXbatWuXyrHLZYmI3H777X4OBxkqtkVHRGT27NkqT5w4UeXHH39cZfv3GADpbf/+/So3a9YsRSPxD1c8AQAAAACeYuIJAAAAAPAUE08AAAAAgKeM4zj+HcwY/w4Gt3zqOE4btx+UWkg/juO4vsYHdZCWOCcgjFpIQrVq1VR+5513VO7SpYvKS5YsUXno0KEqB2kpN14fEMI5AWHF1gJXPAEAAAAAnmLiCQAAAADwFBNPAAAAAICn0modTwAAgHR04sQJlfv27auyvY7nI488orK95jTregJIN1zxBAAAAAB4ioknAAAAAMBTTDwBAAAAAJ5iHU+UhjWZICKs04YIzgkIoxYgIrw+IIJzAsJYxxMAAAAA4D8mngAAAAAATzHxBAAAAAB4yu91PI+KyH4RqRnaDiLGptX36HGDXgtBHZcIdeA3xqZRC8FELfgnqOMSoQ78xtg0aiGYAlMLvn64UOSgxmzxovnYDYzNX0H9mYI6LpFgj62sgvwzMTZ/BflnYmz+CurPFNRxiQR7bGUV5J+JsfkryD8TY0sMb7UFAAAAAHiKiScAAAAAwFOpmnjOStFxE8HY/BXUnymo4xIJ9tjKKsg/E2PzV5B/Jsbmr6D+TEEdl0iwx1ZWQf6ZGJu/gvwzMbYEpKTHEwAAAACQPXirLQAAAADAU75OPI0xdxljdhtj9hpjxvl57BLGM8cYc8QYsz3mezWMMauNMfmhr1enYFzXG2PWGWN2GmN2GGNGBmVsbglSLQS1DkLjoBb8HUsga4E68H0sgayD0DioBX/HQi2kELWQ0LioA3/HEsg6CI0j8LXg28TTGFNBRF4VkbtFpKmI9DPGNPXr+CXIE5G7rO+NE5E1juM0EpE1oey38yLypOM4TUSknYg8GnqugjC2cgtgLeRJMOtAhFrwW54EsxaoA3/lSTDrQIRa8FueUAspQS0kjDrwV54Esw5E0qEWHMfx5Y+ItBeRVTH5KRF5yq/jxxlXjohsj8m7RaROaLuOiOwOwBiXi0jXII4tU2ohHeqAWqAWqAPqgFqgFqgFaoE6oA7StRb8fKvtdSJyMCYXhL4XNLUdxykUEQl9vTaVgzHG5IhIKxHZJAEbWzmkQy0E7rmmFlImUM81dZAygXuuqYWUCdxzTS2kTKCea+ogZQL3XAe1FvyceJpivsdH6sZhjKkqIotFZJTjOCdSPR4XUQtJohYgQh0gilpAGLUAEeoAUUGuBT8nngUicn1MrisiX/l4/EQdNsbUEREJfT2SikEYYyrKhaJZ4DjOkiCNzQXpUAuBea6phZQLxHNNHaRcYJ5raiHlAvNcUwspF4jnmjpIucA810GvBT8nnptFpJExpoExppKIPCgiK3w8fqJWiMjg0PZgufD+aF8ZY4yI/LeI7HQcZ2qQxuaSdKiFQDzX1EIgpPy5pg4CIRDPNbUQCIF4rqmFQEj5c00dBEIgnuu0qAWfm1zvEZE9IrJPRJ5JVWNrzHjeFpFCEfk/ufA/Kg+JyDVy4ROf8kNfa6RgXLfLhbcR/I+IbAv9uScIY8vEWghqHVAL1AJ1QB1QC9QCtUAtUAfUQabUggkNFAAAAAAAT/j5VlsAAAAAQBZi4gkAAAAA8BQTTwAAAACAp5h4AgAAAAA8xcQTAAAAAOApJp4AAAAAAE8x8QQAAAAAeIqJJwAAAADAU/8PSXdIbxrRR2wAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ " 5 0 4 1 9 2 1\n" ] } ], "source": [ "draw_examples(x_train[:7], captions=y_train)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "60000 przykładów uczących\n", "10000 przykładów testowych\n" ] } ], "source": [ "num_classes = 10\n", "\n", "x_train = x_train.reshape(60000, 784) # 784 = 28 * 28\n", "x_test = x_test.reshape(10000, 784)\n", "x_train = x_train.astype('float32')\n", "x_test = x_test.astype('float32')\n", "x_train /= 255\n", "x_test /= 255\n", "print('{} przykładów uczących'.format(x_train.shape[0]))\n", "print('{} przykładów testowych'.format(x_test.shape[0]))\n", "\n", "# przekonwertuj wektory klas na binarne macierze klas\n", "y_train = keras.utils.to_categorical(y_train, num_classes)\n", "y_test = keras.utils.to_categorical(y_test, num_classes)" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "scrolled": true, "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model: \"sequential_1\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "dense_3 (Dense) (None, 512) 401920 \n", "_________________________________________________________________\n", "dropout (Dropout) (None, 512) 0 \n", "_________________________________________________________________\n", "dense_4 (Dense) (None, 512) 262656 \n", "_________________________________________________________________\n", "dropout_1 (Dropout) (None, 512) 0 \n", "_________________________________________________________________\n", "dense_5 (Dense) (None, 10) 5130 \n", "=================================================================\n", "Total params: 669,706\n", "Trainable params: 669,706\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] } ], "source": [ "model = keras.Sequential()\n", "model.add(Dense(512, activation='relu', input_shape=(784,)))\n", "model.add(Dropout(0.2))\n", "model.add(Dense(512, activation='relu'))\n", "model.add(Dropout(0.2))\n", "model.add(Dense(num_classes, activation='softmax'))\n", "\n", "model.summary() # wyświetl podsumowanie architektury sieci" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(60000, 784) (60000, 10)\n" ] } ], "source": [ "print(x_train.shape, y_train.shape)" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 1/10\n", "469/469 [==============================] - 20s 42ms/step - loss: 0.0957 - accuracy: 0.9708 - val_loss: 0.0824 - val_accuracy: 0.9758\n", "Epoch 2/10\n", "469/469 [==============================] - 20s 43ms/step - loss: 0.0693 - accuracy: 0.9793 - val_loss: 0.0807 - val_accuracy: 0.9772\n", "Epoch 3/10\n", "469/469 [==============================] - 18s 38ms/step - loss: 0.0563 - accuracy: 0.9827 - val_loss: 0.0861 - val_accuracy: 0.9758\n", "Epoch 4/10\n", "469/469 [==============================] - 18s 37ms/step - loss: 0.0485 - accuracy: 0.9857 - val_loss: 0.0829 - val_accuracy: 0.9794\n", "Epoch 5/10\n", "469/469 [==============================] - 19s 41ms/step - loss: 0.0428 - accuracy: 0.9876 - val_loss: 0.0955 - val_accuracy: 0.9766\n", "Epoch 6/10\n", "469/469 [==============================] - 22s 47ms/step - loss: 0.0377 - accuracy: 0.9887 - val_loss: 0.0809 - val_accuracy: 0.9794\n", "Epoch 7/10\n", "469/469 [==============================] - 17s 35ms/step - loss: 0.0338 - accuracy: 0.9904 - val_loss: 0.1028 - val_accuracy: 0.9788\n", "Epoch 8/10\n", "469/469 [==============================] - 17s 36ms/step - loss: 0.0322 - accuracy: 0.9911 - val_loss: 0.0937 - val_accuracy: 0.9815\n", "Epoch 9/10\n", "469/469 [==============================] - 18s 37ms/step - loss: 0.0303 - accuracy: 0.9912 - val_loss: 0.0916 - val_accuracy: 0.9829.0304 - accu\n", "Epoch 10/10\n", "469/469 [==============================] - 16s 34ms/step - loss: 0.0263 - accuracy: 0.9926 - val_loss: 0.0958 - val_accuracy: 0.9812\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.compile(loss='categorical_crossentropy', optimizer=keras.optimizers.RMSprop(), metrics=['accuracy'])\n", "\n", "model.fit(x_train, y_train, batch_size=128, epochs=10, verbose=1,\n", " validation_data=(x_test, y_test))" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Test loss: 0.0757974311709404\n", "Test accuracy: 0.9810000061988831\n" ] } ], "source": [ "score = model.evaluate(x_test, y_test, verbose=0)\n", "\n", "print('Test loss: {}'.format(score[0]))\n", "print('Test accuracy: {}'.format(score[1]))" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Warstwa *dropout* to metoda regularyzacji, służy zapobieganiu nadmiernemu dopasowaniu sieci. Polega na tym, że część węzłów sieci jest usuwana w sposób losowy." ] }, { "cell_type": "code", "execution_count": 61, "metadata": { "slideshow": { "slide_type": "notes" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model: \"sequential_22\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "dense_62 (Dense) (None, 512) 401920 \n", "_________________________________________________________________\n", "dense_63 (Dense) (None, 512) 262656 \n", "_________________________________________________________________\n", "dense_64 (Dense) (None, 10) 5130 \n", "=================================================================\n", "Total params: 669,706\n", "Trainable params: 669,706\n", "Non-trainable params: 0\n", "_________________________________________________________________\n", "Epoch 1/5\n", "469/469 [==============================] - 10s 20ms/step - loss: 0.2203 - accuracy: 0.9317 - val_loss: 0.0936 - val_accuracy: 0.9697\n", "Epoch 2/5\n", "469/469 [==============================] - 10s 21ms/step - loss: 0.0816 - accuracy: 0.9746 - val_loss: 0.0747 - val_accuracy: 0.9779\n", "Epoch 3/5\n", "469/469 [==============================] - 10s 20ms/step - loss: 0.0544 - accuracy: 0.9827 - val_loss: 0.0674 - val_accuracy: 0.9798\n", "Epoch 4/5\n", "469/469 [==============================] - 10s 22ms/step - loss: 0.0384 - accuracy: 0.9879 - val_loss: 0.0746 - val_accuracy: 0.9806\n", "Epoch 5/5\n", "469/469 [==============================] - 10s 22ms/step - loss: 0.0298 - accuracy: 0.9901 - val_loss: 0.0736 - val_accuracy: 0.9801\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 61, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Bez warstw Dropout\n", "\n", "num_classes = 10\n", "\n", "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n", "\n", "x_train = x_train.reshape(60000, 784) # 784 = 28 * 28\n", "x_test = x_test.reshape(10000, 784)\n", "x_train = x_train.astype('float32')\n", "x_test = x_test.astype('float32')\n", "x_train /= 255\n", "x_test /= 255\n", "\n", "y_train = keras.utils.to_categorical(y_train, num_classes)\n", "y_test = keras.utils.to_categorical(y_test, num_classes)\n", "\n", "model_no_dropout = keras.Sequential()\n", "model_no_dropout.add(Dense(512, activation='relu', input_shape=(784,)))\n", "model_no_dropout.add(Dense(512, activation='relu'))\n", "model_no_dropout.add(Dense(num_classes, activation='softmax'))\n", "model_no_dropout.summary()\n", "\n", "model_no_dropout.compile(loss='categorical_crossentropy',\n", " optimizer=keras.optimizers.RMSprop(),\n", " metrics=['accuracy'])\n", "\n", "model_no_dropout.fit(x_train, y_train,\n", " batch_size=128,\n", " epochs=5,\n", " verbose=1,\n", " validation_data=(x_test, y_test))" ] }, { "cell_type": "code", "execution_count": 62, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Test loss (no dropout): 0.07358124107122421\n", "Test accuracy (no dropout): 0.9800999760627747\n" ] } ], "source": [ "# Bez warstw Dropout\n", "\n", "score = model_no_dropout.evaluate(x_test, y_test, verbose=0)\n", "\n", "print('Test loss (no dropout): {}'.format(score[0]))\n", "print('Test accuracy (no dropout): {}'.format(score[1]))" ] }, { "cell_type": "code", "execution_count": 63, "metadata": { "slideshow": { "slide_type": "notes" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model: \"sequential_23\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "dense_65 (Dense) (None, 2500) 1962500 \n", "_________________________________________________________________\n", "dense_66 (Dense) (None, 2000) 5002000 \n", "_________________________________________________________________\n", "dense_67 (Dense) (None, 1500) 3001500 \n", "_________________________________________________________________\n", "dense_68 (Dense) (None, 1000) 1501000 \n", "_________________________________________________________________\n", "dense_69 (Dense) (None, 500) 500500 \n", "_________________________________________________________________\n", "dense_70 (Dense) (None, 10) 5010 \n", "=================================================================\n", "Total params: 11,972,510\n", "Trainable params: 11,972,510\n", "Non-trainable params: 0\n", "_________________________________________________________________\n", "Epoch 1/10\n", "469/469 [==============================] - 129s 275ms/step - loss: 0.9587 - accuracy: 0.7005 - val_loss: 0.5066 - val_accuracy: 0.8566\n", "Epoch 2/10\n", "469/469 [==============================] - 130s 276ms/step - loss: 0.2666 - accuracy: 0.9234 - val_loss: 0.3376 - val_accuracy: 0.9024\n", "Epoch 3/10\n", "469/469 [==============================] - 130s 277ms/step - loss: 0.1811 - accuracy: 0.9477 - val_loss: 0.1678 - val_accuracy: 0.9520\n", "Epoch 4/10\n", "469/469 [==============================] - 134s 287ms/step - loss: 0.1402 - accuracy: 0.9588 - val_loss: 0.1553 - val_accuracy: 0.9576\n", "Epoch 5/10\n", "469/469 [==============================] - 130s 278ms/step - loss: 0.1153 - accuracy: 0.9662 - val_loss: 0.1399 - val_accuracy: 0.9599\n", "Epoch 6/10\n", "469/469 [==============================] - 130s 277ms/step - loss: 0.0956 - accuracy: 0.9711 - val_loss: 0.1389 - val_accuracy: 0.9612\n", "Epoch 7/10\n", "469/469 [==============================] - 131s 280ms/step - loss: 0.0803 - accuracy: 0.9761 - val_loss: 0.1008 - val_accuracy: 0.9724\n", "Epoch 8/10\n", "469/469 [==============================] - 134s 286ms/step - loss: 0.0685 - accuracy: 0.9797 - val_loss: 0.1137 - val_accuracy: 0.9679\n", "Epoch 9/10\n", "469/469 [==============================] - 130s 278ms/step - loss: 0.0602 - accuracy: 0.9819 - val_loss: 0.1064 - val_accuracy: 0.9700\n", "Epoch 10/10\n", "469/469 [==============================] - 129s 274ms/step - loss: 0.0520 - accuracy: 0.9843 - val_loss: 0.1095 - val_accuracy: 0.9698\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 63, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Więcej warstw, inna funkcja aktywacji\n", "\n", "num_classes = 10\n", "\n", "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n", "\n", "x_train = x_train.reshape(60000, 784) # 784 = 28 * 28\n", "x_test = x_test.reshape(10000, 784)\n", "x_train = x_train.astype('float32')\n", "x_test = x_test.astype('float32')\n", "x_train /= 255\n", "x_test /= 255\n", "\n", "y_train = keras.utils.to_categorical(y_train, num_classes)\n", "y_test = keras.utils.to_categorical(y_test, num_classes)\n", "\n", "model3 = Sequential()\n", "model3.add(Dense(2500, activation='tanh', input_shape=(784,)))\n", "model3.add(Dense(2000, activation='tanh'))\n", "model3.add(Dense(1500, activation='tanh'))\n", "model3.add(Dense(1000, activation='tanh'))\n", "model3.add(Dense(500, activation='tanh'))\n", "model3.add(Dense(num_classes, activation='softmax'))\n", "model3.summary()\n", "\n", "model3.compile(loss='categorical_crossentropy',\n", " optimizer=keras.optimizers.RMSprop(),\n", " metrics=['accuracy'])\n", "\n", "model3.fit(x_train, y_train,\n", " batch_size=128,\n", " epochs=10,\n", " verbose=1,\n", " validation_data=(x_test, y_test))" ] }, { "cell_type": "code", "execution_count": 64, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Test loss: 0.10945799201726913\n", "Test accuracy: 0.9697999954223633\n" ] } ], "source": [ "# Więcej warstw, inna funkcja aktywacji\n", "\n", "score = model3.evaluate(x_test, y_test, verbose=0)\n", "\n", "print('Test loss: {}'.format(score[0]))\n", "print('Test accuracy: {}'.format(score[1]))" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Przykład: 4-pikselowy aparat fotograficzny\n", "\n", "https://www.youtube.com/watch?v=ILsA4nyG7I0" ] }, { "cell_type": "code", "execution_count": 65, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "def generate_example(description):\n", " variant = random.choice([1, -1])\n", " if description == 's': # solid\n", " return (np.array([[ 1.0, 1.0], [ 1.0, 1.0]]) if variant == 1 else\n", " np.array([[-1.0, -1.0], [-1.0, -1.0]]))\n", " elif description == 'v': # vertical\n", " return (np.array([[ 1.0, -1.0], [ 1.0, -1.0]]) if variant == 1 else\n", " np.array([[-1.0, 1.0], [-1.0, 1.0]]))\n", " elif description == 'd': # diagonal\n", " return (np.array([[ 1.0, -1.0], [-1.0, 1.0]]) if variant == 1 else\n", " np.array([[-1.0, 1.0], [ 1.0, -1.0]]))\n", " elif description == 'h': # horizontal\n", " return (np.array([[ 1.0, 1.0], [-1.0, -1.0]]) if variant == 1 else\n", " np.array([[-1.0, -1.0], [ 1.0, 1.0]]))\n", " else:\n", " return np.array([[random.uniform(-1, 1), random.uniform(-1, 1)],\n", " [random.uniform(-1, 1), random.uniform(-1, 1)]])" ] }, { "cell_type": "code", "execution_count": 67, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "import random\n", "\n", "num_classes = 4\n", "\n", "trainset_size = 4000\n", "testset_size = 1000\n", "\n", "y4_train = np.array([random.choice(['s', 'v', 'd', 'h']) for i in range(trainset_size)])\n", "x4_train = np.array([generate_example(desc) for desc in y4_train])\n", "\n", "y4_test = np.array([random.choice(['s', 'v', 'd', 'h']) for i in range(testset_size)])\n", "x4_test = np.array([generate_example(desc) for desc in y4_test])" ] }, { "cell_type": "code", "execution_count": 68, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6oAAACQCAYAAAABdZZIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAPGUlEQVR4nO3dX4ild33H8c+3uwYhVKTN+ie7WZOLZSUVWswkVXqTXliSYJNeaIm9UKSwKOSyFwsF7WUveiUVQy5C4o22eGGXNq1NhRALymZXVJK2SRdRs0nARkvaVTFm+fVizrMzpLOzc/Y858xvnvN6wZA5c56c5zkn7/09893zzKRaawEAAIBe/Np+HwAAAABsZ1AFAACgKwZVAAAAumJQBQAAoCsGVQAAALpiUAUAAKArhxf5l6vqN5L8TZJbk/wgyR+31v57h+1+kOR/k1xO8kZrbWOR/dIfLTDQAokO2KIFBlog0QF7t+g7qqeTfL21diLJ12e3r+b3W2u/I7LJ0gIDLZDogC1aYKAFEh2wR4sOqg8keXz2+eNJ/mjBx+Pg0gIDLZDogC1aYKAFEh2wR4sOqu9srb2SJLN/vuMq27Uk/1xV56vq1IL7pE9aYKAFEh2wRQsMtECiA/bomj+jWlX/kuRdO9z153Ps5/daay9X1TuSPFlV/9Fae/oq+zuVZIjxjjn2wT6rqtZaq2tstqcW1qGDO+44uE/rhRdeyK9+9av/9/WjR4/m0KFDqar/aq0ducbDzN3CjTfeeMd73/veMZ4CI7hWBxsbG+38+fOvXqMF54c1sKzzw1TXhPPnz+/3ISzNss4PmeiacFC/V1jl+UEHB9tuHVRr7bofuKqeT3J3a+2Vqnp3kqdaayev8e/8RZJLrbW/2sPjX//BsS9aazV2C1PtYJE/ez07efJkXnjhhe+11n577BY2NjbauXPnRjxaluXkyZN56qmncvPNN59P8odxflh7yzg/THVNqLrWTH+gLeX8MNU1YYrfKyzz/KCDg6eqzl/tZ5AXvfT3TJJPzD7/RJK/22HnN1bVrw+fJ/mDJM8uuF/6poU1dv/99yfJb85uamFN3X///Xn88eFHkHTAFVrA+WHNOT+wV4sOqn+Z5ENV9Z9JPjS7naq6uaqemG3zziT/WlXfTXI2yT+01v5pwf3SKS1w+vTpJHmbFtbb6dOn8+STTybJ+6ID4vzAFc4Pa875gb1a6NLfZZvq2/dTtoefQZrbVDvo+c/eona7jGMRU73Mb8qW1cJU14UpW8b5YaprwsQv/bUmzMH3CnM/5iRfsHXtYNF3VAEAAGBUBlUAAAC6YlAFAACgKwZVAAAAumJQBQAAoCsGVQAAALpiUAUAAKArBlUAAAC6YlAFAACgKwZVAAAAumJQBQAAoCsGVQAAALpiUAUAAKArBlUAAAC6YlAFAACgKwZVAAAAumJQBQAAoCsGVQAAALoyyqBaVfdU1fNVdaGqTu9wf1XV52b3f6+q3j/GfumPFhhogZm36YDEmsAV1gQGWmBXCw+qVXUoyeeT3Jvk9iQfq6rb37TZvUlOzD5OJfnCovulW1pgoIU1d/ny5SQ5Hh2wyZpAYk0gzg/szRjvqN6V5EJr7futtdeTfDnJA2/a5oEkX2ybvpXk7VX17hH2TX+0QJLcGC2svbNnzybJL3XAjDWBxJpAnB/YmzEG1aNJXtx2++Lsa/NuwzRogSS5IVpYey+99FKSvL7tSzpYb9YEEmsCcX5gbw6P8Bi1w9fadWyzuWHVqWy+vc80XFcLOpikhVs4fvz4Eg6LZWltx2Xe+YGBNYHEmrCWxjw/6GC6xnhH9WKSW7bdPpbk5evYJknSWnuktbbRWtsY4dhYvVFa0MGB93qW0MKRI0dGP1CW59ixY8nmu+tXvhTnh3VmTSCxJpBxzw86mK4xBtVnkpyoqtuq6oYkDyY586ZtziT5+Oy3d30gyWuttVdG2Df90QJJ8rNoYe3deeedSfJWHTBjTSCxJhDnB/Zm4Ut/W2tvVNVDSb6W5FCSR1trz1XVp2b3P5zkiST3JbmQ5OdJPrnofumWFhhoYc0dPnw4SX4UHbDJmkBiTSDOD+xNXeUa8S5UVb8Hx45aazv9PMFCptpBz3/2FlVV55dxCc7GxkY7d+7c2A/LEi2rhamuC1O2jPPDVNeEqtFfqp5YE+bge4W5H3OSL9i6djDGpb8AAAAwGoMqAAAAXTGoAgAA0BWDKgAAAF0xqAIAANAVgyoAAABdMagCAADQFYMqAAAAXTGoAgAA0BWDKgAAAF0xqAIAANAVgyoAAABdMagCAADQFYMqAAAAXTGoAgAA0BWDKgAAAF0xqAIAANAVgyoAAABdMagCAADQlVEG1aq6p6qer6oLVXV6h/vvrqrXquo7s4/PjLFf+qMFBlpg5m06ILEmcIU1gYEW2NXhRR+gqg4l+XySDyW5mOSZqjrTWvu3N236jdbahxfdH93TAgMtrLnLly8nyfEkt0cHWBPYZE3A+YE9GeMd1buSXGitfb+19nqSLyd5YITH5WDSAklyY7Sw9s6ePZskv9QBM9YEEmsCcX5gbxZ+RzXJ0SQvbrt9Mcnv7rDdB6vqu0leTvJnrbXndnqwqjqV5NQIx8X+GKWF7R0cP348P/zhD5dxrPuqqvb7EJbphiyhhdntkQ+VJXt92+fOD+vNmrBHrbX9PoSl+MpXvpKPfvSj1gTy0ksvJSOdH3QwXWMMqjudId68wn47yXtaa5eq6r4kX01yYqcHa609kuSRJKmqaa7U6+W6WtjewcbGhg6mYeEWrAmT4PzAwJqwZq4ygFsT1tCYLehgusa49Pdiklu23T6Wzb/1uKK19j+ttUuzz59I8paqummEfdMfLZBs/i2pFkg2310f6GC9WRPW3LFjxxJrAtECezPGoPpMkhNVdVtV3ZDkwSRntm9QVe+q2bU5VXXXbL8/GWHf9EcLJMnPogU2vVUHzFgT1tydd96ZWBOIFtibhS/9ba29UVUPJflakkNJHm2tPVdVn5rd/3CSjyT5dFW9keQXSR5sU/0BDLTAQAskyY+iAzZZE9bc4cOHE2sC0QJ7Uz3/93ad+cHTWhv9t1psbGy0c+fOjf2w+26KvwBkm/OttY2xH9SacCBpgSTLOT9MtYOevzdbVFVZE+aghbkfc5Iv2Lp2MMalvwAAADAagyoAAABdMagCAADQFYMqAAAAXTGoAgAA0BWDKgAAAF0xqAIAANAVgyoAAABdMagCAADQFYMqAAAAXTGoAgAA0BWDKgAAAF0xqAIAANAVgyoAAABdMagCAADQFYMqAAAAXTGoAgAA0BWDKgAAAF0ZZVCtqker6sdV9exV7q+q+lxVXaiq71XV+8fYL/3RATO3aoEZLZDE+YErrAkMtMCuxnpH9bEk9+xy/71JTsw+TiX5wkj7pT86IElejRbYpAUGOiCxJrBFC+xqlEG1tfZ0kp/usskDSb7YNn0rydur6t1j7Jvu6IAkuRQtsEkLDHRAYk1gixbY1ap+RvVokhe33b44+xrrRQcMtMBACyQ6YIsWGGhhzR1e0X5qh6+1HTesOpXNt/eZnuvq4Pjx48s8JvaHNYGBFkh0wBYtMNhTCzqYrlW9o3oxyS3bbh9L8vJOG7bWHmmtbbTWNlZyZKzSdXVw5MiRlRwcK2VNYKAFEh2wRQsM9tSCDqZrVYPqmSQfn/32rg8kea219sqK9k0/dMBACwy0QKIDtmiBgRbW3CiX/lbVl5LcneSmqrqY5LNJ3pIkrbWHkzyR5L4kF5L8PMknx9gvXfpmdEByW7TAJi0w0AGJNYEtWmBX1dqOl/13oar6PTh21Frb6ecJFrKxsdHOnTs39sPuu6rRX6qenF/GJTjWhANJCyRZzvlhqh30/L3ZoqrKmjAHLcz9mJN8wda1g1Vd+gsAAAB7YlAFAACgKwZVAAAAumJQBQAAoCsGVQAAALpiUAUAAKArBlUAAAC6YlAFAACgKwZVAAAAumJQBQAAoCsGVQAAALpiUAUAAKArBlUAAAC6YlAFAACgKwZVAAAAumJQBQAAoCsGVQAAALpiUAUAAKArowyqVfVoVf24qp69yv13V9VrVfWd2cdnxtgv/dEBM7dqgRktkMT5gSusCQy0wK4Oj/Q4jyX56yRf3GWbb7TWPjzS/ujXPdEByatJ/iRaQAtscX4gsSawRQvsapR3VFtrTyf56RiPxYGnA5LkUrTAJi0w0AGJNYEtWmBXq/wZ1Q9W1Xer6h+r6rdWuF/6ogMGWmCgBRIdsEULDLSwxsa69Pdavp3kPa21S1V1X5KvJjmx04ZVdSrJqdnNS0meX8kRJjdl8xKEqVnl83rPNe6/7g6qSgeLm0QLsSaMQQvzmWoLOpjPyl6vqlrFbrbTwny0MI5RWtjHDpIVvV7r2kG11kbZQ1XdmuTvW2vv28O2P0iy0Vrr5sRfVedaaxv7fRxjW/Xz0kG/tDAfLYy6v1ujhe7oYD5T7SDRwry0MOr+bo0WutPL81rJpb9V9a6a/VVAVd012+9PVrFv+qEDBlpgoAUSHbBFCwy0wCiX/lbVl5LcneSmqrqY5LNJ3pIkrbWHk3wkyaer6o0kv0jyYBvrrVy6oQMGWmCgBRIdsEULDLTAtYx26e9BV1WnWmuP7PdxjG2qz2tZpvx6Tfm5LcOUX68pP7dlmOrrNdXntSxTfr2m/NyWYcqv15Sf2zJM9fXq5XkZVAEAAOjKKv/3NAAAAHBNaz+oVtU9VfV8VV2oqtP7fTxjqapHq+rHVfXsfh/LQaEFBlNsQQfzm2IHiRauhxYYTLEFHcxvih0k/bWw1oNqVR1K8vkk9ya5PcnHqur2/T2q0TyW5J79PoiDQgsMJtzCY9HBnk24g0QLc9ECgwm38Fh0sGcT7iDprIW1HlST3JXkQmvt+62115N8OckD+3xMo2itPZ3kp/t9HAeIFhhMsgUdzG2SHSRauA5aYDDJFnQwt0l2kPTXwroPqkeTvLjt9sXZ11g/WmCgBRIdsEULDLRAooOVWfdBtXb4ml+DvJ60wEALJDpgixYYaIFEByuz7oPqxSS3bLt9LMnL+3Qs7C8tMNACiQ7YogUGWiDRwcqs+6D6TJITVXVbVd2Q5MEkZ/b5mNgfWmCgBRIdsEULDLRAooOVWetBtbX2RpKHknwtyb8n+dvW2nP7e1TjqKovJflmkpNVdbGq/nS/j6lnWmAw1RZ0MJ+pdpBoYV5aYDDVFnQwn6l2kPTXQrXmkmoAAAD6sdbvqAIAANAfgyoAAABdMagCAADQFYMqAAAAXTGoAgAA0BWDKgAAAF0xqAIAANAVgyoAAABd+T89XtOTyy31ugAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ " s d h s d v v\n" ] } ], "source": [ "draw_examples(x4_train[:7], captions=y4_train)" ] }, { "cell_type": "code", "execution_count": 69, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "x4_train = x4_train.reshape(trainset_size, 4)\n", "x4_test = x4_test.reshape(testset_size, 4)\n", "x4_train = x4_train.astype('float32')\n", "x4_test = x4_test.astype('float32')\n", "\n", "y4_train = np.array([{'s': 0, 'v': 1, 'd': 2, 'h': 3}[desc] for desc in y4_train])\n", "y4_test = np.array([{'s': 0, 'v': 1, 'd': 2, 'h': 3}[desc] for desc in y4_test])\n", "\n", "y4_train = keras.utils.to_categorical(y4_train, num_classes)\n", "y4_test = keras.utils.to_categorical(y4_test, num_classes)" ] }, { "cell_type": "code", "execution_count": 70, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model: \"sequential_24\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "dense_71 (Dense) (None, 4) 20 \n", "_________________________________________________________________\n", "dense_72 (Dense) (None, 4) 20 \n", "_________________________________________________________________\n", "dense_73 (Dense) (None, 8) 40 \n", "_________________________________________________________________\n", "dense_74 (Dense) (None, 4) 36 \n", "=================================================================\n", "Total params: 116\n", "Trainable params: 116\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] } ], "source": [ "model4 = keras.Sequential()\n", "model4.add(Dense(4, activation='tanh', input_shape=(4,)))\n", "model4.add(Dense(4, activation='tanh'))\n", "model4.add(Dense(8, activation='relu'))\n", "model4.add(Dense(num_classes, activation='softmax'))\n", "model4.summary()" ] }, { "cell_type": "code", "execution_count": 71, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "model4.layers[0].set_weights(\n", " [np.array([[ 1.0, 0.0, 1.0, 0.0],\n", " [ 0.0, 1.0, 0.0, 1.0],\n", " [ 1.0, 0.0, -1.0, 0.0],\n", " [ 0.0, 1.0, 0.0, -1.0]],\n", " dtype=np.float32), np.array([0., 0., 0., 0.], dtype=np.float32)])\n", "model4.layers[1].set_weights(\n", " [np.array([[ 1.0, -1.0, 0.0, 0.0],\n", " [ 1.0, 1.0, 0.0, 0.0],\n", " [ 0.0, 0.0, 1.0, -1.0],\n", " [ 0.0, 0.0, -1.0, -1.0]],\n", " dtype=np.float32), np.array([0., 0., 0., 0.], dtype=np.float32)])\n", "model4.layers[2].set_weights(\n", " [np.array([[ 1.0, -1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],\n", " [ 0.0, 0.0, 1.0, -1.0, 0.0, 0.0, 0.0, 0.0],\n", " [ 0.0, 0.0, 0.0, 0.0, 1.0, -1.0, 0.0, 0.0],\n", " [ 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, -1.0]],\n", " dtype=np.float32), np.array([0., 0., 0., 0., 0., 0., 0., 0.], dtype=np.float32)])" ] }, { "cell_type": "code", "execution_count": 73, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "model4.layers[3].set_weights(\n", " [np.array([[ 1.0, 0.0, 0.0, 0.0],\n", " [ 1.0, 0.0, 0.0, 0.0],\n", " [ 0.0, 1.0, 0.0, 0.0],\n", " [ 0.0, 1.0, 0.0, 0.0],\n", " [ 0.0, 0.0, 1.0, 0.0],\n", " [ 0.0, 0.0, 1.0, 0.0],\n", " [ 0.0, 0.0, 0.0, 1.0],\n", " [ 0.0, 0.0, 0.0, 1.0]],\n", " dtype=np.float32), np.array([0., 0., 0., 0.], dtype=np.float32)])\n", "\n", "model4.compile(loss='categorical_crossentropy',\n", " optimizer=keras.optimizers.Adagrad(),\n", " metrics=['accuracy'])" ] }, { "cell_type": "code", "execution_count": 74, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[array([[ 1., 0., 1., 0.],\n", " [ 0., 1., 0., 1.],\n", " [ 1., 0., -1., 0.],\n", " [ 0., 1., 0., -1.]], dtype=float32), array([0., 0., 0., 0.], dtype=float32)]\n", "[array([[ 1., -1., 0., 0.],\n", " [ 1., 1., 0., 0.],\n", " [ 0., 0., 1., -1.],\n", " [ 0., 0., -1., -1.]], dtype=float32), array([0., 0., 0., 0.], dtype=float32)]\n", "[array([[ 1., -1., 0., 0., 0., 0., 0., 0.],\n", " [ 0., 0., 1., -1., 0., 0., 0., 0.],\n", " [ 0., 0., 0., 0., 1., -1., 0., 0.],\n", " [ 0., 0., 0., 0., 0., 0., 1., -1.]], dtype=float32), array([0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32)]\n", "[array([[1., 0., 0., 0.],\n", " [1., 0., 0., 0.],\n", " [0., 1., 0., 0.],\n", " [0., 1., 0., 0.],\n", " [0., 0., 1., 0.],\n", " [0., 0., 1., 0.],\n", " [0., 0., 0., 1.],\n", " [0., 0., 0., 1.]], dtype=float32), array([0., 0., 0., 0.], dtype=float32)]\n" ] } ], "source": [ "for layer in model4.layers:\n", " print(layer.get_weights())" ] }, { "cell_type": "code", "execution_count": 75, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "data": { "text/plain": [ "array([[0.17831734, 0.17831734, 0.17831734, 0.465048 ]], dtype=float32)" ] }, "execution_count": 75, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model4.predict([np.array([[1.0, 1.0], [-1.0, -1.0]]).reshape(1, 4)])" ] }, { "cell_type": "code", "execution_count": 76, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Test loss: 0.7656148672103882\n", "Test accuracy: 1.0\n" ] } ], "source": [ "score = model4.evaluate(x4_test, y4_test, verbose=0)\n", "\n", "print('Test loss: {}'.format(score[0]))\n", "print('Test accuracy: {}'.format(score[1]))" ] }, { "cell_type": "code", "execution_count": 77, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model: \"sequential_25\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "dense_75 (Dense) (None, 4) 20 \n", "_________________________________________________________________\n", "dense_76 (Dense) (None, 4) 20 \n", "_________________________________________________________________\n", "dense_77 (Dense) (None, 8) 40 \n", "_________________________________________________________________\n", "dense_78 (Dense) (None, 4) 36 \n", "=================================================================\n", "Total params: 116\n", "Trainable params: 116\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] } ], "source": [ "model5 = Sequential()\n", "model5.add(Dense(4, activation='tanh', input_shape=(4,)))\n", "model5.add(Dense(4, activation='tanh'))\n", "model5.add(Dense(8, activation='relu'))\n", "model5.add(Dense(num_classes, activation='softmax'))\n", "model5.compile(loss='categorical_crossentropy',\n", " optimizer=keras.optimizers.RMSprop(),\n", " metrics=['accuracy'])\n", "model5.summary()" ] }, { "cell_type": "code", "execution_count": 78, "metadata": { "scrolled": true, "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 1/8\n", "125/125 [==============================] - 0s 3ms/step - loss: 1.3126 - accuracy: 0.3840 - val_loss: 1.1926 - val_accuracy: 0.6110\n", "Epoch 2/8\n", "125/125 [==============================] - 0s 2ms/step - loss: 1.0978 - accuracy: 0.5980 - val_loss: 1.0085 - val_accuracy: 0.6150\n", "Epoch 3/8\n", "125/125 [==============================] - 0s 2ms/step - loss: 0.9243 - accuracy: 0.7035 - val_loss: 0.8416 - val_accuracy: 0.7380\n", "Epoch 4/8\n", "125/125 [==============================] - 0s 2ms/step - loss: 0.7522 - accuracy: 0.8740 - val_loss: 0.6738 - val_accuracy: 1.0000\n", "Epoch 5/8\n", "125/125 [==============================] - 0s 2ms/step - loss: 0.5811 - accuracy: 1.0000 - val_loss: 0.5030 - val_accuracy: 1.0000\n", "Epoch 6/8\n", "125/125 [==============================] - 0s 2ms/step - loss: 0.4134 - accuracy: 1.0000 - val_loss: 0.3428 - val_accuracy: 1.0000\n", "Epoch 7/8\n", "125/125 [==============================] - 0s 2ms/step - loss: 0.2713 - accuracy: 1.0000 - val_loss: 0.2161 - val_accuracy: 1.0000\n", "Epoch 8/8\n", "125/125 [==============================] - 0s 1ms/step - loss: 0.1621 - accuracy: 1.0000 - val_loss: 0.1225 - val_accuracy: 1.0000\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 78, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model5.fit(x4_train, y4_train, epochs=8, validation_data=(x4_test, y4_test))" ] }, { "cell_type": "code", "execution_count": 79, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "data": { "text/plain": [ "array([[3.2040708e-02, 1.0065207e-03, 4.9596769e-04, 9.6645677e-01]],\n", " dtype=float32)" ] }, "execution_count": 79, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model5.predict([np.array([[1.0, 1.0], [-1.0, -1.0]]).reshape(1, 4)])" ] }, { "cell_type": "code", "execution_count": 80, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Test loss: 0.1224619448184967\n", "Test accuracy: 1.0\n" ] } ], "source": [ "score = model5.evaluate(x4_test, y4_test, verbose=0)\n", "\n", "print('Test loss: {}'.format(score[0]))\n", "print('Test accuracy: {}'.format(score[1]))" ] }, { "cell_type": "code", "execution_count": 81, "metadata": { "slideshow": { "slide_type": "notes" } }, "outputs": [], "source": [ "import contextlib\n", "\n", "@contextlib.contextmanager\n", "def printoptions(*args, **kwargs):\n", " original = np.get_printoptions()\n", " np.set_printoptions(*args, **kwargs)\n", " try:\n", " yield\n", " finally: \n", " np.set_printoptions(**original)" ] }, { "cell_type": "code", "execution_count": 82, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[array([[ 0.7, 0.2, -0.7, 0.7],\n", " [-0.5, 0.9, 0.6, 0.6],\n", " [ 1.1, 0.2, 0.1, 0.2],\n", " [ 0.7, 0.1, 0.3, -0.7]], dtype=float32), array([ 0. , 0.1, -0.1, -0.2], dtype=float32)]\n", "[array([[ 0.7, 0.5, -1.1, -1.2],\n", " [ 0.7, 0.9, -0.6, 0.3],\n", " [ 0.1, 1.4, -0.6, 0.8],\n", " [ 1.5, 0.1, -0.1, 0.9]], dtype=float32), array([-0.4, 0.2, -0. , 0.2], dtype=float32)]\n", "[array([[-1. , 1. , -0.7, -0.3, 0.2, 1.3, -0.7, 0.9],\n", " [-0.9, 0.5, 0.8, -1.3, -1.2, 1.3, 0.4, -1. ],\n", " [ 0.9, 0.2, 0.3, 0.4, 1.3, -0.9, -0.1, -0.2],\n", " [-0.4, 0.5, 1.1, -0.6, 1.1, 0.1, -1.5, -1. ]], dtype=float32), array([-0.1, 0.1, 0.1, 0.1, 0.2, -0. , 0.1, 0.2], dtype=float32)]\n", "[array([[ 0.7, -0.5, 0.8, -0.5],\n", " [-0.3, -1.6, -0.2, 0.1],\n", " [-1.5, 0.9, 0.1, -0.5],\n", " [ 0.6, 0.7, 1. , -1.4],\n", " [ 0.7, -1.2, -1.6, 1.2],\n", " [ 1. , -1.2, 0.3, -1.5],\n", " [-0.2, 0. , 0.6, 1.3],\n", " [-0.8, 0.2, -0.6, -1. ]], dtype=float32), array([-0.6, 0.5, -0.3, 0.4], dtype=float32)]\n" ] } ], "source": [ "with printoptions(precision=1, suppress=True):\n", " for layer in model5.layers:\n", " print(layer.get_weights())" ] } ], "metadata": { "celltoolbar": "Slideshow", "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.3" }, "livereveal": { "start_slideshow_at": "selected", "theme": "white" } }, "nbformat": 4, "nbformat_minor": 4 }