umz21/wyk/12_RNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Uczenie maszynowe – zastosowania\n",
    "# 12. Rekurencyjne sieci neuronowe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## RNN – _Recurrent Neural Network_\n",
    "\n",
    "## LSTM – _Long Short Term Memory_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAUDBAcJCAgJCAgKCAgGBwgIBwcJCQYICAgICAgHCAkI\nCQgIChANCAkQCggJDiEODhERHx8TCAsiJCISJBwSExIBBQUFCAcIDwkJDx4NDw0SEhISEhUeEhIS\nEhISEhISEhISEhUVEhISEhISEhISEhISFRUSEhUSEhUSEhISEhUSEv/AABEIAWgB4AMBIgACEQED\nEQH/xAAcAAEAAgMBAQEAAAAAAAAAAAAABQYCBAcDAQj/xABKEAACAgEDAgIHBQUFBgQEBwABAgAD\nBAUREhMhBhQWIjFRU5LTBxUjQZEyYZXU1SQzQlW1VHF1gZShQ0RidDRSsfAIJWRlcpOl/8QAGgEB\nAAMBAQEAAAAAAAAAAAAAAAECAwQFBv/EADQRAAIAAggEBgEFAQADAQAAAAABAhEDBBIhMVFhkRVB\nUvATFHGBocEiMrHR4fFCcqLSBf/aAAwDAQACEQMRAD8A/GUREAREQBERAEREAREQBERAEREAREQB\nERAEREAREQBERAEREAREQBERAEREAREQBERAEREAREQBERAEREAREQBERAEREAREQBERAEREAREQ\nBERAEREAREQBERAEREAREQBERAEREAREQBERAEREAREQBERAEREAREQBERAEREAREQBERAEREARE\nQBERAEREAREQBERAEREAREQBERAEREAREQBERAPu0bS1+guX8Sn5rvpx6C5fxKfmu+nO3h1Y6Gcv\nnqDrRVNo2lr9Bcv4lPzXfTj0Fy/iU/Nd9OOHVjoY89QdaKptG0tfoLl/Ep+a76ceguX8Sn5rvpxw\n6sdDHnqDrRVNo2lr9Bcv4lPzXfTj0Fy/iU/Nd9OOHVjoY89QdaKptG0tfoLl/Ep+a76ceguX8Sn5\nrvpxw6sdDHnqDrRVNo2lr9Bcv4lPzXfTj0Fy/iU/Nd9OOHVjoY89QdaKptG0tfoLl/Ep+a76cegu\nX8Sn5rvpxw6sdDHnqDrRVNo2lr9Bcv4lPzXfTj0Fy/iU/Nd9OOHVjoY89QdaKptG0tfoLl/Ep+a7\n6ceguX8Sn5rvpxw6sdDHnqDrRVNo2lr9Bcv4lPzXfTj0Fy/iU/Nd9OOHVjoY89QdaKptG0tfoLl/\nEp+a76ceguX8Sn5rvpxw6sdDHnqDrRVNo2lr9Bcv4lPzXfTj0Fy/iU/Nd9OOHVjoY89QdaKptG0t\nfoLl/Ep+a76ceguX8Sn5rvpxw6sdDHnqDrRVNo2lr9Bcv4lPzXfTj0Fy/iU/Nd9OOHVjoY89QdaK\nptG0tfoLl/Ep+a76ceguX8Sn5rvpxw6sdDHnqDrRVNo2lr9Bcv4lPzXfTj0Fy/iU/Nd9OOHVjoY8\n9QdaKptG0tfoLl/Ep+a76ceguX8Sn5rvpxw6sdDHnqDrRVNo2lr9Bcv4lPzXfTj0Fy/iU/Nd9OOH\nVjoY89QdaKptG0tfoLl/Ep+a76ceguX8Sn5rvpxw6sdDHnqDrRVNo2lr9Bcv4lPzXfTj0Fy/iU/N\nd9OOHVjoY89QdaKptG0tfoLl/Ep+a76ceguX8Sn5rvpxw6sdDHnqDrRVNo2lr9Bcv4lPzXfTj0Fy\n/iU/Nd9OOHVjoY89QdaKptG0tfoLl/Ep+a76ceguX8Sn5rvpxw6sdDHnqDrRVNo2lr9Bcv4lPzXf\nTj0Fy/iU/Nd9OOHVjoY89QdaKptG0tfoLl/Ep+a76ceguX8Sn5rvpxw6sdDHnqDrRVNo2lr9Bcv4\nlPzXfTj0Fy/iU/Nd9OOHVjoY89QdaKptG0tfoLl/Ep+a76ceguX8Sn5rvpxw6sdDHnqDrRVNo2lr\n9Bcv4lPzXfTj0Fy/iU/Nd9OOHVjoY89QdaKptG0tfoLl/Ep+a76ceguX8Sn5rvpxw6sdDHnqDrRV\nNo2lr9Bcv4lPzXfTj0Fy/iU/Nd9OOHVjoY89QdaOhxET7s+HES3/AGN6bj5WuYePlVLfRbXqBelx\nujGrTM65CR+6ytWH71E1/D/gnIyMOnMuzMDTsbMd6sSzUcp8fzbUkJc1K11WE1o5Cl2CqCdt5jHW\nIYInC9Pmf8M2hoIooU1r8S/krES4YP2baq+TqWKwx8a3Q66rc85OQlNSUWugGQt2xRqhW4tLbj1B\n2DHZT4a14Ezsc4HRfG1GvWrWowb9PuN9NuSliVvjk2V1mq0NYnZ1A2JO/Y7QqzRtytLtT/a8OgpJ\nTsvty/e4q0S85P2Y5y15z1Z+mZJ0fGvyNQpxsy227GXHUtYjo1C833DJuhYBlIJXtvRpajpoaT9L\nKx0UUH6kIiJqZiIiAIiIAiIgCIiAIiIAiIgCIiAIiIAiIgCIiAIiIAiIgCIiAIiIAiIgCIiAIiIJ\nPhMlPE+jPg5Jx3uoyCKca4X41tV9DLk49WQAllZIbbqcd/z47jsQTGSU8U/39f8Aw3SP9J0+Ztu0\nlo/oskrLeq+yLiImhQREQSIiIIEREAREQBERAEREAREQC4/Yrn0Y2u4V2TclFNdeoB7bXrrrU2aX\nn1IGdyACzuij3lgPzklk41GraVoi06hgYWTo2NdgZmLn5deCoDZD3VZdDWja9GRjyCbsGUDZvbOe\nT4Zy0lWtx205O74tf/R00dYlBYamr/mX8HYfEXibTrKfEFNGWtiUeHtE0rDvdum+ovgZNAutqrfZ\nn39c7DfdU39hmr9n3irEwMLw67st1mB4lzbsnEQh8lMa/Eoo64oXdyAGZh27msgTlAP/AHmzpmbf\njX05GPYar8W1LqbV48q7K2DKwDAg7EDsQQfYQQZk6jDYcPf6bJoq5FbT9v8A2mdW8FeGMXDr8SWV\n6xg54fwrrC4tWJbbdfZjscb8fJXpgYjD8IGtzuXt7bhSTyGWzW/tAzsijIoXHwcJdQI+8LcHDpxb\ns0B+pxvtUklTZ6xVOO5333BIlTmlVo44ZxR4uXx7IpWKSByUGCn87iIidZyiIiAIiIAiIgCIiAIi\nbOLihgtllgrpNnBnBqstHqGz8PG6ivZ2AAbsu7oCV33kNyJSNaemNRZYQtVbWMzKoFaO5LNuFUBQ\nSWOx2H57Gevma1TilK8npCWPaVuPPluz0jgq07jYdw5Gx2O5nzMz77eXVtewWPzZWZuBcKEDdP8A\nZBCKqjYdlVQNgAJE2Tceq6RmkKwxMgrZb0UIoyCHuDFDSpC7NZyVhxHfdSNu0JpGaxULiZDGxrVQ\nCjIJdqN+sqgJ6zJsdwPZsd9pocR7h+kcR7h+kX9/6TNG592ZXHn5a7h0PM8+jfx8vvt5jlx26O//\nAIns/fM30jNUsGxMhTW1SuDRkAo1+3RVgU9Vn3GwPt3G280OI9w9/sjiPcP0i/v/AETRILo2cSFG\nJklmuagKMfI5G9FLPQBw3NoUElPaACdpgmlZZAYYt5V67LVIpvIaqk8bbVIXuiEgFh2G/faaXEe4\nfpHEe4fpF/f+kTWXexuvpWWAWOLeFSuu1iabwFquPGq1iV7I5BAY9jt23mbaNnAlTiZIZbloKnHy\nOQvdQyUEcNxaVIIT2kEHaR/Ee4fpHEe4fpF/f+iay72N9NIzWKhcTIY2NaqAUZBLtRv1lUBPWZNj\nuB7NjvtMPuzK48/LXcOh5nn0b+Pl99vMcuO3R3/8T2fvmnxHuH6RxHuHv9kX9/6TNG++kZqlg2Jk\nKa2qVwaMgFGv26KsCnqs+42B9u423n1dGziQoxMks1zUBRj5HI3opZ6AOG5tCgkp7QATtI/iPcP0\njiPcP0iT7/0iay72N1NKyyAwxbyr12WqRTeQ1VJ422qQvdEJALDsN++0PpWWAWOLeFSuu1iabwFq\nuPGq1iV7I5BAY9jt23mlxHuH6RxHuH6Sb+/9FxINo2cCVOJkhluWgqcfI5C91DJQRw3FpUghPaQQ\ndp8TSM1ioXEyGNjWqgFGQS7Ub9ZVAT1mTY7gezY77TQ4j3D9I4j3D9JEn3/omsu9jc+7Mrjz8tdw\n6HmefRv4+X328xy47d
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"https://www.youtube.com/embed/WCUNPb-5EYI\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.YouTubeVideo at 0x7f9868e6cf50>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import IPython\n",
    "IPython.display.YouTubeVideo('WCUNPb-5EYI', width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Rekurencyjna sieć neuronowa – schemat\n",
    "\n",
    "<img style=\"margin: auto\" width=\"20%\" src=\"http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/RNN-rolled.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Rekurencyjna sieć neuronowa – schemat\n",
    "\n",
    "<img style=\"margin: auto\" width=\"80%\" src=\"http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/RNN-unrolled.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Zależności długodystansowe (_long-distance dependencies_) w sieciach rekurencyjnych\n",
    "\n",
    "<img style=\"margin: auto\" width=\"60%\" src=\"http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/RNN-longtermdependencies.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### RNN – typy sekwencji\n",
    "\n",
    "<img style=\"margin: auto\" width=\"80%\" src=\"http://karpathy.github.io/assets/rnn/diags.jpeg\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Prosta sieć RNN – schemat\n",
    "\n",
    "<img src=\"rnn.png\" style=\"margin: auto;\" width=\"100%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### LSTM – schemat\n",
    "\n",
    "<img src=\"lstm.jpg\" style=\"margin: auto;\" width=\"80%\" />\n",
    "\n",
    "* Rekurencyjne sieci neuronowe znajduja zastosowanie w przetwarzaniu sekwencji, np. szeregów czasowych i tekstów.\n",
    "* LSTM są rozwinięciem RNN, umożliwiają „zapamiętywanie” i „zapominanie”."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Co potrafią generować rekurencyjne sieci neuronowe?\n",
    "\n",
    "http://karpathy.github.io/2015/05/21/rnn-effectiveness/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Generowanie tekstu za pomocą LSTM – przykład\n",
    "\n",
    "https://github.com/keras-team/keras/blob/master/examples/lstm_text_generation.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Przewidywanie ciągów czasowych za pomocą LSTM – przykład\n",
    "\n",
    "https://machinelearningmastery.com/time-series-forecasting-long-short-term-memory-network-python/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## GRU – _Gated Recurrent Unit_\n",
    "\n",
    "* Rodzaj rekurencyjnej sieci neuronowej wprwadzony w 2014 roku\n",
    "* Ma prostszą budowę niż LSTM (2 bramki zamiast 3).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### GRU – schemat\n",
    "\n",
    "<img src=\"gru.png\" style=\"margin: auto;\" width=\"50%\" />\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### GRU vs LSTM\n",
    "* LSTM – 3 bramki: wejścia (_input_), wyjścia (_output_) i zapomnienia (_forget_); GRU – 2 bramki: resetu (_reset_) i aktualizacji (_update_). Bramka resetu pełni podwójną funkcję: zastępuje bramki wyjścia i zapomnienia.\n",
    "* GRU i LSTM mają podobną skuteczność, ale GRU dzięki prostszej budowie bywa bardziej wydajna.\n",
    "* LSTM sprawdza się lepiej w przetwarzaniu tekstu, ponieważ lepiej zapamiętuje zależności długosystansowe."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "livereveal": {
   "start_slideshow_at": "selected",
   "theme": "white"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}