umz21/wyk/12_RNN.ipynb

254 lines
34 KiB
Plaintext
Raw Normal View History

{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
2021-04-14 08:03:54 +02:00
"## Uczenie maszynowe zastosowania\n",
"# 12. Rekurencyjne sieci neuronowe"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## RNN _Recurrent Neural Network_\n",
"\n",
"## LSTM _Long Short Term Memory_"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAUDBAcJCAgJCAgKCAgGBwgIBwcJCQYICAgICAgHCAkI\nCQgIChANCAkQCggJDiEODhERHx8TCAsiJCISJBwSExIBBQUFCAcIDwkJDx4NDw0SEhISEhUeEhIS\nEhISEhISEhISEhUVEhISEhISEhISEhISFRUSEhUSEhUSEhISEhUSEv/AABEIAWgB4AMBIgACEQED\nEQH/xAAcAAEAAgMBAQEAAAAAAAAAAAAABQYCBAcDAQj/xABKEAACAgEDAgIHBQUFBgQEBwABAgAD\nBAUREhMhBhQWIjFRU5LTBxUjQZEyYZXU1SQzQlW1VHF1gZShQ0RidDRSsfAIJWRlcpOl/8QAGgEB\nAAMBAQEAAAAAAAAAAAAAAAECAwQFBv/EADQRAAIAAggEBgEFAQADAQAAAAABAhEDBBIhMVFhkRVB\nUvATFHGBocEiMrHR4fFCcqLSBf/aAAwDAQACEQMRAD8A/GUREAREQBERAEREAREQBERAEREAREQB\nERAEREAREQBERAEREAREQBERAEREAREQBERAEREAREQBERAEREAREQBERAEREAREQBERAEREAREQ\nBERAEREAREQBERAEREAREQBERAEREAREQBERAEREAREQBERAEREAREQBERAEREAREQBERAEREARE\nQBERAEREAREQBERAEREAREQBERAEREAREQBERAPu0bS1+guX8Sn5rvpx6C5fxKfmu+nO3h1Y6Gcv\nnqDrRVNo2lr9Bcv4lPzXfTj0Fy/iU/Nd9OOHVjoY89QdaKptG0tfoLl/Ep+a76ceguX8Sn5rvpxw\n6sdDHnqDrRVNo2lr9Bcv4lPzXfTj0Fy/iU/Nd9OOHVjoY89QdaKptG0tfoLl/Ep+a76ceguX8Sn5\nrvpxw6sdDHnqDrRVNo2lr9Bcv4lPzXfTj0Fy/iU/Nd9OOHVjoY89QdaKptG0tfoLl/Ep+a76cegu\nX8Sn5rvpxw6sdDHnqDrRVNo2lr9Bcv4lPzXfTj0Fy/iU/Nd9OOHVjoY89QdaKptG0tfoLl/Ep+a7\n6ceguX8Sn5rvpxw6sdDHnqDrRVNo2lr9Bcv4lPzXfTj0Fy/iU/Nd9OOHVjoY89QdaKptG0tfoLl/\nEp+a76ceguX8Sn5rvpxw6sdDHnqDrRVNo2lr9Bcv4lPzXfTj0Fy/iU/Nd9OOHVjoY89QdaKptG0t\nfoLl/Ep+a76ceguX8Sn5rvpxw6sdDHnqDrRVNo2lr9Bcv4lPzXfTj0Fy/iU/Nd9OOHVjoY89QdaK\nptG0tfoLl/Ep+a76ceguX8Sn5rvpxw6sdDHnqDrRVNo2lr9Bcv4lPzXfTj0Fy/iU/Nd9OOHVjoY8\n9QdaKptG0tfoLl/Ep+a76ceguX8Sn5rvpxw6sdDHnqDrRVNo2lr9Bcv4lPzXfTj0Fy/iU/Nd9OOH\nVjoY89QdaKptG0tfoLl/Ep+a76ceguX8Sn5rvpxw6sdDHnqDrRVNo2lr9Bcv4lPzXfTj0Fy/iU/N\nd9OOHVjoY89QdaKptG0tfoLl/Ep+a76ceguX8Sn5rvpxw6sdDHnqDrRVNo2lr9Bcv4lPzXfTj0Fy\n/iU/Nd9OOHVjoY89QdaKptG0tfoLl/Ep+a76ceguX8Sn5rvpxw6sdDHnqDrRVNo2lr9Bcv4lPzXf\nTj0Fy/iU/Nd9OOHVjoY89QdaKptG0tfoLl/Ep+a76ceguX8Sn5rvpxw6sdDHnqDrRVNo2lr9Bcv4\nlPzXfTj0Fy/iU/Nd9OOHVjoY89QdaKptG0tfoLl/Ep+a76ceguX8Sn5rvpxw6sdDHnqDrRVNo2lr\n9Bcv4lPzXfTj0Fy/iU/Nd9OOHVjoY89QdaKptG0tfoLl/Ep+a76ceguX8Sn5rvpxw6sdDHnqDrRV\nNo2lr9Bcv4lPzXfTj0Fy/iU/Nd9OOHVjoY89QdaOhxET7s+HES3/AGN6bj5WuYePlVLfRbXqBelx\nujGrTM65CR+6ytWH71E1/D/gnIyMOnMuzMDTsbMd6sSzUcp8fzbUkJc1K11WE1o5Cl2CqCdt5jHW\nIYInC9Pmf8M2hoIooU1r8S/krES4YP2baq+TqWKwx8a3Q66rc85OQlNSUWugGQt2xRqhW4tLbj1B\n2DHZT4a14Ezsc4HRfG1GvWrWowb9PuN9NuSliVvjk2V1mq0NYnZ1A2JO/Y7QqzRtytLtT/a8OgpJ\nTsvty/e4q0S85P2Y5y15z1Z+mZJ0fGvyNQpxsy227GXHUtYjo1C833DJuhYBlIJXtvRpajpoaT9L\nKx0UUH6kIiJqZiIiAIiIAiIgCIiAIiIAiIgCIiAIiIAiIgCIiAIiIAiIgCIiAIiIAiIgCIiAIiIJ\nPhMlPE+jPg5Jx3uoyCKca4X41tV9DLk49WQAllZIbbqcd/z47jsQTGSU8U/39f8Aw3SP9J0+Ztu0\nlo/oskrLeq+yLiImhQREQSIiIIEREAREQBERAEREAREQC4/Yrn0Y2u4V2TclFNdeoB7bXrrrU2aX\nn1IGdyACzuij3lgPzklk41GraVoi06hgYWTo2NdgZmLn5deCoDZD3VZdDWja9GRjyCbsGUDZvbOe\nT4Zy0lWtx205O74tf/R00dYlBYamr/mX8HYfEXibTrKfEFNGWtiUeHtE0rDvdum+ovgZNAutqrfZ\nn39c7DfdU39hmr9n3irEwMLw67st1mB4lzbsnEQh8lMa/Eoo64oXdyAGZh27msgTlAP/AHmzpmbf\njX05GPYar8W1LqbV48q7K2DKwDAg7EDsQQfYQQZk6jDYcPf6bJoq5FbT9v8A2mdW8FeGMXDr8SWV\n6xg54fwrrC4tWJbbdfZjscb8fJXpgYjD8IGtzuXt7bhSTyGWzW/tAzsijIoXHwcJdQI+8LcHDpxb\ns0B+pxvtUklTZ6xVOO5333BIlTmlVo44ZxR4uXx7IpWKSByUGCn87iIidZyiIiAIiIAiIgCIiAIi\nbOLihgtllgrpNnBnBqstHqGz8PG6ivZ2AAbsu7oCV33kNyJSNaemNRZYQtVbWMzKoFaO5LNuFUBQ\nSWOx2H57Gevma1TilK8npCWPaVuPPluz0jgq07jYdw5Gx2O5nzMz77eXVtewWPzZWZuBcKEDdP8A\nZBCKqjYdlVQNgAJE2Tceq6RmkKwxMgrZb0UIoyCHuDFDSpC7NZyVhxHfdSNu0JpGaxULiZDGxrVQ\nCjIJdqN+sqgJ6zJsdwPZsd9pocR7h+kcR7h+kX9/6TNG592ZXHn5a7h0PM8+jfx8vvt5jlx26O//\nAIns/fM30jNUsGxMhTW1SuDRkAo1+3RVgU9Vn3GwPt3G280OI9w9/sjiPcP0i/v/AETRILo2cSFG\nJklmuagKMfI5G9FLPQBw3NoUElPaACdpgmlZZAYYt5V67LVIpvIaqk8bbVIXuiEgFh2G/faaXEe4\nfpHEe4fpF/f+kTWXexuvpWWAWOLeFSuu1iabwFquPGq1iV7I5BAY9jt23mbaNnAlTiZIZbloKnHy\nOQvdQyUEcNxaVIIT2kEHaR/Ee4fpHEe4fpF/f+iay72N9NIzWKhcTIY2NaqAUZBLtRv1lUBPWZNj\nuB7NjvtMPuzK48/LXcOh5nn0b+Pl99vMcuO3R3/8T2fvmnxHuH6RxHuHv9kX9/6TNG++kZqlg2Jk\nKa2qVwaMgFGv26KsCnqs+42B9u423n1dGziQoxMks1zUBRj5HI3opZ6AOG5tCgkp7QATtI/iPcP0\njiPcP0iT7/0iay72N1NKyyAwxbyr12WqRTeQ1VJ422qQvdEJALDsN++0PpWWAWOLeFSuu1iabwFq\nuPGq1iV7I5BAY9jt23mlxHuH6RxHuH6Sb+/9FxINo2cCVOJkhluWgqcfI5C91DJQRw3FpUghPaQQ\ndp8TSM1ioXEyGNjWqgFGQS7Ub9ZVAT1mTY7gezY77TQ4j3D9I4j3D9JEn3/omsu9jc+7Mrjz8tdw\n6HmefRv4+X328xy47d
"text/html": [
"\n",
" <iframe\n",
" width=\"800\"\n",
" height=\"600\"\n",
" src=\"https://www.youtube.com/embed/WCUNPb-5EYI\"\n",
" frameborder=\"0\"\n",
" allowfullscreen\n",
" ></iframe>\n",
" "
],
"text/plain": [
"<IPython.lib.display.YouTubeVideo at 0x7f9868e6cf50>"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import IPython\n",
"IPython.display.YouTubeVideo('WCUNPb-5EYI', width=800, height=600)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Rekurencyjna sieć neuronowa schemat\n",
"\n",
"<img style=\"margin: auto\" width=\"20%\" src=\"http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/RNN-rolled.png\"/>"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Rekurencyjna sieć neuronowa schemat\n",
"\n",
"<img style=\"margin: auto\" width=\"80%\" src=\"http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/RNN-unrolled.png\"/>"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Zależności długodystansowe (_long-distance dependencies_) w sieciach rekurencyjnych\n",
"\n",
"<img style=\"margin: auto\" width=\"60%\" src=\"http://colah.github.io/posts/2015-08-Understanding-LSTMs/img/RNN-longtermdependencies.png\"/>"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### RNN typy sekwencji\n",
"\n",
"<img style=\"margin: auto\" width=\"80%\" src=\"http://karpathy.github.io/assets/rnn/diags.jpeg\"/>"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Prosta sieć RNN schemat\n",
"\n",
"<img src=\"rnn.png\" style=\"margin: auto;\" width=\"100%\" />"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### LSTM schemat\n",
"\n",
"<img src=\"lstm.jpg\" style=\"margin: auto;\" width=\"80%\" />\n",
"\n",
"* Rekurencyjne sieci neuronowe znajduja zastosowanie w przetwarzaniu sekwencji, np. szeregów czasowych i tekstów.\n",
"* LSTM są rozwinięciem RNN, umożliwiają „zapamiętywanie” i „zapominanie”."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Co potrafią generować rekurencyjne sieci neuronowe?\n",
"\n",
"http://karpathy.github.io/2015/05/21/rnn-effectiveness/"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Generowanie tekstu za pomocą LSTM przykład\n",
"\n",
"https://github.com/keras-team/keras/blob/master/examples/lstm_text_generation.py"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Przewidywanie ciągów czasowych za pomocą LSTM przykład\n",
"\n",
"https://machinelearningmastery.com/time-series-forecasting-long-short-term-memory-network-python/"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## GRU _Gated Recurrent Unit_\n",
"\n",
"* Rodzaj rekurencyjnej sieci neuronowej wprwadzony w 2014 roku\n",
"* Ma prostszą budowę niż LSTM (2 bramki zamiast 3).\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### GRU schemat\n",
"\n",
"<img src=\"gru.png\" style=\"margin: auto;\" width=\"50%\" />\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### GRU vs LSTM\n",
"* LSTM 3 bramki: wejścia (_input_), wyjścia (_output_) i zapomnienia (_forget_); GRU 2 bramki: resetu (_reset_) i aktualizacji (_update_). Bramka resetu pełni podwójną funkcję: zastępuje bramki wyjścia i zapomnienia.\n",
"* GRU i LSTM mają podobną skuteczność, ale GRU dzięki prostszej budowie bywa bardziej wydajna.\n",
"* LSTM sprawdza się lepiej w przetwarzaniu tekstu, ponieważ lepiej zapamiętuje zależności długosystansowe."
]
}
],
"metadata": {
"celltoolbar": "Slideshow",
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.3"
},
"livereveal": {
"start_slideshow_at": "selected",
2021-04-14 08:03:54 +02:00
"theme": "white"
}
},
"nbformat": 4,
"nbformat_minor": 4
}