271 lines
6.7 KiB
Plaintext
271 lines
6.7 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"\n",
|
|
"<div class=\"alert alert-block alert-info\">\n",
|
|
"<h1> Ekstrakcja informacji </h1>\n",
|
|
"<h2> 15. <i>Sieci Transformer i ich zastosowanie w ekstrakcji informacji</i> [wykład]</h2> \n",
|
|
"<h3> Filip Graliński (2021)</h3>\n",
|
|
"</div>\n",
|
|
"\n",
|
|
""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Modele Transformer\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Atencja\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Atencję w modelach Transformer można interpretować jako rodzaj\n",
|
|
"„miękkiego” odpytywania swego rodzaju bazy danych, w której\n",
|
|
"przechowywane są pary klucz-wartość. Mamy trzy rodzaje wektorów (a\n",
|
|
"właściwie macierzy, bo wektory są od razu upakowane w macierze):\n",
|
|
"\n",
|
|
"- $Q$ - macierz zapytań,\n",
|
|
"- $K$ - macierz kluczy,\n",
|
|
"- $V$ - macierz wartości odpowiadających kluczom $K$.\n",
|
|
"\n",
|
|
"W atencji modeli Transformer patrzymy jak bardzo zapytania $Q$ pasują\n",
|
|
"do kluczy $K$ i na tej podstawie zwracamy wartości $V$ (im bardziej\n",
|
|
"**klucz** pasuje do **zapytania**, tym większy wkład wnosi odpowiednia **wartość**).\n",
|
|
"Ten rodzaj odpytywania można zrealizować z pomocą mnożenia macierzy i funkcji softmax:\n",
|
|
"\n",
|
|
"$$\\operatorname{Atention}(Q,K,V) = \\operatorname{softmax}(QK^T)V$$\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Uproszczony przykład\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Załóżmy, że rozmiar embeddingu wynosi 4, w macierzach rozpatrywać\n",
|
|
"będziemy po 3 wektory naraz (możemy sobie wyobrazić, że zdanie zawiera 3 wyrazy).\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"tensor([[20.5700, 36.2400, 31.1000],\n",
|
|
" [15.1100, 13.9100, 7.9500],\n",
|
|
" [ 2.2100, 7.1800, 7.4000]])"
|
|
]
|
|
},
|
|
"execution_count": 1,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import torch\n",
|
|
"\n",
|
|
"Q = torch.tensor([\n",
|
|
" [0.3, -2.0, 0.4, 6.0],\n",
|
|
" [-1.0, 1.5, 0.2, 3.0],\n",
|
|
" [0.3, -1.0, 0.2, 1.0]])\n",
|
|
"\n",
|
|
"K = torch.tensor([\n",
|
|
" [-0.5, 1.7, 0.3, 4.0],\n",
|
|
" [0.4, -1.5, 0.3, 5.5],\n",
|
|
" [-1.0, -3.5, 1.0, 4.0]])\n",
|
|
"\n",
|
|
"M = Q @ torch.transpose(K, 0, 1)\n",
|
|
"M"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Jak widać, najbardziej pierwszy wektor $Q$ pasuje do drugiego wektora $K$.\n",
|
|
"Znormalizujmy te wartości używać funkcji softmax.\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"tensor([[1.5562e-07, 9.9418e-01, 5.8236e-03],\n",
|
|
" [7.6807e-01, 2.3134e-01, 5.9683e-04],\n",
|
|
" [3.0817e-03, 4.4385e-01, 5.5307e-01]])"
|
|
]
|
|
},
|
|
"execution_count": 2,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import torch\n",
|
|
"\n",
|
|
"Mn = torch.softmax(M, 1)\n",
|
|
"Mn"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Drugi wektor zapytania najbardziej pasuje do pierwszego klucza, trochę\n",
|
|
"mniej do drugiego klucza, o wiele mniej do trzeciego klucza. Te\n",
|
|
"wektory to oczywiście wektory atencji (drugie słowo najbardziej\n",
|
|
"„patrzy” na pierwsze słowo).\n",
|
|
"\n",
|
|
"Teraz będziemy przemnażać przez wektory wartości:\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"tensor([[ 3.9750e+00, 9.9419e-02, 1.0116e-01, 1.5765e-01, 5.8255e-04],\n",
|
|
" [ 9.2517e-01, 6.9357e+00, 2.3313e-02, -3.8112e+00, 9.2174e-01],\n",
|
|
" [ 1.6095e+00, 7.2120e-02, 2.1031e-01, 5.5597e+00, 5.9005e-02]])"
|
|
]
|
|
},
|
|
"execution_count": 3,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import torch\n",
|
|
"\n",
|
|
"V = torch.tensor([\n",
|
|
" [0.0, 9.0, 0.0, -5.0, 1.2],\n",
|
|
" [4.0, 0.1, 0.1, 0.1, 0.0],\n",
|
|
" [-0.3, 0.0, 0.3, 10.0, 0.1]])\n",
|
|
"\n",
|
|
"Mn @ V"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Dodatkowa normalizacja\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"W praktyce dobrze jest znormalizować pierwszy iloczyn przez\n",
|
|
"$\\sqrt{d_k}$, gdzie $d_k$ to rozmiar wektora klucza.\n",
|
|
"\n",
|
|
"$$\\operatorname{Atention}(Q,K,V) = \\operatorname{softmax}(\\frac{QK^T}{\\sqrt{d^k}})V$$\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Skąd się biorą Q, K i V?\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Wektory (macierze) $Q$, $K$ i $V$ w pierwszej warstwie pochodzą z\n",
|
|
"embeddingów tokenów $E$ (właściwie jednostek BPE).\n",
|
|
"\n",
|
|
"- $Q$ = $EW^Q$\n",
|
|
"- $K$ = $EW^K$\n",
|
|
"- $V$ = $EW^V$\n",
|
|
"\n",
|
|
"W kolejnych warstwach zamiast $E$ wykorzystywane jest wyjście z poprzedniej warstwy.\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Literatura\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"[https://arxiv.org/pdf/1706.03762.pdf](https://arxiv.org/pdf/1706.03762.pdf)\n",
|
|
"\n"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"author": "Filip Graliński",
|
|
"email": "filipg@amu.edu.pl",
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"lang": "pl",
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.9.6"
|
|
},
|
|
"org": null,
|
|
"subtitle": "15.Sieci Transformer i ich zastosowanie w ekstrakcji informacji[wykład]",
|
|
"title": "Ekstrakcja informacji",
|
|
"year": "2021"
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 4
|
|
}
|