forked from filipg/aitech-eks-pub
1225 lines
52 KiB
Plaintext
1225 lines
52 KiB
Plaintext
|
{
|
||
|
|
"cells": [
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"## Neurozoo\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"### Funkcja sigmoidalna\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Funkcja sigmoidalna zamienia dowolną wartość („sygnał”) w wartość z przedziału $(0,1)$, czyli wartość, która może być interperetowana jako prawdopodobieństwo.\n",
|
||
|
|
"\n",
|
||
|
|
"$$\\sigma(x) = \\frac{1}{1 + e^{-x}}$$\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 1,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"text/plain": [
|
||
|
|
"tensor(0.6457)"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"execution_count": 1,
|
||
|
|
"metadata": {},
|
||
|
|
"output_type": "execute_result"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"import torch\n",
|
||
|
|
"\n",
|
||
|
|
"def sigmoid(x):\n",
|
||
|
|
" return 1 / (1 + torch.exp(-x))\n",
|
||
|
|
"\n",
|
||
|
|
"sigmoid(torch.tensor(0.6))"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 2,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"text/plain": [
|
||
|
|
"'sigmoid.png'"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"execution_count": 2,
|
||
|
|
"metadata": {},
|
||
|
|
"output_type": "execute_result"
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAAhNUlEQVR4nO3deXyV5Z3+8c83+04gCVsIqyAguwHcWm2tDijuti5YFW39aWvHWtuqHaszdanLtFNn1PKjFpda12opVQpq6zYubMoWQiCGJWHLRvY9554/EpkMBgTMk+cs1/v1Oq/knOdJvI7Gcz3rfZtzDhERiVxRfgcQERF/qQhERCKcikBEJMKpCEREIpyKQEQkwsX4HeBIZWZmuuHDh/sdQ0QkpKxevbrcOZfV3bKQK4Lhw4ezatUqv2OIiIQUM9t+sGU6NCQiEuFUBCIiEU5FICIS4VQEIiIRzrMiMLOFZlZqZhsOstzM7D/NrNDM1pnZNK+yiIjIwXm5R/AkMOsQy2cDozsf1wG/9TCLiIgchGdF4Jx7F6g8xCrnAU+7Dh8B6WY2yKs8IiLSPT/vI8gGirs8L+l8bfeBK5rZdXTsNTB06NBeCSci0hucczS3BahpaqWuqY265rb9Xxta2qlvaaOhuePr8cP68pXR3d4T9qX4WQTWzWvdTo7gnFsALADIzc3VBAoiEpQCAce+hhbK61qoqG+moq6FfQ0tVNa3UNXQyr6Gjq/VjR2PmsZWapvaaGkPHNbvv+G0UWFXBCVATpfnQ4BdPmURETmkptZ2dlY1squqkd3VTeyuamJPTROlNU3srW2itKaZivoW2gPdb6umJcSQnhRH36RY0hJjGdI3kT6JHd+nJsSQmhBLWkIMyXExpCTEkBIfQ3J8DMlx0STFx5AYG010VHfbz1+en0WwGLjRzJ4HZgLVzrnPHRYSEekt1Y2tbC2vp6isjm0VDeyoqGd7ZQPFlY2U1zV/bv2M5DgGpCUwIC2e8YPSyEqNJyslnoyUeDJS4shMiadfchzpibHERAfv1fqeFYGZPQecBmSaWQlwFxAL4JybDywBzgIKgQZgnldZRES6qmlqpWBPLZt217B5bx1bSmspLK2jvK5l/zpRBoP6JDIsI4nTx/Ynu28iQ/omMjg9kcF9EumfFk9CbLSP76LneFYEzrnLvmC5A77v1T9fRAQ6tvLXlVSxrqSaDTurWb+zmpJ9jfuXpybEMLp/Cl8f259j+qcwIjOFEZnJDO2XRFxM8G7F96SQG31URORgnHNsr2hgxdZKVm6r5JPiKgpL6/YvH5aRxOScdC6fOZRxA9MYOyiVgWkJmHlz7D1UqAhEJKTtqmrkvwvL+aCwnA8+raC0tuNYft+kWKYN7cv5UwYzJacvE4f0oU9irM9pg5OKQERCSmt7gJXbKnm7oIy3NpWypXOLPzMljhNHZXLCyH7MGN6PY/qnRPyW/uFSEYhI0GtqbeftgjJez9vD3zeVUt3YSmy0MXNEBpdMz+GU0ZkcOyBVH/xHSUUgIkGptT3Af28pZ/HaXbyet4f6lnb6JMZy+rj+nDl+IKeMziQlXh9hPUH/FkUkqBTsqeWlVcUsWrOT8roW0hJimDNpMOdMHszMkf2IDeLr8UOVikBEfNfU2s5r63bzh4+2s6a4ipgo4/Rx/bn4+BxOHZMVMZdx+kVFICK+2VPdxFMfbuP5FTvY19DKyKxk7jh7HBdMzSYjJd7veBFDRSAivW7TnhoWvFvEX9fuoj3gOHP8QK48cRgnjsrQCV8fqAhEpNds2FnNf/1jC8vy9pIUF83cmcO49pQR5PRL8jtaRFMRiIjnCvbU8tCyAt7M30tqQgw3nT6aeScPJz0pzu9ogopARDxUsq+BX7+xmT9/spOUuBh+dMYYrj55OGkJusM3mKgIRKTHNbS08dhbn7LgvSIAvvuVkdxw6ij6JmsPIBipCESkxzjnWLx2F79csok9NU2cP2UwP5k1luz0RL+jySGoCESkR2yvqOeORRt4b0s5k4b04dG5Uzl+WD+/Y8lhUBGIyJfS1h5gwXtFPPzmFuKio7j7/AnMnTGUKI+mVZSepyIQkaNWWFrLLS+uZW1JNbMnDORfzz2OAWkJfseSI6QiEJEjFgg4Fr6/lQeXFZAcF81jc6dx1sRBfseSo6QiEJEjUlrbxC0vruW9LeV8Y9wAfnnhRLJSNRxEKFMRiMhhe3dzGT96cQ21TW3cd8FELpuRoyEhwoCKQES+UCDgePjvW3j471sYMyCFP37nBI4dmOp3LOkhKgIROaTqxlZufmEN/9hUykXThnDP+RNIjIv2O5b0IBWBiBzUlr21fOfpVezc18jd5x3HFScM06GgMKQiEJFuvbeljO898zHxsdE8f90J5A7XzWHhSkUgIp/zzEfbuWtxHqP7p/D7q6driIgwpyIQkf2cczywtID573zKacdm8V+XTSVVI4WGPRWBiAAdQ0Xc9sp6/rS6hMtnDuUX5x5HjCaKjwgqAhGhsaWdG5/9mL9vKuWm00fzw2+M1knhCKIiEIlw9c1tXPPkSlZsq+Tu8yfw7ROG+R1JepmKQCSC1TS1Mu+JlawpruI3l0zhvCnZfkcSH6gIRCJUdUMrVz6xgryd1Txy2VRma9C4iOXpmSAzm2VmBWZWaGa3dbO8j5n91czWmlmemc3zMo+IdKhpauXKhcvJ31XDb684XiUQ4TwrAjOLBh4FZgPjgcvMbPwBq30f2OicmwycBvzKzDSpqYiH6pvbmPfESvJ21fDY3GmcMX6A35HEZ17uEcwACp1zRc65FuB54LwD1nFAqnVcnpACVAJtHmYSiWhNre1856lVfLJjH/952VS+oRIQvC2CbKC4y/OSzte6egQYB+wC1gM3OecCB/4iM7vOzFaZ2aqysjKv8oqEtdb2ADc8s5qPtlbw629N0UQysp+XRdDdRcjugOf/BKwBBgNTgEfMLO1zP+TcAudcrnMuNysrq6dzioS9QMBx65/W8VZBGfeeP5Hzp+rqIPlfXhZBCZDT5fkQOrb8u5oHvOI6FAJbgbEeZhKJSPcv3cQrn+zkljPGcPnMoX7HkSDjZRGsBEab2YjOE8CXAosPWGcHcDqAmQ0AjgWKPMwkEnF+924RC94t4qoTh3Hj14/xO44EIc/uI3DOtZnZjcAyIBpY6JzLM7PrO5fPB+4GnjSz9XQcSrrVOVfuVSaRSLNk/W7uXZLP2RMHcdc5x2nYCOmWpzeUOeeWAEsOeG1+l+93AWd6mUEkUn28Yx83v7CG44f15VffmkxUlEpAuqehBUXCUHFlA999ahUD0hJY8O3jSYjV1JJycCoCkTBT29TKNU+upC3geGLedDJS4v2OJEFOYw2JhJFAwHHzC2soKq/nD9fMYFRWit+RJARoj0AkjPz6jc28mV/Kz88ex0nHZPodR0KEikAkTLy6bhePvFXIJbk5XHXScL/jSAhREYiEgYI9tfzkpXUcP6wvvzhfl4nKkVERiIS42qZWbnhmNcnxMfx27jTiY3SFkBwZnSwWCWHOOX7y0jq2Vzbw7Hdm0j8twe9IEoK0RyASwn73XhFL8/Zw26yxzByZ4XccCVEqApEQtXp7JQ8sLWD2hIF85ysj/I4jIUxFIBKC9tW38INnPyE7PZEHLp6kk8PypegcgUiIcc7xkz+tpayumZdvOIm0hFi/I0mI0x6BSIhZ+P423swv5fbZ45g0JN3vOBIGVAQiIWTDzmru/1s+3xg3gHknD/c7joQJFYFIiGhsaeefn/+EfslxPKTzAtKDdI5AJETc89pGisrq+eN3ZtI3Oc7vOBJGtEcgEgJez9vDH5fv4LqvjuRkDSYnPUxFIBLkSmubuPXldRw3OI1bzhzjdxwJQyoCkSDmnOP2l9fT0NLOw5dO0ThC4gkVgUgQe3FVMX/fVMqts8ZyTP9Uv+NImFIRiASp4soGfvHXjZw4MoOrNb+AeEhFIBKEAgHHLS+tJcqMf//WZKKidKmoeEdFIBKEnvhgGyu2VnLnOePJTk/0O46EORWBSJDZWl7PQ8s2cfrY/lx8/BC/40gEUBGIBJFAwPHTP60lLjq
|
||
|
|
"text/plain": [
|
||
|
|
"<Figure size 432x288 with 1 Axes>"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"metadata": {
|
||
|
|
"needs_background": "light"
|
||
|
|
},
|
||
|
|
"output_type": "display_data"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"%matplotlib inline\n",
|
||
|
|
"import matplotlib.pyplot as plt\n",
|
||
|
|
"import torch\n",
|
||
|
|
"\n",
|
||
|
|
"x = torch.linspace(-5,5,100)\n",
|
||
|
|
"plt.xlabel(\"x\")\n",
|
||
|
|
"plt.ylabel(\"y\")\n",
|
||
|
|
"plt.plot(x, sigmoid(x))\n",
|
||
|
|
"fname = 'sigmoid.png'\n",
|
||
|
|
"plt.savefig(fname)\n",
|
||
|
|
"fname"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"[[file:# Out[32]:\n",
|
||
|
|
"\n",
|
||
|
|
" 'sigmoid.png'\n",
|
||
|
|
"\n",
|
||
|
|
"]]\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"#### PyTorch\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Funkcja `torch.sigmoid` po prostu stosuje sigmoidę do każdego elementu tensora (*element-wise*).\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 3,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"text/plain": [
|
||
|
|
"tensor([0.6457, 0.7311, 0.0067])"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"execution_count": 3,
|
||
|
|
"metadata": {},
|
||
|
|
"output_type": "execute_result"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"import torch\n",
|
||
|
|
"\n",
|
||
|
|
"torch.sigmoid(torch.tensor([0.6, 1.0, -5.0]))"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Istnieje również `torch.nn.Sigmoid`, które może być używane jako warstwa.\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 4,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"text/plain": [
|
||
|
|
"tensor([0.5000, 0.4502, 0.5987])"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"execution_count": 4,
|
||
|
|
"metadata": {},
|
||
|
|
"output_type": "execute_result"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"import torch.nn as nn\n",
|
||
|
|
"\n",
|
||
|
|
"s = nn.Sigmoid()\n",
|
||
|
|
"s(torch.tensor([0.0, -0.2, 0.4]))"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"##### Implementacja w Pytorchu\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 5,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"text/plain": [
|
||
|
|
"tensor([0.5000, 0.6225, 0.5744])"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"execution_count": 5,
|
||
|
|
"metadata": {},
|
||
|
|
"output_type": "execute_result"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"import torch.nn as nn\n",
|
||
|
|
"import torch\n",
|
||
|
|
"\n",
|
||
|
|
"class MySigmoid(nn.Module):\n",
|
||
|
|
" def __init__(self):\n",
|
||
|
|
" super(MySigmoid, self).__init__()\n",
|
||
|
|
"\n",
|
||
|
|
" def forward(self, x):\n",
|
||
|
|
" return 1 / (1 + torch.exp(-x))\n",
|
||
|
|
"\n",
|
||
|
|
"s = MySigmoid()\n",
|
||
|
|
"s(torch.tensor([0.0, 0.5, 0.3]))"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"#### Wagi\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Funkcja sigmoidalna nie ma żadnych wyuczalnych wag.\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"##### **Pytanie**: Czy można rozszerzyć funkcję sigmoidalną o jakieś wyuczalne wagi?\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"### Regresja liniowa\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"### Softmax\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"W klasyfikacji wieloklasowej należy zwrócić musimy zwrócić rozkład\n",
|
||
|
|
"prawdopodobieństwa po wszystkich klasach, w przeciwieństwie do\n",
|
||
|
|
"klasyfikacji binarnej, gdzie wystarczy zwrócić jedną liczbę —\n",
|
||
|
|
"prawdopodobieństwo pozytywnej klasy ($p$; prawdopodobieństwo drugiej\n",
|
||
|
|
"klasy to po prostu $1-p$).\n",
|
||
|
|
"\n",
|
||
|
|
"A zatem na potrzeby klasyfikacji wieloklasowej potrzeba wektorowego\n",
|
||
|
|
"odpowiednika funkcji sigmoidalnej, to jest funkcji, która zamienia\n",
|
||
|
|
"nieznormalizowany wektor $\\vec{z} = [z_1,\\dots,z_k]$ (pochodzący np. z\n",
|
||
|
|
"poprzedzającej warstwy liniowej) na rozkład prawdopobieństwa.\n",
|
||
|
|
"Potrzebujemy zatem funkcji $s: \\mathcal{R}^k \\rightarrow [0,1]^k$\n",
|
||
|
|
"\n",
|
||
|
|
"spełniającej następujące warunki:\n",
|
||
|
|
"\n",
|
||
|
|
"- $s(z_i) = s_i(z) \\in [0,1]$\n",
|
||
|
|
"- $\\Sigma_i s(z_i) = 1$\n",
|
||
|
|
"- $z_i > z_j \\Rightarrow s(z_i) > s(z_j)$\n",
|
||
|
|
"\n",
|
||
|
|
"Można by podać takie (**błędne**!) rozwiązanie:\n",
|
||
|
|
"\n",
|
||
|
|
"$$s(z_i) = \\frac{z_i}{\\Sigma_{j=1}^k z_j}$$\n",
|
||
|
|
"\n",
|
||
|
|
"To rozwiązanie zadziała błędnie dla liczb ujemnych, trzeba najpierw\n",
|
||
|
|
"użyć funkcji monotonicznej, która przekształaca $\\mathcal{R}$ na $\\mathcal{R^+}$.\n",
|
||
|
|
"Naturalna funkcja tego rodzaju to funkcja wykładnicza $\\exp{x} = e^x$.\n",
|
||
|
|
"Tym sposobem dochodzimy do funkcji softmax:\n",
|
||
|
|
"\n",
|
||
|
|
"$$s(z_i) = \\frac{e^{z_i}}{\\Sigma_{j=1}^k e^{z_j}}$$\n",
|
||
|
|
"\n",
|
||
|
|
"Mianownik ułamka w definicji funkcji softmax nazywamy czasami czynnikiem normalizacyjnym:\n",
|
||
|
|
"$Z(\\vec{z}) = \\Sigma_{j=1}^k e^{z_j}$, wtedy:\n",
|
||
|
|
"\n",
|
||
|
|
"$$s(z_i) = \\frac{e^{z_i}}{Z(\\vec{z})}$$\n",
|
||
|
|
"\n",
|
||
|
|
"Definicja w PyTorchu:\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 1,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"name": "stdout",
|
||
|
|
"output_type": "stream",
|
||
|
|
"text": [
|
||
|
|
"# Out[75]:\n",
|
||
|
|
"tensor([0.1182, 0.0022, 0.0059, 0.8737])"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"import torch\n",
|
||
|
|
"\n",
|
||
|
|
"def softmax(z):\n",
|
||
|
|
" z_plus = torch.exp(z)\n",
|
||
|
|
" return z_plus / torch.sum(z_plus)\n",
|
||
|
|
"\n",
|
||
|
|
"softmax(torch.tensor([3., -1., 0., 5.]))"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"#### Soft vs hard\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Dlaczego *softmax*? Czasami używa się funkcji **hardmax**, która np.\n",
|
||
|
|
"wektora $[3, -1, 0, 5]$ zwróciłaby $[0, 0, 0, 5]$ — to jest po prostu\n",
|
||
|
|
"wektorowa wersja funkcji zwracającej maksimum. Istnieje też funkcja\n",
|
||
|
|
"hard\\*arg\\*max, która zwraca wektor *one-hot* — z jedną jedynką na\n",
|
||
|
|
"pozycji dla największej wartości (zamiast podania największej\n",
|
||
|
|
"wartości), np. wartość hardargmax dla $[3, -1, 0, 5]$ zwróciłaby $[0,\n",
|
||
|
|
"0, 0, 1]$.\n",
|
||
|
|
"\n",
|
||
|
|
"Zauważmy, że powszechnie przyjęta nazwa *softmax* jest właściwie\n",
|
||
|
|
"błędna, funkcja ta powinna nazywać się *softargmax*, jako że w\n",
|
||
|
|
"„miękki” sposób identyfikuje największą wartość przez wartość zbliżoną\n",
|
||
|
|
"do 1 (na pozostałych pozycjach wektora nie będzie 0).\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"##### **Pytanie**: Jak można zdefiniować funkcję *softmax* w ścisłym tego słowa znaczeniu („miękki” odpowiednik hardmax, nie hardargmax)?\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"#### PyTorch\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Funkcja `torch.nn.functional.softmax` normalizuje wartości dla całego tensora:\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 1,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"name": "stdout",
|
||
|
|
"output_type": "stream",
|
||
|
|
"text": [
|
||
|
|
"# Out[5]:\n",
|
||
|
|
"tensor([0.4007, 0.5978, 0.0015])"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"import torch.nn as nn\n",
|
||
|
|
"\n",
|
||
|
|
"nn.functional.softmax(torch.tensor([0.6, 1.0, -5.0]))"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"… zobaczmy, jak ta funkcja zachowuje się dla macierzy:\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 1,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"name": "stdout",
|
||
|
|
"output_type": "stream",
|
||
|
|
"text": [
|
||
|
|
"# Out[6]:\n",
|
||
|
|
"#+BEGIN_EXAMPLE\n",
|
||
|
|
" tensor([[0.4013, 0.5987],\n",
|
||
|
|
" [0.0041, 0.9959]])\n",
|
||
|
|
"#+END_EXAMPLE"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"import torch.nn as nn\n",
|
||
|
|
"\n",
|
||
|
|
"nn.functional.softmax(torch.tensor([[0.6, 1.0], [-2.0, 3.5]]))"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Za pomocą (zalecanego zresztą) argumentu `dim` możemy określić wymiar, wzdłuż którego dokonujemy normalizacji:\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 1,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"name": "stdout",
|
||
|
|
"output_type": "stream",
|
||
|
|
"text": [
|
||
|
|
"# Out[8]:\n",
|
||
|
|
"#+BEGIN_EXAMPLE\n",
|
||
|
|
" tensor([[0.9309, 0.0759],\n",
|
||
|
|
" [0.0691, 0.9241]])\n",
|
||
|
|
"#+END_EXAMPLE"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"import torch.nn as nn\n",
|
||
|
|
"\n",
|
||
|
|
"nn.functional.softmax(torch.tensor([[0.6, 1.0], [-2.0, 3.5]]), dim=0)"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Istnieje również `torch.nn.Softmax`, które może być używane jako warstwa.\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 1,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"name": "stdout",
|
||
|
|
"output_type": "stream",
|
||
|
|
"text": [
|
||
|
|
"# Out[10]:\n",
|
||
|
|
"tensor([0.3021, 0.2473, 0.4506])"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"import torch.nn as nn\n",
|
||
|
|
"\n",
|
||
|
|
"s = nn.Softmax(dim=0)\n",
|
||
|
|
"s(torch.tensor([0.0, -0.2, 0.4]))"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"##### Implementacja w Pytorchu\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 1,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"name": "stdout",
|
||
|
|
"output_type": "stream",
|
||
|
|
"text": [
|
||
|
|
"# Out[48]:\n",
|
||
|
|
"tensor([0.5000, 0.6225, 0.5744])"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"import torch.nn as nn\n",
|
||
|
|
"import torch\n",
|
||
|
|
"\n",
|
||
|
|
"class MySoftmax(nn.Module):\n",
|
||
|
|
" def __init__(self):\n",
|
||
|
|
" super(MySoftmax, self).__init__()\n",
|
||
|
|
"\n",
|
||
|
|
" def forward(self, x):\n",
|
||
|
|
" ex = torch.exp(x)\n",
|
||
|
|
" return ex / torch.sum(ex)\n",
|
||
|
|
"\n",
|
||
|
|
"s = MySigmoid()\n",
|
||
|
|
"s(torch.tensor([0.0, 0.5, 0.3]))"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"###### **Pytanie**: Tak naprawdę wyżej zdefiniowana klasa `MySoftmax` nie zachowuje się identycznie jak `nn.Softmax`. Na czym polega różnica?\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"#### Przypadek szczególny\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Sigmoida jest przypadkiem szczególnym funkcji softmax:\n",
|
||
|
|
"\n",
|
||
|
|
"$$\\sigma(x) = \\frac{1}{1 + e^{-x}} = \\frac{e^x}{e^x + 1} = \\frac{e^x}{e^x + e^0} = s([x, 0])_1$$\n",
|
||
|
|
"\n",
|
||
|
|
"Ogólniej: softmax na dwuelementowych wektorach daje przesuniętą sigmoidę (przy ustaleniu jednej z wartości).\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 1,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"%matplotlib inline\n",
|
||
|
|
"import matplotlib.pyplot as plt\n",
|
||
|
|
"import torch\n",
|
||
|
|
"import torch.nn as nn\n",
|
||
|
|
"\n",
|
||
|
|
"x = torch.linspace(-5,5,100)\n",
|
||
|
|
"plt.xlabel(\"x\")\n",
|
||
|
|
"plt.ylabel(\"y\")\n",
|
||
|
|
"a = torch.Tensor(x.size()[0]).fill_(2.)\n",
|
||
|
|
"m = torch.stack([x, a])\n",
|
||
|
|
"plt.plot(x, nn.functional.softmax(m, dim=0)[0])\n",
|
||
|
|
"fname = 'softmax3.png'\n",
|
||
|
|
"plt.savefig(fname)\n",
|
||
|
|
"fname"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"[[file:# Out[19]:\n",
|
||
|
|
"\n",
|
||
|
|
" 'softmax3.png'\n",
|
||
|
|
"\n",
|
||
|
|
"]]\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 1,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"%matplotlib inline\n",
|
||
|
|
"import matplotlib.pyplot as plt\n",
|
||
|
|
"from mpl_toolkits import mplot3d\n",
|
||
|
|
"import torch\n",
|
||
|
|
"import torch.nn as nn\n",
|
||
|
|
"\n",
|
||
|
|
"x = torch.linspace(-5,5,10)\n",
|
||
|
|
"y = torch.linspace(-5,5,10)\n",
|
||
|
|
"fig = plt.figure()\n",
|
||
|
|
"ax = fig.add_subplot(111, projection='3d')\n",
|
||
|
|
"plt.xlabel(\"x\")\n",
|
||
|
|
"plt.ylabel(\"y\")\n",
|
||
|
|
"X, Y = torch.meshgrid(x, y)\n",
|
||
|
|
"m = torch.stack([X, Y])\n",
|
||
|
|
"z = nn.functional.softmax(m, dim=0)\n",
|
||
|
|
"ax.plot_wireframe(x, y, z[0])\n",
|
||
|
|
"fname = 'softmax3d.png'\n",
|
||
|
|
"plt.savefig(fname)\n",
|
||
|
|
"fname"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"[[file:# Out[27]:\n",
|
||
|
|
"\n",
|
||
|
|
" 'softmax3d.png'\n",
|
||
|
|
"\n",
|
||
|
|
"]]\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"#### Wagi\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Podobnie jak funkcja sigmoidalna, softmax nie ma żadnych wyuczalnych wag.\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"#### Zastosowania\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Podstawowym zastosowaniem funkcji softmax jest klasyfikacja\n",
|
||
|
|
"wieloklasowa, również w wypadku zadań przetwarzania sekwencji, które\n",
|
||
|
|
"mogą być interpretowane jako klasyfikacja wieloklasowa:\n",
|
||
|
|
"\n",
|
||
|
|
"- przewidywanie kolejnego słowa w modelowaniu języka (klasą jest słowo, zbiór klas to słownik)\n",
|
||
|
|
"- przypisywanie etykiet (np. części mowy) słowom.\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"### LogSoftmax\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Ze względów obliczeniowych często korzysta się z funkcji **LogSoftmax**\n",
|
||
|
|
"która zwraca logarytmy pradopodobieństw (*logproby*).\n",
|
||
|
|
"\n",
|
||
|
|
"$$log s(z_i) = log \\frac{e^{z_i}}{\\Sigma_{j=1}^k e^{z_j}}$$\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"#### PyTorch\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 1,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"name": "stdout",
|
||
|
|
"output_type": "stream",
|
||
|
|
"text": [
|
||
|
|
"# Out[25]:\n",
|
||
|
|
"tensor([-1.1971, -1.3971, -0.7971])"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"import torch.nn as nn\n",
|
||
|
|
"\n",
|
||
|
|
"s = nn.LogSoftmax(dim=0)\n",
|
||
|
|
"s(torch.tensor([0.0, -0.2, 0.4]))"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Niektóre funkcje kosztu (np. `NLLLoss`) zaimplementowane w PyTorchu\n",
|
||
|
|
"operują właśnie na logarytmach prawdopobieństw.\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"### Przykład: klasyfikacja wieloklasowa\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Na przykładzie rozpoznawania dyscypliny sportu: git://gonito.net/sport-text-classification.git\n",
|
||
|
|
"\n",
|
||
|
|
"Wczytujemy zbiór uczący:\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 1,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"name": "stdout",
|
||
|
|
"output_type": "stream",
|
||
|
|
"text": [
|
||
|
|
"# Out[23]:\n",
|
||
|
|
"#+BEGIN_EXAMPLE\n",
|
||
|
|
" {'zimowe': 0,\n",
|
||
|
|
" 'moto': 1,\n",
|
||
|
|
" 'tenis': 2,\n",
|
||
|
|
" 'pilka-reczna': 3,\n",
|
||
|
|
" 'sporty-walki': 4,\n",
|
||
|
|
" 'koszykowka': 5,\n",
|
||
|
|
" 'siatkowka': 6,\n",
|
||
|
|
" 'pilka-nozna': 7}\n",
|
||
|
|
"#+END_EXAMPLE"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"import gzip\n",
|
||
|
|
"from pytorch_regression.analyzer import vectorize_text, vector_length\n",
|
||
|
|
"\n",
|
||
|
|
"texts = []\n",
|
||
|
|
"labels = []\n",
|
||
|
|
"labels_dic = {}\n",
|
||
|
|
"labels_revdic = {}\n",
|
||
|
|
"c = 0\n",
|
||
|
|
"\n",
|
||
|
|
"with gzip.open('sport-text-classification/train/train.tsv.gz', 'rt') as fh:\n",
|
||
|
|
" for line in fh:\n",
|
||
|
|
" line = line.rstrip('\\n')\n",
|
||
|
|
" line = line.replace('\\\\\\t', ' ')\n",
|
||
|
|
" label, text = line.split('\\t')\n",
|
||
|
|
" texts.append(text)\n",
|
||
|
|
" if label not in labels_dic:\n",
|
||
|
|
" labels_dic[label] =c\n",
|
||
|
|
" labels_revdic[c] = label\n",
|
||
|
|
" c += 1\n",
|
||
|
|
" labels.append(labels_dic[label])\n",
|
||
|
|
"nb_of_labels = len(labels_dic)\n",
|
||
|
|
"labels_dic"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Przygotowujemy model:\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 1,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"name": "stdout",
|
||
|
|
"output_type": "stream",
|
||
|
|
"text": [
|
||
|
|
"# Out[8]:"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"import torch.nn as nn\n",
|
||
|
|
"from torch import optim\n",
|
||
|
|
"\n",
|
||
|
|
"model = nn.Sequential(\n",
|
||
|
|
" nn.Linear(vector_length, nb_of_labels),\n",
|
||
|
|
" nn.LogSoftmax()\n",
|
||
|
|
" )\n",
|
||
|
|
"\n",
|
||
|
|
"optimizer = optim.Adam(model.parameters())"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Funkcja kosztu to log-loss.\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 1,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"name": "stdout",
|
||
|
|
"output_type": "stream",
|
||
|
|
"text": [
|
||
|
|
"# Out[9]:\n",
|
||
|
|
"tensor(2.3026)"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"import torch\n",
|
||
|
|
"import torch.nn.functional as F\n",
|
||
|
|
"\n",
|
||
|
|
"loss_fn = torch.nn.NLLLoss()\n",
|
||
|
|
"\n",
|
||
|
|
"expected_class_id = torch.tensor([2])\n",
|
||
|
|
"loss_fn(torch.log(\n",
|
||
|
|
" torch.tensor([[0.3, 0.5, 0.1, 0.0, 0.1]])),\n",
|
||
|
|
" expected_class_id)"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Pętla ucząca:\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 1,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"name": "stdout",
|
||
|
|
"output_type": "stream",
|
||
|
|
"text": [
|
||
|
|
"# Out[25]:"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"iteration = 0\n",
|
||
|
|
"step = 50\n",
|
||
|
|
"closs = torch.tensor(0.0, dtype=torch.float, requires_grad=False)\n",
|
||
|
|
"\n",
|
||
|
|
"for t, y_exp in zip(texts, labels):\n",
|
||
|
|
" x = vectorize_text(t).float().unsqueeze(dim=0)\n",
|
||
|
|
"\n",
|
||
|
|
" optimizer.zero_grad()\n",
|
||
|
|
"\n",
|
||
|
|
" y_logprobs = model(x)\n",
|
||
|
|
"\n",
|
||
|
|
" loss = loss_fn(y_logprobs, torch.tensor([y_exp]))\n",
|
||
|
|
"\n",
|
||
|
|
" loss.backward()\n",
|
||
|
|
"\n",
|
||
|
|
" with torch.no_grad():\n",
|
||
|
|
" closs += loss\n",
|
||
|
|
"\n",
|
||
|
|
" optimizer.step()\n",
|
||
|
|
"\n",
|
||
|
|
" if iteration % 50 == 0:\n",
|
||
|
|
" print((closs / step).item(), loss.item(), iteration, y_exp, torch.exp(y_logprobs), t)\n",
|
||
|
|
" closs = torch.tensor(0.0, dtype=torch.float, requires_grad=False)\n",
|
||
|
|
" iteration += 1\n",
|
||
|
|
"\n",
|
||
|
|
" if iteration == 5000:\n",
|
||
|
|
" break"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Model jest tak prosty, że jego wagi są interpretowalne.\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 1,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"name": "stdout",
|
||
|
|
"output_type": "stream",
|
||
|
|
"text": [
|
||
|
|
"# Out[26]:\n",
|
||
|
|
"tensor([[0.0070, 0.0075, 0.0059, 0.0061, 0.0093, 0.9509, 0.0062, 0.0071]])"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"with torch.no_grad():\n",
|
||
|
|
" x = vectorize_text('NBA').float().unsqueeze(dim=0)\n",
|
||
|
|
" y_prob = model(x)\n",
|
||
|
|
"torch.exp(y_prob)"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 1,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"name": "stdout",
|
||
|
|
"output_type": "stream",
|
||
|
|
"text": [
|
||
|
|
"# Out[32]:\n",
|
||
|
|
"#+BEGIN_EXAMPLE\n",
|
||
|
|
" tensor([-2.3693, -2.3421, -2.4205, -2.4353, -2.1499, 2.5163, -2.4351, -2.4546],\n",
|
||
|
|
" grad_fn=<SelectBackward>)\n",
|
||
|
|
"#+END_EXAMPLE"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"with torch.no_grad():\n",
|
||
|
|
" x = vectorize_text('NBA').float().unsqueeze(dim=0)\n",
|
||
|
|
" ix = torch.argmax(x).item()\n",
|
||
|
|
"model[0].weight[:,ix]"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Możemy nawet zaprezentować wykres przedstawiający rozmieszczenie słów względem dwóch osi odnoszących się do poszczególnych wybranych dyscyplin.\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 1,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"name": "stdout",
|
||
|
|
"output_type": "stream",
|
||
|
|
"text": [
|
||
|
|
"# Out[45]:"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAD4CAYAAAAQP7oXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAAaRklEQVR4nO3dfXRV1bnv8e9DQKVcMdKoGMAGvIBCCMQRqIoSrCKK1PAST/EeNFxrM/TWjnMdt6kg1aH22GO1l6o9Sg+HFkGr0iMQEaj4VhrBtwRQQDSKmCqEC0EMqAE1+Nw/sskIcSeZsHf2TsLvM0YGe80195rPJLU/1lxr72XujoiISIhOyS5ARETaD4WGiIgEU2iIiEgwhYaIiARTaIiISLDOyS6gOWlpaZ6RkZHsMkRE2o21a9fudvdTWuv4bTo0MjIyKCsrS3YZIiLthpn9ozWPr+UpEREJ1qFDo6KigszMzKN6b3FxMZs3b45zRSIi7VuHDo2jVVtbq9AQEYnimAmNrVu3kp2dTWlpKeeeey5ZWVlMnDiRTz/9FIDRo0dz6623kpuby29+8xuWLl1KUVERw4YN44MPPkhy9SIibUObvhAeL+Xl5UyZMoV58+Zx7bXX8vvf/57c3Fxuv/127rzzTu6//34Aqqur+fvf/w7A+++/z/jx48nPz09i5SIibUuHC43i9du5b2U5ldX76eF72bZjJ3l5eSxatIjevXtTXV1Nbm4uAAUFBVx11VX17/3Rj36UrLJFRNqFDrU8Vbx+OzMWb2R79X4c2LnvADUczwmpp7JmzZoW39+tW7fWL1JEpB3rUKFx38py9n998PDGTimcMO4WFixYwPLlyzn55JN5+eWXAXj00UfrzzoaO/HEE/nss89au2QRkXalQy1PVVbvj9q+swbWL1vGmDFjmDRpEkVFRdTU1NCvXz/mzZsX9T1TpkzhJz/5CQ8++CBPPfUUZ555ZmuWLiLSLlhbfghTTk6OH8knwkfe8xLbowRHr9SurJn+g3iWJiLSJpnZWnfPaa3jd6jlqaKxA+naJeWwtq5dUigaOzBJFYmIdCwdanlqQnYvgPq7p9JTu1I0dmB9u4iIxKZDhQbUBYdCQkSkdXSo5SkREWldCg0REQkWl9Aws8vMrNzMtpjZ9Cj7R5vZXjN7M/JzezzGFRGRxIr5moaZpQAPAWOAbUCpmS1198ZfEfuyu4+PdTwREUmeeJxpjAC2uPtWd/8KeBLIi8NxRUSkjYlHaPQCPm6wvS3S1th5ZvaWmf3VzAbHYVwREUmweNxya1HaGn/MfB3wPXf/3MzGAcVA/6gHMysECgHOOOOMOJQnIiLxEo8zjW1AnwbbvYHKhh3cfZ+7fx55vQLoYmZp0Q7m7nPcPcfdc0455ZQ4lCciIvESj9AoBfqbWV8zOw6YAixt2MHMepqZRV6PiIz7SRzGFhGRBIp5ecrda83sJmAlkAL8yd3fNrMbIvv/AOQDN5pZLbAfmOJt+ZsSRUQkqg71LbciIsc6fcutiIi0GQoNEREJptAQEZFgCg0REQmm0BARkWAKDRERCabQEBGRYAoNEREJptAQEZFgCg0REQmm0BARkWAKDRERCabQEBGRYAoNEREJptAQEZFgCg0REQmm0BARkWAKDRERCabQEBGRYAoNEREJptAQEZFgCg0REQmm0BARkWAKDRERCabQEBGRYAoNEREJptAQEZFgCg0REQmm0BARkWBxCQ0zu8zMys1si5lNj7LfzOzByP4NZnZOPMYVEZHEijk0zCwFeAi4HBgEXG1mgxp1uxzoH/kpBGbHOq6IiCRePM40RgBb3H2ru38FPAnkNeqTByzwOq8BqWZ2ehzGFhGRBIpHaPQCPm6wvS3SdqR9ADCzQjMrM7OyqqqqOJQnIiLxEo/QsChtfhR96hrd57h7jrvnnHLKKTEXJyIi8ROP0NgG9Gmw3RuoPIo+IiLSxsUjNEqB/mbW18yOA6YASxv1WQpcG7mL6lxgr7vviMPYIiKSQJ1jPYC715rZTcBKIAX4k7u/bWY3RPb/AVgBjAO2ADXA/4x1XBERSbyYQwPA3VdQFwwN2/7Q4LUDP43HWCIikjz6RLiIiARTaIiISDCFhoiIBFNoiIhIMIWGiIgEU2iIiEgwhYaIiARTaIiISDCFhoiIBFNoiIhIMIWGiIgEU2iIiEgwhYaIiARTaIiISDCFhoiIBFNoiIhIMIWGiMTk+uuvZ/PmzQBkZGSwe/duKioqyMzMTHJl0hri8uQ+ETl2zZ07N9klSALpTENEglRUVHDWWWdRUFBAVlYW+fn51NTUMHr0aMrKyur7ffDBB4e9b+vWrWRnZ1NaWsobb7zB+eefT3Z2Nueffz7l5eWJnobESKEhIsHKy8spLCxkw4YNdO/enYcffrh+38GDB/nqq69YsGDBYf0nT57MvHnzGD58OGeddRYlJSWsX7+eu+66i1tvvTUZ05AYaHlKRJpUvH47960sp7J6Pz18L2k90xk5ciQAU6dO5cEHH6zve9ttt7Fv3z7uvPNOPv/8c6qqqsjLy2PRokUMHjwYgL1791JQUMD777+PmfH1118nZV5y9HSmIc1auHAhFRUVyS5DkqB4/XZmLN7I9ur9OLBz3wGqa2opXr+9vo+Z1b/+9a9/TVpaWv32SSedRJ8+fVizZk1922233cZFF13Epk2beOaZZzhw4EBC5iLxo9A4hk2bNo2nnnqqyf2PPfYYH330ERkZGUf1fmnf7ltZzv6vDx7WVrtvF7fPWQzAE088wQUXXNDk+4877jiKi4tZsGABjz/+OFB3ptGrVy8AHnnkkdYpXFqVlqekSVOnTk12CZJEldX7v9XW5bt9+PC1FWRl/Sf9+/fnxhtv5JlnnmnyGN26dWPZsmWMGTOGbt268Ytf/IKCggJmzZrFD37wg9YsX1qJzjSS6IsvvuCKK65g6NChZGZmsnDhQu666y6GDx9OZmYmhYWFuDsAo0eP5uabb2bUqFGcffbZlJaWMmnSJPr3788vf/nL+mPOmjWLzMxMMjMzuf/+++vbFyxYQFZWFkOHDuWaa66pby8pKeH888+nX79+9WcN7k5RURGZmZkMGTKEhQsX1rffdNNNDBo0iCuuuIJdu3Yl4G9JkiU9teu3G83I+lERGzZsYNGiRXznO99h1apV5OTkAHV3WKWlpZGRkcGmTZsASE1NpbS0lLy8PM477zzee+891qxZw69+9SstfbZDOtNIomeffZb09HSWL18O1J26jxkzhttvvx2Aa665hmXLlvHDH/4QqDvdLykp4YEHHiAvL4+1a9fSo0cPzjzzTG6++WYqKiqYN28er7/+Ou7O97//fXJzcznuuOO4++67WbNmDWlpaezZs6e+hh07drB69WreffddrrzySvLz81m8eDFvvvkmb731Frt372b48OGMGjWKV199lfLycjZu3MjOnTsZNGgQ1113XeL/4iQhisYOZMbijYctUZkZRWMHJrEqSTaFRoI1vBvl5K8/Z/uKlfS45RbGjx/PhRdeyKJFi7j33nupqalhz549DB48uD40rrzySgCGDBnC4MGDOf300wHo168fH3/8MatXr2bixIl069YNgEmTJvHyyy9jZuTn59dfpOzRo0d9PRMmTKBTp04MGjSInTt3ArB69WquvvpqUlJSOO2008jNzaW0tJSSkpL69vT0dC0vdHATsuuuPRz63+v3vpfBv69cU98uxyaFRgIduhvl0L/c9nRJ46Sr/y9fnriDGTNmcOmll/LQQw9RVlZGnz59uOOOOw67u+T4448HoFOnTvWvD23X1tbWL2U15u6H3eXSUMPjHHp/U8cBmjyOdEwTsnspJOQwuqaRQI3vRqn97BO+pDOlnTP5+c9/zrp16wBIS0vj888/P+I7k0aNGkVxcTE1NTV88cUXLFmyhAsvvJCLL76Yv/zlL3zyyScAhy1PNXWchQsXcvDgQaqqqigpKWHEiBGMGjWKJ598koMHD7Jjxw7+9re/HeHfgIi0dzGdaZhZD2AhkAFUAP/k7p9G6VcBfAYcBGrdPSeWcdurxnejfF1Vwa5V89h
|
||
|
|
"text/plain": [
|
||
|
|
"<matplotlib.figure.Figure>"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"metadata": {},
|
||
|
|
"output_type": "display_data"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"%matplotlib inline\n",
|
||
|
|
"import matplotlib.pyplot as plt\n",
|
||
|
|
"\n",
|
||
|
|
"with torch.no_grad():\n",
|
||
|
|
" words = ['piłka', 'klub', 'kort', 'boisko', 'samochód']\n",
|
||
|
|
" words_ixs = [torch.argmax(vectorize_text(w).float().unsqueeze(dim=0)).item() for w in words]\n",
|
||
|
|
"\n",
|
||
|
|
" x_label = labels_dic['pilka-nozna']\n",
|
||
|
|
" y_label = labels_dic['tenis']\n",
|
||
|
|
"\n",
|
||
|
|
" x = [model[0].weight[x_label, ix] for ix in words_ixs]\n",
|
||
|
|
" y = [model[0].weight[y_label, ix] for ix in words_ixs]\n",
|
||
|
|
"\n",
|
||
|
|
" fig, ax = plt.subplots()\n",
|
||
|
|
" ax.scatter(x, y)\n",
|
||
|
|
"\n",
|
||
|
|
" for i, txt in enumerate(words):\n",
|
||
|
|
" ax.annotate(txt, (x[i], y[i]))"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"### Zadanie etykietowania sekwencji\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Zadanie etykietowania sekwencji (*sequence labelling*) polega na przypisaniu poszczególnym wyrazom (tokenom) tekstu **etykiet** ze skończonego zbioru. Definiując formalnie:\n",
|
||
|
|
"\n",
|
||
|
|
"- rozpatrujemy ciąg wejściowy tokenów $(t^1,\\dots,t^K)$\n",
|
||
|
|
"- dany jest skończony zbiór etykiet $L = \\{l_1,\\dots,l_{|L|}\\}$, dla uproszczenia można założyć, że etykietami\n",
|
||
|
|
" są po prostu kolejne liczby, tj. $L=\\{0,\\dots,|L|-1\\}$\n",
|
||
|
|
"- zadanie polega na wygenerowaniu sekwencji etykiet (o tej samej długości co ciąg wejściowy!) $(y^1,\\dots,y^K)$,\n",
|
||
|
|
" $y^k \\in L$\n",
|
||
|
|
"\n",
|
||
|
|
"Zadanie etykietowania można traktować jako przypadek szczególny klasyfikacji wieloklasowej, z tym, że klasyfikacji dokonujemy wielokrotnie — dla każdego tokenu (nie dla każdego tekstu).\n",
|
||
|
|
"\n",
|
||
|
|
"Przykłady zastosowań:\n",
|
||
|
|
"\n",
|
||
|
|
"- oznaczanie częściami mowy (*POS tagger*) — czasownik, przymiotnik, rzeczownik itd.\n",
|
||
|
|
"- oznaczanie etykiet nazw w zadaniu NER (nazwisko, kwoty, adresy — najwięcej tokenów będzie miało etykietę pustą, zazwyczaj oznaczaną przez `O`)\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"#### **Pytanie**: czy zadanie tłumaczenia maszynowego można potraktować jako problem etykietowania sekwencji?\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"#### Przykładowe wyzwanie NER CoNLL-2003\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Zob. [https://gonito.net/challenge/en-ner-conll-2003](https://gonito.net/challenge/en-ner-conll-2003).\n",
|
||
|
|
"\n",
|
||
|
|
"Przykładowy przykład uczący (`xzcat train.tsv.xz| head -n 1`):\n",
|
||
|
|
"\n",
|
||
|
|
"O O B-MISC I-MISC O O O O O B-LOC O B-LOC O O O O O O O O O O O B-MISC I-MISC O O O O O B-PER I-PER O B-LOC O O O O O O B-PER I-PER O B-LOC O O O O O O B-PER I-PER I-PER O B-LOC O O O O O B-PER I-PER O O B-LOC O O O O O O B-PER I-PER O B-LOC O O O O O B-PER I-PER O O O O O B-PER I-PER O B-LOC O O O O O B-PER I-PER O B-LOC O B-LOC O O O O O O B-PER I-PER O O O O B-PER I-PER O B-LOC O O O O O O B-PER I-PER O B-LOC O O O O O B-PER I-PER O O O O O B-PER I-PER O B-LOC O O O O O B-PER I-PER O B-LOC O O O O O O B-PER I-PER O O O O O B-PER I-PER O B-LOC O O O O O B-PER I-PER O B-LOC O O O O O O B-PER I-PER O O O O B-PER I-PER I-PER O B-LOC O O O O O O B-PER I-PER O O O O B-PER I-PER O B-LOC O O O O O O B-PER I-PER O B-LOC O O O O O O B-PER I-PER O O O O B-PER I-PER O B-LOC O O O O O O B-PER I-PER O O O O B-PER I-PER O B-LOC O O O O O O B-PER I-PER O B-LOC O O O O O B-PER I-PER O B-LOC O B-LOC O O O O O B-PER I-PER O O O O O\tGOLF - BRITISH MASTERS THIRD ROUND SCORES . </S> NORTHAMPTON , England 1996-08-30 </S> Leading scores after </S> the third round of the British Masters on Friday : </S> 211 Robert Allenby ( Australia ) 69 71 71 </S> 212 Pedro Linhart ( Spain ) 72 73 67 </S> 216 Miguel Angel Martin ( Spain ) 75 70 71 , Costantino Rocca </S> ( Italy ) 71 73 72 </S> 217 Antoine Lebouc ( France ) 74 73 70 , Ian Woosnam 70 76 71 , </S> Francisco Cea ( Spain ) 70 71 76 , Gavin Levenson ( South </S> Africa ) 66 75 76 </S> 218 Stephen McAllister 73 76 69 , Joakim Haeggman ( Swe ) 71 77 </S> 70 , Jose Coceres ( Argentina ) 69 78 71 , Paul Eales 75 71 72 , </S> Klas Eriksson ( Sweden ) 71 75 72 , Mike Clayton ( Australia ) </S> 69 76 73 , Mark Roe 69 71 78 </S> 219 Eamonn Darcy ( Ireland ) 74 76 69 , Bob May ( U.S. ) 74 75 70 , </S> Paul Lawrie 72 75 72 , Miguel Angel Jimenez ( Spain ) 74 72 </S> 73 , Peter Mitchell 74 71 75 , Philip Walton ( Ireland ) 71 74 </S> 74 , Peter O'Malley ( Australia ) 71 73 75 </S> 220 Barry Lane 73 77 70 , Wayne Riley ( Australia ) 71 78 71 , </S> Martin Gates 71 77 72 , Bradley Hughes ( Australia ) 73 75 72 , </S> Peter Hedblom ( Sweden ) 70 75 75 , Retief Goosen ( South </S> Africa ) 71 74 75 , David Gilford 69 74 77 . </S>\n",
|
||
|
|
"\n",
|
||
|
|
"W pierwszym polu oczekiwany wynik zapisany za pomocą notacji **BIO**.\n",
|
||
|
|
"\n",
|
||
|
|
"Jako metrykę używamy F1 (z pominięciem tagu `O`)\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"#### Metryka F1\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"#### Etykietowanie za pomocą klasyfikacji wieloklasowej\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Można potraktować problem etykietowania dokładnie tak jak problem\n",
|
||
|
|
"klasyfikacji wieloklasowej (jak w przykładzie klasyfikacji dyscyplin\n",
|
||
|
|
"sportowych powyżej), tzn. rozkład prawdopodobieństwa możliwych etykiet\n",
|
||
|
|
"uzyskujemy poprzez zastosowanie prostej warstwy liniowej i funkcji softmax:\n",
|
||
|
|
"\n",
|
||
|
|
"$$p(l^k=i) = s(\\vec{w}\\vec{v}(t^k))_i = \\frac{e^{\\vec{w}\\vec{v}(t^k)}}{Z},$$\n",
|
||
|
|
"\n",
|
||
|
|
"gdzie $\\vec{v}(t^k)$ to reprezentacja wektorowa tokenu $t^k$.\n",
|
||
|
|
"Zauważmy, że tutaj (w przeciwieństwie do klasyfikacji całego tekstu)\n",
|
||
|
|
"reprezentacja wektorowa jest bardzo uboga: wektor <u>one-hot</u>! Taki\n",
|
||
|
|
"klasyfikator w ogóle nie będzie brał pod uwagę kontekstu, tylko sam\n",
|
||
|
|
"wyraz, więc tak naprawdę zdegeneruje się to do zapamiętania częstości\n",
|
||
|
|
"etykiet dla każdego słowa osobno.\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"##### Bogatsza reprezentacja słowa\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Można spróbować uzyskać bogatszą reprezentację dla słowa biorąc pod uwagę na przykład:\n",
|
||
|
|
"\n",
|
||
|
|
"- długość słowa\n",
|
||
|
|
"- kształt słowa (*word shape*), np. czy pisany wielkimi literami, czy składa się z cyfr itp.\n",
|
||
|
|
"- n-gramy znakowe wewnątrz słowa (np. słowo *Kowalski* można zakodować jako sumę wektorów\n",
|
||
|
|
" trigramów znakówych $\\vec{v}(Kow) + \\vec{v}(owa) + \\vec{v}(wal) + \\vec{v}(als) + \\vec{v}(lsk) + + \\vec{v}(ski)$\n",
|
||
|
|
"\n",
|
||
|
|
"Cały czas nie rozpatrujemy jednak w tej metodzie kontekstu wyrazu.\n",
|
||
|
|
"(*Renault* w pewnym kontekście może być nazwą firmy, w innym —\n",
|
||
|
|
"nazwiskiem).\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"##### Reprezentacja kontekstu\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Za pomocą wektora można przedstawić nie pojedynczy token $t^k$, lecz\n",
|
||
|
|
"cały kontekst, dla *okna* o długości $c$ będzie to kontekst $t^{k-c},\\dots,t^k,\\dots,t^{k+c}$.\n",
|
||
|
|
"Innymi słowy klasyfikujemy token na podstawie jego samego oraz jego kontekstu:\n",
|
||
|
|
"\n",
|
||
|
|
"$$p(l^k=i) = \\frac{e^{\\vec{w}\\vec{v}(t^{k-c},\\dots,t^k,\\dots,t^{k+c})}}{Z_k}.$$\n",
|
||
|
|
"\n",
|
||
|
|
"Zauważmy, że w tej metodzie w ogóle nie rozpatrujemy sensowności\n",
|
||
|
|
"sekwencji wyjściowej (etykiet), np. może być bardzo mało\n",
|
||
|
|
"prawdopodobne, że bezpośrednio po nazwisku występuje data.\n",
|
||
|
|
"\n",
|
||
|
|
"Napiszmy wzór określający prawdopodobieństwo całej sekwencji, nie\n",
|
||
|
|
"tylko pojedynczego tokenu. Na razie będzie to po prostu iloczyn poszczególnych wartości.\n",
|
||
|
|
"\n",
|
||
|
|
"$$p(l) = \\prod_{k=1}^K \\frac{e^{\\vec{w}\\vec{v}(t^{k-c},\\dots,t^k,\\dots,t^{k+c})}}{Z_k} = \\frac{e^{\\sum_{k=1}^K\\vec{w}\\vec{v}(t^{k-c},\\dots,t^k,\\dots,t^{k+c})}}{\\prod_{k=1}^K Z_k}$$\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"### Warunkowe pola losowe\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Warunkowe pola losowe (*Conditional Random Fields*, *CRF*) to klasa\n",
|
||
|
|
"modeli, które pozwalają uwzględnić zależności między punktami danych\n",
|
||
|
|
"(które można wyrazić jako graf). Najprostszym przykładem będzie prosty\n",
|
||
|
|
"graf wyrażający „następowanie po” (czyli sekwencje). Do poprzedniego\n",
|
||
|
|
"wzoru dodamy składnik $V_{i,j}$ (który można interpretować jako\n",
|
||
|
|
"macierz) określający prawdopodobieństwo, że po etykiecie o numerze $i$ wystąpi etykieta o numerze $j$.\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"#### **Pytanie**: Czy macierz $V$ musi być symetryczna? Czy $V_{i,j} = V_{j,i}$? Czy jakieś specjalne wartości występują na przekątnej?\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Macierz $V$ wraz z wektorem $\\vec{w}$ będzie stanowiła wyuczalne wagi w naszym modelu.\n",
|
||
|
|
"\n",
|
||
|
|
"Wartości $V_{i,j}$ nie stanowią bezpośrednio prawdopodobieństwa, mogą\n",
|
||
|
|
"przyjmować dowolne wartości, które będę normalizowane podobnie jak to się dzieje w funkcji Softmax.\n",
|
||
|
|
"\n",
|
||
|
|
"W takiej wersji warunkowych pól losowych otrzymamy następujący wzór na prawdopodobieństwo całej sekwencji.\n",
|
||
|
|
"\n",
|
||
|
|
"$$p(l) = \\frac{e^{\\sum_{k=1}^K\\vec{w}\\vec{v}(t^{k-c},\\dots,t^k,\\dots,t^{k+c}) + \\sum_{k=1}^{K-1} V_{l_k,l_{k+1}}}}{\\prod_{k=1}^K Z_k}$$\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"metadata": {
|
||
|
|
"kernelspec": {
|
||
|
|
"display_name": "Python 3",
|
||
|
|
"language": "python",
|
||
|
|
"name": "python3"
|
||
|
|
},
|
||
|
|
"language_info": {
|
||
|
|
"codemirror_mode": {
|
||
|
|
"name": "ipython",
|
||
|
|
"version": 3
|
||
|
|
},
|
||
|
|
"file_extension": ".py",
|
||
|
|
"mimetype": "text/x-python",
|
||
|
|
"name": "python",
|
||
|
|
"nbconvert_exporter": "python",
|
||
|
|
"pygments_lexer": "ipython3",
|
||
|
|
"version": "3.9.2"
|
||
|
|
},
|
||
|
|
"org": null
|
||
|
|
},
|
||
|
|
"nbformat": 4,
|
||
|
|
"nbformat_minor": 1
|
||
|
|
}
|