forked from filipg/aitech-eks-pub
2051 lines
132 KiB
Plaintext
2051 lines
132 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"\n",
|
|
"<div class=\"alert alert-block alert-info\">\n",
|
|
"<h1> Ekstrakcja informacji </h1>\n",
|
|
"<h2> 9. <i>Przegląd składowych sieci neuronowych</i> [wykład]</h2> \n",
|
|
"<h3> Filip Graliński (2021)</h3>\n",
|
|
"</div>\n",
|
|
"\n",
|
|
""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Neurozoo\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Kilka uwag dotyczących wektorów\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Wektor wierszowy $\\left[x_1,\\dots,x_n\\right]$ czy kolumnowy $\\left[\\begin{array}{c}\n",
|
|
" x_1 \\\\ \\vdots \\\\ x_n\\end{array}\\right]$?\n",
|
|
"\n",
|
|
"Często zakłada się wektor kolumny, będziemy używać **transpozycji**, by otrzymać wektor\n",
|
|
"wierszowy $\\vec{x}^T = \\left[x_1,\\dots,x_n\\right]$.\n",
|
|
"\n",
|
|
"W praktyce, np. w PyTorchu, może to nie mieć wielkiego znaczenia:\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"tensor([ 1.0000, -0.5000, 2.0000])"
|
|
]
|
|
},
|
|
"execution_count": 1,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import torch\n",
|
|
"x = torch.tensor([1.0, -0.5, 2.0])\n",
|
|
"x"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Musimy tylko uważać, jeśli przemnażamy wektor przez macierz!\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Funkcja sigmoidalna\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Funkcja sigmoidalna zamienia dowolną wartość („sygnał”) w wartość z przedziału $(0,1)$, czyli wartość, która może być interperetowana jako prawdopodobieństwo.\n",
|
|
"\n",
|
|
"$$\\sigma(x) = \\frac{1}{1 + e^{-x}}$$\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"tensor(0.6457)"
|
|
]
|
|
},
|
|
"execution_count": 2,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import torch\n",
|
|
"\n",
|
|
"def sigmoid(x):\n",
|
|
" return 1 / (1 + torch.exp(-x))\n",
|
|
"\n",
|
|
"sigmoid(torch.tensor(0.6))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"'sigmoid.png'"
|
|
]
|
|
},
|
|
"execution_count": 3,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
},
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAAhNUlEQVR4nO3deXyV5Z3+8c83+04gCVsIqyAguwHcWm2tDijuti5YFW39aWvHWtuqHaszdanLtFNn1PKjFpda12opVQpq6zYubMoWQiCGJWHLRvY9554/EpkMBgTMk+cs1/v1Oq/knOdJvI7Gcz3rfZtzDhERiVxRfgcQERF/qQhERCKcikBEJMKpCEREIpyKQEQkwsX4HeBIZWZmuuHDh/sdQ0QkpKxevbrcOZfV3bKQK4Lhw4ezatUqv2OIiIQUM9t+sGU6NCQiEuFUBCIiEU5FICIS4VQEIiIRzrMiMLOFZlZqZhsOstzM7D/NrNDM1pnZNK+yiIjIwXm5R/AkMOsQy2cDozsf1wG/9TCLiIgchGdF4Jx7F6g8xCrnAU+7Dh8B6WY2yKs8IiLSPT/vI8gGirs8L+l8bfeBK5rZdXTsNTB06NBeCSci0hucczS3BahpaqWuqY265rb9Xxta2qlvaaOhuePr8cP68pXR3d4T9qX4WQTWzWvdTo7gnFsALADIzc3VBAoiEpQCAce+hhbK61qoqG+moq6FfQ0tVNa3UNXQyr6Gjq/VjR2PmsZWapvaaGkPHNbvv+G0UWFXBCVATpfnQ4BdPmURETmkptZ2dlY1squqkd3VTeyuamJPTROlNU3srW2itKaZivoW2gPdb6umJcSQnhRH36RY0hJjGdI3kT6JHd+nJsSQmhBLWkIMyXExpCTEkBIfQ3J8DMlx0STFx5AYG010VHfbz1+en0WwGLjRzJ4HZgLVzrnPHRYSEekt1Y2tbC2vp6isjm0VDeyoqGd7ZQPFlY2U1zV/bv2M5DgGpCUwIC2e8YPSyEqNJyslnoyUeDJS4shMiadfchzpibHERAfv1fqeFYGZPQecBmSaWQlwFxAL4JybDywBzgIKgQZgnldZRES6qmlqpWBPLZt217B5bx1bSmspLK2jvK5l/zpRBoP6JDIsI4nTx/Ynu28iQ/omMjg9kcF9EumfFk9CbLSP76LneFYEzrnLvmC5A77v1T9fRAQ6tvLXlVSxrqSaDTurWb+zmpJ9jfuXpybEMLp/Cl8f259j+qcwIjOFEZnJDO2XRFxM8G7F96SQG31URORgnHNsr2hgxdZKVm6r5JPiKgpL6/YvH5aRxOScdC6fOZRxA9MYOyiVgWkJmHlz7D1UqAhEJKTtqmrkvwvL+aCwnA8+raC0tuNYft+kWKYN7cv5UwYzJacvE4f0oU9irM9pg5OKQERCSmt7gJXbKnm7oIy3NpWypXOLPzMljhNHZXLCyH7MGN6PY/qnRPyW/uFSEYhI0GtqbeftgjJez9vD3zeVUt3YSmy0MXNEBpdMz+GU0ZkcOyBVH/xHSUUgIkGptT3Af28pZ/HaXbyet4f6lnb6JMZy+rj+nDl+IKeMziQlXh9hPUH/FkUkqBTsqeWlVcUsWrOT8roW0hJimDNpMOdMHszMkf2IDeLr8UOVikBEfNfU2s5r63bzh4+2s6a4ipgo4/Rx/bn4+BxOHZMVMZdx+kVFICK+2VPdxFMfbuP5FTvY19DKyKxk7jh7HBdMzSYjJd7veBFDRSAivW7TnhoWvFvEX9fuoj3gOHP8QK48cRgnjsrQCV8fqAhEpNds2FnNf/1jC8vy9pIUF83cmcO49pQR5PRL8jtaRFMRiIjnCvbU8tCyAt7M30tqQgw3nT6aeScPJz0pzu9ogopARDxUsq+BX7+xmT9/spOUuBh+dMYYrj55OGkJusM3mKgIRKTHNbS08dhbn7LgvSIAvvuVkdxw6ij6JmsPIBipCESkxzjnWLx2F79csok9NU2cP2UwP5k1luz0RL+jySGoCESkR2yvqOeORRt4b0s5k4b04dG5Uzl+WD+/Y8lhUBGIyJfS1h5gwXtFPPzmFuKio7j7/AnMnTGUKI+mVZSepyIQkaNWWFrLLS+uZW1JNbMnDORfzz2OAWkJfseSI6QiEJEjFgg4Fr6/lQeXFZAcF81jc6dx1sRBfseSo6QiEJEjUlrbxC0vruW9LeV8Y9wAfnnhRLJSNRxEKFMRiMhhe3dzGT96cQ21TW3cd8FELpuRoyEhwoCKQES+UCDgePjvW3j471sYMyCFP37nBI4dmOp3LOkhKgIROaTqxlZufmEN/9hUykXThnDP+RNIjIv2O5b0IBWBiBzUlr21fOfpVezc18jd5x3HFScM06GgMKQiEJFuvbeljO898zHxsdE8f90J5A7XzWHhSkUgIp/zzEfbuWtxHqP7p/D7q6driIgwpyIQkf2cczywtID573zKacdm8V+XTSVVI4WGPRWBiAAdQ0Xc9sp6/rS6hMtnDuUX5x5HjCaKjwgqAhGhsaWdG5/9mL9vKuWm00fzw2+M1knhCKIiEIlw9c1tXPPkSlZsq+Tu8yfw7ROG+R1JepmKQCSC1TS1Mu+JlawpruI3l0zhvCnZfkcSH6gIRCJUdUMrVz6xgryd1Txy2VRma9C4iOXpmSAzm2VmBWZWaGa3dbO8j5n91czWmlmemc3zMo+IdKhpauXKhcvJ31XDb684XiUQ4TwrAjOLBh4FZgPjgcvMbPwBq30f2OicmwycBvzKzDSpqYiH6pvbmPfESvJ21fDY3GmcMX6A35HEZ17uEcwACp1zRc65FuB54LwD1nFAqnVcnpACVAJtHmYSiWhNre1856lVfLJjH/952VS+oRIQvC2CbKC4y/OSzte6egQYB+wC1gM3OecCB/4iM7vOzFaZ2aqysjKv8oqEtdb2ADc8s5qPtlbw629N0UQysp+XRdDdRcjugOf/BKwBBgNTgEfMLO1zP+TcAudcrnMuNysrq6dzioS9QMBx65/W8VZBGfeeP5Hzp+rqIPlfXhZBCZDT5fkQOrb8u5oHvOI6FAJbgbEeZhKJSPcv3cQrn+zkljPGcPnMoX7HkSDjZRGsBEab2YjOE8CXAosPWGcHcDqAmQ0AjgWKPMwkEnF+924RC94t4qoTh3Hj14/xO44EIc/uI3DOtZnZjcAyIBpY6JzLM7PrO5fPB+4GnjSz9XQcSrrVOVfuVSaRSLNk/W7uXZLP2RMHcdc5x2nYCOmWpzeUOeeWAEsOeG1+l+93AWd6mUEkUn28Yx83v7CG44f15VffmkxUlEpAuqehBUXCUHFlA999ahUD0hJY8O3jSYjV1JJycCoCkTBT29TKNU+upC3geGLedDJS4v2OJEFOYw2JhJFAwHHzC2soKq/nD9fMYFRWit+RJARoj0AkjPz6jc28mV/Kz88ex0nHZPodR0KEikAkTLy6bhePvFXIJbk5XHXScL/jSAhREYiEgYI9tfzkpXUcP6wvvzhfl4nKkVERiIS42qZWbnhmNcnxMfx27jTiY3SFkBwZnSwWCWHOOX7y0jq2Vzbw7Hdm0j8twe9IEoK0RyASwn73XhFL8/Zw26yxzByZ4XccCVEqApEQtXp7JQ8sLWD2hIF85ysj/I4jIUxFIBKC9tW38INnPyE7PZEHLp6kk8PypegcgUiIcc7xkz+tpayumZdvOIm0hFi/I0mI0x6BSIhZ+P423swv5fbZ45g0JN3vOBIGVAQiIWTDzmru/1s+3xg3gHknD/c7joQJFYFIiGhsaeefn/+EfslxPKTzAtKDdI5AJETc89pGisrq+eN3ZtI3Oc7vOBJGtEcgEgJez9vDH5fv4LqvjuRkDSYnPUxFIBLkSmubuPXldRw3OI1bzhzjdxwJQyoCkSDmnOP2l9fT0NLOw5dO0ThC4gkVgUgQe3FVMX/fVMqts8ZyTP9Uv+NImFIRiASp4soGfvHXjZw4MoOrNb+AeEhFIBKEAgHHLS+tJcqMf//WZKKidKmoeEdFIBKEnvhgGyu2VnLnOePJTk/0O46EORWBSJDZWl7PQ8s2cfrY/lx8/BC/40gEUBGIBJFAwPHTP60lLjqK+y6cqLuHpVeoCESCyFMfbmPltn3cec5xDNBsY9JLVAQiQWJ7RT0PLN3E147N4qJp2X7HkQiiIhAJAs45bnt5PbFROiQkvU9FIBIEXlhZzIdFFdx+1jgG9dFVQtK7VAQiPttb08S9S/I5YWQ/Lp2e43cciUAqAhEfOee4Y9EGWtoC3H/hJN04Jr7wtAjMbJaZFZhZoZnddpB1TjOzNWaWZ2bveJlHJNj8bcMe3ti4l1vOHMPwzGS/40iE8mxiGjOLBh4FzgBKgJVmttg5t7HLOunAY8As59wOM+vvVR6RYFPd2Mpdi/OYkJ3GNSeP8DuORDAv9whmAIXOuSLnXAvwPHDeAetcDrzinNsB4Jwr9TCPSFB5YOkmKuqauf/CScRE6yit+MfLv75soLjL85LO17oaA/Q1s7fNbLWZXdndLzKz68xslZmtKisr8yiuSO9Zua2SZ5fv4NpTRjAhu4/fcSTCeVkE3Z31cgc8jwGOB84G/gn4uZl9bgom59wC51yucy43Kyur55OK9KLmtnZuf2U92emJ3HyGZhwT/3k5eX0J0PVauCHArm7WKXfO1QP1ZvYuMBnY7GEuEV/9/3eKKCyt44l500mK8/J/QZHD4+UewUpgtJmNMLM44FJg8QHr/AX4ipnFmFkSMBPI9zCTiK+2ltfzyFuFzJk0iK8dq2sjJDh4tjninGszsxuBZUA0sNA5l2dm13cun++cyzezpcA6IAA87pzb4FUmET913DOwnvjoKO6cM97vOCL7ebpf6pxbAiw54LX5Bzx/CHjIyxwiweAva3bxfmEFd58/gf4aWVSCiK5ZE+kFVQ0t3PPaRqbkpDN3xlC/44j8HzpTJdILHlhawL6GVp6+ZqKGkZCgoz0CEY+t3r6P51bsYN5Jwxk/OM3vOCKfoyIQ8VBbe4A7Fm1gYFoCP9Q9AxKkVAQiHnryg23k767hrnPGkxKvI7ESnFQEIh7ZXd3If7yxmdOOzWLWhIF+xxE5KBWBiEfufnUjbQHHL86doKknJaipCEQ88HZBKUvW7+HGrx3D0Iwkv+OIHJKKQKSHNbW2c9fiPEZkJnPdqSP9jiPyhXT2SqSHzX/nU7ZXNPDMtTOJj4n2O47IF9IegUgP2lZez2Nvf8o5kwdzyuhMv+OIHBYVgUgPcc5x1+I84qKjuOPscX7HETlsKgKRHrIsbw/vbC7j5jPGMECDykkI+cIiMLMbzaxvb4QRCVX1zW382183MnZgKledOMzvOCJH5HD2CAYCK83sRTObZbogWuRz/vMfW9hd3cS9F0zQRPQScr7wL9Y5dwcwGvg9cDWwxczuM7NRHmcTCQmb99by+/e28q3cIRw/rJ/fcUSO2GFtujjnHLCn89EG9AX+ZGYPephNJOg55/j5og0kx8dw66yxfscROSpfeB+Bmf0zcBVQDjwO/MQ512pmUcAW4KfeRhQJXovW7GT51kruu2AiGSnxfscROSqHc0NZJnChc2571xedcwEzm+NNLJHgV93Qyr2v5TM5J51Lp+f4HUfkqH1hETjn7jzEsvyejSMSOv799QIq61t4ct4MzTomIU2XN4gchXUlVTyzfDtXnjicCdl9/I4j8qWoCESOUHvAcceiDWQkx/OjMzXrmIQ+FYHIEXp2+XbWlVTz8znjSEuI9TuOyJemIhA5AqW1TTy4rICTj8ng3MmD/Y4j0iNUBCJH4L7X8mluDXD3eZp1TMKHikDkMH1QWM6iNbu4/rRRjMxK8TuOSI9REYgchua2du74ywaGZSTxvdM0uoqEF81QJnIY5r9dRFFZPU9fM4OEWM06JuFFewQiX6CorI5H3y7knMmD+eqYLL/jiPQ4FYHIITjXcc9AfEwUP5+jWcckPKkIRA5h0ZqdfPBpBT+dNZb+qZp1TMKTp0XQOZFNgZkVmtlth1hvupm1m9nFXuYRORL76lu459V8puSkM3fGUL/jiHjGs5PFZhYNPAqcAZTQMcvZYufcxm7WewBY5lUWkaNx35J8qhpb+cMFEzWonIQ1L/cIZgCFzrki51wL8DxwXjfr/QB4GSj1MIvIEfmgsJyXVpdw3VdHMn5wmt9xRDzlZRFkA8Vdnpd0vrafmWUDFwDzD/WLzOw6M1tlZqvKysp6PKhIV02t7fzsz+sZlpHETaeP9juOiOe8LILu9qXdAc9/A9zqnGs/1C9yzi1wzuU653KzsnT5nnjrkX8Usq2igXvPn6h7BiQieHlDWQnQddqmIcCuA9bJBZ7vHLMlEzjLzNqcc4s8zCVyUPm7a5j/zqdcODWbU0Zn+h1HpFd4WQQrgdFmNgLYCVwKXN51BefciM++N7MngVdVAuKXtvYAt768jj6Jsfx8zni/44j0Gs+KwDnXZmY30nE1UDSw0DmXZ2bXdy4/5HkBkd72xPvbWFdSzX9dNpW+yXF+xxHpNZ6ONeScWwIsOeC1bgvAOXe1l1lEDmVbeT2/eqOAb4wbwJxJg/yOI9KrdGexRDznHLe/sp7YqCjuOV/zDEjkURFIxPvj8h18WFTB7WeNY2AfDSMhkUdFIBGtuLKBXy7J55RjMrlsRs4X/4BIGFIRSMRyznHbK+sAuP+iiTokJBFLRSAR67kVxbxfWMHPzh7HkL5JfscR8Y2KQCJScWUD9y3J56RRGVyukUUlwqkIJOIEAo4fv7QWgAcumqRDQhLxVAQScRa+v5XlWyu585zx5PTTISERFYFElC17a3lwWceNY988fojfcUSCgopAIkZre4CbX1xDSnwMv7xQVwmJfMbTISZEgslv3tzMhp01zL9iGlmp8X7HEQka2iOQiLC8qILH3v6Ub+UOYdYEjSUk0pWKQMJedUMrN7+whmH9krjrnOP8jiMSdHRoSMKac46fLVpPaW0zL99wEsnx+pMXOZD2CCSsvbS6hNfW7ebmM8YwOSfd7zgiQUlFIGFry95a7vzLBk4cmcH1p47yO45I0FIRSFhqbGnnxmc/ITkuhocvnUJ0lC4VFTkYHTCVsPSLV/Mo2FvLU9fMoH+a5hgQORTtEUjYWfTJTp5bUcwNp43i1DFZfscRCXoqAgkrBXtquf2V9Uwf3pcfnTHG7zgiIUFFIGGjpqmV659ZTUpCDI9ePo3YaP15ixwOnSOQsOCc4ycvrWVHZQPPffcEnRcQOQLaZJKw8Nt3PmVZ3l5unz2WGSP6+R1HJKSoCCTk/T1/Lw8tK2DOpEFce8oIv+OIhBwVgYS0wtJabnp+DeMHpfHQxZM1tLTIUVARSMiqbmjlu0+vJiE2igVX5pIYF+13JJGQpJPFEpJa2wN879nVlOxr4NnvnkB2eqLfkURClopAQo5zjjv+vIH3Cyt46OJJTB+uk8MiX4YODUnI+e07n/LCqmJ+8PVj+GZujt9xREKeikBCyqvrdvHg0gLOnTxYdw6L9BAVgYSM9wvLufmFNcwY3o8HL56kK4REeoinRWBms8yswMwKzey2bpbPNbN1nY8PzGyyl3kkdK0vqea6p1cxMjOF312ZS0KsrhAS6SmeFYGZRQOPArOB8cBlZjb+gNW2Aqc65yYBdwMLvMojoWtreT1XP7GC9KQ4nr52Bn2SYv2OJBJWvNwjmAEUOueKnHMtwPPAeV1XcM594Jzb1/n0I2CIh3kkBJXsa+CKx5cTcI6nr53BAI0hJNLjvCyCbKC4y/OSztcO5lrgb90tMLPrzGyVma0qKyvrwYgSzPZUNzH38eXUNLXyh2tnMiorxe9IImHJyyLo7kye63ZFs6/RUQS3drfcObfAOZfrnMvNytJEI5GgvK6ZuY9/RHltM09dM4MJ2X38jiQStry8oawE6HqR9xBg14Ermdkk4HFgtnOuwsM8EiLKajtKYGdVI0/Nm8G0oX39jiQS1rzcI1gJjDazEWYWB1wKLO66gpkNBV4Bvu2c2+xhFgkRe6qbuGTBhxRXNrLwqunMHJnhdySRsOfZHoFzrs3MbgSWAdHAQudcnpld37l8PnAnkAE81nlNeJtzLterTBLcdlY1cvnv/vdwkOYVEOkd5ly3h+2DVm5urlu1apXfMaSHFZbWcdXCFdQ0tfLUNTocJNLTzGz1wTa0Neic+G5NcRXznlhBdJTx3HdP0IlhkV6mIhBfvbu5jOufWU1GShx/uGYmwzOT/Y4kEnFUBOKb51bs4I5FGxgzIJWn5k3XhPMiPlERSK9rDzgeWLqJBe8WceqYLB65fCqpCRo2QsQvKgLpVbVNrfzoxbW8sXEvV544jDvnjCcmWoPgivhJRSC9prC0jv/3h1Vsq2jgX88Zz9Unj/A7koigIpBesnTDHn780lriY6J45tqZnDhKN4qJBAsVgXiqua2d+/+2iSfe38bknHTmXzGNQX000bxIMFERiGeKyur4wXOfkLerhnknD+e22WOJj9GEMiLBRkUgPc45x3MrirnntY3ExUTxuytzOWP8AL9jichBqAikR+2pbuLWl9fxzuYyTj4mg4cunszgdB0KEglmKgLpEYGA48VVxdy3JJ/WdscvzjuOK2YOIypKE8yLBDsVgXxphaW1/OyVDazYVsmMEf144KJJjNBQESIhQ0UgR62uuY1H/lHIwv/eSmJcNA9eNIlv5g6hc0hxEQkRKgI5YoGA45VPdvLA0k2U1TZz0bQh3H7WWDJT4v2OJiJHQUUgh805x9uby3hwaQH5u2uYnJPOgm8fz1TNHSAS0lQEclhWbqvkV68X8FFRJTn9EvnNJVM4d/JgnQwWCQMqAjmkj4oqePjNLXxYVEFmShz/es54Lp85jLgYDRQnEi5UBPI57QHHGxv3sODdIj7eUUVWajx3nD2OuTOHkRinO4NFwo2KQParbmzl5dUlPPXhNrZXNDC0XxL/du5xXDI9h4RYFYBIuFIRRDjnHOtKqnluxQ4WrdlJU2uAaUPTuW3WWM48biDROgcgEvZUBBGqrLaZv6zZyYuritm8t47E2GgumJrN3JnDNHm8SIRREUSQ6oZWluXtYfHaXXzwaTkBB1Ny0rn3ggnMmTSYPomaLlIkEqkIwtzemiZe37iX1/P28OGnFbQFHEP7JfG9047hvCmDGT0g1e+IIuIzFUGYaW0PsLa4ircLyniroJS8XTUAjMhM5tqvjGD2hEFMHtJHw0CIyH4qghDX1h4gf3cty7dW8MGnFSwvqqC+pZ3oKOP4oX356axjOX3sAMYMSNGHv4h0S0UQYqoaWlhTXMUnO6r4eMc+PtlRRV1zG9Cx1X/BtGxOHpXJSaMy6ZOkY/4i8sVUBEHKOUdZbTP5e2rZuKuGDTurWb+zmh2VDQBEGYwZkMr5UwczY0QGM4b3Y2CfBJ9Ti0goUhH4LBBw7K5pYmtZPUXldWzZW8fmvbVsKa2jsr5l/3pD+iYyMbsPl0zPYerQdCYNSSclXv/5ROTL0yeJx5xz1DS2sbOqkZ1VjZTsa6C4spEdlQ3sqKxnR2UDTa2B/eunJsQwZkAqZ44fwNiBqYwdlMbYgamkJ8X5+C5EJJypCI5SIOCoamyloq6Z8roWyuqaKa9tprS2mdKaJvbWNrG7uondVU00trb/n59NiI1iWL9khmUk89XRWYzMSmFEZjIjs5Lpnxqvk7oi0qs8LQIzmwU8DEQDjzvn7j9guXUuPwtoAK52zn3sZabPOOdobgtQ39xGfXM7tc2t1DW1UdvURm1zKzWNbdQ0tlLd2EpVYytVDa1UNbSwr6GFqoZW9jW0EHCf/72x0Ub/1AQGpMVz7IBUThvTn8HpCQzqk8iQvolk900kIzlOH/YiEjQ8KwIziwYeBc4ASoCVZrbYObexy2qzgdGdj5nAbzu/9ri3Ckq559WNNLS0dz7aaG3v5pP8AElx0aQnxpKWGEvfpDiO7TxMk5EcR7/OR2ZKPP1T48lMiSc9KVYf8iISUrzcI5gBFDrnigDM7HngPKBrEZwHPO2cc8BHZpZuZoOcc7t7OkyfxFjGDkojOS6apLgYkuKiSY6PISU+Zv/X1ISOr2mJsaQlxJCaEKtx90Uk7HlZBNlAcZfnJXx+a7+7dbKB/1MEZnYdcB3A0KFDjyrMtKF9mXa5plQUETmQl5u73R0fOfBYzOGsg3NugXMu1zmXm5WV1SPhRESkg5dFUALkdHk+BNh1FOuIiIiHvCyClcBoMxthZnHApcDiA9ZZDFxpHU4Aqr04PyAiIgfn2TkC51ybmd0ILKPj8tGFzrk8M7u+c/l8YAkdl44W0nH56Dyv8oiISPc8vY/AObeEjg/7rq/N7/K9A77vZQYRETk0XRspIhLhVAQiIhFORSAiEuGs4zB96DCzMmC73zmOQiZQ7neIXqb3HP4i7f1C6L7nYc65bm/ECrkiCFVmtso5l+t3jt6k9xz+Iu39Qni+Zx0aEhGJcCoCEZEIpyLoPQv8DuADvefwF2nvF8LwPescgYhIhNMegYhIhFMRiIhEOBWBD8zsx2bmzCzT7yxeMrOHzGyTma0zsz+bWbrfmbxiZrPMrMDMCs3sNr/zeM3McszsLTPLN7M8M7vJ70y9xcyizewTM3vV7yw9RUXQy8wsh455nHf4naUXvAFMcM5NAjYDt/ucxxNd5ueeDYwHLjOz8f6m8lwbcItzbhxwAvD9CHjPn7kJyPc7RE9SEfS+/wB+SjczsYUb59zrzrm2zqcf0THxUDjaPz+3c64F+Gx+7rDlnNvtnPu48/taOj4Ys/1N5T0zGwKcDTzud5aepCLoRWZ2LrDTObfW7yw+uAb4m98hPHKwubcjgpkNB6YCy32O0ht+Q8eGXMDnHD3K0/kIIpGZvQkM7GbRvwA/A87s3UTeOtT7dc79pXOdf6HjUMIfezNbLzqsubfDkZmlAC8DP3TO1fidx0tmNgcodc6tNrPTfI7To1QEPcw5943uXjezicAIYK2ZQcdhko/NbIZzbk8vRuxRB3u/nzGzq4A5wOkufG9aici5t80slo4S+KNz7hW/8/SCk4FzzewsIAFIM7NnnHNX+JzrS9MNZT4xs21ArnMuFEcxPCxmNgv4NXCqc67M7zxeMbMYOk6Gnw7spGO+7sudc3m+BvOQdWzNPAVUOud+6HOcXte5R/Bj59wcn6P0CJ0jEC89AqQCb5jZGjOb/0U/EIo6T4h/Nj93PvBiOJdAp5OBbwNf7/xvu6ZzS1lCkPYIREQinPYIREQinIpARCTCqQhERCKcikBEJMKpCEREIpyKQEQkwqkIREQinIpA5Esys+mdcy4kmFly5/j8E/zOJXK4dEOZSA8ws3voGH8mEShxzv3S50gih01FINIDzCyOjjGGmoCTnHPtPkcSOWw6NCTSM/oBKXSMrZTgcxaRI6I9ApEeYGaL6ZiZbAQwyDl3o8+RRA6b5iMQ+ZLM7EqgzTn3bOf8xR+Y2dedc//wO5vI4dAegYhIhNM5AhGRCKciEBGJcCoCEZEIpyIQEYlwKgIRkQinIhARiXAqAhGRCPc/OnN4AJzLYKEAAAAASUVORK5CYII=\n",
|
|
"text/plain": [
|
|
"<Figure size 432x288 with 1 Axes>"
|
|
]
|
|
},
|
|
"metadata": {
|
|
"needs_background": "light"
|
|
},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"%matplotlib inline\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"import torch\n",
|
|
"\n",
|
|
"x = torch.linspace(-5,5,100)\n",
|
|
"plt.xlabel(\"x\")\n",
|
|
"plt.ylabel(\"y\")\n",
|
|
"plt.plot(x, sigmoid(x))\n",
|
|
"fname = 'sigmoid.png'\n",
|
|
"plt.savefig(fname)\n",
|
|
"fname"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"[[file:# Out[32]:\n",
|
|
"\n",
|
|
" 'sigmoid.png'\n",
|
|
"\n",
|
|
"]]\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### PyTorch\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Funkcja `torch.sigmoid` po prostu stosuje sigmoidę do każdego elementu tensora (*element-wise*).\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"tensor([0.6457, 0.7311, 0.0067])"
|
|
]
|
|
},
|
|
"execution_count": 4,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import torch\n",
|
|
"\n",
|
|
"torch.sigmoid(torch.tensor([0.6, 1.0, -5.0]))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Istnieje również `torch.nn.Sigmoid`, które może być używane jako warstwa.\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"tensor([0.5000, 0.4502, 0.5987])"
|
|
]
|
|
},
|
|
"execution_count": 5,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import torch.nn as nn\n",
|
|
"\n",
|
|
"s = nn.Sigmoid()\n",
|
|
"s(torch.tensor([0.0, -0.2, 0.4]))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"##### Implementacja w Pytorchu\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"tensor([0.5000, 0.6225, 0.5744])"
|
|
]
|
|
},
|
|
"execution_count": 6,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import torch.nn as nn\n",
|
|
"import torch\n",
|
|
"\n",
|
|
"class MySigmoid(nn.Module):\n",
|
|
" def __init__(self):\n",
|
|
" super(MySigmoid, self).__init__()\n",
|
|
"\n",
|
|
" def forward(self, x):\n",
|
|
" return 1 / (1 + torch.exp(-x))\n",
|
|
"\n",
|
|
"s = MySigmoid()\n",
|
|
"s(torch.tensor([0.0, 0.5, 0.3]))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Wagi\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Funkcja sigmoidalna nie ma żadnych wyuczalnych wag.\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"##### **Pytanie**: Czy można rozszerzyć funkcję sigmoidalną o jakieś wyuczalne wagi?\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Regresja liniowa\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Iloczyn skalarny — przypomnienie\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"$$\\left[1.0, -0.5, 2.0\\right]\n",
|
|
" \\left[\\begin{array}{c}\n",
|
|
" 3.0 \\\\\n",
|
|
" 1.5 \\\\\n",
|
|
" 0.0\\end{array}\\right]\n",
|
|
" =\n",
|
|
" 1.0 \\cdot 3.0 + -0.5 \\cdot 1.5 + 2.0 \\cdot 0.0 = 2.25$$\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"##### Intuicje\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"- $\\vec{a}^T \\vec{b}$ mierzy jak bardzo $\\vec{a}$ „pasuje” do\n",
|
|
" $\\vec{b}$,\n",
|
|
"- … zwłaszcza gdy znormalizujemy wektory dzieląc przez $|\\vec{a}|$ i $|\\vec{b}|$:\n",
|
|
" $\\frac{\\vec{a}^T \\vec{b}}{|\\vec{a}||\\vec{b}|} = \\cos \\theta$,\n",
|
|
" gdzie $\\theta$ to kąt pomiędzy $\\vec{a}$ and $\\vec{b}$ (podobieństwo kosinusowe!)\n",
|
|
"- co, jeśli if $\\vec{a}^T \\vec{b} = 0$? — $\\vec{a}$ i $\\vec{b}$ są prostopadłe, np.\n",
|
|
" $\\left[1, 2\\right] \\cdot \\left[-2, -1\\right]^T = 0$\n",
|
|
"- a co, jeśli $\\vec{a}^T \\vec{b} = -1$ — wektor są skierowane w przeciwnym kierunku, jeśli dodatkowo $|\\vec{a}|=|\\vec{b}|=1$, np.\n",
|
|
" $\\left[\\frac{\\sqrt{2}}{2},\\frac{\\sqrt{2}}{2}\\right] \\cdot \\left[-\\frac{\\sqrt{2}}{2},-\\frac{\\sqrt{2}}{2}\\right]^T = -1$\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"##### W PyTorchu\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"tensor(2.2500)"
|
|
]
|
|
},
|
|
"execution_count": 7,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import torch\n",
|
|
"x = torch.tensor([1.0, -0.5, 2.0])\n",
|
|
"y = torch.tensor([3.0, 1.5, 0.0])\n",
|
|
"x @ y"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Regresja liniowa jako element sieci neuronowej\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Przypomnijmy sobie wzór na regresję liniową:\n",
|
|
"\n",
|
|
"$$y = w_0 + w_1x_1 + w_2x_2 + \\dots + w_{|V|}x_{|v|}$$\n",
|
|
"\n",
|
|
"Jeśli wprowadzimy sztuczny element wektora $\\vec{x}$ ustawiony zawsze na 1 ($x_0 = 1$), wówczas\n",
|
|
"wzór może przyjąc bardziej zwartą postać:\n",
|
|
"\n",
|
|
"$$y = \\sum_{i=0}^{|V|} w_ix_i = \\vec{w}\\vec{x}$$\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### PyTorch\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"##### Implementacja w PyTorchu\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Zakładamy, że wektor wejściowy **nie** obejmuje dodatkowego elementu $x_0 = 1$.\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"tensor(0., dtype=torch.float64, grad_fn=<AddBackward0>)"
|
|
]
|
|
},
|
|
"execution_count": 8,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import torch\n",
|
|
"import torch.nn as nn\n",
|
|
"\n",
|
|
"class MyLinearRegressor(nn.Module):\n",
|
|
" def __init__(self, vlen):\n",
|
|
" super(MyLinearRegressor, self).__init__()\n",
|
|
" self.register_parameter(name='w', param=torch.nn.Parameter(\n",
|
|
" torch.zeros(vlen, dtype=torch.double, requires_grad=True)))\n",
|
|
" self.register_parameter(name='b', param=torch.nn.Parameter(\n",
|
|
" torch.tensor(0., dtype=torch.double, requires_grad=True)))\n",
|
|
"\n",
|
|
" def forward(self, x):\n",
|
|
" return self.b + x @ self.w\n",
|
|
"\n",
|
|
"regressor = MyLinearRegressor(3)\n",
|
|
"regressor(torch.tensor([0.3, 0.4, 1.0], dtype=torch.double))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"##### Gotowy moduł w PyTorchu\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Możemy skorzystać z ogólniejszej konstrukcji — warstwy liniowej (ale,\n",
|
|
"uwaga!, na wyjściu będzie wektor jednoelementowy).\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 9,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"tensor([-0.0557], grad_fn=<AddBackward0>)"
|
|
]
|
|
},
|
|
"execution_count": 9,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import torch\n",
|
|
"import torch.nn as nn\n",
|
|
"\n",
|
|
"regressor = torch.nn.Linear(in_features=3, out_features=1, bias=True)\n",
|
|
"regressor(torch.tensor([0.3, 0.4, 1.0]))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Zastosowania\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Bezpośrednio możemy zastosować do zadania regresji dla tekstu (np.\n",
|
|
"przewidywanie roku publikacji tekstu).\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"W połączeniu z sigmoidą otrzymamy regresją logistyczną, np. dla zadania klasyfikacji tekstu:\n",
|
|
"\n",
|
|
"$$p(c|\\vec{x}) = \\sigma(w_0 + w_1x_1 + w_2x_2 + \\dots + w_{|V|}x_{|v})\n",
|
|
"= \\sigma(\\Sigma_{i=0}^{|V|} w_ix_i) = \\sigma(\\vec{w}\\vec{x})$$\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"Tak sieć będzie aktywowana dla tekstu <u>aardvark in Aachen</u>:\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"Regresje logistyczną (liniową zresztą też) dla tekstu możemy połączyć z trikiem z haszowaniem:\n",
|
|
"\n",
|
|
"$$p(c|\\vec{x}) = \\sigma(w_0 + w_1x_1 + w_2x_2 + \\dots + w_{2^b}x_{2^b})\n",
|
|
"= \\sigma(\\Sigma_{i=0}^{2^b} w_ix_i) = \\sigma(\\vec{w}\\vec{x})$$ \n",
|
|
"{\\small hashing function $H : V \\rightarrow \\{1,\\dots,2^b\\}$,\n",
|
|
" e.g. MurmurHash3}\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"****Pytanie:**** Jaki tekst otrzyma na pewno taką samą klasę jak <u>aardvark in Aachen</u>?\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Wagi\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Liczba wag jest równa rozmiarowi wektora wejściowego (oraz opcjonalnie\n",
|
|
"obciążenie).\n",
|
|
"\n",
|
|
"Każda waga odpowiada wyrazowi ze słownika, możemy więc interpretować\n",
|
|
"wagi jako jednowymiarowy parametr opisujący słowa.\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Warstwa liniowa\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Mnożenie macierzy przez wektor — przypomnienie\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Mnożenie macierzy przez wektor można interpretować jako zrównolegloną operację mnożenie wektora przez wektor.\n",
|
|
"\n",
|
|
"$$\\left[\\begin{array}{ccc}\n",
|
|
" \\alert<2>{1.0} & \\alert<2>{-2.0} & \\alert<2>{3.0} \\\\\n",
|
|
" \\alert<3>{-2.0} & \\alert<3>{0.0} & \\alert<3>{10.0}\\end{array}\\right]\n",
|
|
" \\left[\\begin{array}{c}\n",
|
|
" \\alert<2-3>{1.0} \\\\\n",
|
|
" \\alert<2-3>{-0.5} \\\\\n",
|
|
" \\alert<2-3>{2.0}\\end{array}\\right]\n",
|
|
" =\n",
|
|
" \\left[\\begin{array}{c}\n",
|
|
" \\uncover<2->{\\alert<2>{8.0}} \\\\\n",
|
|
" \\uncover<3->{\\alert<3>{18.0}}\\end{array}\\right]$$\n",
|
|
"\n",
|
|
"Jeśli przemnożymy macierz $n \\times m$ przez wektor kolumnowy o długości\n",
|
|
"$m$, otrzymamy wektor o rozmiarze $n$.\n",
|
|
"\n",
|
|
"W PyTorchu:\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 10,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"tensor([ 8., 18.])"
|
|
]
|
|
},
|
|
"execution_count": 10,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import torch\n",
|
|
"m = torch.tensor([[1.0, -2.0, 3.0],\n",
|
|
" [-2.0, 0.0, 10.0]])\n",
|
|
"x = torch.tensor([1.0, -0.5, 2.0])\n",
|
|
"m @ x"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"[[file:# Out[19]:\n",
|
|
"\n",
|
|
" tensor([ 8., 18.])]]\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Definicja warstwy liniowej\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Warstwa liniowa polega na przemnożeniu wejścia przez macierz. Można\n",
|
|
"to intepretować jako zrównolegloną operację regresji liniowej (równolegle\n",
|
|
"uczymy czy wykonujemy $n$ regresji liniowych).\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### PyTorch\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Warstwa liniowa, która przyjmuje wektor o rozmiarze 3 i zwraca wektor o rozmiarze 2.\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 11,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"tensor([0.7000, 0.2270], grad_fn=<AddBackward0>)"
|
|
]
|
|
},
|
|
"execution_count": 11,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import torch\n",
|
|
"import torch.nn as nn\n",
|
|
"\n",
|
|
"regressor = torch.nn.Linear(in_features=3, out_features=2, bias=True)\n",
|
|
"regressor(torch.tensor([0.3, 0.4, 1.0]))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Pytanie**: Ile wag (parametrów) ma powyżej użyta warstwa?\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Zastosowania\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Warstwa liniowa jest podstawowym elementem sieci neuronowych —\n",
|
|
"począwszy od prostych sieci neuronowych feed-forward, gdzie warstwy\n",
|
|
"liniowe łączymy używając funkcji aktywacji (np. sigmoidy).\n",
|
|
"\n",
|
|
"Oto przykład prostej dwuwarstwowej sieci neuronowej do klasyfikacji binarnej.\n",
|
|
"\n",
|
|
"\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Softmax\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"W klasyfikacji wieloklasowej należy zwrócić musimy zwrócić rozkład\n",
|
|
"prawdopodobieństwa po wszystkich klasach, w przeciwieństwie do\n",
|
|
"klasyfikacji binarnej, gdzie wystarczy zwrócić jedną liczbę —\n",
|
|
"prawdopodobieństwo pozytywnej klasy ($p$; prawdopodobieństwo drugiej\n",
|
|
"klasy to po prostu $1-p$).\n",
|
|
"\n",
|
|
"A zatem na potrzeby klasyfikacji wieloklasowej potrzeba wektorowego\n",
|
|
"odpowiednika funkcji sigmoidalnej, to jest funkcji, która zamienia\n",
|
|
"nieznormalizowany wektor $\\vec{z} = [z_1,\\dots,z_k]$ (pochodzący np. z\n",
|
|
"poprzedzającej warstwy liniowej) na rozkład prawdopobieństwa.\n",
|
|
"Potrzebujemy zatem funkcji $s: \\mathcal{R}^k \\rightarrow [0,1]^k$\n",
|
|
"\n",
|
|
"spełniającej następujące warunki:\n",
|
|
"\n",
|
|
"- $s(z_i) = s_i(z) \\in [0,1]$\n",
|
|
"- $\\Sigma_i s(z_i) = 1$\n",
|
|
"- $z_i > z_j \\Rightarrow s(z_i) > s(z_j)$\n",
|
|
"\n",
|
|
"Można by podać takie (**błędne**!) rozwiązanie:\n",
|
|
"\n",
|
|
"$$s(z_i) = \\frac{z_i}{\\Sigma_{j=1}^k z_j}$$\n",
|
|
"\n",
|
|
"To rozwiązanie zadziała błędnie dla liczb ujemnych, trzeba najpierw\n",
|
|
"użyć funkcji monotonicznej, która przekształaca $\\mathcal{R}$ na $\\mathcal{R^+}$.\n",
|
|
"Naturalna funkcja tego rodzaju to funkcja wykładnicza $\\exp{x} = e^x$.\n",
|
|
"Tym sposobem dochodzimy do funkcji softmax:\n",
|
|
"\n",
|
|
"$$s(z_i) = \\frac{e^{z_i}}{\\Sigma_{j=1}^k e^{z_j}}$$\n",
|
|
"\n",
|
|
"Mianownik ułamka w definicji funkcji softmax nazywamy czasami czynnikiem normalizacyjnym:\n",
|
|
"$Z(\\vec{z}) = \\Sigma_{j=1}^k e^{z_j}$, wtedy:\n",
|
|
"\n",
|
|
"$$s(z_i) = \\frac{e^{z_i}}{Z(\\vec{z})}$$\n",
|
|
"\n",
|
|
"Definicja w PyTorchu:\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 12,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"tensor([0.1182, 0.0022, 0.0059, 0.8737])"
|
|
]
|
|
},
|
|
"execution_count": 12,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import torch\n",
|
|
"\n",
|
|
"def softmax(z):\n",
|
|
" z_plus = torch.exp(z)\n",
|
|
" return z_plus / torch.sum(z_plus)\n",
|
|
"\n",
|
|
"softmax(torch.tensor([3., -1., 0., 5.]))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"![img](./softmax.png \"Softmax\")\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Soft vs hard\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Dlaczego *softmax*? Czasami używa się funkcji **hardmax**, która np.\n",
|
|
"wektora $[3, -1, 0, 5]$ zwróciłaby $[0, 0, 0, 5]$ — to jest po prostu\n",
|
|
"wektorowa wersja funkcji zwracającej maksimum. Istnieje też funkcja\n",
|
|
"hard\\*arg\\*max, która zwraca wektor *one-hot* — z jedną jedynką na\n",
|
|
"pozycji dla największej wartości (zamiast podania największej\n",
|
|
"wartości), np. wartość hardargmax dla $[3, -1, 0, 5]$ zwróciłaby $[0,\n",
|
|
"0, 0, 1]$.\n",
|
|
"\n",
|
|
"Zauważmy, że powszechnie przyjęta nazwa *softmax* jest właściwie\n",
|
|
"błędna, funkcja ta powinna nazywać się *softargmax*, jako że w\n",
|
|
"„miękki” sposób identyfikuje największą wartość przez wartość zbliżoną\n",
|
|
"do 1 (na pozostałych pozycjach wektora nie będzie 0).\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"##### **Pytanie**: Jak można zdefiniować funkcję *softmax* w ścisłym tego słowa znaczeniu („miękki” odpowiednik hardmax, nie hardargmax)?\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### PyTorch\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Funkcja `torch.nn.functional.softmax` normalizuje wartości dla całego tensora:\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 13,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"<ipython-input-13-e808e5e4899b>:3: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n",
|
|
" nn.functional.softmax(torch.tensor([0.6, 1.0, -5.0]))\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"tensor([0.4007, 0.5978, 0.0015])"
|
|
]
|
|
},
|
|
"execution_count": 13,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import torch.nn as nn\n",
|
|
"\n",
|
|
"nn.functional.softmax(torch.tensor([0.6, 1.0, -5.0]))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"… zobaczmy, jak ta funkcja zachowuje się dla macierzy:\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 14,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"<ipython-input-14-95deaae56e16>:3: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n",
|
|
" nn.functional.softmax(torch.tensor([[0.6, 1.0], [-2.0, 3.5]]))\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"tensor([[0.4013, 0.5987],\n",
|
|
" [0.0041, 0.9959]])"
|
|
]
|
|
},
|
|
"execution_count": 14,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import torch.nn as nn\n",
|
|
"\n",
|
|
"nn.functional.softmax(torch.tensor([[0.6, 1.0], [-2.0, 3.5]]))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Za pomocą (zalecanego zresztą) argumentu `dim` możemy określić wymiar, wzdłuż którego dokonujemy normalizacji:\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 15,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"tensor([[0.9309, 0.0759],\n",
|
|
" [0.0691, 0.9241]])"
|
|
]
|
|
},
|
|
"execution_count": 15,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import torch.nn as nn\n",
|
|
"\n",
|
|
"nn.functional.softmax(torch.tensor([[0.6, 1.0], [-2.0, 3.5]]), dim=0)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Istnieje również `torch.nn.Softmax`, które może być używane jako warstwa.\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 16,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"tensor([0.3021, 0.2473, 0.4506])"
|
|
]
|
|
},
|
|
"execution_count": 16,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import torch.nn as nn\n",
|
|
"\n",
|
|
"s = nn.Softmax(dim=0)\n",
|
|
"s(torch.tensor([0.0, -0.2, 0.4]))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"##### Implementacja w Pytorchu\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 17,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"tensor([0.5000, 0.6225, 0.5744])"
|
|
]
|
|
},
|
|
"execution_count": 17,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import torch.nn as nn\n",
|
|
"import torch\n",
|
|
"\n",
|
|
"class MySoftmax(nn.Module):\n",
|
|
" def __init__(self):\n",
|
|
" super(MySoftmax, self).__init__()\n",
|
|
"\n",
|
|
" def forward(self, x):\n",
|
|
" ex = torch.exp(x)\n",
|
|
" return ex / torch.sum(ex)\n",
|
|
"\n",
|
|
"s = MySigmoid()\n",
|
|
"s(torch.tensor([0.0, 0.5, 0.3]))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"###### **Pytanie**: Tak naprawdę wyżej zdefiniowana klasa `MySoftmax` nie zachowuje się identycznie jak `nn.Softmax`. Na czym polega różnica?\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Przypadek szczególny\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Sigmoida jest przypadkiem szczególnym funkcji softmax:\n",
|
|
"\n",
|
|
"$$\\sigma(x) = \\frac{1}{1 + e^{-x}} = \\frac{e^x}{e^x + 1} = \\frac{e^x}{e^x + e^0} = s([x, 0])_1$$\n",
|
|
"\n",
|
|
"Ogólniej: softmax na dwuelementowych wektorach daje przesuniętą sigmoidę (przy ustaleniu jednej z wartości).\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 18,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"'softmax3.png'"
|
|
]
|
|
},
|
|
"execution_count": 18,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
},
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAAgb0lEQVR4nO3dd3yV9d3/8dcnm4RNwt4QpopIBESL1lHBRfXXu+6FLdpqHb3vVuqotra189Z6i0WLaB0U60ZFLVWr1kEhiECYIRgSVhICYWSd8f39cY5timEIuXKd8X4+HueRc40k7/MgnPe55tecc4iISPJK8TuAiIj4S0UgIpLkVAQiIklORSAikuRUBCIiSU5FICKS5DwrAjObbWYVZrZiP8vNzB4ws2IzW2Zmx3mVRURE9s/LLYLHgUkHWD4ZyI8+pgF/8DCLiIjsh2dF4Jx7D6g+wCpTgCdcxMdARzPr4VUeERFpXpqPv7sXUNZkujw6b8u+K5rZNCJbDeTk5IwZNmxYqwQUEUkUhYWFVc65vOaW+VkE1sy8Zu934Zx7BHgEoKCgwC1evNjLXCIiCcfMSve3zM+zhsqBPk2mewObfcoiIpK0/CyCecAV0bOHxgM1zrkv7BYSERFvebZryMz+DJwC5JpZOXAXkA7gnJsJzAfOAoqBWuBqr7KIiMj+eVYEzrmLD7LcAdd79ftFROTQ6MpiEZEkpyIQEUlyKgIRkSSnIhARiXEbt9fy8Lvr+XB9lSc/388LykREZD9Kt+9l3tLNvL5iKyu37ALgO6cMYsKg3Bb/XSoCEZEYUb23kZeXbuLlpZtZWrYTgIJ+nbjj7OGcObI7fTpne/J7VQQiIj4Khx0frK9i7qIyFhRtozEUZniP9vxo8jDOHdWTnh3beJ5BRSAi4oOaugDPFZbz5Eef8dn2Wjpmp3PJuL5ceHwfhvdo36pZVAQiIq2orLqWR/+xgWcWlVEXCDGmXyduOWMIk47qTmZaqi+ZVAQiIq1gzdbdzHinmNeWbyHF4LxRvbj6xP4c1auD39FUBCIiXlq7bTe/f2sd85dvITs9lWtOGsDVJ/anRwfv9/0fKhWBiIgHynfU8ru/ruWlpZvITk/l+lMGc81JA+iUk+F3tC9QEYiItKCa2gAPvrOOP31YihlcO3EQ104cGJMF8DkVgYhICwiHHX9ZXMav31zDjtpGvnFcb245Y0irnP55pFQEIiJHaFn5Tu58aQWfltdwfP9O3H3eWEb29P8g8KFSEYiIHKbaxiD3LVjLo//YQJe2mdx/4bFMObYnZs0NyR67VAQiIofhw+Iqbn1hGWXVdVwyri+3ThpGhzbpfsc6LCoCEZEvoT4Q4tdvrGH2BxsYkJvD3GnjGT+wi9+xjoiKQETkEBVtruHmuUtZV7GHK0/ox/TJw2mT4c/VwC1JRSAichDOOZ76uJR7Xl1Fp5x0npg6lolD8vyO1WJUBCIiB7C7PsD0F5bz2rItfHVoHr/75rF0juFrAg6HikBEZD+KK3Yz7YlCSqtruXXSMK6dOJCUlPg6I+hQqAhERJqxYOU2bnlmKVnpKcz51jjGxfkB4QNREYiINOGc48G3i/ndgrUc3asDD18+Ji6uDj4SKgIRkajGYJgfvbCc55eUc/7oXtx7wdFkpcf/WUEHoyIQESEyYth3nirkw/XbueX0Idx42uC4u0L4cKkIRCTpbdtVz+WPLqSkci+/+69R/L8xvf2O1KpUBCKS1Eq37+XSWQvZsbeRP00dy4mDc/2O1OpUBCKStFZt2cUVs/9JMBRmzrfHM6pPR78j+UJFICJJaXl5DZc9upA26anMufYE8ru18zuSb1QEIpJ0lpXv5LJZC2mXlc7caePp0znb70i+UhGISFL5vATat4mUQO9OyV0CACl+BxARaS0rNtVw6ayFdMhWCTTlaRGY2SQzW2NmxWY2vZnlHczsFTP71MyKzOxqL/OISPIqrtjDlbP/SbvMNP78bZVAU54VgZmlAjOAycAI4GIzG7HPatcDK51zo4BTgN+ZWWLd1k9EfFdWXctlsxZiZjytEvgCL7cIxgLFzrkS51wjMBeYss86Dmhnkcv32gLVQNDDTCKSZCp3N3DZowupbQzy5DVjGZCb43ekmONlEfQCyppMl0fnNfUgMBzYDCwHbnLOhff9QWY2zcwWm9niyspKr/KKSILZ2xBk6uOLqNjVwONTxzK8R3u/I8UkL4uguZt0uH2mzwSWAj2BY4EHzewL/1LOuUeccwXOuYK8vMQZFUhEvBMIhfnu00tYuWUXMy4dzXF9O/kdKWZ5WQTlQJ8m072JfPJv6mrgBRdRDGwAhnmYSUSSgHOO215YzrtrK/n514/i1GHd/I4U07wsgkVAvpkNiB4AvgiYt886G4HTAMysGzAUKPEwk4gkgRnvFPNsYTk3npbPRWP7+h0n5nl2QZlzLmhmNwBvAqnAbOdckZldF10+E7gHeNzMlhPZlXSrc67Kq0wikvheX76F3/51LeeP7sUtp+f7HScueHplsXNuPjB/n3kzmzzfDHzNywwikjyWl9dwy1+WMqZfJ+694OikGU/gSOnKYhFJCBW76vnWE4vokpPJw5ePSYqRxVqK7jUkInGvMRjmuqcK2V0f5PnvTCC3babfkeKKikBE4t5PXiliycadPHTpcbpW4DBo15CIxLVnFm3k6YUbue7kQZx1dA+/48QlFYGIxK1Py3Zy50tFfCU/lx+cOdTvOHFLRSAicammNsB3n15CXrtMHrhoNKkpOkPocOkYgYjEHecc//3sp1TsrufZ6ybQKUc3LT4S2iIQkbgz6/0N/G3VNn40eTjHJumA8y1JRSAicaWwtJpfvrGaSSO7c/WJ/f2OkxBUBCISN2rqAtz456X07JjFr//rGF053EJ0jEBE4oJzjttfXM7WXfU8e90JtM9K9ztSwtAWgYjEhecKy3l12Ra+f8YQjS3QwlQEIhLzNlTt5a55RYwf2JnrTh7kd5yEoyIQkZgWCIW5ee4nZKSlcP+Ful7ACzpGICIxbcY7xXxaXsNDlx5H9w5ZfsdJSNoiEJGYtbRsJ//3djHnj+6l+wh5SEUgIjGprjHE959ZStd2mdx93ki/4yQ07RoSkZj0qzdWU1K1lznfGkeHNjpV1EvaIhCRmPNxyXYe//AzrprQnwmDc/2Ok/BUBCISU2obg/zwuWX065LNDyfp1tKtQbuGRCSm/Or11WysruWZaePJztBbVGvQFoGIxIyPS7bzp49KuWpCf8YN7OJ3nKShIhCRmFDXGOLW57VLyA/a7hKRmHDf39ZSur2WOd8ep11CrUxbBCLiu2XlO5n1fgkXj+3DhEE6S6i1qQhExFeBUJgfPreM3LaZTJ883O84SUnbXyLiq4ffXc/qrbt55PIxunDMJ9oiEBHflFTu4YG3iznr6O58bWR3v+MkLRWBiPgiMuLYCjLTUrj7XN1LyE8qAhHxxQtLNvFRyXZunTSMru11e2k/qQhEpNVV723kZ6+t5Li+HblkbF+/4yQ9FYGItLpfzF/F7vog915wDCkaccx3nhaBmU0yszVmVmxm0/ezzilmttTMiszsXS/ziIj/FpZs57nCcr49cSBDu7fzO47g4emjZpYKzADOAMqBRWY2zzm3ssk6HYGHgEnOuY1m1tWrPCLiv8ZgmDteWkHvTm248dR8v+NIlJdbBGOBYudciXOuEZgLTNlnnUuAF5xzGwGccxUe5hERnz36jw2sq9jDT84bSZuMVL/jSJSXRdALKGsyXR6d19QQoJOZ/d3MCs3siuZ+kJlNM7PFZra4srLSo7gi4qXyHbU88NY6zhjRjdOGd/M7jjThZRE0dwTI7TOdBowBzgbOBO40syFf+CbnHnHOFTjnCvLy8lo+qYh47ievRPYKa/zh2OPlLSbKgT5NpnsDm5tZp8o5txfYa2bvAaOAtR7mEpFW9vbqbSxYuY3pk4fRq2Mbv+PIPrzcIlgE5JvZADPLAC4C5u2zzsvAV8wszcyygXHAKg8ziUgrqw+EuGteEYO7tmXqiQP8jiPN8GyLwDkXNLMbgDeBVGC2c67IzK6LLp/pnFtlZm8Ay4AwMMs5t8KrTCLS+ma+u56y6jrmfGscGWm6dCkWeXr3UefcfGD+PvNm7jP9G+A3XuYQEX9s3F7LQ39fz7mjejJhsMYZiFWqZxHxzE9eKSI9xbj9LI0zEMtUBCLiibdWbeOt1RXcdHo+3TvopnKxTEUgIi2uPhDip6+uZHDXtlytA8QxTyOUiUiLm/V+CaXba3nqmnGkp+rzZqzTv5CItKhNO+t48J3IqGMn5esAcTxQEYhIi/r5a5EriG8/e4TPSeRQqQhEpMV8UFzF/OVbuf6UwbqCOI6oCESkRQRCYe6eV0Tfztl8e+JAv+PIl6AiEJEW8cRHpayr2MOd54wgK123mI4nKgIROWKVuxu4f8FaJg7J4/ThGl8q3qgIROSI/ebN1dQFQtx17gjMNAZxvFERiMgRWVq2k78sLmfqSQMYlNfW7zhyGFQEInLYwmHHXfOKyGuXyfdOHex3HDlMKgIROWzPLynn07KdTJ80jHZZ6X7HkcOkIhCRw7KrPsCv3ljD6L4dOX/0vsORSzzRvYZE5LD831vr2L63gUevLCAlRQeI45m2CETkSyuu2MNjH3zGN8f0YVSfjn7HkSOkIhCRL8U5x09eKaJNeio/mDTU7zjSAlQEIvKlLFi5jffXVXHzGUPIbZvpdxxpAQctAjO7wcw6tUYYEYlt9YEQ97y2kvyubbnihH5+x5EWcihbBN2BRWb2FzObZLpsUCRp/fG9Esqq67j7vJEacCaBHPRf0jl3B5APPApcBawzs1+Y2SCPs4lIDNm8s44Zfy9m8lHdOXGwBpxJJIdU6c45B2yNPoJAJ+A5M/u1h9lEJIb8/LVVOAe3nz3c7yjSwg56HYGZ3QhcCVQBs4AfOOcCZpYCrAN+6G1EEfHbh8VVvLZ8C7ecPoTenbL9jiMt7FAuKMsFLnDOlTad6ZwLm9k53sQSkVgRCIW5+5Uiendqw7Una8CZRHTQInDO/fgAy1a1bBwRiTVPflTK2m17ePjyMRpwJkHpsL+I7FfVngbu+9tavpKfy9dGdPM7jnhERSAi+/XL11dTHwhx17kjNeBMAlMRiEizCkt38FxhZMCZwV014EwiUxGIyBeEwo4fv7yC7u2zuPHUfL/jiMdUBCLyBXMWllK0eRe3nz2cnEzdrT7RqQhE5D9s39PAb95cw4RBXTjnmB5+x5FWoCIQkf/wy9dXU9sY4ifn6QBxsvC0CKI3qVtjZsVmNv0A6x1vZiEz+4aXeUTkwBZ/Vs2zheVc85UB5Hdr53ccaSWeFYGZpQIzgMnACOBiMxuxn/V+BbzpVRYRObhgKMwdL62gZwcdIE42Xm4RjAWKnXMlzrlGYC4wpZn1vgc8D1R4mEVEDuLxDz9j9dbd/PjckTpAnGS8LIJeQFmT6fLovH8xs17A+cDMA/0gM5tmZovNbHFlZWWLBxVJdltr6rlvwVpOGZrHmSN1BXGy8bIImjvK5PaZvh+41TkXOtAPcs494pwrcM4V5OXltVQ+EYm659WVBMJOB4iTlJfbf+VAnybTvYHN+6xTAMyN/uHlAmeZWdA595KHuUSkiXfWVPDa8i389xlD6Nclx+844gMvi2ARkG9mA4BNwEXAJU1XcM4N+Py5mT0OvKoSEGk9dY0h7nxpBYPycpimW0wnLc+KwDkXNLMbiJwNlArMds4Vmdl10eUHPC4gIt574O11lO+oY+608WSm6RbTycrTUwOcc/OB+fvMa7YAnHNXeZlFRP7Tmq27+eN7JXxjTG/GD+zidxzxka4sFklC4bDjtheX0zYrjdvO0hjEyU5FIJKEnl5YSmHpDu48ewSdczL8jiM+UxGIJJktNXX86o01fCU/lwuO63Xwb5CEpyIQSSLOOX78chHBcJiff/1oXTMggIpAJKm8sWIrC1Zu4/tnDKFvl2y/40iMUBGIJIkdexu58+UijurVnqknDjj4N0jS0J2lRJLET19dyc7aRp6YOpa0VH0GlH/TX4NIEnhr1TZe/GQT3/3qYEb0bO93HIkxKgKRBFdTF+C2F5cztFs7bvjqYL/jSAzSriGRBPezV1dStaeRP15RQEaaPvvJF+mvQiSBvbVqG88WlnPtxIEc07uj33EkRqkIRBLUjr2NTH9hOcO6t+Om0zX0pOyfdg2JJKg7Xl7BztpG/nT1WN1ZVA5IWwQiCeiVTzfz2rIt3Hz6EJ0lJAelIhBJMFtq6rjjpRWM6tORaydqsBk5OBWBSAIJhx3ff+ZTAqEw9194rC4ck0OivxKRBPLH90v4qGQ7d507ggG5Gn9YDo2KQCRBrNhUw2//uoZJI7vzzYI+fseROKIiEEkAexuC3Dj3EzrnZHDvBbq9tHw5On1UJAH8+OUiNlTt5elrxtFJI47Jl6QtApE491xhOc8vKed7p+YzYXCu33EkDqkIROJYccUe7nxpBeMGdOam03T1sBweFYFInKprDHHDnCW0yUjl9xeNJjVFxwXk8OgYgUgccs5x+4vLWbNtN49ddTzdO2T5HUnimLYIROLQUws38sInm7j5tCGcMrSr33EkzqkIROLMJxt38NNXivjq0Dy+d6oGmpEjpyIQiSOVuxv47tNL6N4hi/svHE2KjgtIC9AxApE40RAMcd1TheysDfDcd06gQ3a635EkQagIROKAc447XlxBYekOZlxyHCN7dvA7kiQQ7RoSiQOPffAZzxaWc+Opgzn7mB5+x5EEoyIQiXHvrK7gZ6+t5GsjunHz6UP8jiMJSEUgEsNWbKrh+jlLGN6jPfddeKwODosnPC0CM5tkZmvMrNjMpjez/FIzWxZ9fGhmo7zMIxJPNu+sY+rji+jYJp3ZVx1PTqYO6Yk3PCsCM0sFZgCTgRHAxWY2Yp/VNgAnO+eOAe4BHvEqj0g82VUfYOrji6hrDDH76uPp1l5XDot3vNwiGAsUO+dKnHONwFxgStMVnHMfOud2RCc/Bnp7mEckLtQHQnzr8cWsr9zDQ5cdx7DuGnxevOVlEfQCyppMl0fn7c81wOse5hGJecFQmBvmLGFRaTW/++axfCU/z+9IkgS83OnY3FEt1+yKZl8lUgQn7Wf5NGAaQN++fVsqn0hMCYcdtz6/nL+tquCeKSM5b1RPvyNJkvByi6AcaDpwam9g874rmdkxwCxginNue3M/yDn3iHOuwDlXkJenT0iSeJxz3DWviOeXlHPz6flcfkJ/vyNJEvGyCBYB+WY2wMwygIuAeU1XMLO+wAvA5c65tR5mEYlZzjl++upKnvy4lGkTB2qAGWl1nu0acs4FzewG4E0gFZjtnCsys+uiy2cCPwa6AA9FB9sOOucKvMokEmucc9z7+moe++Azrj6xPz+aPEwDz0urM+ea3W0fswoKCtzixYv9jiFyxJxz/GL+Kv74/gYuH9+Pn04ZqRIQz5hZ4f4+aOsKFREfhMOOO19ewdMLN3LFCf24+1yVgPhHRSDSyoKhMLc+v5znl5Rz7ckDmT5Ju4PEXyoCkVZU1xjixrmfsGDlNr5/xhC+d+pglYD4TkUg0kp27G3kmj8t4pOyndx97giuOnGA35FEABWBSKsoq67lqsf+SdmOOmZcchxnHa0xBSR2qAhEPLbos2que7KQQCjMk1PHMm5gF78jifwHFYGIh55dXMZtLy6nd6dsZl1ZwKC8tn5HEvkCFYGIBwKhMPfOX83sDzZw0uBcZlxynAabl5ilIhBpYVtr6rlhzhIWl+7gqgn9uf3s4aSnajBAiV0qApEW9GFxFTfO/YTaxhAPXDxadxCVuKAiEGkBjcEw/7tgLQ+/t56BuTn8+dvjye/Wzu9YIodERSByhEoq93DzM0tZVl7DxWP7cOc5I8jO0H8tiR/6axU5TKGw47EPNvDbv64hKz2VmZeNYdJR3f2OJfKlqQhEDsP6yj388LllFJbu4LRhXfnFBUdrgHmJWyoCkS+hPhDioXeKmfluCW0yUrnvwlF8/dheul+QxDUVgcghemd1BXfNK2JjdS1fP7Ynt509nK7ttBUg8U9FIHIQa7bu5ufzV/He2koG5uUw51vjmDA41+9YIi1GRSCyH1tq6njgrWKeWbSRtplp3HnOCC4f34+MNF0cJolFRSCyj6o9Dfzh7+t58uNSnHNccUJ/bjotn045GX5HE/GEikAkamtNPX98v4Q5CzfSEAzxjTG9+d6p+fTpnO13NBFPqQgk6a3dtpvZ/9jA80vKCTuYcmxPrv/qYN0pVJKGikCSUijseHdtBY998Bnvr6siMy2FC4/vw7UTB2kLQJKOikCSyrZd9Ty7uIw//7OMTTvr6NY+kx+cOZSLx/als44BSJJSEUjCqw+EWLByGy8sKee9dVWEwo4Jg7owffIwzhzZXWcBSdJTEUhCagiG+Me6Kl5bvoUFRdvY3RCkR4cspk0cyDcL+jAgN8fviCIxQ0UgCWNXfYB311SyYOU23llTwe76IO2z0ph0VHfOH92L8QO7kJKiW0GI7EtFIHErHHas2rqL99ZW8d7aShaXVhMIObrkZDBpZHfOOqYHJw7K1a4fkYNQEUjcCIcd6yr28M8N2/moZDsfl1RTvbcRgOE92jP1pAGcMbwbo/t2IlWf/EUOmYpAYtbO2kaWldfwadlOlmzcQWHpDnbVBwHo2SGLU4bmMWFQLhPzc+mqW0CLHDYVgfjOOcfmmnrWbN3Fys27KIo+NlbX/mudwV3bcvYxPRjTrzPH9+9E387ZuvWzSAtREUiraQiGKKuuZUNVLesr97C+Yg/FlXtYt20PexqC/1qvX5dsjurVnovG9uHY3h05qncH2mel+5hcJLGpCKTFNARDVOxqYNPOOjbtqGPzzjrKdtSysbqWsuo6ttTUEXb/Xj+vXSaD8nK44LheDOnWjqHdIw+96Yu0LhWBHFAo7NhR20j13sijak8DVbsbqNrTSMXueip2N1Cxq4Ftu+rZHj1w21TXdpn06ZzN8f070a9Lbwbk5tCvSzYDc9vSIVtv+CKxwNMiMLNJwO+BVGCWc+6X+yy36PKzgFrgKufcEi8zJZNw2FEbCFHbGGRvQ4i9DUH2NgTZE33srg+yqz4Q+VoXoCb62FUXYGddgJ21AXbVB3Duiz87xSC3bSZd22fSvUMWo/p0pEeHLLq3z6Jnxzb07Bj5mpWe2vovXES+FM+KwMxSgRnAGUA5sMjM5jnnVjZZbTKQH32MA/4Q/RrznHOEwo6Qc4TDEHKOUCgyHQyHI8vCjmDIEYw+D4Qi84PhMIFQZFkgHCYQDBOMLm8IhgmEIvMaQ2Eag5FHQyhMQyAyryEQpj4YoiEQoj4Qpj4Qoj4YeV7XGKIu+uZfHwgf0mtJTzXaZaXToU067duk0yE7g/65OXRsk07H7Ay6tM2gc04GnbMzyG2XSZecDDpmZ+gUTZEE4eUWwVig2DlXAmBmc4EpQNMimAI84ZxzwMdm1tHMejjntrR0mHfXVnLPqysJO4dzNPs18oi8yYej80LhyPJ/v+lHvjb3KdlLGWkpZP7rkUpmeuRrVnoKWWmptMtKIzsjjaz0VNpkpJCdkUab9FSyM1LJzkyjbWYqORlp5GSm0S4r+jUzjfZt0slMS9EZOCJJzMsi6AWUNZku54uf9ptbpxfwH0VgZtOAaQB9+/Y9rDBtM9MY2q0dZpBi9h9fDSPFIDXl3/NTUwwDUlKMFIssT0kxUqPLPl/n8+dpKRZdDmmpKZFlZqSlWmQ6+jw91UhNSSE9JTI/LdXISE0hvcnzjLTIdEZaSnSZ6Y1aRDzjZRE098617+foQ1kH59wjwCMABQUFh/VZfEy/Tozp1+lwvlVEJKF5eROWcqBPk+newObDWEdERDzkZREsAvLNbICZZQAXAfP2WWcecIVFjAdqvDg+ICIi++fZriHnXNDMbgDeJHL66GznXJGZXRddPhOYT+TU0WIip49e7VUeERFpnqfXETjn5hN5s286b2aT5w643ssMIiJyYLpRu4hIklMRiIgkORWBiEiSUxGIiCQ5c619r4QjZGaVQKnfOQ5DLlDld4hWptec+JLt9UL8vuZ+zrm85hbEXRHEKzNb7Jwr8DtHa9JrTnzJ9nohMV+zdg2JiCQ5FYGISJJTEbSeR/wO4AO95sSXbK8XEvA16xiBiEiS0xaBiEiSUxGIiCQ5FYEPzOx/zMyZWa7fWbxkZr8xs9VmtszMXjSzjn5n8oqZTTKzNWZWbGbT/c7jNTPrY2bvmNkqMysys5v8ztRazCzVzD4xs1f9ztJSVAStzMz6AGcAG/3O0goWAEc5544B1gI/8jmPJ8wsFZgBTAZGABeb2Qh/U3kuCPy3c244MB64Pgle8+duAlb5HaIlqQha333AD2lmSM5E45z7q3MuGJ38mMgIdIloLFDsnCtxzjUCc4EpPmfylHNui3NuSfT5biJvjL38TeU9M+sNnA3M8jtLS1IRtCIzOw/Y5Jz71O8sPpgKvO53CI/0AsqaTJeTBG+KnzOz/sBoYKHPUVrD/UQ+yIV9ztGiPB2YJhmZ2d+A7s0suh24Dfha6yby1oFer3Pu5eg6txPZlfB0a2ZrRdbMvITf4gMws7bA88DNzrldfufxkpmdA1Q45wrN7BSf47QoFUELc86d3tx8MzsaGAB8amYQ2U2yxMzGOue2tmLEFrW/1/s5M7sSOAc4zSXuRSvlQJ8m072BzT5laTVmlk6kBJ52zr3gd55WcCJwnpmdBWQB7c3sKefcZT7nOmK6oMwnZvYZUOCci8e7GB4SM5sE/C9wsnOu0u88XjGzNCIHw08DNgGLgEucc0W+BvOQRT7N/Amods7d7HOcVhfdIvgf59w5PkdpETpGIF56EGgHLDCzpWY282DfEI+iB8RvAN4kctD0L4lcAlEnApcDp0b/bZdGPylLHNIWgYhIktMWgYhIklMRiIgkORWBiEiSUxGIiCQ5FYGISJJTEYiIJDkVgYhIklMRiBwhMzs+OuZClpnlRO/Pf5TfuUQOlS4oE2kBZvYzIvefaQOUO+fu9TmSyCFTEYi0ADPLIHKPoXpggnMu5HMkkUOmXUMiLaMz0JbIvZWyfM4i8qVoi0CkBZjZPCIjkw0AejjnbvA5ksgh03gEIkfIzK4Ags65OdHxiz80s1Odc2/7nU3kUGiLQEQkyekYgYhIklMRiIgkORWBiEiSUxGIiCQ5FYGISJJTEYiIJDkVgYhIkvv/iv9SOtogqN4AAAAASUVORK5CYII=\n",
|
|
"text/plain": [
|
|
"<Figure size 432x288 with 1 Axes>"
|
|
]
|
|
},
|
|
"metadata": {
|
|
"needs_background": "light"
|
|
},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"%matplotlib inline\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"import torch\n",
|
|
"import torch.nn as nn\n",
|
|
"\n",
|
|
"x = torch.linspace(-5,5,100)\n",
|
|
"plt.xlabel(\"x\")\n",
|
|
"plt.ylabel(\"y\")\n",
|
|
"a = torch.Tensor(x.size()[0]).fill_(2.)\n",
|
|
"m = torch.stack([x, a])\n",
|
|
"plt.plot(x, nn.functional.softmax(m, dim=0)[0])\n",
|
|
"fname = 'softmax3.png'\n",
|
|
"plt.savefig(fname)\n",
|
|
"fname"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"[[file:# Out[19]:\n",
|
|
"\n",
|
|
" 'softmax3.png'\n",
|
|
"\n",
|
|
"]]\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 19,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"'softmax3d.png'"
|
|
]
|
|
},
|
|
"execution_count": 19,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
},
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAPcAAADuCAYAAADlVZEAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAACLy0lEQVR4nO2dd3hb5dnGf0fLsuS994gd20m8EycQ9t4k7L0pZbVQSqGMjq+D0TJKoZS9y0wChABh7wBxhncS7xnb8h6ytc/3h3xOZFuypcROQvB9XcbBtl69ks59nud9xv0IoigyhznM4cCDYl9vYA5zmMPsYI7cc5jDAYo5cs9hDgco5sg9hzkcoJgj9xzmcIBijtxzmMMBCtU0v5/Lk81hDrMPYTYWnbPcc5jDAYo5cs9hDgco5sg9hzkcoJgj9xzmcIBijtxzmMMBijlyz2EOByjmyD2HORygmCP3HOZwgGKO3HOYwwGKOXLPYQ4HKObIPYc5HKCYI/cc5nCAYo7cc5jDAYo5cs9hDgco5sg9hzkcoJiun3sOswCHw8Ho6ChKpRKlUolKpUIQZqWldw4/YwjT6JbPiTXMIERRxGazYbPZsFgsuL73SqUStVqNSqVCqVTOkf3nhVn5sOfIvZcgiiIWiwWHw4EgCFit1nG/k74kzJH9Z4U5cv9UYbPZZDJLJHUl90TMkf1nhzly/9Tg6oYLgiATUrLi3hJUIrpk9WGO7AcY5sj9U4LD4cBqtcqEdCWfr+SeCInsdXV1REREEBQUhEqlkr/myP6Tw6x8WHPR8hmGKIrY7XbZ7VYoZj7bKN0sHA4HCoUChUIhP6dEapVKhVqtliPyc2T/+WGO3DMIURSxWq3Y7fZJ1trT3+8J6QRBkNcQBEG+kUg3GJvNJv+tRHaVSoVCoZgj+88Ac+SeITgcDlpaWvDz8yMkJGRK8khknK3xye6OAa5kFwRhnBs/R/YDE3Pk3kO4Bs2Gh4cB9juiuCP7xAj+HNkPPMyRew8wMXetUChmzRpPtQdf4Y7sVquV4eFh2tvbSU1NnSP7AYA5cu8mpACW65l3Nl1td5gpwgmCgFKpBGB0dFQusnG17K5ptzmy/zQwR24fMTF37RoN39vkni24kh12eShms1m+iUlkl+ri58i+/2GO3D5gqtw1sE/c8pmEp+j9VGQH5+tWq9Vy6m2O7PsH5sjtBSbmrj1dvFLu+UCHK9mlm5nFYsFisQBOsrvm2Wcj1z+H6TFH7mngS+56X5B7Jj2F3cm7u5bDuu5HIvvIyAh2u53w8PA5su9lzJF7CjgcDrk10xtX86caUJtJTCS70WjEZDIRGBg4Z9n3MubI7QZS0KyhoYGkpCSvL74DJaA201AoFOMsu3RmdyX7xGj8HPYcc+SeAOnCs9vttLa2kpyc7PVjfQmo7Y83gj0th/VmTXc5dlEUMZvNcoBO6nibU6nZM8yR2wWubvju5HK9PXNbLBa2b9+OWq0mLCyM4ODgA9ZaTXfDcEd2h8OByWSSfzbX3rp7mCM3U+eufYE31rivr4+qqiqSk5MRRRGDwUBNTY1M9NDQUAIDA33q9Z4p7A3LPR28Ibv0Gfn7+8+RfQr87Mk9Xe7aF0xFblEUaWxsxGAwUFhYiEqlwuFwEB0dDYDJZKKvr4/W1laGhobQ6XSEhoYSGhqKTqfzmHrb3yF5QbsLd2QfHh6mqamJBQsWAHOW3RN+tuT2NnftCzyR22KxUF5ejl6vp6ioCIVCMUlmSavVEhsbS2xsLKIoMjIyQl9fH/X19YyMjBAYGCiTXavV7tE+PWE2LLfD4RhXALOncI3GK5VK2bJLZbPS7+bI/jMltyf5oz2FuzN3f38/lZWVpKeny1bam3X0ej16vZ6EhAREUWRoaIi+vj62b9+OxWIhODgYs9lMQEDAjOx9trCnltsdJJEK2HVTdu1lnyO7Ez87cs+kGz4RrtFyURRpamqio6ODgoICdDrdbq8rCAJBQUEEBQWRnJyMw+FgYGCA+vp6GhsbaWlpISQkhNDQUEJCQmbUUu4pZssb8HTD8IbsPxdJqp8NuV3d8D0Jmk0FyS23Wq2Ul5fj7+/P0qVLZ/y5FAqFTOTg4GBCQkLo7++nt7eX+vp6lEql7MIHBQV5/fz7Q0DNG7gKRU6Hn7NKzc+C3FIQZrY7mARBwGw2s3HjRtLS0oiJiZmV53F9PnBenBEREURERADOM35fXx8dHR1UV1fj5+cnkz0gIGCvXryzdcPYk4yGJ5Wa9vZ2+T06EMh+wJNbyl0XFxezfPlynz8oby9OURTp6Oigr6+Pgw8+eI/c8D2FRqMhOjpaPuOPjo7S19dHc3Mzw8PD6PV6mez+/v7y6/spWe6Z8oZcyT4wMEBISMgBI0l1wJLbXdDM1w9FoVB4Fe21Wq1UVFQgCAIRERH7lNju4O/vj7+/P3FxcYiiiNFopK+vj9raWrnuOzQ0dFaOKvs7uSeuO7H89acsSXVAknui/NHuvvnelJMODAxQWVlJamoqQUFB1NTUeLX2TF0QvhaxCIJAQEAAAQEBJCYm4nA45Ei8wWDAbDYjiqJs2VWqPbtE9vWZ29d1J9403LnxE1VqJjbB7C9kP+DIPfEuu6dFKZ7KSUVRpKWlhba2NvLy8tDr9YyMjPhUW76nmIk1FAoFwcHBBAcHExQURHd3NxEREfT19dHU1IQgCHIkPjg42OdI/P525p4Kdrt92tfnTrjCnSRVfX09qamp6PX6Gd+ntzhgyO1N7trXC01yyyfCZrNRUVGBWq1m6dKl8oftSzPI/tY0IkGhUBAWFkZYWBjgPHL09/fT3d1NXV0dKpVKtuqBgYHTkmy2LPeeehSe1vX1puFJpeaPf/wj9957L5mZmTO9Ta9xQJDbm9y1t+fniY+ZSMLBwUEqKipISUkhLi5u3O/2x04vX+COiGq1msjISCIjIwEwm8309fWxc+dOhoaG0Gq1Mtn1ev2kx/+U3PKZ8AgkshuNxn1qteEnTm5fRvcolUqfye3qlouiSGtrK62treTm5rqtDPupa6h5Az8/P2JiYoiJiUEURTkS39jYiNFoJCAgYFwk/qcUUJtJGI1GAgMD9+kefrLk9nV0jzRPS61We/0ckrW32WxUVlaiVCrHueET4avlnokLf192hQmCgE6nQ6fTER8fL9cT9PX1UV1djdlsxm63o9Pp8PPzQ6PRzNg+ZyuyP1MYGRnZ51mTnyS5fZU/As/n56kgCALDw8OUl5eTnJxMfHz8tH+/NzXU9peorARBEAgMDCQwMJCkpCQcDgclJSWYTCYqKiqw2+3jymR399z8U7DcoijOSlzAF/ykyC3laIeGhnzOy/pKbqkzq7a2loKCAq8aNLy13Ha7ne3bt2Oz2eTg1UxZtf0JknxSUlIS/v7+2O12+vv7ZTdeEATZhfdFsGK2ztwzmZ7cH45nPxlyS1HI4eFhdu7cKUdzvYUv5LbZbFRVVWGxWFiwYIHXnVfekNtoNFJWVkZcXBz+/v709/ePs2phYWH7rPljtivUlEol4eHhhIeHA85IvJRfr6mpQaPREBoaSlhY2JRlsrPhls8GGfe1Z/WTILfr6B4pMOYrvCX30NAQ5eXlJCUlodFofD6DTnWRdHZ2UltbS05ODjqdDpvNRkhICCkpKbJVk5o/pJRTWFiYT8os+xumumGo1WqioqKIiooCdglWSGWykmBFWFjYuDLZ2XDLZ3rN/eHz2q/J7U7+SFIw8RXekLutrY2mpiZycnIIDAykurrap+fy9IE6HA5qamoYHh5m6dKlqNXqcV1JMNmqSSknSZlFqgeXLnQJB5LMkifBirq6OkZHR+VIvNVqnXFy2+32GVvTYrH4FLidLey35PaUu5ai3r5iKnLb7XaqqqpwOBwsXbpUDoTMRN7abDZTVlZGWFgYhYWF8uuY7oKfmHKS6sGlKHRwcDA2m22/P6vv7g1jKsGKgYEBKioq5OBcaGjoHpNpJhVjpOacfY39jtzTyR/NtFsuRcMTEhJISEgY91y7E2F3hSSGmJmZKbdj7g7c1YMPDAzQ2NhIc3Mz7e3t8nl9f1NSnSlvwFWwYmBggPT0dNm7aWlpQRTFPRKsmEnLPTw8vF8o5OxX5PYmdz2Tlnvnzp00NjbKbrg3j/EGkgpLZ2cnhYWF49zomYAk1jA4OIifnx/h4eH09/fLgSk/Pz9ZSdVd1dhU+/6ptHyqVCr5TA7OIKg7wQopZjEdcWfSco+MjMyR2xXe5q53l3Cuj7Pb7Wzbtg2bzTbODZ+I3XHLRVGktLQUjUYjiyHOFqT3aGKJ6MSqMamlMywsDD8/vyn3PtXv7CLYHSJ2h4ht7Lv8JYroNUqC/dWTHrc3KtQ8CVZIZbKSYEVYWJjbG95MBtSMRuM+L2CB/YDcvsof7Un7psPhkFNR8fHxJCYmTrmerzeS4eFhjEYjqampk+rO9yb8/f3x02rRhUTSY7TQ0tXP5upedvY202+yY0bNiEPFgEmk32wDUcQhgsVqx+ZwgNA2mcBe3uNigvzIiNKTGaUnI1qPechOnggzeYvzJs/tSbBCuuFNDFB60xHmLebccmau79obKBQKOc2SnZ1NUFCQV4/xltzt7e00NDTIogizAVEU6R2xYhiy0GO0UNM8zJAVrJUj9Bgt9Bqt8vfeESs2x2RGKgQI1ooEaayMWuwYRkUOTfInUKdFdNgRRAfBQUGoFAIKBSgVAipBQKlwfqkUu/7t+v+vb9pJ74iVwsQgqg1GvqvrlW8If924gbQIHZnRejKiAsa+6ydZeV/eB1+trCfBCilAKfVjWyyWPQ5S/uzJ7XA4aGtrIyIiYtaJbbfbaW9vx2q1smzZMq/LAr0pJ3U4HOzYsQOTycTSpUvZuHHjbu1RFEUGTTY6Bs3yV+egmdZ+E20DJjoGzfQa3RNWKYBWrUSrVqBRKdBplIToVKiVClQKQf6uEAQUAjhwutet/SYYNRGq0xCvsxLCCEkhauYl+cvDELzF5zt6UCsV3L/SOSjAYnNQ3z3C+h/KsQZEsaPTyJc1vbxd2ik/JjpQM47smdEBJIX5o1JMfy3saZ/+xABlU1PTuIKi4OBguaDI1zLSkZGRn2e03DV3XVtbKxcwzBYkNzwgIICwsDCfPqjpLLfJZKK0tJSoqCiysrKmvOCGTTY6Bk3s7DdR32OkoctIS98IhiGntR002dwSd9o94rSuImCxizhEB1abiNKKRysr/VuyfZ/UDmKyOZ9bwExcUDPx+mbidCIZUXpyk8LJSorCbwqLZhdFXI2pRqUgKyaAoXgVRUVpgPOz7zFa2dE5TLXByA6DkWqDke8b+uTXrlEKpEfqufygBE5cGLlXikEUCgVarZaIiAgSExOx2+0MDAzQ29srl8lK2YigoKBp3XepO25fY6+Se2LuerbR0dFBXV0d2dnZmM1mBgYGfHr8VAG1np4etm/fzoIFCwgNDaXXaKWxx8inTVY+6tlGc98oHQMmeo1WhkxWr86sWpUCvZ+SQD8VIf4qQnVqAvxU6P2U+KuV6DRK6rqMrN/Wzb/OXECin5FAfz9iY2N9el0Snt3Qwr++aODL3xxMz7CVH7a3UN9rotOkotowTHG7CbFuAL4fQKusJyFQQXqEPwvigslNCiczOpBArfMSsjvEaS2uIAhEBGiICAjjkLRd5cNWu9PKVxuMbKjvY12Fgdve2c7bpR3ceUI6KeGzH5xyDagplUq3ghVdXV3U1tZOWz04PDy825/JTGKvkHumR/dI7rKnc5fD4WD79u2YzWa5Iqy7u9vnKLvc8ml30NY/SuXOIXYYhqhs6qalfxSjXcXQ5yWYbQ7Gc7dtt16XyebAZHPQY7RO+7c3r9mGQoC4QDUL4vqYH6UnI9IZxIoP0aLwRrF1bNdKQXC6wykBLE/SkZCQAMCIxU5tl9O6VhuM7Ogc5tvmYdbXGoGdAETplWREBdDcayLYf/cuJ7VSQWZ0AJnRAaSE+bOuwsA5BTF8WNXFmU9v5sqDE7l6eSJa9ezV29vtdo9nbU+CFa2trQwPD6PVauXUo06n261UmCAIJwKPAErgGVEU75vw+2DgFSAJJ28fEEXx+anWnHVyTyd/tDupEol07sg9MjJCWVkZMTExLFiwYFxlmydym612agxGytoGKG8bpK7LSMegmcFRC2a7iENs8rATy6SfqAQI0auJCvQjNkhLVJCWlt4Rvq3r5fYT5hPop0KldLrEosOOWqVApVCMucq4uM2KSW60SiHwRXUPD3xWz10npNPQ3k3zgNPN/XR7t3yD8VcrSI90nmPnR419j9QTopsQwBp7gKf3X6dRkhsfRG78ruCjKIp0Dpmp7jRSuXOAqrZ+aruHaR+y0z5o5jdvbOV3x80nLmz33FLrmHt+3IJIrj88hQc+q+fJb5t5v8LAnSekc1i6bw1D3sKXVJgnwYr6+npeeeUVNm3ahNFo5KCDDiIxMXHa9QRBUAL/AY4DWoFiQRDWiqJY5fJnNwBVoiieJghCJLBDEIT/iaI4+SIcw6ySezr5I8nt9ZXcSqUSu90+6fwsNWYsWrSIkJAQ+ecmi52yncN8tW2ArtJSmnpG6R42M2y2Y7VPtLruoRCc51u1UsBPrUStdBJS8kTFsf+Mmi04HNDWb6KpdxSbXcRqd95U3inZycLYIDKiApgfpSM5RE90kNan1x+mdxJ0+bxQDol2vgexsbGMWOzUdRmp6TJSYxih2jDMZzu6WV3SIT82KkDDfBfCG4Ys8msD7260giAQE6QlJkjL4fPD5Z9f8NwWeowWvqwf4runt3BGupqVi8LkG7u3sQ7pvVIrnS78fSuyODMvhr+tr+H6Nyo4NjOCE6Nnvmd+d1NhEwUr7rnnHq677jrMZjPXXHMNf/nLXygqKppumaVArSiK9WNrvg6sAFzJLQKBgvMDCgB6AdvEhVwxK+T2NnetUql2qz5aIreEwVEza76toGTnCD12LW0bK+gbsWK22t2cdYenXV/AeXE5REgIVOCnUWOzWggK0KNRq1C7WFSJ5JJ17e3uIiY6Uv65SqlgR8cQ39b1EqRV821tD2+XtMvPFeKvIj3SSbj58ncdej9PhTXO7w5RRMmuwhOdRklOfBA5E6xs97CFmjHXumbMvd5Y3I/V5Y15bdNOLiyaWohiOohAWqSepy7M5Z6Panl1ez8buwZYmWhDXVIiV9VNVzEm7Uvl8vulKSGs/sViXvyhlSe/beabWgc7VS1cvDQetXJmMugzVcQiyRtfffXVFBYWevuweKDF5f9bgWUT/uYxYC3O81AgcJ4oilPe5Wac3L7krieS1BMGRixsqO/lh/peKtqHaOkZZvS977HaRTdW1+z5+QQnCYL9VYToNAT6qdD5KdGqlPipnGkkjUqBAviqpoeuYTPHJghkxvhz7NKl6LTT34R+/PFHFi/OGGep3tjUyrd1vTx4djbRQVp29gzy0Q9lNA3Y2Dki0DEyyjulQ4xad31W8cF+4wkfpSc5zB+BsekgTJ8OEgSByEA/IgP9WD5vlztrc4g0947yn68b+XhbN//8tJ53yjr5RUEQubG7F7xyOJzR+ZRwHU9ekMOnO7q5/+M6HigRWZEbyI2HJiBYhscJK0pBK9d2ToncauUEoUalgqsPSeLEhZHc8dZmHvq8gbXlndx1YjpLkkJ2a8/j9z+z5ac+6qe5+yAnXtonACXA0UAa8IkgCN+IojjoadEZJbev8kcSuXuNZr7Y3s239T3UdBoxDJsxmu27lRqaCnYRhsx2hsx2Wvsn3wQUAqjGcsJWmwOrQ+SZSqCyD9UX35ASrmN+VAAZY8UYGdEBJIT4o3CJEruLsCvHfm93QG9vLw3bt3H6QQvQ6/VYrVZ6e3vp7umhqWuYHrsf3TYNO41Q1z3KN7W7ikFUCqerCtDaN0rqbo7pVikE5kXoSI/U8/G2bh48cwEPfFrPbR+1c/S8AO46NYKoQM9lqu5gE0U5iCcIAsdlRXLIvDD+/NaPvF9h4PPqHn51RArnFmahEJwVY729veOmnoSFhWEyS265eysaG6Th5sX+DAamcN/HtVzxchmn50RxyzHzCNfvfvHJTDeO+JjnbgVcD+cJSBHLXbgCuE90Xly1giA0AFmAx8KKGSO3ZLFhahVSV5gdAov/+YNPz6NRQIAGQrUKwrQKUuMiCdP7EaRV8Y9Pajk1J5oLlybKASjJdbZZLTTU1ZCfm4tKOfZzxXiXWiJpV1cXd79TQUk33H1IEEJoPDWdw1QbhinfOciHlbsKMfzVCtIidxGeATvxgybiwlS7FEjG1m1qaUEx0svixYvRaDRYLBb8/PzkHubssbbG3t5eent7cSRDQFAcQ4KO9lEFtd0jbKjvo2PQzK/erOSMRcFclB8++U3yFmM3jWOzIjgsPYyH11eyqnKA057YxHWHJXFRkfdur8Mhyq9Tgk6j5JwMNdeekM89H9Vyz0e1vF3SwV0npZMXH4RO54zMS1NPent7qW8yANDe2kKEKmrSIATJIzwqI5yDUkN46ttmXvihlS9qernpyBTOLoidtA/v9j+zteU+Wu5iYL4gCKk4Uy3nAxdO+Jtm4BjgG0EQooFMoH6qRWeM3NLZ2pdGizerjOPXYCxwJUh3bgEHIg5RxG531jdbHNBrgl6TgzocFHe0j1tjXXknpa2DY0GjAOZHBzA/KoCEUH8GtQIxwZ7NnSiK1NXV0dfXR1xMNJu7OkkMUpKfEwM5u/7OaLY5U0Sdw9QYnAUZX9f0sGarcy/3/fAjITo1GWN7MFqccY+hYSPHLivyGLl3bWtMSUnBZrPR19eH0NuLaBogOdyPGI2eezuNHDE/nFUVPXxeN8ytx6k5NTvK58CkdKgRAH+1kkvyQzkhI5jnS4Z48LMG1pR0cMcJ6RycGjrtWnY35JYwL0LH0xfm8NG2bv75SR0Xv1DCmXkx3HRUCmF6zbipJwlD/lBaTWhI0LhBCK5acxIJ/dVKbjoqldNyovn7+hr+tr6Wd0o7ufukdBbF+iYrPJO15SaTyadOQFEUbYIg3Ah8hDMV9pwoipWCIFw79vsngL8CLwiCUI7zI7tdFMXuqdadUbfc1y4qrVqJABydGTGWEhJ2pYWUu9I/kqtsMY3S19uDzt+PkKAgQoICx37v/Ju/vL+dzKgA4kL8qTEM81VND3aHFKARiNYJ5DaUO4k/RvrEUH+UCgGLxUJ5eTmBgYEsWbKEzz+uxeoQ3ZJQ76ciLyGYvITgcT/vNVp4/9utWHURNPRZqDEM83ZJO0aLM67wQpWVtPRR0iK9c9lUKtWkbq+qLc0AHBE+xCEHaXl1h4071+7gzS3t3HF8Ggt9uKhFN6mwuCANj52XzVc1Pdz/SR3XvFrOcVkR/O7YecROcWO0i5PJ7XotCILAiQsjOSwtlCe+beaVjW18uqObm45M4SwXayuduaMiwokJctbou8ovDQ0NYbfbZR09rVbLvAgdz1yUyweVXfzz0zoueG4r5y2O41dHphCk9e4Sn0nLvTvnd1EUPwA+mPCzJ1z+vRM43pc192njiE6jRAQyogPIjAkkI8pZWzzRFZRkioaGzOQev5zW1lb8/f0nVQE9/GktmTGB/PX0XfXNDT0jTuvaMcTG6lYq2wdZX9UpX9h+KgUpYVrClGbyUiIpDAwjYMCMWilgs/umYhmm15AdpSEzMxadTkdPTw/btm1jhz2ef3zezLaOYU5//AcuXpbIdYcno/XxWvL39yc8LAzoZNHCRSiGOpgfbuSLRhOragY5/7mtnLYojFuOnU94wPRnZpHxkRzX13rE/HAOSg3lhR9aeOa7Fr6p7eUXhyRx2UEJ+Kkmb9zhEJkQA3ObWtP7qfjtMfNYkRvNPR/V8tf1tawp7eDuE+eTHReIzU1AzVV+yWg0UlNTg81mY/v27VgsFlmk4YSsMA5PD+Oxrxp5bdNOPtpm4N/nLCJ/wk3YHWbqzL0/qJ5K2Kfk9h+rOHri60Y5NKhWCqSOBa7So/SkhPph62kmKyGSxYsXy+6/O4vqp1Zitu2KvmtUuyqfyIlhg76L5cuXyznh6s4httZ3sK19kLoRJd9/3wHfO/PCGqWAzSGyvdvMYh9ek1Q919DQQFdXF0VFRYzWDwDNPH5+Hh9WdvLiD82sLW3nxiOSOTPftzOizBWX/Oot+fFcPjTKvz+v5e3KXj7d8SPnLNBxXmEskRHhaLXuLa5DFJnoybuS0U+l4JeHJnNaTjQPfFrPo1818k5ZB78/Lm1cjhucEXh3ltvTUSE9Us+zF+XyYVUXD3xaz4XPb+Wsghhig5x79XTWF0VRlkuWtNGlOnBpcOF580PJDE/kTx+18MvXynnjysJpS1jnBBKnga8vyF/jJPf6Xx3MyFiVWI1hmFqDkZLWAd6v2BW48lONkhbZQ3qknig/G8khGg7WhREfrJUDYX4qBSbr9AUOOo2ShTF66G0iI1PDwjMOR6lUMjBqpcZgpNYwzNslOylpHeQvG0bYOlzBrcemT3led8X27dvR6XQsWbIExdgxAyDQX8VfTl/A+UXx/O2DHfzlw1re2tLB7censThpeusCyKmwiQgL9OfPK3K4ZLmRez+u48Xyfr5saebCzHbmB4vuZZNF9zmYiYgL1vLQWQvZUN/HvR/XcsOblRyRHsbtx6eRGOo8WzpEJpW8TlcUIwgCJy+K4vD0MB7/uolXi9tkUnuqU5/Y7inlzyVFFkku2a/Tee2IDge/eKWE5y5cSEJEsMf9zFRn4myIU+wu9rFb7nx6B7AwNoiFsc4CDIfDQW1tLYbeAfyjU2nst1A7Frja2OiMFgPwyU50GiVpY7ngYZONXqPHajwZIyMjlJaWTtJNC/ZXsyQ5hCXJIYxY7JS0DnJSqoqPqgx8us3A1YemcNXyZPmm5G7dnp4eUlJSSEtLk3+uFKRUmNM/WRgbxMuXF7KurJ2HPm/g8pdLOXFhJLccnTrluRZci1ic3ye6gWmRep6+MIfPd/Twz0/ruPfHEY7LCufqyIBxsslhYWGYLZZxF+J0F+byeaGs+cViXtnYxn+/aWLlk5u44uBErlqe6Dag5u2FHuCn4rbj0liZF8ONb1TQPmhmTUkHFy+dXFgznYWV5JLjzH5AH9cdmsh/vmnlxjcrubVQRURIkFxMMxvikmazeUq1m72JfUvusSqsUcsuV9pkMslqoYcsW4IgCJPc4pqmNrbv7GdUHUK1YZjaLme0unvYQueQmbvereLmo9OIdJOrlXTGsrOzCQ72bC1VY2e+k1PU/HZFIQ98UsOjX9Szaksbtx43n1Oyo8dduF1dXVRXVxMaGjpJDHFXnntygOnIjHCe/76F575v5cvqHq48OJHLD06QjywTIT2jRBx3ZzxBEDgmK4JD0kJ5/odWnt3Qwje1fVy1PJErDl4MdmduvX+gE1EUqaqqIiwszKuCIrVSwRUHJ3Lyoige+txZ9722rBOTzeH2zO2Lq5sRpee0nGie+q6Z+z+pIzpIw3FZkeP+xlv32WxzenC5iWH888wgbl5VyRstwfxlQRyD/X1UVFTgcDjk8/pMnZX3F+VTmFn1G5/dEclyj1qdF1V3dzebN28mLS2N9PR0j+sF+6vJitBw7pJ47j45kxcuK+S73x3O4qRgogL9eLe0neP/vYEnvm7ANLa2KIqYTCZaWlooKiqaktiw68xnEyEx1J9Hzs3llSsWE6rT8NtVFVzw7CbK2gbk9FljYyNFRUX4+flNUcQy+QLyVyu5/vAU1l67hCPmh/P4N02seGIT66u6PBIXJpcvuYNWreS6w5JZe+0SDksP4z9fN7HiyU182zhETEwMERGRKBQCCQkJmEwmOjo6qK+vl9OBU3XRRQf5cf/KBTx3cS56PyUjFjubmgfG7Xl3XFS7KKJWCuTFB/H7d7azuXl8m6637cKWMXL7qRQclRHOHcen81VNL49/byA5OZnCwkLy8/MJCQmht7eXkZERtm7dSlNTE0NDQ7tN9v1FhQVmmNy+QqqfXr1lJ5Xbq2loaGDJkiWyML8neCpbDfBTERmoYd0NB3PwvDAe/qyOkx79nne3tlJcXIwgCBQWFnrljknRWtcquaKUUFZds5S/r1hAS98o5zxVzDXPfkvHgEkuTJmyQm3Cz13/Li5YywNnOskSpFXxu7e3ccUrZWzvGF8LL1/WPlx70pn5mYty8FcruHlVFde+XkH/iBUB5Lx6XFwcKSkpBAUFYTAY2LRpE2VlZbS2tjIyMuL2gi9KDuGtqxejUgjUdI3w6FeN8u92p2/fanegVip47NxFxIVo+dVbldQYdtVDeOsNmMcaUDRjkf3zl8Rx5cGJvLGlnWe/d5ZxS6KK6enp6PV6Fi5ciEajoaWlhY0bN1JRUcHOnTsZHR31ev/7iwoL7GO3XO/n7HBaU9LOtzVKbjsxiyVeEM9jtFylwGJzkBKu4/EL8vihoZe/vb+N297ZTnaMjpXJKq8vNsly2yc8jVIhcHZhPIcm67n/vRI+abKwqdPANUM6rlieNE356fifu9tLUXIIb1xVyOqSdh79spHzntvCWfmx3HhEMmF6jXzm3h27siwllDevKuTNLe3856tGvjfbUSgEHGOlo9JkStfc+sjIyLgyUUl+KDQ0VK6fV411x2VE6Xn6uxbC9RouKorfLctttTtFH0J0ap68IIeLXyjh2tfL+d/l+cQEaX12y13TdjcdlUL7oIlHvmgkJsiPU7Od4onSTci1WtDTIAgpKOlpCIIkvrg/YJ+65XaL8454cWEUYYH+3Lq6knOfLmZzc/+Uj/NkuZ2pMOeHKooisYoh7lys4E8npdMxbONvP5r57apydvabpt2bq1s+EQaDgfodVfzfmYV88KuDOTQtjH99XsfJj33PhhbTpBvPxIAaTP1eKRUC5xbGse66Ii5cEs+aknZOfWITr2xswzF24xAnfPcWaqWCi4riWXddEWkROuwOkX9+Wu9xHalENDc3lyVLlhAdHc3g4CAlJSVs2bKFhoYGBgcHcYhwWHoYR2eEc//HdXxYadhNcjtkrykuWMt/z89mxGLn2tcqGBi1ek3uXW75rudXCAJ/OzWTouRg/vBeNT829gHui04knbXExETy8vJYsmQJUVFRDA4OUlZWxubNm6mvr6e/v3/c5707brkgCCcKgrBDEIRaQRB+7+FvjhQEoUQQhEpBEL7yZt19Yrmlc2qvwVlHPD8ujDtPy+Gd0nYe/qyWC5/dxAkLo7j1uHSSwibnJz2SW6XAbHNgs9morKxEpVJx0LJlLFcoOL0ggT+/sYGPqgx8sq2LK5Yncc2hKR5bKzXKXc0eE/fd39/PkiVL0Gg0BAGPnu/0Eu5dX80/vuvhowYTfzpNw6I4Z/R/qjP3VAj2V3P78WmcXRDDPz6t5/5P6ogJcgYJHexZLjVMr2F5WiiNvaO8srGNEH8Vx0/T9alQKAgJCZF75SVt8NbWVmwOkb7uLm5eFkffiIU71+7gwdPTiPFxjza7OC7HnRkdwCNnL+La18v59VuV/N+REfLNciqYbZIm2/gbgUal4F9nL+Kyl0q4eVUVL16aT3KwatobhqeUW2dnJ9XV1fj5+WG326murvbJco9dx1MKNQiCEAI8DpwoimKzIAheCQ/u9TO32Wxm8+bNOBwOigryAGdATakQOKsgjo9/fQi/OnIeX9d0c/Jj33Pv+moGRsfLDk3llpssdoqLi4mIiGDRokXyhxbgp+K8Bf68d10Rxy2I5ImvGznh3xtYtaXNLelkyz32O6vVytatW3E4HPL52hUHpYax5pfLuHFZGE19Js56aiN3vFNJ15AZ6fqaeOb2FmmRep44P5tHz1nk1BUH1ld27dZarhBF58V+anYUj33VxAfVQz7dMCRt8KwFzorAsLBQlDj45UKI0cFt79Wy3TDq04QYq0Oc1O65NCWEe07PZEvLIPd93YnoRXbe4sYtlxCkVfH4edno1Equf72cnf2jPpeLSim3zMxMli5dSkZGBv39/axZs4Znn32Wyy67jOLi4mnXGVPLrRVFsX5MVUUSanDFhcAaURSbAURRNHizx71axNLb28u2bdvIyMggMjISs9XZUDHikgrTaZTceNQ8zlkczyOf1/HiD828XbKT64+Yx4VFCWhUCo+W22YeZdRqY9GiQre65AqFgphADQ+encMly5K4d301d727jVd+bOGOEzNYlrqr51kOqIlOV6usrIx58+YRExPj8fUpFQInZQRxcnYMq7YZefnHZtZXGji70Fkj7Xo/6u7uprW1ldDQUMLDw6fNjQqCwJEZ4UAGv3qrkleK24j2i+KYebt/vpPKT/9yagbDZhtPbuolIljPeT7ONZNujv5+fnLl2PMZo1zywlb+udGIik0khfjJzR9TjTiy2h1uC1hOXBhF17CFf3xSzxMbe/h7QvyU15t0PNO4ITdAbLCWx8/P5rKXSrnlnRruOmjPRj75+/tz9NFHy6OMli9f7pU2fltbG0wv1JABqAVB+BKnUMMjoii+NN3ae8VyS+5sbW0tixcvloM1fmoVSmFXKswV0UF+3LNyIe9cu4xFsUHcu76aU//zPZ9sM0zSE5cEES2jRmwOPLbbuVr8/MRgXr96CQ+enU3/qJVLX9jCDa+V0tQzAuyy3GaLjbKyMnJycqYktgRBENBrFNx+wnw5av/SD87PrrHHiCiKNDQ00NDQQHx8PDabjaqqKoqLi6mtrZ02BaUZO0Muig3g4e8MfNtk9Pi300J0FsWolQr+ecYCFkZquPeLNr6p7fVpGYncrkUsMcH+PLwyHYVC4LFyCIufh0qlorGxkY0bN1JVVUVHR4fcJizBOsEtd8UlSxM4c2EQ720f4JkNLW7/RoLZ5nD2509R2psZHcBDZy2koXeUhzYOyxJPewKj0UhwcDBLliwhIyNj2r/3EOuY+EMVsBg4Badowx8EQZh28Vknt8ViYfPmzdhsNpYsWTKpztlPyZQlo1kxgTx3aQFPXZSPSqngxtfLuPylUmp7nReFyWRi06ZNaDQakhPicIh4FHmY6M4LgsCpOTGs/9Vybj46jQ31vZzyn++5b3015rEbjslqZ8mSJV7357q2vUpR+/tWLgTgv1818v63WxkZGWHx4sUEBweTlJREQUEBBQUFhISEjEtBtbW1YTJNDP45L9ZfH5lKVqSWf3xj4Ns638goQXRxcLVqJXceFs68MC23rK5ia8vAlI91xS5yj/95QrCGuw8LY8Bk46a3q9GHRpKdnc3SpUtJSEhgdHSUiooKNm3aJOfWbS4BNXe4NC+IY9KD+PeXjbxT2uHx7yx2B34qxbTe5PJ5odx2ZALlBgt/er96j4tZRkdHfTpzjynNTifU0AqsF0XRONbm+TWQN93as0ruvr4+iouLSU5OJjMz023Qwk8pjHPL3UEQBI7IiGDtdcv486lZ1PcY+csPZn796mY+/raYtLQ05s2bh5/KeW4ye7hZeDqra9VKrjsilY9+vZwVebG88H0zN75eAoBSrfFpkIG7VNji5BDnP0Q7f/tuAH1MqsdBdpmZmRQVFZGeni57JMXFxdTU1NDb24swdlP3Uyn4+/HxJAVr+M2qqknFHt5AFMcfpXRqgX+ckkx0kB83vFnJjs7p9eYAWSnGXW15epiGR85eSFPvKL96s5JRq13uW09NTZWLSaTcek/fABbTqJxbnwgBuP3IeA5ODeHP71fztQcvw2xzeHTJJ+KEjGAuzAnkvXLDuDz97sDXaPmYeOJ8QRBSBUHQ4BRqWDvhz94FDhMEQSUIgg6n275turVnJRUmiiL19fVUV1dTWFgou+HuoFG6d8vdQaVUcEFRAh//ajknJgl8XtPH3d9beXFrH8MmG37qMVfa5p7c0832jgr04/dHJ/Lng7WkhDnPYB8327D5EBCaeGQAMA4PAXBxUSwICi5/cTNt/Z4LIyRFzcTERPLz8yksLCQsLIzu7m5qqqsB6DQY0Ah2/npMNDHBftz4ZgVV7UNe7xOc9ekTDVuYTs1TFzqLXX75WjktfdMXcDjcuOWwq0LtoNRQ7l2RRUnrIL9bs22SZyXl1jMzM/EPCCRQ78yQ1NbWsnHjRnbs2EFXVxc2mw2Hw4GfWsnDZy0kIzqAW9dUUd42WUbMYnPg56WKjN1u5/ycEM7Kj+Hp71p4a0v79A/yAF+njYwZDkmoYRvwpiTU4CLWsA1YD5ThlFV6RhTFiunWnnHLbbVa2bJlCxaLhaKiomkVKfyUTGu5XWGz2ajdXsGKVIH1v17O8QujePKbRo7/9wZKxlxJT+R2RzxXtLe3U15ezqmH5vPguU6vZ0evgz+v2yFfwNNhouVua2ujvq4WgKTIEJ6/tBCj2c7lL27FMORZzNEVSqWS8PBwMjIyyMrKkn/e0dHBYFcbdxwUgF6t4NrXyqnvnmztPEH0EHeOC9by1AU52Bwiv3i1fNp9SmSdmKJyzXOfsCCSu05M56vaXv5vCvfXaneS1zW3LuWXS0pK6OjooLOzE7t5hP+cu4gwvYYb3qykqXf8TcgXyy3lue8+aT6HpYXxt/U1fF3T49VjJ2J3BhKIoviBKIoZoiimiaL497GfPTFBrOGfoiguFEUxWxTFf3mz7oySWxRFtm7dSmJiIllZWV4VG/ipBEYtU8ovyxgaGmLjxo3ExMSg1WpJCNXxwFnZvHVNEakROtaWOc9gjT3ug0wKhcJtlF1yfzs6Oli6dCkBAQFoxlz8gigVq0s6+OsHO7w6j0nkltY0GAwUFuQDThIsiA3kmUsK6DFauObVcq+62Mav7/weERFJUlISCQkJZCVF88fDQ7HbbVz+wmaKq+oxGo3T73eCW+5KxrRIPf89L5teo4VfvlY+KR3pCqmwZrqusPMWx3HdYUm8U9bJv75odLvWxICalF9OS0tjyZIlsvpKa2sr9VUl/HaxHw67nV++Wkb38K730mxzuE2DuYMksaRSCDxw5gKyogO49e1tVO70zROC/WdOGMyCW15UVOTTcD+tSjGuK8wTdu7cSXl5Obm5uZMUWHLjg3nlisVcdUgyALe/XUmrG3dSqVROuuClgJ9arSY/P18+X0tFLItjNVy6NI5Xi1u576OaaQkjCAI2m40tW7agUqnIz8/Hb6xUUbL+eQnBPHFhHjsHzNMSxxNEdinMhoWFcXhBFs9fthi7oOCuTzvYXFVLcXHxOJd28hpT93PnxAfx73MW0dQ7yg1vVHr0sDwF1NxVqF13WDLnFsby3PctvPRj66S1bG7y3K4QBIGIiAgWLlzI0qVLOTg7jT8dHU3XsJnLn/uRsqod9PT0YLbafbLckiHSaZT857xswnRqrn+zwqtjiSsOWHIDPhcDTBdQczgcVFVVYTAYZKsqYaJG17IxIb9hs51LX9g8qcx0YkBtYGCA4uJiuf/a9UKUa8tFgZuPTOaSZYm88H0zD31aNyXBLRYLTU1NJCYmyp1tkkVzPWsuTQnl4bMXUts1wvWvV2A0e+e9yLXlbraQEaXnv+fn0G9y8PBWK+kL88a5tFLX0/DwMKIoOtdw4ZG713VQaij/WLmA8p2D/GZ1ldt0kd1Ly+3cv8CdJ6RzXFYE//y0nvfKO8f9Xmoc8QRXIkoloscVzudfZ2fTOizycPEwnV09dPcNYDOP0tzcLL9eT5gojhgRoOG/F+Rgd4hcP9Zg4y2Gh4d9VT6dNezTrjBwXhBNfaP82DA56jk6OkpxcTE6nY68vLxxUWt3hSySG3b78fMZNNm49IXNtA/sIrirW97W1kZVVRX5+fluA367yO28SO86KYPzFsfz1LeN/OfLBrevxWAw0NLSQmxsLNHR0bv2OnaBOyZcYIfMC+WBMxdQ2T7EjWOR5OkgRaQ9Xaq58UE8dm42zb2j3PBmFRpdoOzSSl1PjY2NFBcX09fXBw7HOKvuLnV0bFYEfzp5Phvq+7jj3R2TKvokvk+MlnuqA1cqBO5bkcXS5GD+uK56XF7d6ZZ7ttye1jwsPYw/n5LBplYjL+1w4KcLIDhQL+fWi4uL2bZtG52dnfJAyqnWTA3X8eg5ixi22PnGh1Sj2Wz2KGu1t7HPya1VKXA4RK75Xwk/uBC8p6eHLVu2kJGRQUpKyqSLzl1aSyJ3XIiW5y4poG/EwqUvbKZz0CQ/xm63s23bNlnfzFNOUrrA7OKueWZ/PjWLM/NjefTLep76plH+W6lIp6mpibS0tEkdQ+4st4RjMiO4Z0UWm5sH+M2qKrls0hNcxRpcv7tiaUoID565kG0dzpuG1NMudT1lZ2ezZMkS/HU6RES5CWRwcNBja+eZ+bHccnQqH23r4u/ra8f9javCrCumahzRqBQ8cs4i5kfq+e2aKsrGIt6eKtS8WXNlXgy/PjKF9ysMtPSN4q9WERcXR3Z2NkVFRcTFxTEyMkJ5eTmbNm2SGz88iSMWJAajVgr80NDncT/u9jeTWmx7gn2+i2B/JRqVgsRQf375vxK+r+uhrq6O+vp6lixZIhfqT4Q7y60dC4JZbA5yE4J59pJCeowWLn1hC52DZhwOB42NjWi12kmewES4Wm7pJqJQCPxtxUJOzYnhwU9reWFDE3a7ndLSUiwWC4sXL0atVnts+fQUcT95URR/PmU+39X38bu3t01ZKeXqlk9VoHFkRjh/P9150/jtmslrKhQK1GoNaqWKJUuWkJ2djVKpxGAwyBVkE63cFQcncuXBiby11dmOKkEit8IHcoOz3v/x87OJCNBw/RsV1HePTGocmYjpusKuXp7IeYtj6TZa6XSJ8guCQHBwsJxbz8vLIzAwkI6ODjo6OmhsbKS1tXVS77bZ6vB6dPD+pHwKs0BuXzuVtColJquDFy4tJDHEn2v+t5XNrcMsXrx4ynprd5ZbCqBIqbD8xGCeubgAw5CZS54rpnRHA2FhYaSmpk4/6misR9mV3NLP7z9jIScsjOLej2q4d9V3REREsGDBAhQKxZT93FONRzozP5bfH5/G59U93PXeZNd3F7xXYjklO4q7T0rn69pe7lw7eU3RRSFRo9Gg0+lITU2VK8ikccibN2+WWztvOjLZmQ/e0MKLYwExOVruo0AiOM+3T16Qg1qh4JpXy7BMU6E2HbkFQeCO49PxVyuoNhhZX+W+wUaauZ2VlUV4eLg8are6unpcbn3UavcoeTXVHvYH7FOxBnAqoIqAaB3lxmwH/yr1429f95CY2M/BLsPrJmKqM7fJxbUtTArhnhPiuf2DZh6r0nBvvPeieCqlAptj8h1ZpVTwh2MS6Ozq5uUqCxlpAuc659W7nboiGbSJZ+6JuKgonlGrnUe+aMRfpeRPp8yfdI6dKqDmDucWxjFstvPw5w3oNUr+dPJ8l2KjybrlUgRemnySmpoqzzNrbW1laGiIs5L1GPoDeODTeoK0KjKinEcbdwE1b1zUxFB/nrggm8tfLsVotk/ZGutNP7dSIRCoVaHTiNy5djthOjVLU0KmXFOv1xMZGSmPNxoYGKCnpweT1UF/dyeNjQp5Qqkn8u5PyqewH7jlujEl0ZLyKg4ryufVq5eRHK7jl/8rYUOd50KCqc7cUvmpw+GgsrKSOPUoT12Uj2HYyh0ft9Mz7F1uWa0UJllugJaWFupqq3n68oM4fH44f1y3jbdLnOXA7gplpIi5N4MNr16exDWHJLGmtIP7P54cmZfP3F69AieuPDiRXyxPZHVJBw9+1rDrvI53VkatVhMdHS2nn1KSk/nNQaEsClfyp3XVfLC5bmxvk2WkvL3YM6MD+Pc5ixCBz3b0eMygeLum1S5yeHoYyWH+3LRq6lLaiTcMKbeekJyKCCTGx8i59Y0bN1JZWUl7eztm8/jintHRUZ/GCAGsX7+e6YQaAARBKBIEwS4Iwtnerr1P3XK73Y5lxFkokJmdi16vJ0yv4cXLCkkO13Htq6V854Hgbs/caunMbcdkMlFcXIxeryc3N5eD0iL4x2nz6Bi2cflLW7wqHlErFePILaXlent7KSoqIihAx6Pn5XJwahh3vlPF++UdHhVJFcL4ls+pcOMRyVyyNJ5XN+3kkS8bx62ncDHdnp7LHX51ZAoXLInjxR+dM67BSW5XY+ttkU5gYCDz01J57sqDyI4N4H9lzmBYfW2NrDtmNpt9tmRLxrTbu4Yt/HaN+7SbtIfpYLY5CNSq+O/52eg0Sq57vYKdA+4VeDwF1KSGpgCthpiYGPnmlpycjMVikbv5ampq6Onpoaury2ehhhtuuAHgJGAhcIEgCAvdvF4lcD/OElWvsc8s98jICMXFxQT6O8/VVseuD0wieEq4juteLeXb2skEn8py9w0Z2bx5M+np6eMi7UVJwdx1WBhNPSNc/uIW+kamJrhaKchuuVTsotVqyc3NlfOiWrWSxy/IY3FSCL9bU8mXdf1uSaLy0nKD8+L93bHzOKcglmc3tPDUd827fjf23dfpxoIg8Pvj0zg9J4r/fN3EKxvbnORz83feQqdR8vj5OUSPqcP4RyXJAwyrqqpoaWnBYDBM28YqQXp/js4I59u6Pv70fvW0RxlPkGrLY4K0PHF+Diabg2tfK3ebs/Y028s0Nr3GX72LJlJuPTk5mYKCArnuv6amhjPPPJPt27fz8MMPUz3WAzAVNm7cSHp6OtMINQD8ClgNeCXSIGGfkLurq4utW7eSlZVFdIQzGj4xxxum1/DCZYWkRui47rXJBHdnuVUK58Xf3tlFYWHhJBVVhUJBTpSGxy/Io6FnhCte3DJlgYJmzHIbjUa52GXevHmTCOCvUfLERfnkxgfxhw8aKG6bXNUkCRG6YipLKQgCd5+ULqukyNVceyCQqBAE/u/UTI7JDOf+T+po7h2d3DniI0J0am45eh4A//6yEYNJIbexxsY6Z6ZN38bqhDQEsCAxiBuPSOa9cgMPfea+pmAq2BwiNocoB1jnR+n59zkLaes3ua0n8HSOH7E4b0hTRculuv+DDjqIF198kcWLFxMYGMj3338/7T7b2trkQN4YWoFxYleCIMQDZwBP4CP2qlsuiiI1NTU0NTVRVFRESEiIrIDqrgRVIvi8MYJ/40LwiZbbbrdTWVmJWgnhUTFuzz5SV9ih6eH85/xcaruMXPnSFo/ln2qlApPFRktLC3l5eVN2twX4qXj64gIyonT8Y0PfuL2C03K7BoocDgcOhwOr1YrVapX/3xUKQeCvp2XK1VxvbWmflOf2FSqFwD9WLuDg1BBK24bkHLi05u4EhIL8nXFZQYBrXi2TC4ek9NN0bazS65bccJVCwTWHJMnHiOe/n1qYYSLcSSwtSQrh/pULKGsb5La3t0/yoty9bum90XpZxmo0GomOjubqq6/msssum/bvvRRq+BfOcb3ed1eNYa9ZbsmtBcZpkElTR0Y8VGeF6pwET4vQcf1rpXxd4xxJ7Gq5pUq24OBg/NUqPGk/uAa7Dp8fwX/Oz6PaMMyVL21lcALBRVFEtFkZNVtISUnxql44UKvisbOzSAhUccNrpXxfv6soRyE43XJRFLHb7TgcDtRqNSqVSi6usdvtWK1W+ffgJOP9K7M4LC2Mv35YI1dLSVfA7pBcEgkM9VfRY7Ty3W6KPUiQ8ve3H5vGiMXONa85G2JcbxZTtbFu2rSJ0tJSmlrbAOdxSDpGnLQwkoc+b5hSmGEiLB4klo7NiuCOE9L5sqaHv6+fvk9Ayrp4Gh81Eb72cickJNDSMu7G5U6oYQnwuiAIjcDZwOOCIKz0Zv29Qu7+/n5ZtGH+/Pnj7pKS5TZNUXoZqtPw/GWFpEfqueH1Mr6u6ZYJ0dvby5YtW8jMzCQpKQmNWuFxrYn93EdkRPDv83LZ0TnEVS9vZcjkLMO02WyUlJSgVIBa6++TNQv2V3P3ocEkh/lz3aslbGpyVjc5LbfTOtvtTsEChUKBSqVCo9Gg0WhQq9XyHiWi22w2lAI8dNYClqaE8J+vmpxPNE0Ry3TQaZQsSQ5BrRS4eVUVW1oGdttySx7JvAgdj52XTceAmWtfr8BosXtcz7WNVRIYlOJnrc2NVFdX09vTw19OSZeFGb6s9q4N051muYQLlsTxi+WJrNrawRPfNE/6vSuk68j1zD0VfG33LCoqoqamhqmEGkRRTBVFMUUUxRRgFXC9KIrveLP+rLrloijS3NzM9u3bKSgocOvWTuWWuyJUp+H5S50Ev/61UopbjfT09FBTU8PixYvlSjY/ldJjP7e7INzRmZH869wcqtqHuPrlrRj6BikuLiY6OppAvT92N3nu6V5/oEbg+csKiQ3W8otXSihpGXBabrtDPt+5K6dVKpWo1Wq0Wi0ajQalUokgCM7GBhw8uDKD+VFOIYMKH4UZ3EGlEIgJ8iMm2I8b3qigod+75pWJkJRYlAqBwsRgHjxrITUGI/d804PF7t175+/vT0SUsx4/fd48IiIi6O/vp7y0hCsy7MwL03Drmiqq+6b3Ti12z+QGZ+ZgRW40j3/TxKqtnoUZRqUzt8p7y+1LtFylUvHYY4/BFEINe4JZs9x2u53y8nIGBwcpKipCp3M/HzlA6yS3FLyYCiE6Nc9fWsj8KD13f9TCxtYRioqKxhXqa8e0y93Bk8zSsVlRPHxODuVtA1zxwiaS0zOJi4tDrVRgc4heRXolSOmpiAA/XrhsMZEBGq5+eSsOUcTuEN0S29Ne1Wr1OKseoFXz++NSAXjhhxY2tQzuUcmjQ3QS8ukLcwj0U3HPhkEaen1rcYTJAomHp4fxt9MyqTRY+L9PWrzOEkgBNY3KWTCSnp5OUVERBdkL+OtxcYRpBR7ZYuaTjZUe21hhassNzs/oTyfP59C0UP76YQ0lBvc3jFEpWu6lW7477Z4nn3wy0wk1SBBF8XJRFFd5u/askNtoNLJx40bCwsLkmmVPkMg9lVvuCg1Wrl/gIClYzWMl5kkaWn7TkNudWIMoimQFmLguX0vDgINb19ZjNNvG0leTi1imgmuFWnSQH89dkk+QVkW30UKfD62DE9dUKpVoNBrCA503yXC9hrs/aqV+0Kl+Y7FYxp3VvYGkxBITpOXpi3JRCPDrNTW0eTGRxRXuWj5PyY7iqoJAvmsc8jqlJQXUJtaWa7VaFqUl8cwlBfipFNyzYYDa9l63bawwPbml53jwzIUsiA7giTILJa2TpZqkPLe3AbWRkZH9pt0TZoHcNptTCnjRokWSsuOU0GudgTVPATVX9PT0sHXrVoryFvLEeQtJDlZz4+ulfLFjV/2wxkfLLVWxDQwMcP1pB/PA2dlsbennl/8rGRcE8xaS5ZYCZ9GBGp67JA+FIPB9Qx+1XXsgRex8BgBOSoJwvZo/ftZBdbdp3FndYrHIemNTwbX5JDnMnzuXB2K2OvjFq2V0eSkBBS6NIxM8khPTdFy1LIa1ZZ08MMXIIgm2McvtqbY8NkjD74r8Mdvhng2DpC3MG9fGunHjRmdbZ5fzbD6dWINOo+TfZ2cRqhW48c2KSRJVMrm9PHMbjUaPHuq+wIyTW61Ws2zZMq8E2cFZp61STH3mlrS+6+rqWLJkiXOkjU7DHw8PJTM6kF+9USYTXKtWYra5X2uiO2w2m9m0aRMBAQGyh3Fydgz/PCubzc39VHUMYbH77pZLaS1pwFximI64YD8E4MqXS2js8V7nbCJGR52PTYmP4aUrlhCoVfHL18pp6rfIZ3Wp200Kyk2MwI/br8u/EwOV/OvMDLqHLVzjoeDDHTy1fDocDq5cGsNFRXG8vLGNp7+bOqVldckQuIMoiiQFq3js3EW0D5i5/vUK7IJKbmOV2joHhp3vUVNdLY2NjVOO5A3WKrl9WQBKhcB1r4/Xi5Py4TofouUHtOWG3VFj8ayAarPZKC0tZXR0lCVLlsidYkqlEn+lyPOXFsgE/3xHl9Mtn0IHXcLg4CCbNm1i3rx5k/rFT82J4b4zFtE/YqVtwOz1kUGCyWRiaMgZ8JLW9VcrWZwUgiiKXPFSibOAxEf09vZSW1MDQEBAAHHBWp6/pACNUsGVL5fS2DMin9X9/Pxkoksey8RUm+hG/TQ7NoBHz11Ec+8o13mpELOr5XP8z6XGkduOS+PU7Cge/aqRNzZPzPTsglW23O4vSykYWZgYzANnLGBbxxA3r6qU3XmFQuGcxBnpDMxlZaSh1Wppbm722MbqcDiICXSOF+oftXH96xVy1mTUakc5zWADV+xPEkuwHzSOgFNqyR25pRLVyMhIFi5cOKm43+FwEOSv5vlLC8iKCeTXb5TRP2r16JZLaG9vp6Kigvz8fCI8jM5ZkRdLfkIQJpvI377q9org0sWckpJCXV0dP/74I9u3b6enpweFIKBVK3j24nzMNgdXvLzVp7Ptzp07qaurY9FCZ+mxZIeSwvx59pJ8HKLIlS+XjFtzYqpNisBLRwa7wwHiLs9Esm7LUkJ5wI3YgydI52lPYg0KQeAvp2Zw5Pww/r6+lg8r3VdRWqdxy10ryY7MCOdPp2TwfUM/d723Y9yZXvr89Vo/YmJiWLRo0ZRtrIIgsCg2kIfPWkBd9wg3r6rEYnNgGuvl9jY9uDvKp7OJ/YLcGuWutIMEqUR14cKFxMdPHj/pWsQS5K/muUsKWBATSGnrAP0eKs5EUcRkMtHe3s7SpUunTVvMi9ATpFVS1mnmhtfL5Ckk7mC32+XobVxcHPn5+SxbtozIyEi6urowjRrpGxhEbxvgv+ctwGi2c+XLW+kYnJrgkspLV5ezpNbPzxmjcL2Y0yP1PHNxHiMW55rupIgVCsUkqy61d0oW3fU4cdQ0Yg+usHk4c7vmzaWRRYVJwdy5dofbKSk2DwE1CRPLRM/Ii+Hmo1L5sLKLf3xSN2VAzXUQwuLFi8nNzUWn09HR0UFvby+VlZWk+pv504nz2Ng0wF3v7WDEYvf6vA0/E8vtayGEU7vcSQzpYm5sbKSoqIjg4GC3j5kY+Q7yV/PcpYWE+qvpHrbw6bbx1kFSJAUoKCjwaoqIWqlArVRw7ZIgvq3t4VdvlE2SQXKtOJPI4rrH8PBwsrKyCAoMQKv1d56BDQ3cnK+ie8jM5S9sweCB4A6Hg4qKCux2+7hmFefzjv/bBTGBPHlhLj1GK1e+XDJt15tCoUDEWUij0Wjo6OhAFEVUKpVM9hOywrjrxDSPYg+79jm25jRiDVq1kkfHpJV+s6qKktaBcX8vWe6pztwTr60rD07gkqXx/K94pzw/zFOFmiukNtbk5GSio6NJSkrCbDaTJHZyToaa9VVdFDf1+STU8LMgt6/QKhWMWu1yZZjVanU7JtcV7qaHBGpVHLsgCqUAN71ZzidjBJdSc3FxcWi1Wq9vPmqlgM0ucnSylr+clsVXNT382oXgkj65O2JPhEqhAIWSlJQUlixZwllHLeH+k5PoHDJz4dM/sGFLOQaDQbb+0nCHkJAQMjIyXMo4Pe83LyGY/56fw85+E1e9UuqFZLIzFVZbW8vw8DCFhYXodLpxBTRn5kbx6yOSWF/VxV8+qHabSrRP45a7IlCr4r8XZBMd5Mf1b4zvs5YCat645RIEQeDWY+dxanYU//6ykdVb271Khcl7t9tRqVQEBgaSkpJCYWEhd6xcwpnZoTT3mTGOmsa1sU6Fn0VAzVdo1QJdgyZ++PFH57xnLwYaeOplDvBToVIqyI4L4uY3y1n1Qy0lJSVkZ2fLeufeprbUSgXWsaqy85Yk8KdTsviiupub3nS66J4stjsoBMZZPrVazbH583jqonx6zQL3/TBMW1cfmzdvpri4mO+//56YmJiJXUMIksySh5dQlBLKv8/Npr7byC9fLZsyIOYQRUymUeeZ02WW+cQCml8cmsJVByewprSTBz+rn5Rq81VDLVyvkUcWXfvaLm1wbwNqEyGd6Q+ZF8pfPqyhcqx6zxtyu1tTrVbzp9OzmRfuT1xYwLg21qmmsVqt1ikNkjtMJ9YgCMJFgiCUjX1tEAQhz9u19wu3/LRUFQ29Jt5s1hEVPf2Y3KmeQ6tWYLE7ePqiPOaHa/jj+kb6AlLk1JwvAgcqpYDVJc994dIE/nhyJp/v6OaWt8qx2OxeERucBR7u3NqilFAeOz+H5j4L92wYIi4lDYvFQlxcHF1dXfzwww/OOuux7ilZq2GKps9D08N56OxFVO4c4rrXytwGK61WKwMDg6hUykn1/q6QCmhuOTadC4viebm4ned+3DnurG4Zm7Ou8EGJRRpZZHU4uObVcrqGzNPmuaeSWFIrFTx01kKyYwN5v8LpsXkzlGCiZrn8ugWBEJ0avUZJQEDAtNNY29vbfVY+9VKsoQE4QhTFXOCvwFPerr9PLbfUAprgb+W65XF8tL2HW1dX7tGcZD+VAlGEmh3buPOQYHLig7jtne18VOUUv/dUguoOTsstjnNFLyiK5/fHp/Hpjm5uf2dy66AneCI3wPJ5YTxybjY7Ooe4/vVKsrLzSU9Pp6CggKKiIkJDQ+ns7OTHH39kx/btAFitU6eojsmM5L4zFrC5eYCb3qwYFyswmUxs3boVP60WP41nEUpXCILAnSfOZ0VuDP/5upk3thrkslhJ8kF02MdVyk3XiCKNLOoxOvPqgybnMcKXM7crpGkhgVpnPOWZ75qnj/RPccMwuVE+9TSN9dprr6W1tZXf/va3fPHFF1M+pwRvxBpEUdwgiqKkrfwDzs4xr7DPyC2dKUVRJDIyksuXxXH78fP5sLKT37xVPq1+tycocD5OFxjE0oJcnr2kkNz4IH7zVgXrKzunnfTpClm73KVV0263c8myRG4/Pp2Pt3Vx+9vbsHmxntKNWIMEURRJUg3yq0IdTUMiN63ZIWuIKZVKIiMjWbBgAQcddBBJSU43vbmlheLiYhoaGjwWaZySHc1fTsvi27pefjt20xweHqakpIT58+ePRcy9eiuAsf7y0zM5NiuCez+q4d0y5/spJbj9tX6oVCpEUcRms8lEn6osNic+iEfGRha9usnZxOGrW+6KEJ2a47IiUArwn6+bWPnkJj7d3u3RW5uS3DbHlNFy1zbWd955h/j4eI4//njKysqm3KMEb8QaJuAq4EOvFmcfkXt4eJji4mLi4+PJyMiQI7RXHpLMnSdm8Mm2Ln795uTI9HQYGBjAsNOpWBIZEwdAgFbFM5cUkBsfxC2rKijumL4sU4J0kVnsjkmBs8sOSuTWY9NYX2XgjnemJ7hScC+zJOmymUwmrjqxiH+cuZCS1gFueL1sktWRJH4AUlJSyMvLw8/Pj4aGBn744Qd52IKrp3FWQSx3njifz3Z0c+tbpZSVl5OdnU1oaOgk9VNvoFIoeODMRSyfF8of3tvOx9sMskeiVjm72tRqNdXV1URFRaFWq3GMTTXxVCl3cGoo96/MkkUePAUCvSE3gFKhIFCr4rmLc9H5qfjN6ip+8Wq529JfT245OPsdvNUsl6SaTjzxRG666SavHuOlWAMAgiAchZPct3u1OPvgzN3R0UFZWRk5OTnExDjP164568sOTuKPp2TyxY5ubny9dMrcMux6g3bu3ElVVRVpKUnA+DG+AX5OguclBPHvTcOs91BEMRESuUdGzRiNxknn6yuXJ3HLMfN4v8LAne9un1KS151bbrVaKSkpISAgQA4inrgwintXLmBjYz+/frNiUintLiUWp9Z4XFwcubm5LFu2jOjoaPr6+iguLmbr1q20trZiMpm4eGkCv1gWxSfV/azrCEI3lt/f3f5tjUrBv8/NIS8+iFtXV9HYM4LAmCDFWMZD6uryVEAzMSh3XFYkx2Q5C4rOeXYLz2xonlSM5C25LWPje4uSQ3jzqkLuPCGdbR3DnP30Zu77uHbczWOqNUetDp96uX2tK/dSrAFBEHKBZ4AVoih6PVt4r1luURSprq6mra2NoqKicSmDiXpoFy1N5K+nLeDr2h6uf63U47lJCups376dzs5O57p6p7ySaUIJqiyDFKbmzveq+aCi092S46AeO/tFx8WzY8cONm7cSE1NDf39u0QQrz4kmZuOSmVdeSd3r/VMcKVCkFNG4FSP2bJlC/Hx8SQnJ48j2Wk5MbI7/Zu3KuX+ZOdrHns/J6yvUDjbJDMyMjjooIPIzMyUm2K++eYbluh6uXRxJO+UG7h3bFqpiO+WW4JOo+S/F+aSHqVnfaUBheBU29m6dSuxsbHj3E3XXnXXslgYX/+eHuH87A5KCeGRLxpZ+eQmPnNxqb29GZntu8b3qhQCFyyJ4/3riji7IJbXNu3ktCc28daWduwOcVrL7W2e29debvBOrEEQhCRgDXCJKIrTqy66YK+QW5JYEgSBwsLCybO03IgdnrsknntWLOS7+l5++b8StzrWgiCwdetWeVSuSqWS2/PcufQBfiruPiyU7LgAbl1dwQcVnqV7RFFEITgvqsjoGAoLC1m8eDHBwcG0tbXxww8/UFlZicFg4KqDE/jVkam8W9bBH97b7vZs7Wq5BwYGKCkpcQpEugwMdMVZBbH88eQMvqzp4Xcu0zWnS4VJkM6CQUFBTini+fM5NRmOS1Lyv41t/O29cuwu0ffdQZBWzdMX5RHgp8Iuwh/e/JHohKRJI5YnYmKlnKRAY7U7UArw4BmZ/Pe8hfipFNw85lJXG4w+We6JabAQnZq7T5rPG1cWkhruz18+rOGC57ZSZTC5XVMURUatDp/aPX0tYPFSrOGPQDhOeaUSQRA2eb2+T7vxEq5316GhIcrLy0lPT/c4t9sduQHOLIhDqRD4/duVXPPKVp68KB/9mOba8PAww8PDZGZmjrMSE0cKTYTeT8XDK9K57f1Gbl1dCcDJ2bvSb66FKZoxt1yunFKpiIqKIioqClEUGRgYoKuri4aGBpYFqOnLD+WVkg6UCqfKqGvFlpPczkmgDQ0N5OfnTytgf/6SeKx2B/d+VMsd72zj/jMWepUKg11neY1GQ15eHoIgEBMTw8MLHdz9TiWvlXQT6S8QolXQ2NhIZGQkOp3OZzc9XK/hmIxQ3inv4r16Gz8YavnNMQ5Oz42ZVLHmDhKxlEolIs6KQKVSycGpobx2eRCrSzr577fNnPPMZk5MD+CqZTFMfetwfvYaD0G5rJgAXrgkj4+2dfHAp/Xc9cUwx+8Uue14f1miWVoDplY+dcXuWG7YJdbg+jNXoQZRFK8GrvZ5YWZ5nFB7ezsNDQ3k5uZOeVdTKpVYLO7LJVfkxaJSCPxuTSVXv7KVpy8qYHSoj+rqaoKCgibJF+8aKeTelVcoFGhVAk9dlM8v/1fCb1dVIIpwSk7MpIoztUoit5tWSUFwtp6GhABONzs83MCQcYTVW9sZHhrkjydnEBIcPNY8AWaLlZaWFrfeiydcsiwRi83Bg5/Vo1EpuOkopxLLVJZb6qkPDw8nOTl50uv/2xnZOBTbWFvWiUKlRFQoqampwWQyERoaSkREBKGhoV5ZycHBQQb7e9FplDx1YR73fVzLne9u59XiNm4/Pp3FSSFevU4Aq8M5BFB6b1QqBxcvS+SkRZH895tm3trawTdN9Vx7mJ3zFsfip3Z/+ZrdWG5XCILAiQujODw9nPve3cL7tf18U1/MNYckccmyBPxUCkbHjnXenrl3l9yziVkhtyRfOzo6ytKlS6et4/ZkuSWckhODUiHw21UVXPTM9/ym0I/lRUVs377d49SR6QQb9H4qnhwj+K2rK3CIIicvihoXEZdSYVYvdMD8/f1JTk7mnqQkIj6r49kNLYjvbePcNJHg4GAG+vqxOxwUFBT4POL1qkOSsdhFHv2ywdnJNQXMZjOlpaUkJSXJActJ74Eg8LfTs/iurpfOIQvXvtvGrcemcXROGP39/XR1dVFdXY1OpyMiIoKIiAi3Qxl7e3uprq4mLDwCZVcP+YnBvHplIe9XdPLQp/Vc8sJWTlwYxW+PnUd8yPRjdqx2ByqXAhbpfYoM0nHxAg1LQoJ5qw7++VkDq0o6uPXoFA5ODZGlq6S/lwJq00GnUXJOlpZLD03nse+c013WlHbwu2PTyBzTqvP2zL2/dYTBLJHbZDKh0WjIzMz0rnprGnIDHJcVwc1LA/nXxkEeLdeQny9MOXXEUxrN9TF6PxVPXVzANa9s5bY1ldjtdk7NidnVyaTwbLk9QRAEbjkmDRB4dkMzoaGxnKgcABzY7A5KS0uJjIwkIiLCpyHt1x2egsXm4Mlvneqn7gJ3RqOR8vJyMjIyCAvzPEQRnCmteRF6wvRWRBFuequCxUnB3H58OtlZWYiiiNFopLu7m/LychwOB+Hh4URGRhIYGEhXVxeNjY0UFBTw0aeNcuGJQhA4LSeGYzIjef77Zp79rpnPd3Rz+cGJ/OKQJPlY5Q5WuzipOk0qdLJarZx8SAEnHwKfV3dz/0e13PDWNo5ID+U3RyaTFOYvjwWSRgl5A7vdTlK4nn+dvYgN9X3c/0kdv36rkoIEZ0WjLyos+1NdOcxSQE2v17udzOEJ05Fbmvt1YnYsj52fx47OYS5/cTNGG5MeN92Ze+INwV+t4D/nZbM4KZg73t3OOpcounSh2bxU8JTgJPg8LimK5Y0t7axt9SMmKgqlSk1mZqY8QGHjxo3U1dUxOOid0OGvj0rlgiXO/P3/ilvZ3rGr6WJgYECWt5qO2BJEUSTEX82aXy7hz6dk0tgzwrnPbOb371TROWQmICBAbnQpKChAr9fT3NzMN998w7Zt20hMTHR+dg5x0vlap1FywxGpfHDjMk5YGMlT3zZx0n9+ZE1Ju8dinomzuUVRZMeOHTgcDhYuXCh7VMdkRrLu+mX89ph5FDcPcvZzpfz76xZGrM5iI7PNjlqBx2EPrnAN0i2fF8qqqwu57bh5bB9raPH2zL2/SSzBftI4MhW5+/qczRRS4OyozEgevyCPuu4R/vBFD93D4zt1JBlaT+kzV3JLPdh6PxX/vSCPouQQ7nhnG2vLnFF0tdJ3yy1heHiYI4J7OTc/krcr+yhtHcDmENHpdCQnJ7N48WKZME1NTR6LUFwhCAI3HJECQMegmbOeKubutdvZ1riT7du3k5+f75P1kFJhKoWCcxfH8eGNB3H1IUmsr+zi5Md+5NEv6jGOteKq1WpiYmIICAggICCAnJwchoeH2bx5M4bubhAdkwbXg1N88f4zFvLalYXEB2u5e+12zn1mE5ua+if9rdVlNrcoilRVVaFUKt16gBqVgqsOSebDG5dxWm40L/zQyoqnNrOuqgezTUSrVk457EHCxCGAaqWCS5YmcP/KLMC3M/fPwi33NeLqidytra20trZSWFg4LrJ8+PwInrwwj2v/V8Kv367llavCiAhwngn9vLDcriWRkjXQaZQ8fkEu179Wxp3vbgMgIcTpNntz5nZFd3c3tbW15OXlcbBOh0ZTyysbW1ErhXFVTxJhYmJicDgc8nm3rq4OrVYru++u512F4Hx91x2WQt+Ilf9tbGVdeTtXHpxIttK7IJ0EUQTB5doN8FNxyzFpnLc4joc/q+e/Y7reNx09j9Nzommor8NsNpOfny/n1QFWtZSj6Otn+/btmM1mwsLCiIyMJDg4WCZOXoLzPP5BpYEHP63j0he3csLCSH57TBoJof7y+6wau/lWVFQQEBDAvHnzpnwNkQF+/P30BZy/OJ57P6rhD+/tQKUQiAzQULrTSEq4P6H+qnHBUtehEIDbGIgUbffWco+MjEybAtzbmNVoubeQyk8lOBwOduzYgdlspqioyG2RwfK0cP5ybCx/+qydS57fzIuXLyYq0M+rM7fkrk2sOPNX7yL4He9s4/oxK+mL5W5tbaWjo4PCwkK5/e+OE9LZ3NzPto5hTv7Pj/z6qFROGwsSuu4rLCxMJozRaKSrq4vy8nJEUZTPu6Jy103srDSB/IAgPmzT8MS3Lawq6eTXR6ZyRn7suLV9RXyIPw+ctYiLlyZw38e13L12O898XcuV+cGcdVjO5Ju3QomfRk1BQYE8Baa9vZ3t27cTEBBAREQE4eHhaDQaTsmO5ujMCF74voVnvmviix09XHpQAr88NNlpuRUCZWVlhIaGTor0T4Wc+CD+d0Uh71cYuP2dKra2DnLZSyUABPurSIvQkxqhIzXcn5RQLclhWiL8FXJZ7MSgnK/RcqPR+POIloNvrZWuqioWi4XS0lJZwWQqL6AgXs+9JyZy18dtToJfVkh4gJNQEyvUwOnq6fV6ecZ2ZGQkkZGR43pwJYLf8Ho5//mqEXCmaKaDFPgxm80UFBSMuyEJgsAh88KoNRiJDNBw57vbefGHVn53bBrL09yfj/V6PXq9npSUFCwWCz09PTQ0NNDV7zwLtrS2YAkO5biDCzheEChpGeAfn9Tyx3U7eHmjc+1D08Pdru26Z4Xg+eLNTwzmf5fn89T6zby+3cQfv+zhi50V/O64NFLCd50v7Q4R5djnJDW6REZGIooiw8PDdHV1UVpaCkBERASRkZFce1gyZ+bH8q/P63nmu2beLukgVKdCtJqJiIj1ShZ7IgRB4NScaP60bjunZEdz/IJI6rtHqOseoaHbyBc7ulntouiqVkBSqB/p7TWkhGmdxA/zJyVcJysD+XLm/lm45b5Ccsu9KXiZ+LhFUX48e0kBV7+ylYvHCK4QJrvlUqdSQEAABx10kGwZS0tLEQRBvuj0ej3+aiX/OT+HK18qobRtkDc376QwMZgwvftGfLvdTkVFBXq9nuzsbLc3JOcIX3jtqsWsrzTwr8/rufp/pRwyL5TfHptOVoznC0Oj0RAbG0tsbCy9Q6Pw5Q84HCL9/f2UlZU5BSSjI/jfFYV8vK2Lhz6r45pXyzg0LYxbj00jI9r92g436qeukPLlp+bGcsXxcbz8YytPfdvE6f/dyAVF8Vx3eAoh/mocojhJ+RTGRisFBhIYGMi8efOwWCx0d3dTX1+P0WgkJCSE3x4ayXmFMfzj03pKWgdRCvD7TwykRQ6TGq4jNULPvAgdccFar7wRZ0DNQbhew6Hp4ZNucP0jVqo7B/lq6w6MygA6jCKV7cN8vK1bLgtSCBA0Fm1XYZ/kxrvDz+bM7SsEQcBqtVJeXj5twYsrJBe7MDWEFy4t5MqXt3LJC5vRKMcPJnCnmOJqGc1mM93d3XIhh+QC/9+pGax8chPf1fVywqM/cNUhSVy6LHGcjrXZbKasrIy4uDi3Qo4SpNpyATg5O5pjsyJ5bVMb//26kbOeKmZFXgy/PiqVmCDP6TGLxUJFeTkACYmJHHRQ4ribFEBGRASvXbKItdv6eeKbJs58qpgz82P51ZGpRAaOz1VPVVsueVCJiYlyvvwXhyZzRn4sj37ZwP82tvJuaQfXH56Cxe7wSv5XanSJi4uTYwzd3d2M9vRwdYqJewaUWEUFggCfbu8eN6FFo1SQEu5PaoSOeWOETw3XkRKuG/d52BwiDtGzCoteDXTVcdmh4w2IyWqnsWeU+m4j9d0jfLaji/5RGzo/tRyEs9lsstDkRKL/bPLc4L1bLgkiWiwWli9f7nXlFowPxOUmBPPiZYVc8dIWzHYHXUNmrzXO/Pz8iI+PJz4+HrvdTk9PD21tbTR0OgX8riyKomHAzr+/aOC14jZuPDKVM/JjMI2MUFFRwfz58ydVyk3a69hTO0TnvzUqBZcdlMjKvBie/raJVza28WGlgUuXJXD1IcmT8rSSJG9q6jygUp4WIkWvU1NTZcvY0tRAJiM8dmIE6xpsrCnt4P0KA1cuT+SKg5NkMnhqxDCZTJSWlpKWljZJ+jkiQMP/nZrJRUXx/OOTWu77uBatWkGIv5pNTf2kRugI06mnl50aizHo9Xr6+vrIyJhPwPZmtIKNGxdYCQ+PQaUPoceqorFnhPruERq6R9jWMcwn27pwPSnFBvsxb8zKS0HQziEz39T2YB6TKDbb7IyYrdQ3taIPCuG7oSHMtgFMNgdmqwOTzT723YHJaqd72FkxGaTTolHt0n2XvkvXnXRWn3PLJ8Bms1FeXo6/vz86nc4nYsPknPWiuCBeunwxFzy7iQ8qOgjwU3LNIYlEBXkviqhUKuX68cgkE3z7PWrRwsUpZg6LDOStGit/WreD575r5LRkkUuOyvUq/SS5lHZRROliL4P91dx6XDoXFCXwyBf1PP1dM6u2tnPd4SmcuzgOjVLBwMAAVVVVLFq0CKXWedZ1d9ucaBl7e3s5X9NNQYCadxrgP1818ubmnfz6qHmszHNfvSYVwmRlZcmlte6QER3A0xfl8U1tL79ZVUHHoJlLX9wKOF1ap0utY16E07rOi9CRGOo/Lo9tMpkoKSmRi24EZTvhwYEsXryAnp4eurs7sQwOkhkYyKG5kYSFJaNWq7HYHDT17iJ8fY/z++qt7bKk1Bubd04xAMHgHDagVCDg1JKTLL4r/FQKOTXnaq0llRnX1FpDQ4NXirp7E/tsNyMjI3KZZHx8PBs2bPB5DXcptKyYQD761cE89mU9b23ZyTul7Vx+UCJXLE8iYIrqKHfQjOXMQ8MjOGhZIjlGI0vnGXi/tJXVNSYeKxH4xrCd207IoCDRvQSzvFeJ3A4R3MRo4kO0/OOMhVx+UCL//KSWe9bX8MrGVq4uiiTeYZAbTeTuuGm8IoVCIZeOZmaKHD08zNdVLTzxQxd/eG87z31bjx0FobpdN9TBwUEqKyvJycnxygoJgsDh88PJTwxmYMTKzUfPo2HMyjb2jPBdXS/vlO7qvFMpBBJDnQGrpBA1CmM3By1MQeHvvDlKeW6VSkV0dDTR0dGIosjg4CDd3d00NTU5p4qEhRMQHEZBYjALYgIx2+yYrE6Lu71ziHs/qmNZSijRgRp6jBY6B03s7B9l1LbrpmgXwT6NGMjKvBgP8ZNdzS4Oh4PHHnuMkJAQr4uH9hZm1S33hN7eXrZt28aiRYtk6yDN2PKl7trd1E5RFAnTqfjDSfO57KAEHv2ykf9+08Trm3fyy0OTOX9JvFd1x7CrQk2aMa3T6bDb7RyWGsRlx83n9R8aeH5TFxc9v4VlCVpuOjKVvNRot69dIrdjmsj7wthAnrskn29qe7lv/Xb++FEzufGB3BZvpjDJH+lo60vmXQpsnbJsIScvFVlXupOHv2igY8hE54CJK579nqRQPwJEI4fnZWJTeqerJsHhcBaNuAtgDZlsNPaMjCN9rWGYb2pGsYnwfGUNUEOYTo3RYmfIZOOyF7distqdLvUEt9lscyAyBDROuacfG/um/L0EP5VAhF5DWpSew9PDOHFhtMfA6USIosjTTz/Nd999x5dffumz8ulsY69b7ubmZtrb21m8ePG42mrpLugLuSfqobkGzhQKBSnheh48axFXHpzEQ5/Vcd/Htbz0Yyu/OjKVU3Oip42+ulaoScIHfn5+5ObmIggCvzhmERcdZufF75t5dkMzF/9vG4fFVXNRXggZSTGEh4fLKTEpVeStoGKiapC/HaqjzpHMY181cfELWzk2K4Ibj3B2hXm5zCQIgsBp+fGckB3LCY/+gM3hYOeQjU07TdhFeL7S2QYb6q8idSw3nBLuDF7Ni9ARH6KdpHFmd4geA2qBWhU58UHkxDtrtYeGhqioqGDhoiUM2FQ0dBup73GS/t3SDud7LYroNEr8NUoconN9i83OsNnOwKiVEYsdq326plcnFDjrw6OCNCyMDuSIjAiOzAgjULtnRBRFkeeff56PP/6Yt99+e78jNuxFcjscDrZt24bdbmfJkiWTClOUSiU2m82nc4tkuacLnC2KC+TZS/LZUN/Lw5/Vc8e723j++2Z+c/Q8Dp8f7tHLkJRYzBbntJLo6OhJOuI6jZLrjkjlvCXxPPFNE69vauPHzj5Oz7RwRHQdoQH+REZGgui8CU03p1oURbZv344oiuTn5VGoUHBKTiwv/dDCMxua+Xx7NwCtfaN0D1sI108fvHIHjUpBqF5NiEbk+mwFi3Jy6RyyUtXaQ1VrD3WGYTqMw3zWNUy/aZd35Opap4b7kxqhZ9BkQ6tWUt42OBbAsjst7lgwS/r/geERdnZ2ERQazvqvd+6yzlY7o1YHdtFZn1DSMoAvRYFqhUCAVkVMgJrUYAXzAqwsChMQ7FZSU1NJSEjYrfdoKrzyyiu8++67rF271qcGoL0JYZqI9m7ah/Gzs6Q2xKioqEmSQhIkRU5fqnxMJhOVlZXk5+d7PRzAIYp8VNXFI1/U09w7yuKkYH57TBr5Hs7MuX/7khOSldx2YpaTpNOguXeUf39RzweVBkJ1aq5aFsthcQpWb23npSozr5yTSGZSrNvXabfbKS8vl2daTXwt3cMWHv2ynre2tMs/C/RTySmilDAdKWMpouQw/2kLME599DtCNA5evHq52ypAk8lEd3c3jTs7aeo1Myzo6LNr2DnsTBs19Y74XJqrEJw3CBGn9yGKkwNZEyHgtL5heg1JoVpy4oI4JD2M3Pgg/FST922xWNiyZQvh4eGYTCaGh4cJDg6WK+V8nUI7EW+88QYvvfQS69atm6mqtJm984xh1i334OAg5eXlZGZmepyoCe7HA00HQRAYHR2VO3K8uTsrBIGTFkVxbFYEq7e285+vGrnw+S0cnRnBzUfPIz1y14fV39+PUhAJDgv3itjgnLr5wFmLuPzgRB78tI4HvmjmjVAtB6WGAe0olKpJ+fTg4GCsVitlZWXExsZ6zJdHBGj4w8kZvLWlndNzo1kUG0hjzygNPSNsbOxnbdmujjYBiA3WkhLuL7vVqeFO8kcHaqivq8PusBMSHOLxYtdqtSQkJJCQkMDBdjuGrm52Grro6RtCk6LHPyCaQbTc+3EdKoXAIWlhDJts9I5YMQyb6Rm2MGSyjbPCDnFXDMMVSgXoNSoiAjTMi9BRmBRMUXIomdF65ygmL2E2m2VDIaUnHQ4HAwMDdHd309DQgFqtluv2p1PDmYg1a9bw/PPP8/777+935aYTMauWu7W1lfr6evLy8qZ9I6qqqoiNjSU0NNSr9aWKs507d2IwGLBarURERBAVFUVAQID3Y1ctdl76sYVnv2tm1GpnZV4sNxyRgjDaT3NzM7/+0sRpuTHcdWLG9Iu52eO3db089FkdOzqdsronL4pkYWwQyaFagpVmVOYBjIMDWK1WEhMTSUlJmdKy2B0iOX/7khuPSOH6sfO362tp6nWeXxu6R2gYs64N3SMYXTToNEqIC1DRbRIJ1KooTAwe5z5PdKelgJavVtoVAs467VC9mqRQfzKiAlgYG0h+YhDxIf5eSTJNB1diTxW5Hh0dpbu7m66uLiwWC+Hh4URERIxrdHGHdevW8cgjj/D+++9PmSbcDcyK5Z41cu/cuZPGxkby8vK8Okfv2LFDfpOng7uKM5vNRnd3NwaDAaPRSFhYGFFRUYSEhHhF9L4RC09908Srm9pAFDk+1Y/fn17Ayqe2cExWBH8+JXP6F+0BDlHk/97fwVtb2gnWqhgw7ZoWohQgwl8gJdyfaJ2CYIWZpFA/spMiyUqOnaSA4hBFsv/6JTcckcINE8jtCaIo0j1soa7LyIbyGrotSvpsajbU96EQBKKD/NCqnPplggCI4EDEahcZtdgZtTq/zDZxavlmnBVgkQFqUiMDSApSEs4wpxySR2SQ/4yfe11hNpvZunWrV0IVrpAaXbq6uhgYGJAbXSIiIsbVXXz00Ufcf//9vP/++9MWLO0GflpueVRUlLMwYYYEG2DqqZoqlWpc+6RrZ1JQUJC8H09WMVSn4XfHpbEsdJQ3txn5oM7E149vAkSGTbbd1vgG51GgKDmUt7a089pViwnVqWnqHaW8yUBpfTtmdRDN/RZK20cw2RyAFTYM46dsIEavIDnMn/kxwcyPCSYlTGqPdNA3YhmzsLtSRaNjQarRsaosyQKPWGw0t7aj0vqj0ekJsTpFBFVKAbPVQfuAaVrLrBQgVKcmMkBDXIiWeRE6MqICmB+pJynMH72fCpvNRk9PD83NzQwN9TpLPM3DOBx+e3zW9YTdJTZ4bnQpKSmR1XWVSiUvvvgiH3zwwWwQe9Ywa5bb4XBgtU43QnYXGhoa8PPzIy4uzv1GfBiXO/FxAwMDGAwGent78ff3JyoqatKdWaptDw8PJykpiRqDkX99Xs+XNU4N+CCtipRwnfMMG6YjOdyf5LHves3098gPKzv57eoq1l63lPRIPe3t7bS2tpKXlyenURyiiGHITEP3CE29ozT2jFDXZaSha5j2IevufxhjEHA2sEhVWRONsICzPTI2SEtimJb0yACSw/1JCPEnIVRLhF7j1fve0tJCV1cXubm5Mll6e3vls25kZOSMRZilKrfMzEyvj3TewmKxcN999/H6668TEBDAsccey8MPPzwbHshPy3L7iqkst1TqJ1lPX95cV5VSSRfMYDDId+SoqCgCAwPZsWMHqampcjNBRnQAj1+QS1nbIFtbBsbOsqOTAlcAUYGacYSXbgIJof5y079coWZ30NjYSF9fHwUFBeOOLApBICZIS0yQloMnaBRY7A7qDMOUNnTyl09bSQ0SSA/3Q6Hyw6ZQMWJxMGiyMjBqo2/UitE8+b3UaZREB/kRHehHdJAfUYF+xAb7kRji3GtMsJ9HSWBv0djYyMDAgCzo4KoQOzIyQnd3N5WVTr066RgWFBS0W4SZTWIDFBcX8/HHH7NhwwYiIiIoKyub1aPFTGPWLLcoih7lit2hra0Nq9VKSkrKpHV2l9jTwWQy0dzcTGtrK/7+/sTExMhtn1M9z6jVTvOYZZUsbFPPCI29o+M6mRSCs6w0OUyHUiHwVU0Pp8zXo1dBYGiYHKya6EKbJgS0TFbn7zwVwAhAsJ9AZKCG+BA9sSH+RAVqCPGD4a42Fi9IY358xJTihHsKURSpr69nZGRk3KxvT7BarWP1490MDQ0RHBxMZGTklEcnV0jEnq4GfnexceNGbr75Zt57771JtQ2zgJ9WQM1Xcnd0dGA0GklLSxu3hpQr91UO2BtIAwJyc3NRqVR0dXXR1dU1pkG+K03lyw2lf9RK0xjpJcI7o9dGTLZdb6daKeCvVuKnUqBVK9GqFWjH/Xvsu4ffhenVRAf6ERXkR2SABotpVN4/QGBgID09PeTl5c16t5IkVGGz2ViwYIHPN2ApVSW5735+frL77k5SebaJvWXLFm644QbefffdScZmlnBgk7urq2us9S9jt8/XvuytubmZ7u5ucnNzJ3WjSW2fXV1dDA4OEhwcLAfkducmIw3804dFkZwYj1al3CMZpOnQ2dkp645brVY5c+DrjcobSBV1giB4LWU9HSRJ5e7ubux2uyykERAQILejzhaxy8rKuOaaa1i9ejXz58+f8fU94KdFbnBGMb1Fb28vnZ2dZGVlzSqxHQ4H1dXV2O12FixYMC1ZXYULe3t70ev1ckDOmxSfdDG6nudnEwaDQU5B+vn5yakeg8HA4OAgQUFBREZGzkilljSyyM/Pj/T09Fk5j1qtVjknPTQ0JB/dJFnlmURVVRVXXnklb775JllZWTO69jQ4sMk9MDBAS0sLCxYsmDViS/3jwcHBbss7p4MoigwNDdHV1UV3dzcajWZK93F4eJiKiopZszIT0dbWRnt7O3l5eW57413nm/X09MgKqxN15LyBLwqlM4HR0VFKSkpISEhgZGSEvr4+/P39Zau+p40bO3bs4LLLLuPVV18lOzt7hnbtNX565LZYLF6LJA4NDVFZWcnChQt3ayDddDCZTJSVlZGYmDhjErQjIyPyOVcURSIjI4mKikKn08mjdnJycvZKmWJjYyP9/f3k5OR4bdEkiabubueYXKnCb7r9SjXwviqU7i5GR0cpLS1lwYIFBAc7ewBEURz3/gNy8YkvFYoAtbW1XHzxxbz88svk5eXNymuYBgcuuUVRxGq1snPnTrq6urDZbDJRZoIYUpthVlbWrKRMwPlau7q65Ao5aUpGeLjnrrOZgCiK1NbWYjabWbhw4W4HHqX9d3V1YTKZZO3xiRV+drtdbgLaHYVSX+GO2J72L53TjUYjoaGhREZGTjvQsKmpifPPP5/nnnuOxYsXz8ZL8AYHHrk9Bc6kc5bBYGB0dFS2KIGBgT4TRRoQsLcsaHNzM11dXcTGxtLT08Pw8DChoaFyKexMRv2lNlqVSkVGRsaM3UQmlmRK5/Tg4GDKy8uJj4/fKwL8km7cwoULCQoK8vpxDoeDvr4+OUir0+nkRhFX9721tZVzzz2XJ598kmXLls3GS/AWPz1yS+L/bhf2MiJut9tlog8PD/tUMy4NCMjNzZ31ZnopHWSxWMZZUOlCMxgM9Pf3ExgYSFRU1B4HtCQ55aCgIFJSUmbNO5DO6R0dHezcuROdTkdiYqLHyZ8zBUmGa9GiRT4ReyKkwiXp+AHOtKter+cPf/gD//73vznssMNmatu7iwOH3Lub6pJqxg0GAwMDAwQHBxMdHT3J9XIdELBw4cJZq2l23VdlZSVarXbKqLGkB2YwGOSAlhR59+XmY7PZKC0tJTo6eq+4xlK31bx589DpdJPOuTN1fJIgETs7O3vGJ2daLBZeeuklHn30UQBOPfVU7rzzTq9bemcJBwa5Z6riTEpRGQwG+vr6ZIsYEhLCtm3b0Ov1pKWlzXq5oES0yMhIkpKSfHqsVArb3d2NQqGQI9dT9RhbLBZKSkqmnL89k5ioUDpxL5JXNdU53RcYjUbKyspmhdjgrKc466yzuOeeezj88MP58ssvOeyww/Z1b/ZPj9w2m21cvfhsVZxJFrG9vV12HZOTk4mMjJxVuVkpAp+cnEx0dPQeryVZRJvNNs4iSkSRgkve6KTPBKQzrzepPE/ndF+OH7NN7N7eXs4880z+9Kc/ccopp8z4+hIkKbH4+HjWrVvnzUN+2uR214M9k5Byyunp6Wi1Wjo7O+VcdFRU1IzkQt0932w0LbgGFEdGRggPDycwMJCGhgYWLly4V3LmEtF258w7MZ8+XTmp6/PNFrH7+/s566yzuP3221m5cuWMr++Khx56iE2bNjE4OHhgk9tms81qxRkg55Szs7Mn1VGPjIxgMBjo6upCEAR54MCetBz29zvH1bp7vpmGq6KNRqORUzzeNljsDqTUobf65dPBXT7dtUFHIvZMPd9EDA4OcvbZZ3PTTTdxzjnnzPj6rmhtbeWyyy7jrrvu4qGHHjqwyW2xWGaV2Dt37qStrY3c3Nxpo7eS62swGHA4HLI18eW8JTWb5OXl7RXVy56eHmpqauTnk+IMUimslOLxdVqLJ0jTTXJzc2flHCqd06UGnYCAAPr7+8nLy5sViz08PMw555zDL3/5Sy688MIZX38izj77bO644w6GhoZ44IEH9im5Z7Wf+8YbbyQ6OpqVK1eSkeG7BtlUkGaMGY1GCgsLvbJiWq2WxMREEhMT5Yusuroai8Xilf5aS0sLBoOBwsLCGSPTVOjs7KSpqYmCggL5xhUaGkpoaKisGiL1pqtUKrnwZ3dTVH19fezYsUOebjIbcB15NDAwQFlZGUFBQXJab6bq3sHptZ1//vlcccUVe4XY69atIyoqisWLF/Pll1/O+vNNh1m13F1dXbz77rusWbMGg8HAiSeeyMqVK3erLdAVrgMC5s+fv8cegav+2sjIyKQuKulGMjIyQnZ29qy0n05Ea2srnZ2dbrvW3GF0dFT2SnwpJZXQ09NDbW3tXvNIhoeH5amuer1+t87pU2F0dJQLLriAs88+m2uuuWYWXsFk3HHHHbz88suoVCpMJhODg4OceeaZvPLKK9M99Kfnlruir6+PtWvXsmbNGpqbmznuuOM444wzyMnJ8YksFouFsrIytwMCZgITu6hCQkIYGRlBr9fPWEvjdNidOnFXTExRhYeHExUV5VHxROoky8/P3yuTM6Qz/VSu/3Tn9KlgNpu56KKLOOWUU7j++uv3iXrKl19+uc/d8r1GbldIUcQ1a9ZQU1PDMcccw8qVKyksLJyS6FJqJi0tba8UHVgsFrZu3YogCNjtdq+EFvcEnqrc9gRSb7rBYGBoaGhSzXVHRwctLS3k5+fvlaOGN8SeiInn9Kny6RaLhcsuu4wjjzySm2++eZ/JIv1sye2KkZERPvjgA1avXk1FRQVHHnkkK1asYNmyZeMI1N/fz7Zt22YtVTIR0pQUqVjEVWixp6cHvV5PdHQ04eHhM5JLn6068YnP4Vr4I+nWFRQU7BVXfHeIPRGe8ulhYWGIoshVV11FUVERt912209J7+zAJLcrTCYTH3/8MW+99RZbt27l0EMPZeXKldTU1BAeHs6JJ564Vy5CaUa1J6lc12BWd3c3fn5+ci59d6zf3qoTd0VzczMdHR2EhITQ19c3a/UAEiRi5+XlodPpZmRN13P6XXfdRW1tLZmZmTz55JN7pXpvBnHgk9sVFouFTz/9lD/+8Y90dXVx1FFHcdZZZ3H44YfPqvsopYJ88RCkMtKuri5ZUdXbqPXerhOHXQqlrvEO6Ywr1QNIkfeZiJpLc79nktiusNvt3HjjjQQGBpKamsqHH37Iu+++u69LSn3Bz4vcAG+//TaffPIJDz74IN999x2rVq3im2++YfHixaxcuZKjjjpqRjuTurq6qK+vJzc3d7cv6olR68jISKKjo92uJ9WJz0T5qjfwVqHUbDbLr2F3xzRJGBwclPPms0Fsh8PBzTffTGhoKPfff/+sZDJaWlq49NJL6ejoQKFQcM0113DTTTfN5FP8/Mgt7W2iWMA333zD6tWr+eKLL8jJyWHlypUce+yxe2RlpPZQTxJFuwOLxYLBYJBJ4ipAIWmr7a068d1VKN2TMU17g9i33XYbKpWKf/3rX7OWomxvb6e9vZ3CwkKGhoZYvHgx77zzDgsXLpypp/j5kXs62O12fvjhB1avXs2nn35KRkYGZ5xxBscdd5zXZYySNRseHiY7O3vWSjpd68WNRiMWi4X58+cTFxc362dsSaFUoVDsUbBuYsvtVNmDgYEBtm3bRl5e3qwUxDgcDu6++25MJhOPP/74Xqk9kLBixQpuvPFGjjvuuJlaco7cU8HhcLBlyxbeeust1q9fT2pqKqeffjonn3yyx8YHh8MhX/R7K4c9MDBAZWUl8fHxDA0NMTQ05PPQQl8gKZRqtdoZbYGdakzTyMjIrBJbFEX+7//+j+7ubp5++ulZ79d3RWNjI4cffrgcAJ0hzJHbWzgcDsrLy3nrrbf48MMPiYmJ4fTTT+fUU0+VO7jsdjtlZWWEhITstQi1a524dNG7E6DYE410V0gKpVKgabbgOqaps7OT0dFRkpOTiY+Pn/HshiiK3HvvvTQ1NfHCCy/sVWIPDw9zxBFHcNddd3HmmWfO5NJz5N4diKLItm3bWLVqFevWrSMkJIRjjjmGjz76iMcee2xvTZSQ68SnqgITRXFcY0hAQICcS/f1IpZuXmFhYXtFoRR2dctlZWUxODhIV1cXdrt9XIPOntxERVHkwQcfpKqqildeeWVWe/Unwmq1cuqpp3LCCSdwyy23zPTyc+TeU4iiyOeff86ll15KamoqGo2G008/ndNPP53o6OhZs96+1olLex0aGpJz6Z6mk7qD3W6npKRkr6bXJGLn5+ePs9ZWq3VGxjSJosijjz5KcXExr7/++l6ppnN97ssuu4ywsDD+9a9/zcZTzJF7JvCPf/yDY489loKCAhobG1m9ejVvv/02SqWS0047jZUrV85YkEsUxXE55T1xIV2LZlQqlZxLn+gFWK1WSktL95pCKXgm9kRMHNMUEhIiV5dNdQQRRZEnn3ySL7/8klWrVu2V+ndXfPvttxx22GHj6gLuueceTj755Jl6ijlyzxZEUaStrU0musVi4bTTTmPFihUkJyfvFtFno05cwujoqJxikwQoIiMjUSqVezVvDuPbRH05X08c0xQQECD3pru626Io8txzz/HBBx/w9ttv75UKxX2AOXLvDYiiSGdnJ2vWrGHNmjUMDg5yyimnsHLlSq/nYe2NOnEJZrMZg8FAR0cHQ0NDREVFkZqauleqs3aX2BMxsZxXo9EQEBCAVqvliy++YPXq1axdu3bWesz3A8yRe1+gq6uLd955hzVr1tDV1cVJJ53EihUrPBaCSKN2goOD91oUXlIonTdvHjabDYPBgNlsJjw8nOjo6N2qLJsOkrTVnhLbHUZGRiguLubWW2+ls7OTW265hQsuuGBWI/77GHPk3tfo6+uTxSdaWlo4/vjjOeOMM2QBB6vVKvea761AlieFUpvNRk9PD52dnRiNRrmneybG+ErEdlWImWmsWbOGp59+mueff54vvvgCrVbLRRddNCvPtR9gjtz7EwYGBuSe9NraWg455BA2bNjA888/T2Zm5l7Zg7cKpe4EKKKioqado+UOvb291NTUkJ+fP2vEfu+993j00Ufl1OVsYP369dx0003Y7Xauvvpqfv/738/K83iJOXLvr6iqquLUU08lMzOT1tZWjjrqKFasWMHSpUv3O4VST8McvMmlS1JMs0ns9evX889//pP333/fbbvtTMBut5ORkcEnn3xCQkICRUVFvPbaazNZK+4rfnoCiT8XVFRU8Prrr7N06VJGR0f5+OOPeeGFF7jppps47LDDWLFiBcuXL5+xogvXum1fGzIUCgVhYWGyuIE03qi+vh6dTifn0ifuVSJ2QUHBrKWiPvvsM+677z4++OCDWSM2wMaNG0lPT5fnip9//vm8++67+5Lcs4I5yz2LMJvNfPbZZ6xatYoff/yRgw8+mJUrV3LYYYftdhGGFKGe6bptdxFrKcU2NDQ068T++uuvufvuu3n//fdnPY23atUq1q9fzzPPPAPAyy+/zI8//shjjz02q887BWbFcu+9Vhov8cADDyAIgjyR8acMPz8/Tj75ZJ577jlKSko477zzWLduHcuXL+e6667jo48+wmw2e71eT0+PHMia6bSQIAgEBgaSlpbGsmXLyMjIwGq1smnTJnket6eJrXuK7777jjvvvJO1a9futb72ifgJSTJ5jf3KLW9paeGTTz7xeaDeTwFqtZrjjjuO4447DpvNxrfffsuqVav4wx/+QG5uLitXruSYY47xSFpJoXQ2racr9Ho9o6OjqFQqioqK6Ovro7KyUh7mEBUVNSM92j/++CO33XYba9euJS4ubgZ2Pj0SEhJoaWmR/7+1tXWvPffexH7llp999tn84Q9/YMWKFWzatImIiIi9+fT7BHa7ne+//17uSc/KymLlypUcf/zxciFKe3s7ra2te02hFJz5/YaGhkmNLhaLRVZpkYY5REdH71ZTyJYtW7jhhht4991391oDDzjThBkZGXz22WfEx8dTVFTEq6++yqJFi/baHibgwA6orV27lvj4ePLy8vb1VvYqlEolhx56KIceeigOh4PNmzfz1ltv8Y9//IN58+YRGRkpd0PtrS4oT8QG58SQ+Ph44uPjZZWWuro6uSlkKn10V5SWlnL99dezZs2avUpsAJVKxWOPPcYJJ5yA3W7nyiuv3JfEnjXsVct97LHH0tHRMennf//737nnnnv4+OOP5cqun4vl9gSHw8Fdd93F6tWrCQoKknvSTznllBmfKuoKidgFBQU+eQnu9NElAYqJufTKykquuuoq3nrrrb1WE7Cf46dvuT/99FO3Py8vL5eH64HzDFRYWMjGjRt/ahK1Mwar1YrNZqO8vByNRkNVVRWrVq1i5cqVhIWFsWLFCk499dQZvQG6nut9df9dVV8dDgd9fX10dnayY8cOWY4pMDCQxsZGrrrqKl577bU5Ys8y9qszt4Q5y+0ZUrfZqlWrWLt2LTqdjhUrVnDaaaftUU+6wWCQxSRm8lwvyTHt2LGDq666CovFws0338y111476+OPf0L4eaTCZhK/+93vyMrKIjc3lzPOOIP+/v59vaU9hiAIZGRkcOedd/L999/zzDPPYLFYuOSSSzjppJN4/PHH2blzp9t0jyfMFrGl/YaEhMgNLPfddx/9/f3cdtttM/o8c5iM/dJyzxQ+/vhjjj76aFQqFbfffjsA999//z7e1exAFEVaW1vlnnSbzcZpp53G6aefPmVPemdnJ83NzbMaiW9paeG8887jySefZNmyZbPyHK743e9+x3vvvYdGoyEtLY3nn39+xmrU//CHPxARESHrlt91111ER0fz61//ek+Wnast3xO8/fbbrFq1iv/973/7eiuzDlEU6ejokHvSh4eHOeWUU1ixYsW4nnSJ2AUFBbMWid+5cyfnnHMOjz76KIceeuisPMdEzOZNvbGxkTPPPJMtW7bgcDiYP38+Gzdu3FPt+Z9+QG1f4rnnnuO8887b19vYKxAEgdjYWG644QZuuOEGurq6ePvtt7n99tvp6enhpJNOQqVSYbPZuPXWW2eN2B0dHZx33nk8/PDDe43YAMcff7z874MOOohVq1bN2NopKSmEh4ezdetWOjs7KSgo2CtDJXYHP3nLPVV6bcWKFfK/N23axJo1aw7IMkNf0Nvby1133cW7775LfHw8xxxzDGecccaU44V2B11dXZx11lnce++9Myne7zNOO+00zjvvPC6++OIZW/ONN95gw4YNdHR0cNlll82EltqcW747ePHFF3niiSf47LPPZmWkzU8NAwMDXH755bz00ks4HA7WrVvH6tWrqaur47jjjmPlypXk5+fvEdF7eno466yz+POf/zyTIoLjsC9v6haLhZycHKxWKzU1NTPR1jtHbl+xfv16brnlFr766isiIyP39Xb2awwPD/PBBx+watUqtm3bxtFHH82KFSsoKiry6eLt7+/nzDPP5I477pBJti8w2zf1a6+9lpCQEO67776ZWG523ElRFKf6+kkjLS1NTEhIEPPy8sS8vDzxl7/85W6v9eGHH4oZGRliWlqaeO+9987gLvc/jIyMiG+//bZ40UUXiYsWLRKvu+46cf369eLAwIBoNBo9frW3t4vLly8X33rrrX26/w8//FBcsGCBaDAYZmV9u90u5uXlidXV1TO15HQ83K2vA5rcMwWbzSbOmzdPrKurE81ms5ibmytWVlbu623tFZhMJnHdunXi5ZdfLi5atEi8+uqrxXXr1on9/f3jiN3Z2Skefvjh4quvvrqvtzyjN/WJqKysFFNTU8VbbrllxtYUZ4ncB7RbPlP4/vvv+fOf/8xHH30EwL333gvAHXfcsS+3tddhtVplqeFvv/2WoqIiVq5cybJly7jooou47LLLuOyyy/b1Nn+KmKtQ21doa2sjMTFR/v+EhATa2tr24Y72DdRqNccffzxPPvkkpaWlXH755XzyySfk5ORw/PHHzxF7P8PPJs+9J3Dn3fzcU2oqlYojjzySI488koceemivj/iZw/SYI7cX+Lkod+wuZksJdQ57hjm33AsUFRVRU1NDQ0MDFouF119/ndNPP31fb+tniQNJY2+2MUduL+Cq3LFgwQLOPffcGVHuaGlp4aijjmLBggUsWrSIRx55ZAZ2e+DiQNbYmw3MRcv3Idrb22lvb6ewsJChoSEWL17MO++8c8DpZ88UDmCNvblo+YGG2NhYCgsLAQgMDGTBggU/yyi8N/i5auztCeYCavsJGhsb2bp1617pd95f4Y3G3hy8x5xbvh9geHiYI444grvuuoszzzxzX29nv0N5eTnHHHOMXCMuZSsOII29ucaRAxFWq5VTTz2VE044gVtuuWVfb+cngQNQY2/uzH2gQRRFrrrqKhYsWDBH7DnMOOYs9z7Et99+y2GHHUZOTo7cP33PPffMSA+03W5nyZIlxMfHs27duj1ebw6zijmZpQMNhx56qE8qpb7gkUceYcGCBQwODs7K+nPY//Gzd8uLi4vJzc3FZDJhNBpZtGgRFRUV+3pbe4TW1lbef/99rr766n29lTnsQ0znlv8sIAjC3wAt4A+0iqJ47z7e0h5BEIRVwL1AIHCrKIqn7uMtzWEfYM4td+IvQDFgAvZIgHpfQxCEUwGDKIqbBUE4ch9vZw77ED97t3wMYUAATkun3cd72VMcApwuCEIj8DpwtCAIr+zbLc1hX2DOLQcEQViLkwipQKwoijfu4y3NCMYs95xb/jPFz94tFwThUsAmiuKrgiAogQ2CIBwtiuLn+3pvc5jDnmDOcs9hDgco5s7cc5jDAYo5cs9hDgco5sg9hzkcoJgj9xzmcIBijtxzmMMBijlyz2EOByjmyD2HORygmCP3HOZwgOL/AaJgSf7ygNsrAAAAAElFTkSuQmCC\n",
|
|
"text/plain": [
|
|
"<Figure size 432x288 with 1 Axes>"
|
|
]
|
|
},
|
|
"metadata": {
|
|
"needs_background": "light"
|
|
},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"%matplotlib inline\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"from mpl_toolkits import mplot3d\n",
|
|
"import torch\n",
|
|
"import torch.nn as nn\n",
|
|
"\n",
|
|
"x = torch.linspace(-5,5,10)\n",
|
|
"y = torch.linspace(-5,5,10)\n",
|
|
"fig = plt.figure()\n",
|
|
"ax = fig.add_subplot(111, projection='3d')\n",
|
|
"plt.xlabel(\"x\")\n",
|
|
"plt.ylabel(\"y\")\n",
|
|
"X, Y = torch.meshgrid(x, y)\n",
|
|
"m = torch.stack([X, Y])\n",
|
|
"z = nn.functional.softmax(m, dim=0)\n",
|
|
"ax.plot_wireframe(x, y, z[0])\n",
|
|
"fname = 'softmax3d.png'\n",
|
|
"plt.savefig(fname)\n",
|
|
"fname"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"[[file:# Out[27]:\n",
|
|
"\n",
|
|
" 'softmax3d.png'\n",
|
|
"\n",
|
|
"]]\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Wagi\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Podobnie jak funkcja sigmoidalna, softmax nie ma żadnych wyuczalnych wag.\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Zastosowania\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Podstawowym zastosowaniem funkcji softmax jest klasyfikacja\n",
|
|
"wieloklasowa, również w wypadku zadań przetwarzania sekwencji, które\n",
|
|
"mogą być interpretowane jako klasyfikacja wieloklasowa:\n",
|
|
"\n",
|
|
"- przewidywanie kolejnego słowa w modelowaniu języka (klasą jest słowo, zbiór klas to słownik, np. klasą początku tekstu *Dzisiaj rano kupiłem w piekarni* może być *bułki*)\n",
|
|
"- przypisywanie etykiet (np. części mowy) słowom.\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### LogSoftmax\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Ze względów obliczeniowych często korzysta się z funkcji **LogSoftmax**\n",
|
|
"która zwraca logarytmy pradopodobieństw (*logproby*).\n",
|
|
"\n",
|
|
"$$log s(z_i) = log \\frac{e^{z_i}}{\\Sigma_{j=1}^k e^{z_j}}$$\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### PyTorch\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 20,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"tensor([-1.1971, -1.3971, -0.7971])"
|
|
]
|
|
},
|
|
"execution_count": 20,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import torch.nn as nn\n",
|
|
"\n",
|
|
"s = nn.LogSoftmax(dim=0)\n",
|
|
"s(torch.tensor([0.0, -0.2, 0.4]))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Niektóre funkcje kosztu (np. `NLLLoss`) zaimplementowane w PyTorchu\n",
|
|
"operują właśnie na logarytmach prawdopobieństw.\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Przykład: klasyfikacja wieloklasowa\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Na przykładzie rozpoznawania dyscypliny sportu: git://gonito.net/sport-text-classification.git\n",
|
|
"\n",
|
|
"Wczytujemy zbiór uczący:\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 21,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"{'zimowe': 0,\n",
|
|
" 'moto': 1,\n",
|
|
" 'tenis': 2,\n",
|
|
" 'pilka-reczna': 3,\n",
|
|
" 'sporty-walki': 4,\n",
|
|
" 'koszykowka': 5,\n",
|
|
" 'siatkowka': 6,\n",
|
|
" 'pilka-nozna': 7}"
|
|
]
|
|
},
|
|
"execution_count": 21,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import gzip\n",
|
|
"from pytorch_regression.analyzer import vectorize_text, vector_length\n",
|
|
"\n",
|
|
"texts = []\n",
|
|
"labels = []\n",
|
|
"labels_dic = {}\n",
|
|
"labels_revdic = {}\n",
|
|
"c = 0\n",
|
|
"\n",
|
|
"with gzip.open('sport-text-classification/train/train.tsv.gz', 'rt') as fh:\n",
|
|
" for line in fh:\n",
|
|
" line = line.rstrip('\\n')\n",
|
|
" line = line.replace('\\\\\\t', ' ')\n",
|
|
" label, text = line.split('\\t')\n",
|
|
" texts.append(text)\n",
|
|
" if label not in labels_dic:\n",
|
|
" labels_dic[label] =c\n",
|
|
" labels_revdic[c] = label\n",
|
|
" c += 1\n",
|
|
" labels.append(labels_dic[label])\n",
|
|
"nb_of_labels = len(labels_dic)\n",
|
|
"labels_dic"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Przygotowujemy model:\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 22,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import torch.nn as nn\n",
|
|
"from torch import optim\n",
|
|
"\n",
|
|
"model = nn.Sequential(\n",
|
|
" nn.Linear(vector_length, nb_of_labels),\n",
|
|
" nn.LogSoftmax()\n",
|
|
" )\n",
|
|
"\n",
|
|
"optimizer = optim.Adam(model.parameters())"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Funkcja kosztu to log-loss.\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 23,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"tensor(2.3026)"
|
|
]
|
|
},
|
|
"execution_count": 23,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import torch\n",
|
|
"import torch.nn.functional as F\n",
|
|
"\n",
|
|
"loss_fn = torch.nn.NLLLoss()\n",
|
|
"\n",
|
|
"expected_class_id = torch.tensor([2])\n",
|
|
"loss_fn(torch.log(\n",
|
|
" torch.tensor([[0.3, 0.5, 0.1, 0.0, 0.1]])),\n",
|
|
" expected_class_id)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Pętla ucząca:\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 24,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"/usr/lib/python3.9/site-packages/torch/nn/modules/container.py:119: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.\n",
|
|
" input = module(input)\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"0.04162006452679634 2.081003189086914 0 0 tensor([[0.1248, 0.1249, 0.1252, 0.1248, 0.1248, 0.1253, 0.1251, 0.1251]],\n",
|
|
" grad_fn=<ExpBackward>) MŚ w hokeju: [...]\n",
|
|
"1.3499293327331543 1.4718239307403564 4950 5 tensor([[0.1089, 0.0784, 0.1235, 0.1255, 0.0868, 0.2295, 0.1285, 0.1189]],\n",
|
|
" grad_fn=<ExpBackward>) Liga Letnia NBA: [...].\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"iteration = 0\n",
|
|
"step = 50\n",
|
|
"closs = torch.tensor(0.0, dtype=torch.float, requires_grad=False)\n",
|
|
"\n",
|
|
"for t, y_exp in zip(texts, labels):\n",
|
|
" x = vectorize_text(t).float().unsqueeze(dim=0)\n",
|
|
"\n",
|
|
" optimizer.zero_grad()\n",
|
|
"\n",
|
|
" y_logprobs = model(x)\n",
|
|
"\n",
|
|
" loss = loss_fn(y_logprobs, torch.tensor([y_exp]))\n",
|
|
"\n",
|
|
" loss.backward()\n",
|
|
"\n",
|
|
" with torch.no_grad():\n",
|
|
" closs += loss\n",
|
|
"\n",
|
|
" optimizer.step()\n",
|
|
"\n",
|
|
" if iteration % 50 == 0:\n",
|
|
" print((closs / step).item(), loss.item(), iteration, y_exp, torch.exp(y_logprobs), t)\n",
|
|
" closs = torch.tensor(0.0, dtype=torch.float, requires_grad=False)\n",
|
|
" iteration += 1\n",
|
|
"\n",
|
|
" if iteration == 5000:\n",
|
|
" break"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Model jest tak prosty, że jego wagi są interpretowalne.\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 25,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"tensor([[0.0926, 0.1013, 0.0889, 0.0887, 0.0975, 0.3423, 0.0921, 0.0965]])"
|
|
]
|
|
},
|
|
"execution_count": 25,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"with torch.no_grad():\n",
|
|
" x = vectorize_text('NBA').float().unsqueeze(dim=0)\n",
|
|
" y_prob = model(x)\n",
|
|
"torch.exp(y_prob)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 26,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"tensor([-0.6260, -0.6487, -0.6373, -0.6369, -0.5877, 0.6338, -0.6376, -0.6468],\n",
|
|
" grad_fn=<SelectBackward>)"
|
|
]
|
|
},
|
|
"execution_count": 26,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"with torch.no_grad():\n",
|
|
" x = vectorize_text('NBA').float().unsqueeze(dim=0)\n",
|
|
" ix = torch.argmax(x).item()\n",
|
|
"model[0].weight[:,ix]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Możemy nawet zaprezentować wykres przedstawiający rozmieszczenie słów względem dwóch osi odnoszących się do poszczególnych wybranych dyscyplin.\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 27,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD5CAYAAAAqaDI/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAAbi0lEQVR4nO3df3QV1b338feXAMqDIlBQwo824uJHMQFiAw8PVECFi8XWgOBTXdUbrJhaa++9/YFF0a6qvS1Kr1rXY3FxVQRtkRY0UuSWq6hFqbaJooD0piCmlYRCAANiQEn4Pn9kOA3x5BczSQ6Zz2uts87Mnr1n730O5uPMOWfG3B0REYmvDm09ABERaVsKAhGRmFMQiIjEnIJARCTmFAQiIjGnIBARibmOUezEzC4Ffg6kAY+4+/w6278G/CBYPQR8093fbmy/vXr18oyMjCiGKCISC2+88cZed+/dnDahg8DM0oCHgMnATqDQzFa5+9Za1d4DJrj7B2b2JWAR8L8b23dGRgZFRUVhhygiEhtm9tfmtoni1NBoYLu773D3T4CngNzaFdz9D+7+QbD6OtA/gn5FRCQCUQRBP+D9Wus7g7L6XA/8VwT9NqqkpITMzMyTaltQUMDWrVsbrygicoqLIggsSVnS61aY2UXUBMEPkm0P6uSbWZGZFZWXl0cwvOarqqpSEIhIbEQRBDuBAbXW+wNldSuZ2XDgESDX3ffVtzN3X+TuOe6e07t3sz7vaNCOHTvIzs6msLCQMWPGMHz4cKZPn84HH9ScsZo4cSK33XYbEyZM4J577mHVqlXMmTOHkSNH8u6770Y2DhGRVBNFEBQCg8zsXDPrDFwFrKpdwcw+CzwNXOvuf4mgz2YpLi5mxowZLF68mOuvv5577rmHTZs2kZWVxZ133pmoV1FRwe9//3vmzZvH5ZdfzoIFC3jrrbc477zzWnvIIiKtJvS3hty9ysxuBtZS8/XRx9z9HTO7Mdj+MPBD4DPAL8wMoMrdc8L2nUzBxlIWrC2mrOIwPf0AO3ftJjc3l5UrV9K/f38qKiqYMGECAHl5eVx55ZWJtl/96ldbYkgiIiktkt8RuPsaYE2dsodrLc8GZkfRV0MKNpZy69ObOXy0GoDdB49QyWmc3v1sNmzY0Ogf+q5du7b0EEVEUk67+mXxgrXFiRBI6JDG6VN/wNKlS3nuuefo0aMHr7zyCgBPPPFE4uigrjPPPJMPP/ywpYcsItLmIjkiSBVlFYeTlu+uhI2rVzN58mSuuOIK5syZQ2VlJQMHDmTx4sVJ21x11VXccMMNPPjgg6xYsUKfE4hIu2WpfIeynJwcb84vi8fNf5HSJGHQr3sXNsy9OMqhiYikJDN7o7mfwbarU0NzpgyhS6e0E8q6dEpjzpQhbTQiEZHU165ODU3LrvlB8/FvDfXt3oU5U4YkykVE5NPaVRBATRjoD7+ISNO1q1NDIiLSfAoCEZGYUxCIiMScgkBEJOYUBCIiMacgEBGJOQWBiEjMKQhERGJOQSAiEnMKAhGRmFMQiIjEnIJARCTmFAQiIjGnIBARiblIgsDMLjWzYjPbbmZzk2wfamavmdnHZvb9KPoUEZFohL4fgZmlAQ8Bk4GdQKGZrXL3rbWq7Qf+BZgWtj8REYlWFEcEo4Ht7r7D3T8BngJya1dw9z3uXggcjaA/ERGJUBRB0A94v9b6zqDspJhZvpkVmVlReXl56MGJiEjDoggCS1LmJ7szd1/k7jnuntO7d+8QwxIRkaaIIgh2AgNqrfcHyiLYr4iItIIogqAQGGRm55pZZ+AqYFUE+xURkVYQ+ltD7l5lZjcDa4E04DF3f8fMbgy2P2xmfYAioBtwzMz+DRjm7gfD9i8iIuGEDgIAd18DrKlT9nCt5b9Tc8pIRERSjH5ZLCIScwoCEZGYUxCIiMScgkBEJOYUBCIiMacgEBGJOQWBiEjMKQhERGJOQSAiEnMKAhGRmFMQiIjEnIJARCTmFAQiIjGnIBARiTkFgYhIzCkIRERiTkEgIhJzCgIRkZhTEIiIxFwkQWBml5pZsZltN7O5SbabmT0YbN9kZhdE0a+IiIQXOgjMLA14CPgSMAy42syG1an2JWBQ8MgHFobtV0REohHFEcFoYLu773D3T4CngNw6dXKBpV7jdaC7maVH0LeIiIQURRD0A96vtb4zKGtuHQDMLN/MisysqLy8PILhiYhIQ6IIAktS5idRp6bQfZG757h7Tu/evUMPTkREGhZFEOwEBtRa7w+UnUQdERFpA1EEQSEwyMzONbPOwFXAqjp1VgH/HHx7aAxwwN13RdC3iIiE1DHsDty9ysxuBtYCacBj7v6Omd0YbH8YWANMBbYDlcB1YfsVEZFoRPI7Andf4+6D3f08d//3oOzhIAQIvi30rWB7lrsXRdGvSFzNnj2brVu3ApCRkcHevXspKSkhMzOzjUcmp6LQRwQi0voeeeSRth6CtCO6xIRICispKWHo0KHk5eUxfPhwZs6cSWVlJRMnTqSo6B8H1u++++4J7Xbs2EF2djaFhYX86U9/YuzYsWRnZzN27FiKi4tbexqS4hQEIimuuLiY/Px8Nm3aRLdu3fjFL36R2FZdXc0nn3zC0qVLT6g/Y8YMFi9ezKhRoxg6dCjr169n48aN3HXXXdx2221tMQ1JYTo1JJJiCjaWsmBtMWUVh+npB+jVpy/jxo0D4JprruHBBx9M1L3jjjs4ePAgd955J4cOHaK8vJzc3FxWrlzJ+eefD8CBAwfIy8tj27ZtmBlHjx5tk3lJ6tIRgUgKKdhYyq1Pb6a04jAO7D54hIrKKgo2libqmP3j95k/+clP6NWrV2L9rLPOYsCAAWzYsCFRdscdd3DRRRexZcsWfvvb33LkyJFWmYucOhQEIilkwdpiDh+tPqGs6uAefrjoaQCWLVvGF7/4xXrbd+7cmYKCApYuXcqvfvUroOaIoF+/miu6PP744y0zcDmlKQhEUkhZxeFPlXX6zADee30Nw4cPZ//+/Xzzm99scB9du3Zl9erV3H///Tz77LPccsst3HrrrYwbN47q6uoG20o8mXvSS/6khJycHK/9zQiR9m7c/BcprRUGVQd2s2fFnYz63mI2zL24DUcmpwoze8Pdc5rTRkcEIilkzpQhdOmUdkKZmTFnypA2GpHEgb41JJJCpmXXnMs//q2hz30ug/+3dkOiXKQlKAhEUsy07H76wy+tSqeGRERiTkEgIhJzCgIRkZhTEIiIxJyCQEQk5hQEIiIxpyAQEYk5BYGISMyFCgIz62lmz5vZtuC5Rz31HjOzPWa2JUx/IiISvbBHBHOBde4+CFgXrCfzOHBpyL7kFLV8+XJKSkraehgiUo+wQZALLAmWlwDTklVy9/XA/pB9SQqaNWsWK1asqHf7k08+yd/+9jcyMjJOqr2ItLyw1xo6x913Abj7LjM7O4IxSTtyzTXXtPUQRKQRjR4RmNkLZrYlySO3JQZkZvlmVmRmReXl5S3RRbvy0UcfcdlllzFixAgyMzNZvnw5d911F6NGjSIzM5P8/HyO33Ni4sSJfOc732H8+PF8/vOfp7CwkCuuuIJBgwZx++23J/Z53333kZmZSWZmJg888ECifOnSpQwfPpwRI0Zw7bXXJsrXr1/P2LFjGThwYOL/7t2dOXPmkJmZSVZWFsuXL0+U33zzzQwbNozLLruMPXv2tMKrJCINcveTfgDFQHqwnA4UN1A3A9jSnP1/4QtfcGnYihUrfPbs2Yn1iooK37dvX2L9mmuu8VWrVrm7+4QJE/yWW25xd/cHHnjA09PTvayszI8cOeL9+vXzvXv3elFRkWdmZvqhQ4f8ww8/9GHDhvmbb77pW7Zs8cGDB3t5ebm7e6KPvLw8nzlzpldXV/s777zj5513XmJckyZN8qqqKv/73//uAwYM8LKyMl+5cmWivLS01M866yz/zW9+0yqvlUgcAEXezL/lYU8NrQLygPnB87Mh9ydNVLCxlAVri/nrjn3sXbmafUdv4jvXX82FF17IypUruffee6msrGT//v2cf/75fOUrXwHg8ssvByArK4vzzz+f9PR0AAYOHMj777/Pq6++yvTp0+natSsAV1xxBa+88gpmxsyZMxM3Su/Zs2diLNOmTaNDhw4MGzaM3bt3A/Dqq69y9dVXk5aWxjnnnMOECRMoLCxk/fr1ifK+ffty8cW665ZIWwv7YfF8YLKZbQMmB+uYWV8zW3O8kpktA14DhpjZTjO7PmS/sVawsZRbn95MacVhOvbsR+9r7+f1ijPI/5fvcdddd3HTTTexYsUKNm/ezA033MCRI0cSbU877TQAOnTokFg+vl5VVZU4jVSXu2NmSbfV3s/x9vXtB6h3PyLSNkIFgbvvc/dL3H1Q8Lw/KC9z96m16l3t7unu3snd+7v7o2EHHmcL1hZz+GjNTcirPtxHh06n0XnoBDzzy7z55psA9OrVi0OHDjX7Gznjx4+noKCAyspKPvroI5555hkuvPBCLrnkEn7961+zb98+APbvb/hLYOPHj2f58uVUV1dTXl7O+vXrGT16NOPHj+epp56iurqaXbt28dJLL53EKyAiUdIdyk5BZbVubn60vIQ9Ly8GM6xDR5787a8oKCggKyuLjIwMRo0a1ax9X3DBBcyaNYvRo0cDMHv2bLKzswGYN28eEyZMIC0tjezsbB5//PF69zN9+nRee+01RowYgZlx77330qdPH6ZPn86LL75IVlYWgwcPZsKECc1/AUQkUtbQIXxby8nJ8aKiorYeRsoZN/9FSmuFwXH9undhw1ydcxeJMzN7w91zmtNG1xo6Bc2ZMoQundJOKOvSKY05U4a00YhE5FSmU0OnoOM3Nl+wtpiyisP07d6FOVOG6IbnInJSFASnqGnZ/fSHX0QioVNDIiIxpyAQEYk5BYGISMwpCEREWkhJSQmZmZlNrj979my2bt1a7/aJEyfSEl+p14fFIiIp4pFHHmmTfnVEICLSgqqqqsjLy2P48OHMnDmTyspK1q1bR3Z2NllZWXz961/n448/Bv7xf/zV1dXMmjUrcRn3+++//4R9Hjt2jLy8PG6//XaOHDnCddddR1ZW1vGrAJzZ3DEqCEREWlBxcTH5+fls2rSJbt26cd999zFr1iyWL1/O5s2bqaqqYuHChSe0eeuttygtLWXLli1s3ryZ6667LrGtqqqKr33tawwePJgf//jHPPTQQwBs3ryZZcuWAWSY2enNGaOCQEQkQgUbSxk3/0XOnfscMxb+gV59+jJu3Dig5o5969at49xzz2Xw4MEA5OXlsX79+hP2MXDgQHbs2MG3v/1tfve739GtW7fEtm984xtkZmYyb948oOaS78dvFDV06FCAT4DBzRmzgkBEJCK1LxHvwO6DR6iorKJgY2mz9tOjRw/efvttJk6cyEMPPcTs2bMT28aOHctLL72UuLx8FNeLUxCIiESk9iXij6s6uIcfLnoagGXLljFp0iRKSkrYvn07AE888cSnrsK7d+9ejh07xowZM7j77rsTl5cHuP7665k6dSpXXnklVVVVjB8/nl/+8pcA/OUvfwHoTM3dI5tM3xoSEYlIWZKrAnf6zADee30Nw4f/J4MGDeLnP/85Y8aMSfwhHzVqFDfeeOMJbUpLS7nuuus4duwYAD/96U9P2P7d736XAwcOcO211/Loo49y0003kZWVRceOHQFK3P3j5oxbl6EWEYlIKlwiXpehFhFpQ6fqJeJ1akhEJCKn6iXiFQQiIhE6FS8RH+rUkJn1NLPnzWxb8NwjSZ0BZvaSmf3ZzN4xs38N06eIiEQr7GcEc4F17j4IWBes11UFfM/dPw+MAb5lZsNC9isiIhEJGwS5wJJgeQkwrW4Fd9/l7m8Gyx8CfwZOreMmEZF2LGwQnOPuu6DmDz5wdkOVzSwDyAb+2ECdfDMrMrOi8vLykMMTEZHGNPphsZm9APRJsmleczoyszOAlcC/ufvB+uq5+yJgEdT8jqA5fYiISPM1GgTuPqm+bWa228zS3X2XmaUDe+qp14maEPiluz990qMVEZHIhT01tArIC5bzgGfrVjAzAx4F/uzu94XsT0REIhY2COYDk81sGzA5WMfM+prZmqDOOOBa4GIzeyt4TA3Zr4iIRCTUD8rcfR9wSZLyMmBqsPwqYGH6ERGRlqNrDYmIxJyCQEQk5hQEIiIxpyAQEYk5BYGISMwpCEREYk5BICIScwoCEZGYUxCIiMScgkBEJOYUBCIiMacgEBGJOQWBiEjMKQhERGJOQSAiEnMKAhGRmFMQiIjEnIJARCTmFAQiIjEXKgjMrKeZPW9m24LnHknqnG5mfzKzt83sHTO7M0yfIiISrbBHBHOBde4+CFgXrNf1MXCxu48ARgKXmtmYkP2KiEhEwgZBLrAkWF4CTKtbwWscClY7BQ8P2a+IiEQkbBCc4+67AILns5NVMrM0M3sL2AM87+5/rG+HZpZvZkVmVlReXh5yeCIi0piOjVUwsxeAPkk2zWtqJ+5eDYw0s+7AM2aW6e5b6qm7CFgEkJOToyMHEZEW1mgQuPuk+raZ2W4zS3f3XWaWTs3/8Te0rwozexm4FEgaBCIi0rrCnhpaBeQFy3nAs3UrmFnv4EgAM+sCTAL+J2S/IiISkbBBMB+YbGbbgMnBOmbW18zWBHXSgZfMbBNQSM1nBKtD9isiIhFp9NRQQ9x9H3BJkvIyYGqwvAnIDtOPiIi0HP2yWEQk5hQEIiIxpyAQEYk5BYGISMwpCEREYk5BICIScwoCEZGYUxCIiMScgkBEJOYUBCIiMacgEBGJOQWBiEjMKQhERGJOQSAiEnMKAhGRmFMQiIjEnIJARCTmFAQiIjGnIBARiblQQWBmPc3seTPbFjz3aKBumpltNDPduF5EJIWEPSKYC6xz90HAumC9Pv8K/DlkfyIiErGwQZALLAmWlwDTklUys/7AZcAjIfsTEZGIhQ2Cc9x9F0DwfHY99R4AbgGOhexPREQi1rGxCmb2AtAnyaZ5TenAzL4M7HH3N8xsYhPq5wP5AJ/97Geb0oWIiITQaBC4+6T6tpnZbjNLd/ddZpYO7ElSbRxwuZlNBU4HupnZk+5+TT39LQIWAeTk5HhTJiEiIicv7KmhVUBesJwHPFu3grvf6u793T0DuAp4sb4QEBGR1hc2COYDk81sGzA5WMfM+prZmrCDExGRltfoqaGGuPs+4JIk5WXA1CTlLwMvh+lTRESipV8Wi4jEnIJARCTmFAQiIjGnIBARiTkFgYhIzCkIRERiTkEgIhJzCgIRkZhTEIiIxJyCQEQk5hQEIiIxpyAQEYk5BYGISMwpCEREYk5BICIScwoCEZGYUxCIiMScgkBEJOYUBCIiMacgEBGJuVA3rzeznsByIAMoAf6vu3+QpF4J8CFQDVS5e06YfkVEJDphjwjmAuvcfRCwLlivz0XuPlIhICKSWsIGQS6wJFheAkwLuT8REQFKSkrIzMw8oezll1/my1/+cmNNO5vZlub0FTYIznH3XQDB89n11HPgv83sDTPLb2iHZpZvZkVmVlReXh5yeCIi0phGg8DMXjCzLUkeuc3oZ5y7XwB8CfiWmY2vr6K7L3L3HHfP6d27dzO6EBFpn3bs2EF2djaFhYWJsh/96Ef87Gc/S6xnZmZSUlJyfLWjmS0xs01mtsLM/ldD+280CNx9krtnJnk8C+w2s3SA4HlPPfsoC573AM8AoxvrV0REoLi4mBkzZrB48WJGjRrV1GZDgEXuPhw4CNzUUOWwp4ZWAXnBch7wbN0KZtbVzM48vgz8E9Cs81ciInFQsLGUcfNf5Ny5zzFj4R/YuWs3ubm5PPnkk4wcObI5u3rf3TcEy08CX2yoctggmA9MNrNtwORgHTPra2ZrgjrnAK+a2dvAn4Dn3P13IfsVEWlXCjaWcuvTmymtOIwDuw8eoZLTOL372WzYsOFT9Tt27MixY8cS60eOHKm92etUr7t+4r5Oftjg7vuAS5KUlwFTg+UdwIgw/YiItHcL1hZz+Gj1iYUd0jh96g9YuvSnnHHGGfTt2zexKSMjg9WrVwPw5ptv8t5779Vu+Vkz+z/u/hpwNfBqQ33rl8UiIimgrOJw0vLdlbB69Wruv/9+Dhw4kCifMWMG+/fvZ+TIkSxcuJDBgwfXbvZnIM/MNgE9gYUN9W3uDR4xtKmcnBwvKipq62GIiLS4cfNfpDRJGPTr3oUNcy9u8n7M7I3m/nBXRwQiIilgzpQhdOmUdkJZl05pzJkypMX7DvUZgYiIRGNadj+g5rOCsorD9O3ehTlThiTKW5KCQEQkRUzL7tcqf/jr0qkhEZGYUxCIiMScgkBEJOYUBCIiMacgEBGJuZT+QZmZlQN/betxNKAXsLetBxEBzSN1tIc5QPuYx6k6h8+5e7Ou4Z/SQZDqzKyoPdx6U/NIHe1hDtA+5tEe5tBUOjUkIhJzCgIRkZhTEISzqK0HEBHNI3W0hzlA+5hHe5hDk+gzAhGRmNMRgYhIzCkIGmFmPc3seTPbFjz3qKfeY2a2x8y21Cn/kZmVmtlbwWNq64z8U+MLO48mtW9JzZjDpWZWbGbbzWxurfI2fS/qG1et7WZmDwbbN5nZBU1t21pCzqHEzDYHr32b3mikCfMYamavmdnHZvb95rQ9Jbm7Hg08gHuBucHyXOCeeuqNBy4AttQp/xHw/XYwjya1b+s5AGnAu8BAoDPwNjCsrd+LhsZVq85U4L8AA8YAf2xq21SfQ7CtBOjVFq//SczjbGAU8O+1/82kynsR9UNHBI3LBZYEy0uAackquft6YH8rjelkhJ1Hk9q3sKaMYTSw3d13uPsnwFNBu7bWlHHlAku9xutAdzNLb2Lb1hBmDqmk0Xm4+x53LwSONrftqUhB0Lhz3H0XQPB89kns4+bgMPmxtjilEgg7jyheh7CaMoZ+wPu11ncGZce11XvR2LgaqtOUtq0hzBwAHPhvM3vDzPJbbJSNC/N6psp7ESndmAYwsxeAPkk2zYtg9wuBu6n5j+Bu4D+Ar0ew309p4Xm0igjmYEnKjn81rtXei2aOq7E6TWnbGsLMAWCcu5eZ2dnA82b2P8ERaGsL83qmynsRKQUB4O6T6ttmZrvNLN3ddwWHuHuaue/dtfb1n8Dqkx9po3212DyAsO2bJII57AQG1FrvD5QF+26196I542pCnc5NaNsawswBdz/+vMfMnqHmNEtbBEFT5tESbVOWTg01bhWQFyznAc82p3Gd86PTgS311W1hoeYRQfsoNGUMhcAgMzvXzDoDVwXt2vq9qHdctawC/jn45s0Y4EBwCqwpbVvDSc/BzLqa2ZkAZtYV+Cfa7r+FMK9nqrwX0WrrT6tT/QF8BlgHbAueewblfYE1teotA3ZR8+HSTuD6oPwJYDOwiZp/MOmn6DyStk/ROUwF/kLNtzvm1Spv0/ci2biAG4Ebg2UDHgq2bwZyGptTG7wHJzUHar5l83bweKct59DEefQJ/v0fBCqC5W6p9F5E+dAvi0VEYk6nhkREYk5BICIScwoCEZGYUxCIiMScgkBEJOYUBCIiMacgEBGJOQWBiEjM/X/f77bhPki9GAAAAABJRU5ErkJggg==\n",
|
|
"text/plain": [
|
|
"<Figure size 432x288 with 1 Axes>"
|
|
]
|
|
},
|
|
"metadata": {
|
|
"needs_background": "light"
|
|
},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"%matplotlib inline\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"\n",
|
|
"with torch.no_grad():\n",
|
|
" words = ['piłka', 'klub', 'kort', 'boisko', 'samochód']\n",
|
|
" words_ixs = [torch.argmax(vectorize_text(w).float().unsqueeze(dim=0)).item() for w in words]\n",
|
|
"\n",
|
|
" x_label = labels_dic['pilka-nozna']\n",
|
|
" y_label = labels_dic['tenis']\n",
|
|
"\n",
|
|
" x = [model[0].weight[x_label, ix] for ix in words_ixs]\n",
|
|
" y = [model[0].weight[y_label, ix] for ix in words_ixs]\n",
|
|
"\n",
|
|
" fig, ax = plt.subplots()\n",
|
|
" ax.scatter(x, y)\n",
|
|
"\n",
|
|
" for i, txt in enumerate(words):\n",
|
|
" ax.annotate(txt, (x[i], y[i]))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Zadanie etykietowania sekwencji\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Zadanie etykietowania sekwencji (*sequence labelling*) polega na przypisaniu poszczególnym wyrazom (tokenom) tekstu **etykiet** ze skończonego zbioru. Definiując formalnie:\n",
|
|
"\n",
|
|
"- rozpatrujemy ciąg wejściowy tokenów $(t^1,\\dots,t^K)$\n",
|
|
"- dany jest skończony zbiór etykiet $L = \\{l_1,\\dots,l_{|L|}\\}$, dla uproszczenia można założyć, że etykietami\n",
|
|
" są po prostu kolejne liczby, tj. $L=\\{0,\\dots,|L|-1\\}$\n",
|
|
"- zadanie polega na wygenerowaniu sekwencji etykiet (o tej samej długości co ciąg wejściowy!) $(y^1,\\dots,y^K)$,\n",
|
|
" $y^k \\in L$\n",
|
|
"\n",
|
|
"Zadanie etykietowania można traktować jako przypadek szczególny klasyfikacji wieloklasowej, z tym, że klasyfikacji dokonujemy wielokrotnie — dla każdego tokenu (nie dla każdego tekstu).\n",
|
|
"\n",
|
|
"Przykłady zastosowań:\n",
|
|
"\n",
|
|
"- oznaczanie częściami mowy (*POS tagger*) — czasownik, przymiotnik, rzeczownik itd.\n",
|
|
"- oznaczanie etykiet nazw w zadaniu NER (nazwisko, kwoty, adresy — najwięcej tokenów będzie miało etykietę pustą, zazwyczaj oznaczaną przez `O`)\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### **Pytanie**: czy zadanie tłumaczenia maszynowego można potraktować jako problem etykietowania sekwencji?\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Przykładowe wyzwanie NER CoNLL-2003\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Zob. [https://gonito.net/challenge/en-ner-conll-2003](https://gonito.net/challenge/en-ner-conll-2003).\n",
|
|
"\n",
|
|
"Przykładowy przykład uczący (`xzcat train.tsv.xz| head -n 1`):\n",
|
|
"\n",
|
|
"O O B-MISC I-MISC O O O O O B-LOC O B-LOC O O O O O O O O O O O B-MISC I-MISC O O O O O B-PER I-PER O B-LOC O O O O O O B-PER I-PER O B-LOC O O O O O O B-PER I-PER I-PER O B-LOC O O O O O B-PER I-PER O O B-LOC O O O O O O B-PER I-PER O B-LOC O O O O O B-PER I-PER O O O O O B-PER I-PER O B-LOC O O O O O B-PER I-PER O B-LOC O B-LOC O O O O O O B-PER I-PER O O O O B-PER I-PER O B-LOC O O O O O O B-PER I-PER O B-LOC O O O O O B-PER I-PER O O O O O B-PER I-PER O B-LOC O O O O O B-PER I-PER O B-LOC O O O O O O B-PER I-PER O O O O O B-PER I-PER O B-LOC O O O O O B-PER I-PER O B-LOC O O O O O O B-PER I-PER O O O O B-PER I-PER I-PER O B-LOC O O O O O O B-PER I-PER O O O O B-PER I-PER O B-LOC O O O O O O B-PER I-PER O B-LOC O O O O O O B-PER I-PER O O O O B-PER I-PER O B-LOC O O O O O O B-PER I-PER O O O O B-PER I-PER O B-LOC O O O O O O B-PER I-PER O B-LOC O O O O O B-PER I-PER O B-LOC O B-LOC O O O O O B-PER I-PER O O O O O\tGOLF - BRITISH MASTERS THIRD ROUND SCORES . </S> NORTHAMPTON , England 1996-08-30 </S> Leading scores after </S> the third round of the British Masters on Friday : </S> 211 Robert Allenby ( Australia ) 69 71 71 </S> 212 Pedro Linhart ( Spain ) 72 73 67 </S> 216 Miguel Angel Martin ( Spain ) 75 70 71 , Costantino Rocca </S> ( Italy ) 71 73 72 </S> 217 Antoine Lebouc ( France ) 74 73 70 , Ian Woosnam 70 76 71 , </S> Francisco Cea ( Spain ) 70 71 76 , Gavin Levenson ( South </S> Africa ) 66 75 76 </S> 218 Stephen McAllister 73 76 69 , Joakim Haeggman ( Swe ) 71 77 </S> 70 , Jose Coceres ( Argentina ) 69 78 71 , Paul Eales 75 71 72 , </S> Klas Eriksson ( Sweden ) 71 75 72 , Mike Clayton ( Australia ) </S> 69 76 73 , Mark Roe 69 71 78 </S> 219 Eamonn Darcy ( Ireland ) 74 76 69 , Bob May ( U.S. ) 74 75 70 , </S> Paul Lawrie 72 75 72 , Miguel Angel Jimenez ( Spain ) 74 72 </S> 73 , Peter Mitchell 74 71 75 , Philip Walton ( Ireland ) 71 74 </S> 74 , Peter O'Malley ( Australia ) 71 73 75 </S> 220 Barry Lane 73 77 70 , Wayne Riley ( Australia ) 71 78 71 , </S> Martin Gates 71 77 72 , Bradley Hughes ( Australia ) 73 75 72 , </S> Peter Hedblom ( Sweden ) 70 75 75 , Retief Goosen ( South </S> Africa ) 71 74 75 , David Gilford 69 74 77 . </S>\n",
|
|
"\n",
|
|
"W pierwszym polu oczekiwany wynik zapisany za pomocą notacji **BIO**.\n",
|
|
"\n",
|
|
"Jako metrykę używamy F1 (z pominięciem tagu `O`)\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Metryka F1\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Etykietowanie za pomocą klasyfikacji wieloklasowej\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Można potraktować problem etykietowania dokładnie tak jak problem\n",
|
|
"klasyfikacji wieloklasowej (jak w przykładzie klasyfikacji dyscyplin\n",
|
|
"sportowych powyżej), tzn. rozkład prawdopodobieństwa możliwych etykiet\n",
|
|
"uzyskujemy poprzez zastosowanie prostej warstwy liniowej i funkcji softmax:\n",
|
|
"\n",
|
|
"$$p(l^k=j) = s(W\\vec{v}(t^k))_j = \\frac{e^{(W\\vec{v}(t^k))_j}}{Z},$$\n",
|
|
"\n",
|
|
"gdzie $\\vec{v}(t^k)$ to reprezentacja wektorowa tokenu $t^k$.\n",
|
|
"Zauważmy, że tutaj (w przeciwieństwie do klasyfikacji całego tekstu)\n",
|
|
"reprezentacja wektorowa jest bardzo uboga: wektor <u>one-hot</u>! Taki\n",
|
|
"klasyfikator w ogóle nie będzie brał pod uwagę kontekstu, tylko sam\n",
|
|
"wyraz, więc tak naprawdę zdegeneruje się to do zapamiętania częstości\n",
|
|
"etykiet dla każdego słowa osobno.\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"##### Bogatsza reprezentacja słowa\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Można spróbować uzyskać bogatszą reprezentację dla słowa biorąc pod uwagę na przykład:\n",
|
|
"\n",
|
|
"- długość słowa\n",
|
|
"- kształt słowa (*word shape*), np. czy pisany wielkimi literami, czy składa się z cyfr itp.\n",
|
|
"- n-gramy znakowe wewnątrz słowa (np. słowo *Kowalski* można zakodować jako sumę wektorów\n",
|
|
" trigramów znakówych $\\vec{v}(Kow) + \\vec{v}(owa) + \\vec{v}(wal) + \\vec{v}(als) + \\vec{v}(lsk) + \\vec{v}(ski)$\n",
|
|
"\n",
|
|
"Cały czas nie rozpatrujemy jednak w tej metodzie kontekstu wyrazu.\n",
|
|
"(*Renault* w pewnym kontekście może być nazwą firmy, w innym —\n",
|
|
"nazwiskiem).\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"##### Reprezentacja kontekstu\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Za pomocą wektora można przedstawić nie pojedynczy token $t^k$, lecz\n",
|
|
"cały kontekst, dla *okna* o długości $c$ będzie to kontekst $t^{k-c},\\dots,t^k,\\dots,t^{k+c}$.\n",
|
|
"Innymi słowy klasyfikujemy token na podstawie jego samego oraz jego kontekstu:\n",
|
|
"\n",
|
|
"$$p(l^k=j) = \\frac{e^{(W\\vec{v}(t^{k-c},\\dots,t^k,\\dots,t^{k+c}))_j}}{Z}.$$\n",
|
|
"\n",
|
|
"Zauważmy, że w tej metodzie w ogóle nie rozpatrujemy sensowności\n",
|
|
"sekwencji wyjściowej (etykiet), np. może być bardzo mało\n",
|
|
"prawdopodobne, że bezpośrednio po nazwisku występuje data.\n",
|
|
"\n",
|
|
"Napiszmy wzór określający prawdopodobieństwo całej sekwencji, nie\n",
|
|
"tylko pojedynczego tokenu. Na razie będzie to po prostu iloczyn poszczególnych wartości.\n",
|
|
"\n",
|
|
"$$l = (l^1,\\\\dots,l^k), p(l) = \\prod_{k=1}^K \\frac{e^{(W\\vec{v}(t^{k-c},\\dots,t^k,\\dots,t^{k+c}))_{l^k}}}{Z_k} = \\frac{e^{\\sum_{k=1}^K (W\\vec{v}(t^{k-c},\\dots,t^k,\\dots,t^{k+c}))_{l^k}}}{\\prod_{k=1}^K Z_k}$$\n",
|
|
"\n",
|
|
"Reprezentacja kontekstu może być funkcją embeddingów wyrazów\n",
|
|
"(zakładamy, że embedding nie zależy od pozycji słowa).\n",
|
|
"\n",
|
|
"$$\\vec{v}(t^{k-c},\\dots,t^k,\\dots,t^{k+c}) = f(\\vec{E}(t^{k-c}),\\dots,\\vec{E}(t^k),\\dots,\\vec{E}({t^{k+c}}))$$\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Warunkowe pola losowe\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Warunkowe pola losowe (*Conditional Random Fields*, *CRF*) to klasa\n",
|
|
"modeli, które pozwalają uwzględnić zależności między punktami danych\n",
|
|
"(które można wyrazić jako graf). Najprostszym przykładem będzie prosty\n",
|
|
"graf wyrażający „następowanie po” (czyli sekwencje). Do poprzedniego\n",
|
|
"wzoru dodamy składnik $V_{i,j}$ (który można interpretować jako\n",
|
|
"macierz) określający prawdopodobieństwo, że po etykiecie o numerze $i$ wystąpi etykieta o numerze $j$.\n",
|
|
"\n",
|
|
"**Pytanie**: Czy macierz $V$ musi być symetryczna? Czy $V_{i,j} = V_{j,i}$? Czy jakieś specjalne wartości występują na przekątnej?\n",
|
|
"\n",
|
|
"Macierz $V$ wraz z macierzą $W$ będzie stanowiła wyuczalne wagi w naszym modelu.\n",
|
|
"\n",
|
|
"Wartości $V_{i,j}$ nie stanowią bezpośrednio prawdopodobieństwa, mogą\n",
|
|
"przyjmować dowolne wartości, które będę normalizowane podobnie, tak jak to się dzieje w funkcji Softmax.\n",
|
|
"\n",
|
|
"W takiej wersji warunkowych pól losowych otrzymamy następujący wzór na prawdopodobieństwo całej sekwencji.\n",
|
|
"\n",
|
|
"$$p(l) = \\frac{e^{\\sum_{k=1}^K (W\\vec{v}(t^{k-c},\\dots,t^k,\\dots,t^{k+c}))_{l^k} + \\sum_{k=1}^{K-1} V_{l^k,l^{k+1}}}}{\\prod_{k=1}^K Z_k}$$\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Algorytm Viterbiego\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"W czasie inferencji mamy ustalone wagi funkcji $\\vec{v}(\\dots)$ oraz\n",
|
|
"macierz $V$. Szukamy sekwencji $y$ która maksymalizuje prawdopodobieństwo estymowane przez model:\n",
|
|
"\n",
|
|
"$$y = \\underset{l}{\\operatorname{argmax}} \\hat{p}(l|t^1,\\dots,t^K)$$\n",
|
|
"\n",
|
|
"Naiwne podejście polegające na obliczeniu prawdopodobieństw wszystkich możliwych sekwencji miałoby\n",
|
|
"nieakceptowalną złożoność czasową $O(|L|^K)$.\n",
|
|
"\n",
|
|
"Na szczęście, możemy użyć **algorytmu Viterbiego** o lepszej złożoności\n",
|
|
"obliczeniowej, algorytmu opartego na idei programowania dynamicznego.\n",
|
|
"\n",
|
|
"W algorytmie będziemy wypełniać dwuwymiarowe tabele $s[i, j]$ i $b[i, j]$:\n",
|
|
"\n",
|
|
"- $s[i, j]$ — będzie zawierać maksymalne prawdopodobieństwo (właściwie: nieznormalizowaną wartość,\n",
|
|
" która jest monotoniczna względem prawdopodobieństwa)\n",
|
|
" dla ciągów o długości $i$ zakończonych etykietą $l_j$,\n",
|
|
"- $b[i, j]$ — będzie zawierać „wskaźnik” wsteczny (*backpointer*) do podciągu o długości $i-1$, dla którego\n",
|
|
" razem z $l_j$ jest osiągana maksymalna wartość $s[i, j]$.\n",
|
|
"\n",
|
|
"Inicjalizacja:\n",
|
|
"\n",
|
|
"- $s[1, j] = (W\\vec{v}(t^k,\\dots,t^{k+c}))_j$,\n",
|
|
"- $b[1, j]$ — nie musimy wypełniać tej wartości.\n",
|
|
"\n",
|
|
"Dla $i > 1$ i dla każdego $j$ będziemy teraz szukać:\n",
|
|
"\n",
|
|
"$$\\underset{q \\in \\{1,\\dots,|V|\\}}{\\operatorname{max}} s[i-1, q] + (W\\vec{v}(t^{k-c},\\dots,t^k,\\dots,t^{k+c}))_j + V_{q, j}$$\n",
|
|
"\n",
|
|
"Tę wartość przypiszemy do $s[i, j]$, z kolei do $b[i, j]$ — indeks\n",
|
|
"$q$, dla którego ta największa wartość jest osiągnięta.\n",
|
|
"\n",
|
|
"Najpierw obliczenia wykonujemy wprzód wypełniając tabelę dla coraz większych wartości $j$.\n",
|
|
"W ten sposób otrzymamy największą wartość (nieznormalizowanego) prawdopodobieństwa:\n",
|
|
"\n",
|
|
"$$\\underset{q \\in \\{1,\\dots,|V|\\}}{\\operatorname{max}} s[K, q]$$\n",
|
|
"\n",
|
|
"oraz ostatnią etykietę:\n",
|
|
"\n",
|
|
"$$y^K = \\underset{q \\in \\{1,\\dots,|V|\\}}{\\operatorname{argmax}} s[K, q]$$\n",
|
|
"\n",
|
|
"Aby uzyskać cały ciąg, kierujemy się *wstecz* używając wskaźników:\n",
|
|
"\n",
|
|
"$$y^i = b[i, y^{i+1}]$$\n",
|
|
"\n",
|
|
"\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Złożoność obliczeniowa\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Zauważmy, że rozmiar tabel $s$ i $b$ wynosi $K \\times |L|$, a koszt\n",
|
|
"wypełnienia każdej komórki to $|L|$, a zatem złożoność algorytmu jest wielomianowa:\n",
|
|
"$O(K|L|^2)$.\n",
|
|
"\n",
|
|
"**Pytanie:** Czy gdyby uzależnić etykietę nie tylko od poprzedniej\n",
|
|
"etykiety, lecz również od jeszcze wcześniejszej, to złożoność\n",
|
|
"obliczeniowa byłaby taka sama?\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Przykład\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Rozpatrzmy uproszczony przykład tagowania częściami mowy:\n",
|
|
"\n",
|
|
"- słownik $V=\\{\\mathit{Ala}, \\mathit{powieść}, \\mathit{ma}\\}$,\n",
|
|
"- zbiór etykiet $L=\\{\\mathit{C}, \\mathit{P}, \\mathit{R}\\}$,\n",
|
|
"- kontekst nie jest uwzględniany ($c = 0$).\n",
|
|
"\n",
|
|
"(To, że liczba słów i etykiet jest taka sama, jest przypadkowe, nie ma znaczenia)\n",
|
|
"\n",
|
|
"Zakładamy, że słowa reprezentujemy wektorowo za pomocą prostej reprezentacji one-hot.\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 28,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"tensor([0., 1., 0.])"
|
|
]
|
|
},
|
|
"execution_count": 28,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import torch\n",
|
|
"\n",
|
|
"vocab = ['Ala', 'ma', 'powieść']\n",
|
|
"labels = ['C', 'P', 'R']\n",
|
|
"\n",
|
|
"onehot = {\n",
|
|
" 'Ala': torch.tensor([1., 0., 0.]),\n",
|
|
" 'ma': torch.tensor([0., 1., 0.]),\n",
|
|
" 'powieść': torch.tensor([0., 0., 1.])\n",
|
|
" }\n",
|
|
"\n",
|
|
"onehot['ma']"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Przyjmijmy, że w czasie uczenia zostały ustalone następujące wartości\n",
|
|
"macierzy $W$ i $V$ (samego procesu uczenia nie pokazujemy tutaj):\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 29,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"tensor([0.4983, 0.0034, 0.4983])"
|
|
]
|
|
},
|
|
"execution_count": 29,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import torch\n",
|
|
"import torch.nn as nn\n",
|
|
"\n",
|
|
"matrixW = torch.tensor(\n",
|
|
" [[-1., 3.0, 3.0], # C\n",
|
|
" [0., 2.0, -2.0], # P\n",
|
|
" [4., -2.0, 3.0]]) # R\n",
|
|
" # Ala ma powieść\n",
|
|
"# rozkład prawdopodobieństwa, gdyby patrzeć tylko na słowo\n",
|
|
"nn.functional.softmax(matrixW @ onehot['powieść'], dim=0)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 30,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"tensor([0.1027, 0.1386, 0.7587])"
|
|
]
|
|
},
|
|
"execution_count": 30,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import torch\n",
|
|
"import torch.nn as nn\n",
|
|
"\n",
|
|
"matrixV = torch.tensor(\n",
|
|
" [[-0.5, 1.5, 2.0], # C\n",
|
|
" [0.5, 0.8, 2.5], # P\n",
|
|
" [2.0, 0.8, 0.2]]) # R\n",
|
|
"# C P R\n",
|
|
"\n",
|
|
"# co występuje po przymiotniku? - rozkład prawdopodobieństwa\n",
|
|
"nn.functional.softmax(matrixV[1], dim=0)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Algorytm Viterbiego:\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 31,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"[4.0, 3.5, 4.5]"
|
|
]
|
|
},
|
|
"execution_count": 31,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"d = ['Ala', 'ma', 'powieść']\n",
|
|
"\n",
|
|
"s = []\n",
|
|
"b = []\n",
|
|
"\n",
|
|
"# inicjalizacja\n",
|
|
"s.append(matrixW @ onehot[d[0]])\n",
|
|
"b.append(None)\n",
|
|
"\n",
|
|
"# wprzód\n",
|
|
"i = 1\n",
|
|
"os = []\n",
|
|
"ob = []\n",
|
|
"for j in range(0, len(labels)):\n",
|
|
" z = s[i-1] + matrixV[:,j] + matrixW @ onehot[d[i]]\n",
|
|
"\n",
|
|
" ns = torch.max(z).item()\n",
|
|
" nb = torch.argmax(z).item()\n",
|
|
"\n",
|
|
" os.append(ns)\n",
|
|
" ob.append(nb)\n",
|
|
"\n",
|
|
"os"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"author": "Filip Graliński",
|
|
"email": "filipg@amu.edu.pl",
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"lang": "pl",
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.9.6"
|
|
},
|
|
"org": null,
|
|
"subtitle": "9.Przegląd składowych sieci neuronowych[wykład]",
|
|
"title": "Ekstrakcja informacji",
|
|
"year": "2021"
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 4
|
|
}
|