![\"Rys.](\"perceptron.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Zasada działania perceptronu\n",
"\n",
"1. Ustal wartości początkowe $\\theta$ (wektor 0 lub liczby losowe blisko 0).\n",
"1. Dla każdego przykładu $(x^{(i)}, y^{(i)})$, dla $i=1,\\ldots,m$\n",
" * Oblicz wartość wyjścia $o^{(i)} = g(\\theta^{T}x^{(i)}) = g(\\sum_{j=0}^{n} \\theta_jx_j^{(i)})$\n",
" * Wykonaj aktualizację wag (tzw. *perceptron rule*):\n",
" $$ \\theta := \\theta + \\Delta \\theta $$\n",
" $$ \\Delta \\theta = \\alpha(y^{(i)}-o^{(i)})x^{(i)} $$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"$$\\theta_j := \\theta_j + \\Delta \\theta_j $$\n",
"\n",
"Jeżeli przykład został sklasyfikowany **poprawnie**:\n",
"\n",
"* $y^{(i)}=1$ oraz $o^{(i)}=1$ : $$\\Delta\\theta_j = \\alpha(1 - 1)x_j^{(i)} = 0$$\n",
"* $y^{(i)}=-1$ oraz $o^{(i)}=-1$ : $$\\Delta\\theta_j = \\alpha(-1 - -1)x_j^{(i)} = 0$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Czyli: jeżeli trafiłeś, to nic nie zmieniaj."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"$$\\theta_j := \\theta_j + \\Delta \\theta_j $$\n",
"\n",
"Jeżeli przykład został sklasyfikowany **niepoprawnie**:\n",
"\n",
"* $y^{(i)}=1$ oraz $o^{(i)}=-1$ : $$\\Delta\\theta_j = \\alpha(1 - -1)x_j^{(i)} = 2 \\alpha x_j^{(i)}$$\n",
"* $y^{(i)}=-1$ oraz $o^{(i)}=1$ : $$\\Delta\\theta_j = \\alpha(-1 - 1)x_j^{(i)} = -2 \\alpha x_j^{(i)}$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Czyli: przesuń wagi w odpowiednią stronę."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Zalety perceptronu\n",
"\n",
"* intuicyjny i prosty\n",
"* łatwy w implementacji\n",
"* jeżeli dane można liniowo oddzielić, algorytm jest zbieżny w skończonym czasie"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Wady perceptronu\n",
"\n",
"* jeżeli danych nie można oddzielić liniowo, algorytm nie jest zbieżny"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"slideshow": {
"slide_type": "notes"
}
},
"outputs": [],
"source": [
"def plot_perceptron():\n",
" plt.figure(figsize=(12,3))\n",
"\n",
" plt.subplot(131)\n",
" plt.ylim(-0.2,1.2)\n",
" plt.xlim(-0.2,1.2)\n",
"\n",
" plt.title('AND')\n",
" plt.plot([1,0,0], [0,1,0], 'ro', markersize=10)\n",
" plt.plot([1], [1], 'go', markersize=10)\n",
"\n",
" ax = plt.gca()\n",
" ax.spines['right'].set_color('none')\n",
" ax.spines['top'].set_color('none')\n",
" ax.xaxis.set_ticks_position('none')\n",
" ax.spines['bottom'].set_position(('data',0))\n",
" ax.yaxis.set_ticks_position('none')\n",
" ax.spines['left'].set_position(('data',0))\n",
"\n",
" plt.xticks(np.arange(0, 2, 1.0))\n",
" plt.yticks(np.arange(0, 2, 1.0))\n",
"\n",
"\n",
" plt.subplot(132)\n",
" plt.ylim(-0.2,1.2)\n",
" plt.xlim(-0.2,1.2)\n",
"\n",
" plt.plot([1,0,1], [0,1,1], 'go', markersize=10)\n",
" plt.plot([0], [0], 'ro', markersize=10)\n",
"\n",
" ax = plt.gca()\n",
" ax.spines['right'].set_color('none')\n",
" ax.spines['top'].set_color('none')\n",
" ax.xaxis.set_ticks_position('none')\n",
" ax.spines['bottom'].set_position(('data',0))\n",
" ax.yaxis.set_ticks_position('none')\n",
" ax.spines['left'].set_position(('data',0))\n",
"\n",
" plt.title('OR')\n",
" plt.xticks(np.arange(0, 2, 1.0))\n",
" plt.yticks(np.arange(0, 2, 1.0))\n",
"\n",
"\n",
" plt.subplot(133)\n",
" plt.ylim(-0.2,1.2)\n",
" plt.xlim(-0.2,1.2)\n",
"\n",
" plt.title('XOR')\n",
" plt.plot([1,0], [0,1], 'go', markersize=10)\n",
" plt.plot([0,1], [0,1], 'ro', markersize=10)\n",
"\n",
" ax = plt.gca()\n",
" ax.spines['right'].set_color('none')\n",
" ax.spines['top'].set_color('none')\n",
" ax.xaxis.set_ticks_position('none')\n",
" ax.spines['bottom'].set_position(('data',0))\n",
" ax.yaxis.set_ticks_position('none')\n",
" ax.spines['left'].set_position(('data',0))\n",
"\n",
" plt.xticks(np.arange(0, 2, 1.0))\n",
" plt.yticks(np.arange(0, 2, 1.0))\n",
"\n",
" plt.show()"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA7YAAAEXCAYAAACZJdj6AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAm8UlEQVR4nO3deXRU9f3/8dfNnpCQEGgVDesIrVUOiyxWQEArqLS2Gpe6piqtiIDy07rUpUfleKhaPUlEVKBKFI5CAat1ibQGIa1lacBavwoYwiKiNgkhA2FJyOf3x2RuEpjsQ5LPzPNxTo5h5nPv/Qzcz9v7mnvv5zrGGCMAAAAAACwV0dEdAAAAAACgLQi2AAAAAACrEWwBAAAAAFYj2AIAAAAArEawBQAAAABYjWALAAAAALAawRYAAAAAYDWCLQAAAADAagRbAAAAAIDVCLZhaNasWXIcR47j6Mc//nGT7Xfs2OG2dxxH119/fZPL+Ns+9NBDJ7y3evXqeuvz/8TExOh73/ueBgwYoMsuu0yPPfaYNm3a1KrPCAAny5YtW/TQQw9p5MiROvXUUxUTE6NTTjlFI0aM0IMPPqgvvvii0eVfeeWVgDUwIiJCycnJGjRokKZOnaqCgoJ2+kQAcKL//e9/6tGjhxzH0bBhw1RVVdVo+08//VQxMTFyHEdXXnnlCe9XVVVp8eLFuuaaa+TxeJSYmKjExER5PB5dddVVevXVV1VZWdlkv/r27RuwhsbFxem0007TpEmT9Pzzz6uioqLVnx2WMggrlZWV5vvf/76R5P5s2bKl0WWKiorqtY+IiDD//e9/G13G3/bBBx884b28vLx662vqZ9SoUWbdunVt+twA0FZHjhwx06dPN1FRUY3WrMjISHPHHXeYI0eOBFzPyy+/3KzaFxERYR577LF2/pQAUOvVV191a9ITTzzRYLuqqiozfPhwI8mkpqaab775pt77a9asMQMHDmyy7g0cONCsWbOm0T716dOnWTV0wIABZvv27UH5e4AdOGMbZt577z1999139V7Lyclp0Tqqq6v1yCOPBKU/L7zwgrxer7xer8rLy7V7925t2LBBc+fO1bhx4yRJ69at0+jRo/Xiiy8GZZsA0FIHDx7UxRdfrOeee05VVVU666yz9OKLL+qLL75QSUmJtmzZovnz52vQoEE6duyY5s6dq4svvrjJMwbvvvuuWwP37dungoICPfroo4qLi3Nr7YoVK9rpUwJAfTfccIMuvfRSSdJjjz2mrVu3Bmz3zDPPaOPGje7vp5xyivveypUrddFFF2nr1q2KiorS7bffrtWrV+vrr7/W3r17tWbNGt1xxx2KiorS1q1bddFFF+nNN99ssm9jxoxx66fX69WuXbv07rvvauTIkZKkbdu26fLLL1d1dXUb/xZgjY5O1mhf6enpRpLp0aOHOe+884wk07t3b1NdXd3gMnXP2Ho8HiPJOI5jCgoKGlzG376pM7Yvv/xyo/3Nzc01PXr0cM+CvP/++83+rAAQLDfffLNbt2655RZz9OjRgO2OHj1qbr31VrftrbfeekKbumds8/LyAq7nz3/+s9vm3HPPDeZHAYAW2bVrl0lKSjKSzJgxY044Zty6dauJj483ksykSZPqvff555+bhIQE90zu+vXrG9zOhg0bTGpqqpFkunTpYr744ouA7fxnbMeNGxfw/aqqKjN48GC3hnLsGD44YxtGSktL9fbbb0uSrr32Wt16662SpF27dikvL69Z67jjjjvUvXt3GWP08MMPn7S++k2cOFFvvfWWoqKidOzYMd1555188wagXa1evVovv/yyJOnCCy/UggULFB0dHbBtdHS05s+frwsuuECStHDhQn300Uct3mZ6eroGDBggSdqwYYOOHj3ayt4DQNv06tVLc+bMkSTl5+fr+eefd98zxujXv/61Dh06pMTERL300kv1lp06daoqKirkOI6WL1+uESNGNLid4cOHa/ny5XIcRwcPHtTUqVNb1d/IyEjde++97p//8Y9/tGo9sA/BNoy8/vrr7sHRjTfeqKuuukrx8fGSpEWLFjVrHUlJSW6xeOedd/Svf/3r5HS2jh//+Me6+eabJfkmbfnrX/960rcJAH5PP/20+3t2drYcx2m0veM4eu655wIu3xJnnnmmJOnYsWMqLS1t1ToAIBhuv/12jR07VpL0wAMPaNeuXZJ8t5T5v7ybM2eOevfu7S6zceNG972rr75a48ePb3I748eP19VXXy3J96Xiv//971b1118/JZ1wCx5CF8E2jPjD6w9/+EONGDFCSUlJ+sUvfiFJWrFihQ4ePNis9UyfPl2nnnqqJLXLWVtJ7tllSfrggw/aZZsAcPDgQeXm5kqSxo4dW+9gqTFnnnmmxowZI0l6//33m11f66p7dUpycnKLlweAYHEcRwsWLFBcXJy8Xq9uu+027dq1S/fdd58k3/2u06ZNq7fM8uXL3d9/85vfNHtbv/71rwOuoyXq1s+UlJRWrQP2IdiGiS+++ELr16+X5Dtb63fTTTdJkg4cONDs4pGQkKAHHnhAkvS3v/1Na9asCXJvTzRs2DDFxcVJkj7++OOTvj0AkKR//etf7iMu/BPaNZe/fVVVldatW9fibfsfG+TxeNyrawCgowwcOFC///3vJfm+sDv//PPl9XoVFxenhQsXnnA1S35+viTfLRrnnXdes7czevRoRUVF1VtHS9V97NrZZ5/dqnXAPgTbMOE/W+s4jm644Qb39Ysuukg9e/as16Y5brvtNvXq1UuSAj6rNtiio6PVt29fSdLevXtP+vYAQJK2b9/u/n7WWWe1aNm67YuKilq07LJly/Tll19K8l0lAwCdwT333KOhQ4dKknbu3ClJ+v3vf6+BAwee0NZfP/v37++enGiOuLg49e/fX1LLa6fku33jySeflCSdcsopAZ+pi9BEsA0D1dXVeu211yT5ziDUvf8hMjJS119/vSQpLy/PvWeiKbGxsW6gXbt2rXup3snkv5Rk3759J31bACCp3r2t3bp1a9GydduXlJQEbHPo0CEdOHBABw4c0P79+7V582Y9/vjj7tU01157rWbMmNGKngNA8EVFRemXv/yl++dTTz1V99xzT8C2/vrZ0tpZd5mGaqfkC7D++nngwAHt3r1b77//vkaPHq3//Oc/6tq1q958880WhWrYjWAbBv7+97/rq6++klT/MmQ//wGUMUavvvpqs9d7yy23uN+oBeu5to0xxkhSkxO3AEBn0JxademllyopKUlJSUlKSUnR0KFD9cgjj+jYsWNasmSJlixZosjIyHboLQA07euvv9YTTzzh/vmbb77RW2+9FfTtNKd+5ufnu/UzKSlJvXv31iWXXKJ169ZpwoQJ2rZtm84999yg9w2dF8E2DPgvMY6Pjw94OcagQYM0ZMgQSVJOTk6z1xsVFeXea7F+/fqTUtjq2r9/vyQpNTX1pG4HAPzqnmkoKytr0bJ1ry5pad2qrKzUb3/7W23btq1FywHAyTRt2jTt379fsbGx6tevnyTf7RL+Y7S6/PWzpbVTqq2frT3my8vLa/WM9LAXwTbEeb1erVy5UpI0YsQIbd26VRs3bjzhx39T/9atW1v0CJ/rr79eP/zhDyX5ztr6z6oGW2VlpXufxWmnnXZStgEAx/MfuEnS//3f/7Vo2brt666nrry8PBljZIzRgQMHtHHjRl133XWSpD179ujyyy/X4cOHW9FzAAiuN954Q3/5y18k+Y75Fi1aJMdxtHfvXv32t789ob2/7hUWFurIkSPN3s6RI0fc+3Mbqp2S7/Y6f/08duyY9uzZo4ULF7pzxzz11FMtuhIR9iPYhrhly5apoqJCkrRmzRqNGDEi4E/dh223ZBKpyMhIPfroo5KkTz75RH/+85+D+wFq/Pvf/3aLYktm1gOAtjj33HPdS4FXr17domX97aOiopp1OVyXLl10zjnnaPHixZoyZYok6bPPPuOsA4AOV1JSopkzZ0qShgwZonvvvVdjx47VbbfdJklasGCB+8xaP/8jzyorK/XPf/6z2dv6xz/+ocrKynrraEpERIROO+003XLLLcrLy1NiYqIk6a677gp4NhmhiWAb4loSUv3eeOONFn2zdtVVV2nw4MGSfDPj1X12WLAsXLjQ/X3ixIlBXz8ABJKUlOTWnLVr12rLli3NWm7Lli3uYyomTZqkLl26tGi7zz77bL2zDnUnsQKA9nbXXXfpu+++U2RkpBYuXOg+jucPf/iDTj/9dBlj9Jvf/KbeFSbp6enu7/Pnz2/2thYsWBBwHc31gx/8wD3pUlpaqqeeeqrF64CdCLYhrKioSGvXrpUkzZgxw71co6Ef/9nWffv2teh+Wcdx9Nhjj0mSPv/8cy1evDion+Pjjz/WK6+8Ikn60Y9+pEsuuSSo6weAxtx9992SfBPY+WtpY45v19CMoY1JTEx05zAoLy/Xs88+2+J1AEAwvPfee+7TNe6++24NGzbMfa9r166aN2+eJN/tbP7jQUkaOXKke8b19ddfd49JG7N27Vq9/vrrknyXGg8fPrxVfZ4+fbr7FJDs7GyeqBEmCLYhLCcnxz2w8t+z1ZjJkyera9eu7rItcdlll2nkyJGS5H5LFgyrVq3SZZddpqqqKkVFRSkrK4tZkQG0qwsvvNCdPX7VqlWaOnWqqqqqAratqqrS1KlTtWrVKkm+2ePHjx/fqu3ecsst7vO7s7OzuZwOQLvzer2aOnWqJGnAgAEBj/F+9rOf6eqrr5bku8LkP//5j/veCy+8oPj4eBljdMUVV6igoKDBbW3atElXXHGFjDFKSEjQCy+80Op+x8TEuI+lLC8vV2ZmZqvXBXsQbEOUMcYNpx6Pp1n3d8XFxbmXfLz//vv67rvvWrTNxx9/XJJvkoDmOnLkiPv8Ma/Xqz179qigoEDz5s3ThAkTNHHiRBUXFys6Olrz5s3ThRde2KI+AUAwzJ07V+eff74k6aWXXtLQoUO1YMECbdu2TaWlpdq2bZv+9Kc/6ZxzztFLL70kSZowYYKys7Nbvc3o6Gg9/PDDknyzwnNgBqC93X///dq1a5ccx9H8+fMbfCZsdna2UlNTVVVVpSlTpujYsWOSpLPOOks5OTmKiYlRcXGxRo0apTvuuENr167VN998o2+//Vb5+fmaOXOmRo0apeLiYsXGxuq1115zJydtrZtvvtl9LGVmZqbKy8vbtD5YwCAkrVmzxkgyksxDDz3U7OVWrVrlLvfMM88YY4wpKipyX5s/f36jy48dO9ZtK8k8+OCDJ7TJy8ur16apn3PPPdds2LChZX8BABBkhw8fNrfffruJjIxstGZFRkaaadOmmcOHDwdcz8svv+y2zcvLa3SblZWV5owzzjCSTGpqqikvLz8JnwwATrR27VrjOI6RZG677bYm27/yyitubfvjH/9Y773Vq1ebAQMGNHnMN2DAAPPRRx81up0+ffoYSWbcuHFN9qluvZ09e3aT7WE3ztiGqLqTRjXnMmS/Cy64wJ2wpDUTT82ePbvFy/hFR0ere/fu8ng8+ulPf6pHH31Umzdv1scff9zqeywAIFhiY2P1/PPP67///a8eeOABDR8+XN/73vcUHR2tHj166JxzztH999+vTz/9VHPnzlVsbGybtxkVFaVHHnlEkm8SlOeee67N6wSAphw+fFhTpkyRMUann366nnzyySaXycjI0EUXXSTJ9zigHTt2uO+NGzdOn332mXJycnTllVeqX79+SkhIUEJCgvr27av09HTl5OTos88+c6+OCYYbb7xRAwcOlOSblO/AgQNBWzc6H8eYk/TgUQAAAAAA2gFnbAEAAAAAViPYAgAAAACsRrAFAAAAAFiNYAsAAAAAsBrBFgAAAABgNYItAAAAAMBq1gdbY4zKy8vFU4sAoD7qIwA0jBoJhBbrg63X61VycrK8Xm9HdwUAOhXqIwA0jBoJhBbrgy0AAAAAILwRbAEAAAAAViPYAgAAAACsRrAFAAAAAFiNYAsAAAAAsBrBFgAAAABgNYItAAAAAMBqBFsAAAAAgNUItgAAAAAAqxFsAQAAAABWI9gCAAAAAKxGsAUAAAAAWI1gCwAAAACwGsEWAAAAAGA1gi0AAAAAwGoEWwAAAACA1Qi2AAAAAACrEWwBAAAAAFYj2AIAAAAArEawBQAAAABYjWALAAAAALAawRYAAAAAYDWCLQAAAADAagRbAAAAAIDV2j3YbtmyRdnZ2frVr36lQYMGKSoqSo7jaPbs2S1fmTFSSYnv95IS35+BMGSMUXFFsXaU7VBxRbEMY8FKwayPxhiVVPjqY0lFCfsEwho10n5BPX4UNRJwGSMVF0s7dvj+a/FYiGrvDc6bN0+ZmZltW0lZmbRokZSdLRUW+l7r31/yeKQZM6SMDCklpa1dBTq9ssNlWrR5kbLXZ6twX6H7uqebRzNGzlDGkAylxKV0XAfRIsGoj/X2ib2+faJ/Zn95erJPIPxQI0NHUI4fRY0EXIHylGR1nmr3M7Znn3227rnnHi1evFiff/65brzxxpatIDdXSkuTZs2Stm+v/9727b7X09J87YAQlvtlrtKeSdOs3Fnavq/+WNi+b7tm5c5S2jNpyv2SsWCLttZH9gmgFuMhtLT5+FHsE4ArRPNUu5+xnTJlSr0/R0S0IFvn5kqTJ/tOkQc6Te5/7dAhX7t33pEmTWpDb4HOKffLXE1eMlnGGBmdOBb8rx2qPKTJSybrneve0aQzGAudXVvqI/sEUIvxEHradPwo9gnAFcJ5yp7Jo8rKpPR03192dXXjbaurfe3S033LASGk7HCZ0pemyxijajU+FqpVLWOM0pemq+xwWft0EO2OfQKoxXjA8dgngBohnqfsCbaLFkkVFU3/I/hVV/va5+Sc3H4B7WzR5kWqqKxo8n/OftWqVkVlhXI+YSyEKvYJoBbjAcdjnwBqhHiesiPYGuO7sbk1srKsnt0LqMsYo+z1rRsLWeuymPUxBLFPALUYDzge+wRQIwzylB3BtqTEN1tXS/9CjfEtV1p6cvoFtLOSQyUq3FcY8P6gxhgZFe4rVOkhxkKoYZ8AajEecDz2CaBGGOQpO4LtgQNtW97rDU4/gA524GjbxoL3KGMh1LBPALUYDzge+wRQIwzylB3BNjGxbcsnJQWnH0AHS4xp21hIimEshBr2CaAW4wHHY58AaoRBnrIj2Hbv7ntYsOO0bDnH8S2Xmnpy+gW0s+7x3eXp5pGjlo0FR4483TxKjWcshBr2CaAW4wHHY58AaoRBnrIj2DqONGNG65adObPl/4BAJ+U4jmaMbN1YmDlqphzGQshhnwBqMR5wPPYJoEYY5Ck7gq0kZWRICQlScx/IHRHha3/TTSe3X0A7yxiSoYToBEU0c/hGOBFKiE7QTYMZC6GKfQKoxXjA8dgngBohnqfsCbYpKdLy5b5vC5r6x4iI8LVbscK3HBBCUuJStPzq5XIcp8n/SUcoQo4crbhmhVLiUtqng2h37BNALcYDjsc+AdQI8TzlmHZ+QFdBQYGmTZvm/rmwsFDFxcVKS0vT6aef7r6+cuVK9ezZ88QV5OZK6em+hwVLKjdGyZL2S+rqP0WekOD7R5g48SR+EqBj5X6Zq/Sl6aqo9I2Fuo8y8N9LlBCdoBXXrNBED2PBBm2tjyfsE4eNNEfS/ZITxz6B8EKNDC1tPn4UNRJwHZen6j0CyOI81e7BdvXq1ZowYUKT7YqKitS3b9/Ab5aVSTk5UlaWygsLa4Otx+O7BjwjQ0pODmKvgc6p7HCZcj7JUda6LBXuK3Rf93TzaOaomcoYnKHkOMaCLYJRH+vtE3sL3YM2T0/2CYQfamToCMrxo6iRgKtOnlJhbX2UxXmq3YNtUBmj8p07ldyvn/YXFalrnz5W3NgMBJsxRqWHSuU96lVSTJJS41OZ8CLMGWO089ud6tezn4r2FqnPKX3YJxC2qJE4HjUSqGGMVFrqe05tUpJv9mNLx4LdwVZSeXm5kpOTtX//fnXt2rWjuwMAnQb1EQAaRo0EQos9k0cBAAAAABAAwRYAAAAAYDWCLQAAAADAagRbAAAAAIDVCLYAAAAAAKsRbAEAAAAAViPYAgAAAACsRrAFAAAAAFiNYAsAAAAAsBrBFgAAAABgNYItAAAAAMBqBFsAAAAAgNUItgAAAAAAqxFsAQAAAABWI9gCAAAAAKxGsAUAAAAAWI1gCwAAAACwGsEWAAAAAGA1gi0AAAAAwGoEWwAAAACA1Qi2AAAAAACrEWwBAAAAAFYj2AIAAAAArEawBQAAAABYjWALAAAAALAawRYAAAAAYDWCLQAAAADAagRbAAAAAIDVCLYAAAAAAKsRbAEAAAAAViPYAgAAAACsRrAFAAAAAFiNYAsAAAAAsBrBFgAAAABgNYItAAAAAMBqBFsAAAAAgNUItgAAAAAAqxFsAQAAAABWI9gCAAAAAKxGsAUAAAAAWI1gCwAAAACwGsEWAAAAAGA1gi0AAAAAwGoEWwAAAACA1Qi2AAAAAACrEWwBAAAAAFYj2AIAAAAArEawBQAAAABYjWALAAAAALAawRYAAAAAYDWCLQAAAADAagRbAAAAAIDVCLYAAAAAAKsRbAEAAAAAViPYAgAAAACsRrAFAAAAAFiNYAsAAAAAsBrBFgAAAABgNYItAAAAAMBqBFsAAAAAgNUItgAAAAAAqxFsAQAAAABWI9gCAAAAAKxGsAUAAAAAWI1gCwAAAACwGsEWAAAAAGA1gi0AAAAAwGoEWwAAAACA1Qi2AAAAAACrEWwBAAAAAFYj2AIAAAAArEawBQAAAABYjWALAAAAALAawRYAAAAAYDWCLQAAAADAagRbAAAAAIDVCLYAAAAAAKsRbAEAAAAAViPYAgAAAACsRrAFAAAAAFiNYAsAAAAAsBrBFgAAAABgNYItAAAAAMBqBFsAAAAAgNUItgAAAAAAqxFsAQAAAABWI9gCAAAAAKxGsAUAAAAAWI1gCwAAAACwGsEWAAAAAGA1gi0AAAAAwGoEWwAAAACA1Qi2AAAAAACrEWwBAAAAAFYj2AIAAAAArEawBQAAAABYjWALAAAAALAawRYAAAAAYDWCLQAAAADAagRbAAAAAIDVCLYAAAAAAKsRbAEAAAAAViPYAgAAAACsRrAFAAAAAFiNYAsAAAAAsBrBFgAAAABgNYItAAAAAMBqBFsAAAAAgNUItgAAAAAAqxFsAQAAAABWI9gCAAAAAKxGsAUAAAAAWI1gCwAAAACwGsEWAAAAAGA1gi0AAAAAwGoEWwAAAACA1Qi2AAAAAACrEWwBAAAAAFYj2AIAAAAArEawBQAAAABYjWALAAAAALAawRYAAAAAYDWCLQAAAADAagRbAAAAAIDVCLYAAAAAAKsRbAEAAAAAViPYAgAAAACsRrAFAAAAAFiNYAsAAAAAsBrBFgAAAABgNYItAAAAAMBqBFsAAAAAgNUItgAAAAAAqxFsAQAAAABWI9gCAAAAAKxGsAUAAAAAWI1gCwAAAACwGsEWAAAAAGA1gi0AAAAAwGoEWwAAAACA1Qi2AAAAAACrEWwBAAAAAFYj2AIAAAAArEawBQAAAABYjWALAAAAALAawRYAAAAAYDWCLQAAAADAah0WbJctW6bx48erW7du6tKliwYPHqwnn3xSlZWVzV+JMVJJie/3khLfn4FwZIxUXCzt2OH7L2PBatRHILiMMSquKNaOsh0qriiWYTxYKyj1UaJGAjVCqT46pgN6f9dddykzM1NRUVG64IILlJiYqA8//FBlZWUaM2aMPvjgA8XHxze8grIyadEiKTtb5YWFSpa0X1JXj0eaMUPKyJBSUtrnwwAdqc5YUGFh7euMBWtRH4HgKTtcpkWbFyl7fbYK99XWSE83j2aMnKGMIRlKiUvpuA6iRdpcHyVqJFAjFOtjuwfbN998U5dffrkSExP10UcfadiwYZKk4uJiXXDBBfr0009199136+mnnw68gtxcKT1dqqiQJJUbU1uUHMfXJiFBWr5cmjTp5H8goKMcNxbqfdvMWLAS9REIntwvc5W+NF0Vlb7xYFRbIx35xkNCdIKWX71ck85gPHR2ba6PEjUSqBGq9bHdg+3IkSO1YcMGzZ49Ww8++GC99/Lz8zV27FjFxsbq22+/VXJycv2Fc3OlyZN9B/DV1ZKkcqm2KPnbRUT4DuzfeYfChNAUYCwExFiwCvURCI7cL3M1eclkGWNUrYZrZIQi5DiO3rnuHasO3sJRm+qjRI0EaoRyfWzXYLtnzx6lpaVJkrZv365+/fqd0KZ3797avXu3lixZomuvvbb2jbIyKS1NOnSo3oF8wKIk+QpTfLz01VdcUoLQ0sBYaBBjwQrURyA4yg6XKe2ZNB2qPNToQZtfhCIUHx2vr/7fV9Zddhcu2lQfJWokUCPU62O7Th61adMmSVJqamrAoiRJw4cPr9fWtWiR79KR5hzIS752FRVSTk6r+wt0SoyFkER9BIJj0eZFqqisaNZBmyRVq1oVlRXK+YTx0Fm1qT5K1EigRqjXx3YNtkVFRZJ836o1pFevXvXaSvJdNpKd3bqNZmUx0x1CB2MhZFEfgbYzxih7fevGQ9a6LKtnAw1lra6PEjUSqBEO9TGqOY2MMfJ6vW3e2P/+9z9JUmxsrMrLywO2iY6OliSVlpbWtikpqT/jax3lx/23HmN8y+3cKaWmtqHnQCfRyFhoVCcaC0lJSXL8k3SEAOoj0HmUVJSocG/La6SRUeHeQu38dqdSEzpuPFAfA2t1fZSokUCNcKiPzbrHtry8PPCN+ADQQvv371fXrl2bbmgJ6iOAYKE+AkBgzamPzQq2wfrG7YUXXtB9992nQYMGKT8/P2Cbe++9Vy+++KJ+/vOfK8d/b0NJidS/f8D25ZJ6Sdqt4278r6uoiG/bEBoaGQvN0gnGAmckAqM+Am1XUlGi/pmtr5FFdxZ1+jMSNunw+ihRI4Ea4VAfm3UpsuM4QfkG8cwzz5Qkff311w2u77vvvpMkDRgwoLZNUpLk8Ujbtzd4r0NXBShKjuMrZn361D7XE7BZM8ZCQIyFk4b6CHQeSUlJ8vT0aPu+7fWey9gUR476d+uvPqf0Calg2dE6vD5K1EigRjjUx3adPGro0KGSpJKSkhNv7q+xceNGSXIfvC3JV1BmzGjdRmfOpCAhdDAWQhb1EWg7x3E0Y2TrxsPMUTM7/UFbuGp1fZSokUCNcKiP7Rps09LSNGLECEnSkiVLTng/Pz9fu3fvVmxsrC699NL6b2ZkSAkJvmeLNUdEhK/9TTe1tdtA58JYCEnURyA4MoZkKCE6QRHNPMSJcCKUEJ2gmwYzHjqrNtVHiRoJ1Aj1+tiuwVaSfve730mS5syZo4KCAvf1kpISTZs2TZI0ffr0EycbSEmRli/3fXPWVGGKiPC1W7GCB2sj9DAWQhb1EWi7lLgULb96uRzHafLgLUIRcuRoxTUrlBKX0j4dRKu0uj5K1EigRqjXx2ZNHhVsd955p7KyshQdHa0LL7xQXbp00d///neVlZVp9OjRWrVqleLj4wMvnJsrpaf7HpwtqdwYJUvaL6mr/xR5QoKvIE2c2C6fB+gQx42FevcOMRasRX0EgiP3y1ylL01XRaVvPNS9p8yRbzwkRCdoxTUrNNHDeLBBm+qjRI0EaoRqfeyQYCtJS5cu1dy5c7V582ZVVlbK4/Hohhtu0KxZsxQTE9P4wmVlUk6OlJWl8sLC2qLk8fjuh8jIkJheHuGgzlio95w+xoLVqI9AcJQdLlPOJznKWpelwn21NdLTzaOZo2YqY3CGkuMYDzZpU32UqJFAjVCsjx0WbIPCGJXv3Knkfv20v6hIXZm5DuHKGKm0VPJ6fTNApqYyFsId9RFwGWNUeqhU3qNeJcUkKTU+1YqJUHASUSMBSaFVH5v1uJ9Oy3Fqny3GgTzCmeNI3bv7fgCJ+gjU4TiOuid0V/cEaiRqUCMBSaFVH9t98qhgWbZsmcaPH6/evXtLks477zw9+eSTqqys7OCeAe1ny5Ytys7O1q9+9SsNGjRIUVFRchxHs2fP7uiuoYMtW7bMnR301FNP1eDBg6mRCCvURzSEY0iEu1Ctj1aesb3rrruUmZmpqKgonX/++frwww/11Vdf6b777tPbb7+tDz74oPHJA4AQMW/ePGVmZnZ0N9DJ+GtkZGSkJOknP/mJ1q5dS41EWKE+IhCOIYHQrY/WnbF98803lZmZqcTERK1bt04rV66UJG3atEmDBg1Sfn6+Hn744Q7uJdA+zj77bN1zzz1avHixPv/8c914440d3SV0sLo18sMPP5Qkvfbaa9q2bRs1EmGF+ojjcQwJ+IRqfbTujO0TTzwhSbr//vs1bNgwlZeXS5K6d++u559/XmPHjtVzzz2nhx9+OPCzzIAQMmXKlHp/jmjuw+cRsurWyCFDhriv9+jRgxqJsEJ9xPE4hgR8QrU+WvUp9uzZow0bNkiSrrvuuhPeHzNmjHr16qUjR47o3Xffbe/uAUCHokYCQGDURyD0WRVsN23aJElKTU1Vv379ArYZPnx4vbYAEC6okQAQGPURCH1WBduioiJJcmexC6RXr1712gJAuKBGAkBg1Ecg9FkVbL1erySpS5cuDbZJTEyUJPe+CQAIF9RIAAiM+giEPquCLQAAAAAAx7Mq2CYlJUmSDh482GCbAwcOSJK6du3aLn0CgM6CGgkAgVEfgdBnVbDt27evJGn37t0NtvG/528LAOGCGgkAgVEfgdBnVbAdOnSoJKmkpKTBG/s3btwoSRo2bFi79QsAOgNqJAAERn0EQp9VwTYtLU0jRoyQJC1ZsuSE9/Pz87V7927Fxsbq0ksvbe/uAUCHokYCQGDURyD0WRVsJel3v/udJGnOnDkqKChwXy8tLdW0adMkSdOnT1dycnKH9A8AOlLdGrl582b39ZKSEmokgLDGMSQQ2hxjjOnoTrTUnXfeqaysLEVHR2vcuHH629/+puTkZO3fv1+jR4/WqlWrFB8f39HdBE66goIC93/GklRYWKji4mKlpaXp9NNPd19fuXKlevbs2RFdRAfw18ioqChVVVXpsssu05o1a1RWVkaNRNigPiIQjiGB0K2PVgZbSVq6dKnmzp2rTZs2yev16swzz1RGRoZmzZqlmJiYju4e0C5Wr16tCRMmNNmuqKiIyTDCzNKlS5WZmal//vOfiouL0xlnnKEbbriBGomwQX1EQziGRLgL1fpobbD1Ky8vd79pY3p2AKhFfQSAhlEjgdBi3T22AAAAAADURbAFAAAAAFiNYAsAAAAAsBrBFgAAAABgNYItAAAAAMBqBFsAAAAAgNUItgAAAAAAq1n/HFtjjLxer5KSkuQ4Tkd3BwA6DeojADSMGgmEFuuDLQAAAAAgvHEpMgAAAADAagRbAAAAAIDVCLYAAAAAAKsRbAEAAAAAViPYAgAAAACsRrAFAAAAAFiNYAsAAAAAsNr/B+PU5KM38gioAAAAAElFTkSuQmCC\n",
"text/plain": [
"![\"Rys.](\"reglin.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"#### Uczenie regresji liniowej:\n",
"* Model: $$h_{\\theta}(x) = \\sum_{i=0}^n \\theta_ix_i$$\n",
"* Funkcja kosztu (błąd średniokwadratowy): $$J(\\theta) = \\frac{1}{m} \\sum_{i=1}^{m} (h_{\\theta}(x^{(i)}) - y^{(i)})^2$$\n",
"* Po obliczeniu $\\nabla J(\\theta)$ - zwykły SGD."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Perceptron a dwuklasowa regresja logistyczna\n",
"\n",
"![\"Rys.](\"reglog.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"#### Uczenie dwuklasowej regresji logistycznej:\n",
"* Model: $h_{\\theta}(x) = \\sigma(\\sum_{i=0}^n \\theta_ix_i) = P(1|x,\\theta)$\n",
"* Funkcja kosztu (entropia krzyżowa): $$\\begin{eqnarray} J(\\theta) &=& -\\frac{1}{m} \\sum_{i=1}^{m} \\big( y^{(i)}\\log P(1|x^{(i)},\\theta) \\\\ && + (1-y^{(i)})\\log(1-P(1|x^{(i)},\\theta)) \\big) \\end{eqnarray}$$\n",
"* Po obliczeniu $\\nabla J(\\theta)$ - zwykły SGD."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Perceptron a wieloklasowa regresja logistyczna\n",
"\n",
"![\"Rys.](\"multireglog.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"#### Wieloklasowa regresja logistyczna\n",
"* Model (dla $c$ klasyfikatorów binarnych): \n",
"$$\\begin{eqnarray}\n",
"h_{(\\theta^{(1)},\\dots,\\theta^{(c)})}(x) &=& \\mathrm{softmax}(\\sum_{i=0}^n \\theta_{i}^{(1)}x_i, \\ldots, \\sum_{i=0}^n \\theta_i^{(c)}x_i) \\\\ \n",
"&=& \\left[ P(k|x,\\theta^{(1)},\\dots,\\theta^{(c)}) \\right]_{k=1,\\dots,c} \n",
"\\end{eqnarray}$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* Funkcja kosztu (**przymując model regresji binarnej**): $$\\begin{eqnarray} J(\\theta^{(k)}) &=& -\\frac{1}{m} \\sum_{i=1}^{m} \\big( y^{(i)}\\log P(k|x^{(i)},\\theta^{(k)}) \\\\ && + (1-y^{(i)})\\log P(\\neg k|x^{(i)},\\theta^{(k)}) \\big) \\end{eqnarray}$$\n",
"* Po obliczeniu $\\nabla J(\\theta)$, **c-krotne** uruchomienie SGD, zastosowanie $\\mathrm{softmax}(X)$ do niezależnie uzyskanych klasyfikatorów binarnych."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* Przyjmijmy: \n",
"$$ \\Theta = (\\theta^{(1)},\\dots,\\theta^{(c)}) $$\n",
"\n",
"$$h_{\\Theta}(x) = \\left[ P(k|x,\\Theta) \\right]_{k=1,\\dots,c}$$\n",
"\n",
"$$\\delta(x,y) = \\left\\{\\begin{array}{cl} 1 & \\textrm{gdy } x=y \\\\ 0 & \\textrm{wpp.}\\end{array}\\right.$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* Wieloklasowa funkcja kosztu $J(\\Theta)$ (kategorialna entropia krzyżowa):\n",
"$$ J(\\Theta) = -\\frac{1}{m}\\sum_{i=1}^{m}\\sum_{k=1}^{c} \\delta({y^{(i)},k}) \\log P(k|x^{(i)},\\Theta) $$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* Gradient $\\nabla J(\\Theta)$:\n",
"$$ \\dfrac{\\partial J(\\Theta)}{\\partial \\Theta_{j,k}} = -\\frac{1}{m}\\sum_{i = 1}^{m} (\\delta({y^{(i)},k}) - P(k|x^{(i)}, \\Theta)) x^{(i)}_j \n",
"$$\n",
"\n",
"* Liczymy wszystkie wagi jednym uruchomieniem SGD"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Perceptron – podsumowanie\n",
"\n",
"* W przypadku jednowarstowej sieci neuronowej wystarczy znać gradient funkcji kosztu.\n",
"* Wtedy liczymy tak samo jak w przypadku regresji liniowej, logistycznej, wieloklasowej logistycznej itp. (wymienione modele to szczególne przypadki jednowarstwowych sieci neuronowych).\n",
"* Regresja liniowa i binarna regresja logistyczna to jeden neuron.\n",
"* Wieloklasowa regresja logistyczna to tyle neuronów, ile klas."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Funkcja aktywacji i funkcja kosztu są **dobierane do problemu**."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## 10.2. Funkcje aktywacji"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"* Operacje, które każdy neuron wykonuje, to dodawanie i mnożenie. Są to **funkcje liniowe**."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* Złożenie funkcji liniowych jest funkcją liniową."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* Gdyby nie było funkcji aktywacji, cała sieć neuronowa działałaby jak regresja liniowa."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* **Najważniejszym celem** stosowania funkcji aktywacji jest **wprowadzenie nieliniowości**, dzięki której sieć może uczyć się zależności bardziej skomplikowanych niż tylko liniowe."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Jakie cechy powinna mieć dobra funkcja aktywacji?"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* **różniczkowalna** (co najmniej **przedziałami różniczkowalna**), aby można było obliczyć jej gradient"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* **prosta obliczeniowo**, ponieważ podczas uczenia sieci jej wartość obliczana jest wiele razy"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"* odporna na **problem zanikającego gradientu** (wyjaśnienie kilka slajdów dalej)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"slideshow": {
"slide_type": "notes"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-05-04 10:23:21.183242: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA\n",
"To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
"2023-05-04 10:23:22.176521: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory\n",
"2023-05-04 10:23:22.176596: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n",
"2023-05-04 10:23:24.881062: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory\n",
"2023-05-04 10:23:24.881370: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory\n",
"2023-05-04 10:23:24.881390: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n"
]
}
],
"source": [
"%matplotlib inline\n",
"\n",
"import math\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"import random\n",
"\n",
"import keras\n",
"from keras.datasets import mnist\n",
"from keras.models import Sequential\n",
"from keras.layers import Dense, Dropout, SimpleRNN, LSTM\n",
"from keras.optimizers import Adagrad, Adam, RMSprop, SGD\n",
"\n",
"from IPython.display import YouTubeVideo"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"slideshow": {
"slide_type": "notes"
}
},
"outputs": [],
"source": [
"def plot(fun):\n",
" x = np.arange(-3.0, 3.0, 0.01)\n",
" y = [fun(x_i) for x_i in x]\n",
" fig = plt.figure(figsize=(14, 7))\n",
" ax = fig.add_subplot(111)\n",
" fig.subplots_adjust(left=0.1, right=0.9, bottom=0.1, top=0.9)\n",
" ax.set_xlim(-3.0, 3.0)\n",
" ax.set_ylim(-1.5, 1.5)\n",
" ax.grid()\n",
" ax.plot(x, y)\n",
" plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Funkcja logistyczna"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"$$ g(x) = \\frac{1}{1 + e^{-x}} $$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"#### Funkcja logistyczna – wykres"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAABLoAAAJpCAYAAACq4lvbAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABYv0lEQVR4nO3de5zddWHn//dM5n6fScj9yl3lYsJFukDLRSteoGhXu4vW0tV1t6xVxK6Cl12sVuyu7Ra1uN3VVldl7cVisVJXVOpPpMZAEFEkCCQkIZDbJHOfyczk/P6YycCYBBIyyUm+83w+Hudx5nzP93zP5/j4OE5efr7fU1EqlUoBAAAAgGNcZbkHAAAAAABTQegCAAAAoBCELgAAAAAKQegCAAAAoBCELgAAAAAKQegCAAAAoBCELgAAAAAKQegCAAAAoBCELgAAAAAKoayha82aNfnUpz6Vq6++OqeffnqqqqpSUVGRj370oy/oeDfeeGMqKiqe8/bwww9P8acAAAAA4GhQVc43/8xnPpObb755yo975pln5qUvfek+n2ttbZ3y9wMAAACg/Moauk477bT8wR/8QZYvX54VK1bkYx/7WL74xS8e8nGvvPLK3HjjjYc+QAAAAACOGWUNXW9729smPa6sdMkwAAAAAF4YZQkAAACAQijriq7DZfXq1bn++uvT2dmZ1tbWLF++PJdffnmam5vLPTQAAAAADpNChq6vf/3r+frXvz5pW2traz75yU/mLW95y/O+fmhoKENDQxOPd+/enc7OzsycOTMVFRVTPl4AAACA6a5UKqWnpyfz589/wZe3KlToOuGEE/Kxj30sr3rVq7JkyZIkyUMPPZSPf/zj+cd//Mf8zu/8TmbMmJE3velNz3mcm266KR/+8IePxJABAAAAeJYNGzZk4cKFL+i1FaVSqTTF43nBrr766nzhC1/IRz7ykXzwgx+c0mO/853vzKc+9akcd9xx2bhxY2pqava77y+v6Orq6srixYvzyCOPpKOjY0rHBUeT4eHh3HXXXbn44otTXV1d7uHAYWGeM12Y60wX5jrTgXnOdNHZ2ZmTTz45O3fuTGtr6ws6RqFWdD2XG2+8Mbfccku2bt2alStX5sILL9zvvrW1tamtrd1re0dHR2bOnHk4hwllNTw8nIaGhsycOdP/gFJY5jnThbnOdGGuMx2Y50w3h3LZqGnzrYsdHR2ZPXt2kmTjxo1lHg0AAAAAU23ahK7R0dF0dXUliW9fBAAAACigaRO6br/99vT396eioiJnn312uYcDAAAAwBQ75kLXpz/96Zx66ql5y1veMmn7+vXr86UvfSmDg4N7veZrX/ta3va2tyVJ3vSmN2Xu3LlHZKwAAAAAHDllvRj96tWrc80110w8fuyxx5Ikf/EXf5F//Md/nNh+2223Zd68eUmSbdu2Zc2aNXvFqs7Ozvz2b/92fu/3fi/Lly/PggULMjAwkIceeii/+MUvkiQXX3xxPvOZzxzujwUAAABAGZQ1dHV3d2flypV7bd+4ceOkC8YPDQ0977EWLVqU973vfVm1alUeffTRrF69Ort27cqsWbPy2te+NldddVV+67d+K5WVx9wiNgAAAAAOQFlD10UXXZRSqXRQr7nxxhtz44037rV95syZ+fjHPz5FIwMAAADgWGN5EwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhlDV1r1qzJpz71qVx99dU5/fTTU1VVlYqKinz0ox89pON++9vfzqtf/erMmjUr9fX1OfXUU/OBD3wgvb29UzRyAAAAAI42VeV888985jO5+eabp/SY/+N//I9cd911qaioyIUXXpg5c+bk+9//fj72sY/lq1/9au6+++7MmjVrSt8TAAAAgPIr64qu0047LX/wB3+QL3/5y/n5z3+e3/7t3z6k491///15z3vekxkzZuQb3/hGvve97+Vv/uZv8thjj+XSSy/NmjVr8h//43+cotEDAAAAcDQp64qut73tbZMeV1YeWne76aabUiqV8ru/+7t51ateNbG9oaEhn/vc53L88cfnq1/9ah5++OGceuqph/ReAAAAABxdCnMx+l27duUb3/hGkuSqq67a6/klS5bk/PPPT5LcdtttR3RsAAAAABx+hQldjzzySPr7+5MkZ5999j732bP9/vvvP2LjAgAAAODIKEzoWrt2bZKkra0tzc3N+9xn0aJFk/YFAAAAoDjKeo2uqdTT05MkaWxs3O8+TU1NSZLu7u7nPNbQ0FCGhoYmHu/Zf3h4OMPDw4c6VDhq7Znf5jlFZp4zXZjrTBfmOtOBec50MRVzvDChayrddNNN+fCHP7zX9rvuuisNDQ1lGBEcWXfeeWe5hwCHnXnOdGGuM12Y60wH5jlFt+eSVIeiMKFrz+mKfX19+92nt7c3SdLS0vKcx7rhhhty3XXXTTzu7u7OokWLcvHFF2fmzJlTMFo4Og0PD+fOO+/MK17xilRXV5d7OHBYmOdMF+Y604W5znRgnjNdbN++/ZCPUZjQtXTp0iTJzp0709PTs8/rdG3YsGHSvvtTW1ub2travbZXV1f7pcK0YK4zHZjnTBfmOtOFuc50YJ5TdFMxvwtzMfpTTjll4rTCe++9d5/77Nm+YsWKIzYuAAAAAI6MwoSumpqavOY1r0mS3HrrrXs9/8QTT+See+5Jkrzuda87omMDAAAA4PA75kLXpz/96Zx66ql5y1vestdz119/fSoqKvJXf/VX+eY3vzmxvb+/P29961szOjqa3/zN38ypp556JIcMAAAAwBFQ1mt0rV69Otdcc83E48ceeyxJ8hd/8Rf5x3/8x4ntt912W+bNm5ck2bZtW9asWZO5c+fudbwVK1bkT/7kT3Ldddfl1a9+dX7t134ts2fPzve///089dRTOeWUU/I//+f/PMyfCgAAAIByKGvo6u7uzsqVK/favnHjxmzcuHHi8dDQ0AEf893vfndOP/30/Mmf/El+9KMfpa+vL4sXL84NN9yQG264YZ8XqQcAAABgapVKpQwO707frpH0DY2kb2g0/btG0js0kv5do2P3QyPp2zWaweHRXH3WrEN+z7KGrosuuiilUumgXnPjjTfmxhtvfM59Xv7yl+flL3/5IYwMAAAAYHoplUoZGtmdnsGxGNU7OJKeoeH0Do6kb9dIeodGJ8JU39DIeLTas20sZO2JWv3jP+8+iOzzppd2HPJnKGvoAgAAAODQlEqlDAyPjoep8UA1OJLeoeFJ0ap36JnnnwlZI+kZHJ54PHIwZeogNNTMSGNtVRprZqShpipNtVVpqJ28bfcUvLfQBQAAAFBGQyOj6R4YSffgcLoHhtM9OJLugeF0DQyPb3vmub3C1Xikmso+VVGRNNVUpamuKs11VWmsHQtTjTXjcaqmaiJQNdZWpbH2WfFqYtszz9dXz0hlZcXzvu/27dsPeexCFwAAAMAh2DWyOz2De8LUyHisGv6lePXM466ByUFraGT3lIyjsiJprqtOU+1YoGqqHYtVkx7XVk8ErOa9nh97ruEAw9TRSOgCAAAASDI4PJqd/cPZ0b8rO/uH0zWwKzv6h7Ozfzg7B3ala9JzYz93D4xkYHj0kN+7oiJprq1KS311Wuqq01JfNX5fndbxbc11Y8//csjaE6zqq2ekouLYDFRTRegCAAAACuWFBKud/Ye+sqqptiot4zFqX8Gqpa5qLFrt47nm2qpjdhXV0UToAgAAAI5au3eXsnNgOJ19uybddvTv2mvbntuhrLCaUVmRtvrqtDZUp72hZuLntvqatDdUp62hOq3j29vGt7fUj62uqppROYWfnBdC6AIAAACOmMHh0WzrHcqOvuF09u9KZ99QOvuGJ90/89yu7Ozf9YIutF5VWTEWpeqr0zYRpmrG49R4pJp4PH7fMHZa4HQ//e9YJnQBAAAAL1ipVErv0Ei29e7Ktt6hbO8dytbeXdnWMzT+eGz72G1XeodGXtD7tNRVpaOxZuLW3lCTjqaadDTUTNre0ViT9saaNAtW05LQBQAAAEyy53TBbb1DY8Gq75lwNTlejd0f7LWtamZUTgSpmc++b6hJR2N1Ohpr095YnZnj9+0NNal2WiAHQOgCAACAaWJ4dHe29gxlS89QtnQPjt33DGVrz2C2dI9v7xnMtt5dGT3I8wUba2ZkVnNtZjXVZmZjzcTPxzXVZGbT2M+zmsa2W23F4SJ0AQAAwDFucHh0PFQN7hWxnv24s2/XQR23raF6LFo11WZWc22O+6WINatp/Lmm2tTXzDhMnw4OnNAFAAAAR6ndu0vZ2jOU9b3Jt3++Jdv6hvNU12Ce7h7M012D2TwesHoGD/y6V1WVFTmuuTazm2tzXHNdZreM/Ty7uS6zm2szp6UuxzXXpqOxJjVVThfk2CJ0AQAAQBkMjYytwnq6ezBPdQ1mc9f4ffdgnuoayObuoWzuHszI7lKSquTBHz/n8eqqKydi1Vi8qpsIWrNbxrc316a9oSaVlU4bpJiELgAAAJhig8OjebprME/uHMimnQN5+lmrsPbcbz/A0wgrK5LmqlKWzGnNvNb6zGutz5yWusxtrc2cPSuyWupc9woidAEAAMBBKZVK2d63K5vGI9aTOwcnft7zeFvv0AEdq6aqMvNa6zKnpS7zWusyt6Uuc59931qXttrKfOv/fTOvfvV5qa6uPsyfDo5tQhcAAAA8y+DwaJ7qGhyPVs8OWAPZNB61hkZ2P+9x6qtnZEF7fea1Pjti1Wdua23mttRnbmtd2huqn3cV1vDw8FR9NCg8oQsAAIBpZWDXaJ7c2Z8NnQPZsKM/G3cMZENn/0TU2tb7/KcUVlQks5trM7+tPvPb6rOgrT7zW+smPW47gIgFTC2hCwAAgELZNbI7T3UNTISsDZ3jMWvHWNw6kNMKG2pmjMWriXA1OWLNaanzjYRwFBK6AAAAOKaM7i5lc/dgNnT2Z8P4aqw9IWtjZ3+e7h7M7tJzH6O5tioLOxqyqL0+izoasrB9LGAtGL9vrbcaC45FQhcAAABHncHh0azv7M8T2/vzxPa+sfvO/qzf3pcndw5kePS5S1ZddWUWtk8OWYvaG7KooyGL2hvSUu8bCqGIhC4AAADKomtgOOu392fd9r7xqNWXddv7s3772Kqs51I9oyLz2/bEq/osbB+PWeMha1ZTjZAF05DQBQAAwGFRKpWytXdoPGaNrcZa96yVWTv6n/vbBJvrqrJ0ZmMWz2zI0pkNWdIx9vPijobMaanLjEohC5hM6AIAAOCQdA0MZ922vqzd1pfHx+/XbuvN2q196ds1+pyvndVUm6UzG8ZjVmOWzGzIkpmNWdLR4FsLgYMmdAEAAPC89lwz6/GtzwpZ41FrW++u/b6usiKZ11qfpbMasrijcWxl1njMWtzRkMZa/ywFpo7fKAAAACQZ+zbDTTsHJgLWMyu0erNxx0BKz3H99+Oaa3P8rMYcf1xjls1qzLJZTVk2qzGLOupTWzXjyH0IYFoTugAAAKaZweHRPL61L49u7c1jW3on7h/f1pddI7v3+7qm2qpnhayx2/GzmrJ0VkOa66qP4CcA2DehCwAAoKC6+ofz6NaePLqld+L22Na+bNjRv9/VWdUzKrJk5p6INX5/3NjqLN9kCBzthC4AAIBjWKlUylNdg3lsa+8vBa3e57x2VktdVU6c3TRxO+G4sfsFbfWpmlF5BD8BwNQRugAAAI4BpVIpT+4cyC8292bN5p48snlspdZjW3qf85sN57XWTYSsE2Y35cTxoGV1FlBEQhcAAMBRpFQqZXP3UB4Zj1ljt978YnPPfoNWVWVFlsxsmFiVted2/HFNafKthsA04jceAABAmWzrHQ9aT/fkkS29Y/ebe9I9OLLP/atnVOT4WU05eW5zTp7dlJPmjAWtxR2NqalyuiGA0AUAAHCYdQ0MZ83TPXut0urs2/c1tGaMr9A6ZU5zTprTnFPmNOfkOU1ZOqsx1a6fBbBfQhcAAMAUGRndnXXb+/Pzp7rz8NPdefipnjz8dE+e3Dmwz/0rKpLFHQ05aXZzTpnblJPnNOfkOc05/rjG1FbNOMKjBzj2CV0AAAAvQGffrjz8VHd+/nRPHn6qOw+Pr9gaGtm9z/3nt9bl5LnNk1ZpnTi7KfU1ghbAVBG6AAAAnsOukd15fFtvHn6qJz8fX6X186e6s6VnaJ/711fPyClzm/Oiec05dW5LTp07dt/aUH2ERw4w/QhdAAAA43b07cpDT3XnZ5u68vBTPXnoqe48trU3w6Olfe6/uKNhLGTNa8mL5jbnRfNasrijIZWVFUd45AAkQhcAADANlUqlbOoazM+e7MrPNnXnZ5u689CmrmzqGtzn/k21VeNBayxmnTq3JafMbU5TrX9SARxN/FYGAAAKbXR3KY9v7R0PWmNh66GnurOzf3if+y/uaMhL5rfkRfNaxqNWcxa216eiwiotgKOd0AUAABTG4PBoHn66ZyJo/WxTd9Y83Z3B4b0vEF9VWZETZzflJfNb85L5LXnx+K2lzrW0AI5VQhcAAHBM6hsayQPrd+SuTRX5zt8+mIc39+SxrX0Z3b339bQaambkRfNa8pL5Y7cXz2vNSXOaUlftGw8BikToAgAAjnp9QyN56Knu/GRjV376ZFd+snFnHt/Wl1IpSWYkeWpi35mNNXnx/JaJlVovmd+SJTMbM8MF4gEKT+gCAACOKv27RvLQpu48+GRXHtzYlQef7MqjW3vHo9Zkc1tqc9yMgVyy/KScubg9L5nfmtnNta6nBTBNCV0AAEDZDOwazUNPjQWtnzw5tlrr0S292cfZh5nbUpfTFrTmjIWtOX1Ba05b0Jq2usrccccdefXFJ6S62rW1AKY7oQsAADgiBodH89BT3ROrtB7c2JVfbOnZZ9Sa3VybMxa2ToSt0xa0ZnZz3V77DQ/v+5sTAZiehC4AAGDKje4u5dEtvXlgw878eOPOPLBhZx5+umefF4qf1VQ7sUrr9AWtOX1ha+a07B21AOD5CF0AAMAhKZVKeaprcFLUenBjV/p2je6176ymmmcFrbacvqA1c1pcUwuAqSF0AQAAB6VrYDgPbuzKjzfsyI83dOWBjTuztWdor/0aambk9AWteemitpw5fpvfWidqAXDYCF0AAMB+DY2M5udP9eSBDTsnVmw9vrVvr/1mVFbklDnNOXNRW166qDVnLmrLSbObM6NS1ALgyBG6AACAJGOnIK7b3p/71+8Yi1obduahp7ozPLr3dbUWddTnpYvac+bCsRVbL5nfmvqaGWUYNQA8Q+gCAIBpqndoJD/ZsDOr1+/I6vU7c//6HdnRv/e3GLY3VI+deriwLS9d1JYzFrZmZlNtGUYMAM9N6AIAgGmgVCrl8W19Wf3Ejty/YWdWP7Ejj2zuyS9/CWLNjMqctqBlbLXWotYsX9SeRR31rqsFwDFB6AIAgALqGRzOAxu6xldr7cj963ema2Dv1VoL2uqzfHFbli9uz4rFbXnx/JbUVjkFEYBjk9AFAADHuN27S3l8W29WP7Ez92/YkdVP7MwjW3pS+qXVWrVVlTljYetE1Fq+uD1zWurKM2gAOAyELgAAOMZ0Dw7nx+ufubbWj9fvSPfgyF77LWyvz4pnRa0XzWtJTVVlGUYMAEeG0AUAAEexUqmUDZ0DufeJztz7xI7ct27HPldr1VVX5oyFbVm+uC0rFrdn+eK2zG62WguA6UXoAgCAo8jw6O78bFN37l3Xmfue2JF7n9iRrT1De+23ZGZDli9qy4ol7VmxuD2nzG1O9QyrtQCY3oQuAAAoo67+4axev2Nsxda6HXlg484MDu+etE/1jIqctqA1Zy9pz1lLOnLWkvYc11xbphEDwNFL6AIAgCOkVCplfWd/7l03tlLrvic688jm3r32a2uozlmL23PW0vacvaQjZyxsTV21b0IEgOcjdAEAwGGya2R3frapa+wUxPG4ta1379MQl81qzFlL2nP2kvacvbQ9x89qSmVlRRlGDADHNqELAACmSFf/cO5b3zkRtR7YsDNDI5NPQ6yZUZnTFrTk7KVjpyCetaQ9s5qchggAU0HoAgCAF+iproH8aG1nfrS2M6vW7fs0xPaG6vGg1ZGzl7bn9AVOQwSAw0XoAgCAA1AqlfL4tr6sGg9bP1rXmY07Bvba7/g9pyEuHYtbJxzXmIoKpyECwJEgdAEAwD6M7i7l5091T6zYuveJzmzr3TVpn8qK5CXzW3Puso6cs7Qj5yxtz0ynIQJA2QhdAACQZHB4ND/Z2JVV6zqzcm1nVj+xI71DI5P2qamqzPJFbRNha8WS9jTV+pMaAI4W/lcZAIBpqWdwOPc9sSOr1o2t2HpgQ1d2jU6+cHxzbVXOXtqec5Z15NylHTl9YWtqq1xfCwCOVkIXAADTwrbeobHra42HrZ8/1Z3dpcn7zGqqzcuWjZ2CeM6yjpw6tyUzKl1fCwCOFUIXAACFUyqVsnHHwMS3If5obWce39a3136LOxpy7vhqrXOWdWTpzAYXjgeAY5jQBQDAMa9UKmXttr6sXNuZHz6+PT9a25mnugYn7VNRkZwypznnLO0Yi1vLOjKnpa5MIwYADgehCwCAY06pVMpjW/uycu32/PDxzqx8fHu29AxN2qeqsiKnL2zNueNh6+wlHWltqC7TiAGAI0HoAgDgqFcqlfLolt78cHzF1srHO7Otd3LYqplRmeWL2/Ky42fmvGUdWb64PfU1LhwPANOJ0AUAwFFn9+5SfrGld3zF1tipiNt6d03ap6aqMisWt+W842fmZctmZvnittRVC1sAMJ0JXQAAlN3u3aWs2dyTlY+PnYr4o3Wd6eybHLbqqiuzYnH7eNjqyJmLhC0AYDKhCwCAI2737lJ+/nR3Vj4+fvH4dZ3Z2T88aZ/66hk5a0l7zju+Iy87fmbOWNia2iphCwDYP6ELAIDDbnR3KT9/qjs/HF+xtWpdZ7oGJoethpo9YWtmzju+I6cvaEtNVWWZRgwAHIuELgAAptzo7lIe2jQWtlau3Z6VazvTMzgyaZ/Gmhk5e2nH2KmIx3fk9AWtqZ4hbAEAL5zQBQDAIRsZ3Z2fTYStzqxa25meoclhq7m2KmcvHb/G1vEzc9r8llQJWwDAFBK6AAA4aMOju/PTJ7vyw8c7s3Lt9ty7bkd6fzls1VXl3Get2HrxPGELADi8jorQ9bd/+7f58z//8zzwwAPZtWtXTjzxxLzpTW/Ku9/97lRXVx/wcT7/+c/nd3/3d59zn3/6p3/KZZdddqhDBgCYVoZHd+cnG7smVmzdu64z/btGJ+3TUleVc5eNXV/rvONn5kXzWjKjsqJMIwYApqOyh65rr702N998c6qqqnLJJZekqakp3/3ud/O+970vX//61/Otb30r9fX1B3XME044IRdccME+n1uwYMFUDBsAoNB2jezOTzbufFbY2pGB4clhq62hetKKrVPnClsAQHmVNXR97Wtfy80335ympqZ873vfy4oVK5Ik27ZtyyWXXJK77747H/rQh/KJT3zioI57wQUX5POf//xhGDEAQDENjYzmgQ1dExePv++JHRkc3j1pn47GmvGw1ZGXHT8zp8xpTqWwBQAcRcoauj72sY8lSa6//vqJyJUks2bNyi233JILL7wwn/70p/OhD30ora2t5RomAEDhDA6P5scbxldsPd6Z1et3ZGhkctia2ViTl42fhviyZTNz0uwmYQsAOKqVLXQ9+eSTWbVqVZLkqquu2uv5Cy64IIsWLcqGDRtyxx135N/+2397pIcIAFAYg8OjWb1+R1Y+3pkfPr4992/YmV2/FLZmNdVOhK3zlnXkxNlNqagQtgCAY0fZQtf999+fJOno6MiyZcv2uc/ZZ5+dDRs25P777z+o0PXoo4/mgx/8YLZs2ZKmpqacdtppueKKKzJr1qwpGTsAwNFuYNdo1nRV5JHvPJpV63bmxxt2Ztfo5LA1u7k2Lzt+7OLxL1s2Mycc1yhsAQDHtLKFrrVr1yZJFi9evN99Fi1aNGnfA/WDH/wgP/jBDyZtq6ury4033pj3ve99z/v6oaGhDA0NTTzu7u5OkgwPD2d4ePigxgLHkj3z2zynyMxziqp/10hWr+/Kj9Z15kdrd+SBjV0Z2T0jeejxiX3mNNfm3GXtedmyjpy7tD1LZzZMClsjIyPlGDocEr/XmQ7Mc6aLqZjjZQtdPT09SZLGxsb97tPU1JTkmdD0fObOnZsPfOADueKKK3L88centrY2a9asyac+9al88YtfzPXXX5/R0dG8//3vf87j3HTTTfnwhz+81/a77rorDQ0NBzQWOJbdeeed5R4CHHbmOce6odHk8Z6KPNpdkUe7KrK+L9ldmrwaq62mlBNbnrnNqhtJRUVfsmVjfr4l+XmZxg6Hg9/rTAfmOUXX399/yMco68Xop9pll12Wyy67bNK2s88+O1/4whdy5pln5j3veU/+8A//MG9961szZ86c/R7nhhtuyHXXXTfxuLu7O4sWLcrFF1+cmTNnHrbxQ7kNDw/nzjvvzCte8YpUV1eXezhwWJjnHKt6h0Zy3xM78qN1O7Jy7Y78dFN3RneXJu0zv7UuL1vWnnOXdWTFwub8fNXd+fVfN9cpNr/XmQ7Mc6aL7du3H/Ixyha6mpubkyR9fX373ae3tzdJ0tLScsjv9653vSs33XRTtm3blm9961v57d/+7f3uW1tbm9ra2r22V1dX+6XCtGCuMx2Y5xztugeHc++6zomLx+8rbC1srx//RsSxC8gv6nhm5fnw8HAerjDXmT7MdaYD85yim4r5XbbQtXTp0iTJhg0b9rvPnuf27HsoZsyYkZNOOinbtm3Lxo0bD/l4AABTqWtgOKvWdmbl2u354eOd+dmmrvxS18rijoaJC8e/7PiOLGx3SQUAgGcrW+havnx5krFlaWvXrt3nNy/ee++9SZIVK1ZMyXvuWQK3ZzUZAEC57OzflZVrx1ZsrVy7PQ891Z3SL4WtpTMbxlZsjcet+W315RksAMAxomyha+HChTnnnHOyatWq3HrrrfnABz4w6fm77747GzZsSG1tbV796lcf8vutXr06jzzySJLk3HPPPeTjAQAcjO29Q/nR2s6sXDt2KuKazT17ha3jZzXmZcd3jJ+OODNzW+vKM1gAgGNUWS9G//73vz+ve93r8vGPfzyvetWrJlZubd++Pddcc02S5B3veEdaW1snXnPbbbflhhtuyIIFC/Kd73xnYnt/f3/+6q/+Km95y1v2WrH1//1//19+53d+J0lywQUXCF0AwGG3tWcoK9dun1ix9cjm3r32OXF20zOnIi7ryOwWYQsA4FCUNXRdeeWVeec735lPfvKTOe+883LppZemsbEx3/nOd7Jz586cf/75+chHPjLpNV1dXVmzZk0GBwcnbd+1a1fe8Y535D3veU+WL1+exYsXZ2RkJI888kh++tOfJklOP/30/M3f/M0R+3wAwPSxpXswPxxfrbXy8e15bOveX7hzypzmiRVb5y7ryKymvb/8BgCAF66soStJbr755px//vn58z//89xzzz0ZHh7OCSeckOuvvz7vfve7U1NTc0DHaWhoyIc+9KHce++9efjhh/Ozn/0sAwMDaW9vz8tf/vK84Q1vyNVXX33AxwMAeC5PdQ1MfCPiyrWdWbttctiqqEhOndsy8Y2I5y7rSEejv0MAAA6nsoeuJHnjG9+YN77xjQe079VXX52rr756r+01NTX5wz/8wykeGQDAmI07+ieFrfWd/ZOer6xIXjy/JS9bNjPnHT8z5yxtT1uDsAUAcCQdFaELAOBoUiqVsqFzID8cv8bWDx/fnid3Dkzap7IiOX1Ba152/Nj1tc5e2pHW+uoyjRgAgEToAgBIqVTKuu39WTm+WuuHj2/PU12Trwc6o7IiZyxsHbtw/PEdOXtJe5rrhC0AgKOJ0AUATDulUimPbe2b9K2Im7uHJu1TPaMiZyxsm/hWxLOWtKex1p9OAABHM3+tAQCFt3t3KWs29+RHazvzo3WdWfl4Z7b1Tg5bNTMq89JF42Hr+JlZsbg99TUzyjRiAABeCKELACic4dHdefDJrqxa25kfre3MqnWd6R4cmbRPTVVlVixum7h4/PLFbamrFrYAAI5lQhcAcMwb2DWa+zfsGFuxtbYz96/fmYHh0Un7NNTMyFlL2vOyZR05Z2lHXrq4LbVVwhYAQJEIXQDAMadrYDj3PdGZlWs7s2ptZx58sivDo6VJ+7Q3VOecpR05d9nY7cXzWlI1o7JMIwYA4EgQugCAo96WnsGsWrsjq9aNxa2Hn+5OaXLXytyWuomode6yjpx4XFMqKyvKM2AAAMpC6AIAjiqlUikbdwxMrNb60brOrN3Wt9d+y2Y15tylHTlnWUdetqwjC9vrU1EhbAEATGdCFwBQVrt3l/Lo1t6J62utWteZp7oGJ+1TUZGcOrdl4vpa5yxrz+zmujKNGACAo5XQBQAcUbtGdudnm7py3xM7JsLWjv7hSftUVVbkjIWtE6u1zlrSkdb66jKNGACAY4XQBQAcVl0Dw1m9fkfuWzd2ja0HNu7M4PDuSfvUVVdmxeL2setrLe3I8sXtqa/xjYgAABwcoQsAmDKlUilP7hzIvet25N4nOnPvuh1Zs7lnrwvHtzVU5+wl7Tl7/FsRT5vfmpoq34gIAMChEboAgBdsZHR3Hn66J/eu68y9T+zIvet25Onuwb32WzqzIWct6cg5S9tz9tL2HD/LNyICADD1hC4A4ID1DY3kxxt2ZtW6ztz3xI6sfmJH+naNTtqnqrIiL1nQmrOXtOecpe05a0lHjmuuLdOIAQCYToQuAGC/NncP5t7xa2vd98SOPPRUd0Z3Tz4Psbm2KiuWtE+civjSRW2urwUAQFkIXQBAkmR0dymPbO6ZWKm16onObOgc2Gu/BW31OXvpM2Hr5DnNmeE0RAAAjgJCFwBMUzv7d+X+9Tuzev2OrF6/Iw9s6Erv0MikfSorklPntoydgri0I2cvac/8tvoyjRgAAJ6b0AUA08Du3aX8YkvvWNR6YixsPba1b6/9Gmtm5KWL23LW4rHVWssXt6W5rroMIwYAgIMndAFAAXUNDOfHG3ZORK0fb9iZnsGRvfZbNqsxKxa3Z8WStqxY3O40RAAAjmlCFwAc43bvLuXxbX0TUWv1+h35xZbelCZfMz711TPy0kVtE1Fr+eL2dDTWlGfQAABwGAhdAHCM6R0ayY+fdW2t+9fvTNfA8F77Le5oyFlL2rNicVuWL27PqXObUzWjsgwjBgCAI0PoAoCj2Mjo7jyyuTc/3rAzP94wdgrivlZr1VVX5owFbVnxrLB1XHNteQYNAABlInQBwFGiVCplU9dgHtiwcyxsrd+ZB5/sysDw6F77Lmirn1ittWJJe140ryXVVmsBADDNCV0AUCY9g8P5ycau8dVaY7etPUN77ddcW5UzFrXmpYvacubCtrx0cVtmN9eVYcQAAHB0E7oA4AgYGd2dh5/uyY837JxYsfXo1r1PQZxRWZFT5zbnpYvaJm4nHNeUSt+ECAAAz0voAoApViqVsnHHQB7Y+EzUevDJrgwO795r3wVt9Xnp4rYsH49aL5nfmvqaGWUYNQAAHPuELgA4BKVSKZu7h/LAxp15cGNXfvJkVx7cuDM7+vf+FsTmuqqxUw/Ho9aZi9pcMB4AAKaQ0AUAB2Frz1AefHJnfrKxayJs7eu6WtUzKnLq3Jacuag1L13UnpcuasvxsxqdgggAAIeR0AUA+7Gjb1cefLIrDz7ZlZ+Mr9ja1DW4134zKity0uymnLGwNWcsbMsZC1tzytzm1FY5BREAAI4koQsAknQPDuenT46v0trYlZ88uTMbOgf22q+iIjnhuPGotaA1py9sy4vntbiuFgAAHAWELgCmnf6R5F8e356HN/flZ5u68+DGrjy+rW+f+y6b1ZjTF7TmjIWtOX1Ba16yoDVNtf7nEwAAjkb+Ugeg0LZ0D+anm7rysye7x++7snFnVbLqvr32XdhePx60xk4/PG1Ba1rrq8swagAA4IUQugAohFKplA2dA/nZpq6xoLWpOz99sjvbeve+UHySLGqvz2kLxmLWS+a35IyFbelorDnCowYAAKaS0AXAMWd0dymPb+2dtFLroU3d6R4c2WvfyvFrau0JWqfOacyGB3+Yf33FhamutloLAACKROgC4KjWOzSSNU9356GnevLzp7onboPDu/fat2ZGZU6Z25yXzG/JS8bD1ovmTr5Q/PDwcLb//Eh+AgAA4EgRugA4KpRKpWzcMZCHxkPWw0/15OdPd+eJ7f373L+hZkZePK8lpy1ozYvnt+S0+a05cXZTaqoqj/DIAQCAo4XQBcARN7BrNGs2T16h9fBTPekZ2vvUwySZ01KbF81rmbi9ZH5Lls5szIzKiiM8cgAA4GgmdAFw2JRKpTzVNfisoDUWt9Zu70uptPf+1TMqctLs5pw6rzkvflbYcpF4AADgQAhdAEyJnf27subpnjyyuSdrNvfkkad7s2ZzT7oGhve5/6ymmmet0mrOi+a15ITjmlI9w6mHAADACyN0AXBQ+neN5Bebe8dj1njU2tyTzd1D+9y/qrIiJxzXNBGzXjSvJafOa87s5rojPHIAAKDohC4A9ml4dHfWbuvLw09PDlrrO/v3edphkixoq88pc5vHbnOac/Kc5pwwuzG1VTP2/QIAAIApJHQBTHMjo7vzRGd/Ht3Sm19s7smazb155OmePL6tN8Oj+y5as5pqcvJ4yDp1bnNOntuck2Y3pbmu+giPHgAA4BlCF8A0MTg8mrXb+saC1pbePLalN7/Y0pO12/r2G7Saaqty8pymnDJ3LGrtuZ/VVHuERw8AAPD8hC6AgukdGhmPWL15dEtvHt3Sk0e39GZ9Z3927+eUw/rqGTlhdmNOmr0naDXllLktmd9al4qKiiP7AQAAAF4goQvgGLWjb1ce3dqbX2zuHV+l1ZPHtvRmU9fgfl/TUleVk+Y058TjmnLSnKacMLspJx7XlAVt9amsFLQAAIBjm9AFcBQbGhnN+u39eWxrXx7f1pvHt/bl8a29WbutLzv6h/f7ullNtTlpdlNOnD0WtE48riknzmnKcU21VmgBAACFJXQBlFmpVMqWnqE8tnVPyHomam3csf/TDZOxbzk8cU/QGr8/cXZT2hpqjtwHAAAAOEoIXQBHSN/QSNZu68vj28ZWZe0JWmu39qVv1+h+X9dUW5Xjj2vM8bMac/xxTVk2qzHHH9eYZbMa01Dj1zgAAMAe/oUEMIV6h0byxPa+rNvWn3Xb+8Z+3t6fJ7b3ZXP30H5fN6OyIos7GsYi1njQOv64saDldEMAAIADI3QBHKSugeFnAta2Z0LWuu392da7/5iVJDMba8ZXZzVl2bNWaS3uaEhNVeUR+gQAAADFJHQB/JJSqZSd/cPjK7L6J92ve56LwCdjMWvJzIYsndmYpbMan/l5ZmNaG6qP0KcAAACYfoQuYFoaHB7Nxh39Wd/Znw2dA+P3/dmwYyAbOvvTOzTynK8/rrk2S2c2ZMnMsWtl7YlZi2c2pKVOzAIAACgHoQsopNHdpTzdPTgWr54VsfYErS09z32KYZLMbambCFhLZj2zKmvJzIY01vr1CQAAcLTxLzXgmFQqlbKjfzhP7olXE6uzxm5P7hzI8GjpOY/RVFuVRR0NWdRen8UdDVnU0TB+X5+F7Q2pq55xhD4NAAAAU0HoAo5KI6O7s7lnKE/uGMiTO/uzaedgNu4YyJM7B7Jp50Ce3DGQgeHR5zxGVWVFFrbXZ1FHQxa2PxOxFnc0ZFF7Q9oaqn2bIQAAQIEIXUBZDOwazZM7x8LVkzvG49X4z0/uHMjT3YMZ3f3cK7KSsWtlPXtF1qLxiLV4ZkPmttRlRqWQBQAAMF0IXcCUGxndna29Q3mqazBPdw3mqa7ByTFr50A6+3Y973GqZ1RkXmt9FrTVZ35bfRa012fh+P38tvrMa61zeiEAAAAThC7goAyP7s7m7mcC1sR990Ce6hrMUzsHs6VnMAewGCtNtVVZMB6unh2zFrSN3Y5rrrUiCwAAgAMmdAEThkZ2Z/tg8qN1ndnWN/KskDUwEbS29g6ldAARq6qyInNa6jK3tS7zxm9jUathIm611FW5RhYAAABTRuiCaWDXyNiphJu7B7OlezBbesZ+3tw9lC09QxPbxk4nrEruv/c5j1czozJzWmszr6U+89rGY1ZLXea21k9ErZlNVmMBAABwZAldcAybHLCGsqVncOLnzXsFrANTXVHK/PbGzGury/zW+okVWXsi1tzWunQ01KRSxAIAAOAoI3TBUWZ0dyk7+ndlW+9QtvXsytbewWzrGXu8tWcoW8fvDzpgzajI7Oa6zGmpfea+pS6zm2szp6Uuc1rq0l5fmXvu+nZe85oLUl1dfRg/JQAAAEw9oQuOgN27S+l8VrzaE6229T4Trrb1jm3f3jt0QBdy32NPwJrdUps5e+7HA9bslrGgNae5Lm0N1c97Pazh4eG4ZBYAAADHKqELXoBSqZS+XaPp7N2V7X1jK6u29+1K5/htW+9YuNoTszr7dmX0YOpVko7Gmsxqqsmsptoc11ybWU17bjU57lmrsNrqq51GCAAAABG6IMnYiqvuweFJsWrPbXvvrnT2DU16bnvfruwa2X3Q7/N88WrP9o7GmlTPqDwMnxQAAACKS+iicHbvLqVncCQ7+ndl58Dw2H3/ruzsH86O/uHs7B9ffdX7TLTa0X/wK66SpK66MjMbx8JUR2NNZu65F68AAADgiBO6OGqVSqUMDI9OxKmxUDV2v7N/V3aMP+569vaBsedeQLNKkjTVVu0VrTqa9vxc+8y2xprMbKpJQ43/CgEAAMDRwr/SOeyGRkbTNTCc7oGRsfvB4XQPDI9vG5703DPBaixkvZDTA/doqJmRtvrqtDXUpL1x7L6tvjrtDc+Eqo7GybfaqhlT+MkBAACAI0no4nmVSqX0DI3sFaX2GawGRyY97hoYztAhxKokqaqsGItVDdVpa6ie+Lm9oSat4/ft49vb9myvr05dtWgFAAAA04nQVXAjo7vTOzSSnsGx29jPw+kdGkn34Eh6B595/Mw+wxM/74lZL/RUwD0qKpLm2qq01Fentb46LXXj9/VVzzxumLzqqq2hOu2NNWmsmZGKCt8qCAAAADw3oesoNTy6O/1Do+ndNZK+oV+KVOMRqueXHu95vmdoZGLbwPDolI2ppqpyPEqNx6n9RKs92yaer69OU21VZlSKVQAAAMDhI3RNgT0XTe8bGk3f0Ej6do2M/TweqfqHRtM7NJL+XSPpHRodvx/bvmefZ+/ft2v0kK5NtS911ZVpqh2LVE11VWmuq0pTbVWa68Yi1DPbq8e3j92eHa2cCggAAAAczYSug/D7X3kgo1X16dv17Hg1FqhKh3hq3/7UzKhMQ+2M8TBVPRagxkPUs8PURKiqrZ4IWc3j+zfWVqWmqvLwDBAAAADgKCF0HYS7H92eytqG59ynsWZGGmurxm8z0lAztnKqoWbG+H1VmmpnpGHPPnv2rxnbv/FZ2xtqBCoAAACAAyV0HYT/8tpTMmfWrIlw9eyg1VhTlfrqGal0HSoAAACAsjgqlgv97d/+bS666KK0t7ensbExZ555Zv7bf/tvGR4efkHHu++++/KGN7whc+bMSV1dXZYtW5bf//3fz5YtWw5pnK976YJcfub8XHzq7Lzs+Jk5bUFrls1qzOzmujTWVolcAAAAAGVU9tB17bXX5o1vfGN+8IMf5Nxzz81ll12W9evX533ve18uueSSDAwMHNTx/u7v/i7nnXde/u7v/i5LlizJb/zGb6SysjKf/vSnc8YZZ+TRRx89TJ8EAAAAgHIqa+j62te+lptvvjlNTU1ZuXJl/t//+3/56le/ml/84hc5/fTTc/fdd+dDH/rQAR9v06ZN+Z3f+Z2MjIzkL/7iL/KjH/0of/3Xf51HHnkkb37zm7N58+ZcddVVKR2uK8cDAAAAUDZlDV0f+9jHkiTXX399VqxYMbF91qxZueWWW5Ikn/70p9PV1XVAx/uzP/uz9Pf35+Uvf3ne/va3T2yfMWNGPvOZz6S1tTWrVq3Kt771rSn8FAAAAAAcDcoWup588smsWrUqSXLVVVft9fwFF1yQRYsWZWhoKHfccccBHfO2227b7/GamppyxRVXJEn+/u///oUOGwAAAICjVNlC1/33358k6ejoyLJly/a5z9lnnz1p3+fS09Mzcf2tPa87lOMBAAAAcGypKtcbr127NkmyePHi/e6zaNGiSfs+l3Xr1k38vL9jHujxhoaGMjQ0NPG4u7s7STI8PPyCvwkSjgV75rd5TpGZ50wX5jrThbnOdGCeM11MxRwvW+jq6elJkjQ2Nu53n6ampiTPhKYDOd5zHfNAj3fTTTflwx/+8F7b77rrrjQ0NDzvWOBYd+edd5Z7CHDYmedMF+Y604W5znRgnlN0/f39h3yMsoWuo9kNN9yQ6667buJxd3d3Fi1alIsvvjgzZ84s48jg8BoeHs6dd96ZV7ziFamuri73cOCwMM+ZLsx1pgtznenAPGe62L59+yEfo2yhq7m5OUnS19e33316e3uTJC0tLQd8vD3HbG1tfcHHq62tTW1t7V7bq6ur/VJhWjDXmQ7Mc6YLc53pwlxnOjDPKbqpmN9luxj90qVLkyQbNmzY7z57ntuz73NZsmTJxM/r168/5OMBAAAAcGwpW+havnx5krFlafu7OPy9996bJFmxYsXzHq+lpSUnnnjipNcdyvEAAAAAOLaULXQtXLgw55xzTpLk1ltv3ev5u+++Oxs2bEhtbW1e/epXH9AxX/e61+33eL29vfn617+eJHn961//QocNAAAAwFGqbKErSd7//vcnST7+8Y9n9erVE9u3b9+ea665Jknyjne8Y9L1tm677baceuqpufTSS/c63rXXXpuGhoZ8+9vfzv/+3/97Yvvo6Giuueaa7Ny5M+ecc05+/dd//XB9JAAAAADKpKyh68orr8w73/nO9Pb25rzzzsurXvWq/Ot//a9z4okn5sEHH8z555+fj3zkI5Ne09XVlTVr1uSxxx7b63jz58/P5z//+cyYMSNvf/vbc9555+Xf/Jt/k5NPPjlf/OIXM2fOnNx6662pqKg4Uh8RAAAAgCOkrKErSW6++eb89V//dX7lV34l99xzT+64444sXLgwH//4x/Pd73439fX1B3W8N7zhDVm5cmVe//rX5/HHH89tt92W0dHR/Kf/9J/ywAMPTFzHCwAAAIBiqSr3AJLkjW98Y974xjce0L5XX311rr766ufc56yzzspXv/rVKRgZAAAAAMeKsq/oAgAAAICpIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhCFwAAAACFIHQBAAAAUAhlDV09PT15//vfn1NOOSX19fWZNWtWXvOa1+S73/3uCzreRRddlIqKiv3e5s6dO8WfAAAAAICjRVW53njLli258MIL88gjj2TevHm5/PLLs3nz5vzTP/1T/umf/ik333xzfv/3f/8FHfuVr3zlPqNWa2vroQ4bAAAAgKNU2ULX29/+9jzyyCO59NJLc/vtt6ehoSFJcscdd+SKK67Itddem1/7tV/LGWeccdDHvv7663PRRRdN8YgBAAAAOJqV5dTFhx56KP/wD/+QGTNm5HOf+9xE5EqSV7/61bn66quze/fu3HTTTeUYHgAAAADHoLKErttuuy1Jcv7552fJkiV7PX/VVVclSb7+9a9neHj4iI4NAAAAgGNTWU5dvP/++5MkZ5999j6f37O9r68vv/jFL/LiF7/4oI5/22235Wtf+1oGBgYyZ86c/Kt/9a/y67/+66ms9CWTAAAAAEVVltC1du3aJMnixYv3+XxLS0taWlrS3d2dtWvXHnTo+uQnP7nXtpNPPjlf+tKXcs455xz8gAEAAAA46pUldPX09CRJGhsb97tPU1NTuru7093dfcDHvfDCC/PmN785v/qrv5qFCxdmx44dWbVqVT74wQ/mZz/7WV7+8pfnhz/8YV70ohc953GGhoYyNDQ08XjPGIaHh51KSaHtmd/mOUVmnjNdmOtMF+Y604F5znQxFXP8oEPXe9/73tx+++0H/Uaf/exnc8EFFxz06w7GRz7ykUmPGxoasmDBgrzqVa/KhRdemFWrVuWGG27I1772tec8zk033ZQPf/jDe22/6667Jl04H4rqzjvvLPcQ4LAzz5kuzHWmC3Od6cA8p+j6+/sP+RgHHbo2bdqUNWvWHPQb9fb2Tvzc3NycZOwaXM+3f0tLy0G/1y+rra3NBz7wgVx55ZX55je/meHh4VRXV+93/xtuuCHXXXfdxOPu7u4sWrQoF198cWbOnHnI44Gj1fDwcO6888684hWveM7/jsCxzDxnujDXmS7MdaYD85zpYvv27Yd8jIMOXV/60pfypS996ZDedOnSpVm9enXWr1+/z+effcri0qVLD+m99thzuuLQ0FC2bduWefPm7Xff2tra1NbW7rW9urraLxWmBXOd6cA8Z7ow15kuzHWmA/OcopuK+V2WryFcsWJFkuTee+/d5/N7tjc2Nubkk0+ekvd8dhXcs6IMAAAAgOIoS+i68sorkyQ/+MEP9rmq69Zbb02SXH755VNWq7/yla8kGVvZ1dTUNCXHBAAAAODoUZbQ9ZKXvCS/8Ru/kdHR0bz1rW/NwMDAxHP/9E//lM9//vOprKzMDTfcsNdr3/KWt+TUU0/Npz/96Unb77rrrvzzP/9zSqXSpO27du3Kxz/+8XzqU59KkrznPe85DJ8IAAAAgHI76Gt0TZX/9b/+Vx566KF8+9vfzgknnJALL7wwW7Zsyfe+972USqXcfPPNOeOMM/Z63fr167NmzZps27Zt0vYHHngg7373uzNnzpy89KUvzcyZM7N169b85Cc/yebNm5Mkf/AHf5C3vvWtR+TzAQAAAHBklS10zZ49O/fee29uuummfPWrX80//MM/pLGxMa985SvzB3/wB7n00ksP6ni/9mu/lt/7vd/Lfffdl5/85Cfp7OxMZWVl5s+fn1e+8pX5D//hP+Rf/at/dZg+DQAAAADlVrbQlSQtLS256aabctNNNx3wa/75n/95n9uXL1+eW265ZYpGBgAAAMCxpizX6AIAAACAqSZ0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhSB0AQAAAFAIQhcAAAAAhVC20HXHHXfkxhtvzOWXX5758+enoqIiFRUV2bhx4yEdd9euXfnjP/7jnHnmmWlsbEx7e3suuuii/N3f/d0UjRwAAACAo1FVud74qquuSldX15Qes7+/P694xStyzz33pK2tLZdddll6e3vz3e9+N9/73vfynve8J5/4xCem9D0BAAAAODqULXS9/vWvz0knnZQVK1ZkxYoVmT179iEf8/3vf3/uueeenH766fnud7+bWbNmJUnuu+++XHTRRfmTP/mTXHTRRXnta197yO8FAAAAwNGlbKHrL//yL6f0eDt27MhnPvOZJMlnPvOZiciVJGeddVbe97735UMf+lD+6I/+SOgCAAAAKKDCXIz+jjvuyK5du7J48eKcf/75ez1/1VVXJUl++MMfZtOmTUd6eAAAAAAcZoUJXffff3+S5Oyzz97n88cff3w6OjqSJD/+8Y+P1LAAAAAAOELKduriVFu7dm2SZPHixfvdZ+HChens7JzYd3+GhoYyNDQ08bi7uztJMjw8nOHh4SkYLRyd9sxv85wiM8+ZLsx1pgtznenAPGe6mIo5XpjQ1dPTkyRpbGzc7z5NTU1JnglX+3PTTTflwx/+8F7b77rrrjQ0NBzCKOHYcOedd5Z7CHDYmedMF+Y604W5znRgnlN0/f39h3yMgw5d733ve3P77bcf9Bt99rOfzQUXXHDQryuHG264Idddd93E4+7u7ixatCgXX3xxZs6cWcaRweE1PDycO++8M694xStSXV1d7uHAYWGeM12Y60wX5jrTgXnOdLF9+/ZDPsZBh65NmzZlzZo1B/1Gvb29B/2ag9Hc3Jwk6evre94xtLS0POexamtrU1tbu9f26upqv1SYFsx1pgPznOnCXGe6MNeZDsxzim4q5vdBX4z+S1/6Ukql0kHfLrvsskMe7HNZunRpkmT9+vX73Wfjxo2T9gUAAACgOArzrYsrVqxIktx77737fP7xxx9PZ2dnkmT58uVHbFwAAAAAHBmFCV2vfvWrU1NTk/Xr1+cHP/jBXs/feuutSZLzzjsv8+fPP9LDAwAAAOAwO+ZC16WXXppTTz01t91226Tt7e3t+b3f+70kyTXXXDPpAmarV6/OH//xHydJPvCBDxy5wQIAAABwxBz0xeinykc+8pF84xvf2Gv7FVdckZqamiRjpyPecsstk55/7LHH8sQTT6Srq2uv137sYx/Lj370o/zLv/xLTjrppFxyySXp6+vLd77znQwPD+e6667La1/72sPzgQAAAAAoq7KFrsceeywrV67ca/v9998/8XNdXd1BHbOhoSH//M//nD/90z/Nl7/85dxxxx2pqanJr/zKr+Qd73hH3vCGNxzyuAEAAAA4OpUtdH3+85/P5z//+YN+3bp1657z+Zqamlx//fW5/vrrX9jAAAAAADgmHXPX6AIAAACAfRG6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACgEoQsAAACAQhC6AAAAACiEsoWuO+64IzfeeGMuv/zyzJ8/PxUVFamoqMjGjRtf8DGXLl06cZx93c4777wp/AQAAAAAHE2qyvXGV111Vbq6ug7LsX/zN38zTU1Ne20/4YQTDsv7AQAAAFB+ZQtdr3/963PSSSdlxYoVWbFiRWbPnj1lx/7EJz6RpUuXTtnxAAAAADj6lS10/eVf/mW53hoAAACAAnIxegAAAAAKoWwrug6nv/qrv0pnZ2dGRkYyf/78/Nqv/Vp+9Vd/tdzDAgAAAOAwKmTo+sM//MO9tp1zzjm59dZbc+KJJz7v64eGhjI0NDTxeM9F8zs7O6dukHAUGh4eTn9/f7Zv357q6upyDwcOC/Oc6cJcZ7ow15kOzHOmiz3dpVQqveBjFCp0veY1r8kFF1yQc889NwsWLMjTTz+d73//+/ngBz+YVatW5aKLLsrq1auf98L3N910Uz784Q/vtf3kk08+XEMHAAAAIMn27dvT2tr6gl5bUTrITPbe9743t99++0G/0Wc/+9lccMEF+x9IRUWSZMOGDVm4cOFBH/+5dHZ25qyzzsq6devyrne9K3/2Z3/2nPv/8oqunTt3ZsmSJVm/fv0L/g8ajgXd3d1ZtGhRNmzYkJaWlnIPBw4L85zpwlxnujDXmQ7Mc6aLrq6uLF68ODt27EhbW9sLOsZBr+jatGlT1qxZc9Bv1Nvbe9CvmSodHR259tprc+211+brX//684au2tra1NbW7rW9tbXVLxWmhZaWFnOdwjPPmS7MdaYLc53pwDxnuqisfOHfnXjQr/zSl76UUql00LfLLrvsBQ9yKrzoRS9KkmzcuLGs4wAAAADg8HjhiewYs3379iRJc3NzmUcCAAAAwOEwbULXV77ylSTJueeee9Cvra2tzX/9r/91n6czQpGY60wH5jnThbnOdGGuMx2Y50wXUzHXD/pi9IfLgV6M/tJLL82TTz6Zm266Ka973esmtv/DP/xDFi5cmLPOOmvS/j09PfnQhz6Um2++OUly55135uUvf/lh+AQAAAAAlNNBX4x+qnzkIx/JN77xjb22X3HFFampqUmSrFixIrfccsuk5x977LE88cQT6erqmrT9rrvuys0335zFixfn9NNPT1tbWzZt2pQf//jH2bFjR6qqqvKJT3xC5AIAAAAoqLKFrsceeywrV67ca/v9998/8XNdXd0BH+/KK69Mb29vVq9enXvvvTednZ2pqanJ4sWL81u/9Vu55pprcvrpp0/J2AEAAAA4+hw1py4CAAAAwKGYNhejBwAAAKDYhK4XoLu7Ox/60Ifymte8JieccEJaW1tTU1OT+fPn5zd+4zf2ee0xONZs2bIl/+f//J9cddVVOemkk1JXV5eGhoaceuqpeec735l169aVe4gwZe64447ceOONufzyyzN//vxUVFSkoqIiGzduLPfQ4KD97d/+bS666KK0t7ensbExZ555Zv7bf/tvGR4eLvfQYEqsWbMmn/rUp3L11Vfn9NNPT1VVVSoqKvLRj3603EODKTM8PJzvfOc7+c//+T/nnHPOSVtbW6qrqzN37txcccUV/s1JoXz5y1/OW97ylpx55pmZPXt2qqur09ramnPPPTc33XRTent7D+p4Tl18AR599NGcdNJJaWpqymmnnZZ58+alsrIyjz76aB544IEkyTXXXJM///M/L/NI4YV785vfnC9/+cuprKzMaaedllNOOSV9fX1ZtWpVtm7dmsbGxtx22215xSteUe6hwiFra2vb60tOkuf/JmA42lx77bW5+eabU1VVlUsuuSRNTU357ne/m507d+aCCy7It771rdTX15d7mHBI9szzX/aRj3wkH/zgB8swIph63/72tyf+zp47d27OOuusNDY25qGHHspPf/rTJMnb3/72/M//+T9TUVFRzqHCIbvgggtyzz335EUvelEWLVqUjo6ObN68Of/yL/+SgYGBnHjiifne976X+fPnH9DxhK4XoLe3Nz/96U9z9tlnp6pq8vX877rrrrz2ta9Nf39/vvnNb+aVr3xlmUYJh+ad73xnZs2albe+9a1ZsGDBxPbe3t78+3//7/OVr3wlHR0defTRR9Pe3l7GkcKh+3f/7t/lpJNOyooVK7JixYrMnj07idDFseVrX/taXve616WpqSnf+973smLFiiTJtm3bcskll+TBBx/Me97znnziE58o80jh0Hz2s5/NmjVrsnz58qxYsSIf+9jH8sUvflHoolC++93v5pZbbsm73vWuXHjhhZOe++u//uu86U1vyujoaL7whS/kLW95S5lGCVNj5cqVOemkk9LR0TFp+/bt23PllVfm7rvvzr/5N/8m//f//t8DOp7QdRi89a1vzV/+5V/muuuuy5/8yZ+Uezgw5fr7+zN37tz09PTki1/8Yt785jeXe0gwpfb8P6NCF8eSc889N6tWrcpHP/rRfOADH5j03N13350LL7wwtbW12bx5c1pbW8s0Sph6V199db7whS8IXUwrb3vb2/K5z30ul156ab797W+Xezhw2Hz/+9/Pr/7qr6ajoyPbt28/oNe4RtdhsGeVV21tbZlHAodHQ0NDTjnllCRjIQCA8nryySezatWqJMlVV1211/MXXHBBFi1alKGhodxxxx1HengATLHly5cn8bc4xfdC+orQNcVWrVqVv/7rv05FRUUuv/zycg8HDovh4eGJi9HPmzevvIMBIPfff3+SpKOjI8uWLdvnPmefffakfQE4dv3iF79I4m9xiq2npyc33nhjkuSKK6444NdVPf8uPJf/8l/+S9avX5+BgYGsXbs2q1atSk1NTT75yU/mV37lV8o9PDgsPve5z2Xbtm2pr6/Pq171qnIPB2DaW7t2bZJk8eLF+91n0aJFk/YF4Nj09NNP5/Of/3yS5Dd/8zfLOxiYQt/61rdy6623Zvfu3RMXo+/p6clll12WP/7jPz7g4whdh+j222+f+KbFJGlqasqf/umf5t/9u39XxlHB4fPggw/mP//n/5wk+dCHPpQ5c+aUeUQA9PT0JEkaGxv3u09TU1OSpLu7+4iMCYCpNzIykje/+c3p6urK6aefnv/wH/5DuYcEU+ahhx7KF77whUnbrrrqqvzpn/7pQV1fdNqFrve+9725/fbbD/p1n/3sZ3PBBRfstf3HP/5xkrE/GtesWZM/+7M/y9vf/vZ85Stfyde+9rU0Nzcf6pDhoE31PN9j48aNufzyy9Pb25srrrgi119//aEMEw7Z4ZrrAABHo//4H/9jvvOd72TmzJn5u7/7u9TU1JR7SDBlrr322lx77bUZHh7O+vXr8w//8A/56Ec/mm9+85u57bbb8qu/+qsHdJxpF7o2bdqUNWvWHPTrent7n/P5lpaWnHPOOfnyl7+ctra23HLLLfnwhz/sK7wpi8Mxz59++ulceumleeKJJ/LKV74yf/M3fzPxzXRQLofrdzoca/b8H2t9fX373WfPvG9paTkiYwJgar3rXe/K5z73ubS3t+fOO+/MySefXO4hwWFRXV2dE044Idddd13OP//8/Mqv/Ere/OY3Z82aNamvr3/e10+7i9F/6UtfSqlUOujbZZdddsDv8bu/+7tJkttuu+1wfQx4TlM9z7ds2ZJLLrkkjzzySF7+8pfna1/7mm8V5ahwJH6nw7Fg6dKlSZ7727f2PLdnXwCOHe95z3vyyU9+Mm1tbfnWt7418a2LUHQve9nL8uIXvzgbNmzIvffee0CvmXah60jYc32MLVu2lHkkcOi2bt2aSy65JD//+c9z6aWX5vbbb09dXV25hwXAs+z5B8/27dv3e7H5PX8crlix4oiNC4BD9973vnfiGkXf+ta3Jr5FF6aLg20sQtdh8J3vfCdJLCXlmLdt27Zccskl+dnPfpZLL700X//61w9oqSgAR9bChQtzzjnnJEluvfXWvZ6/++67s2HDhtTW1ubVr371kR4eAC/Q9ddfn//+3/97Wltbc+edd078rofpYtu2bRNfAHigjUXoegFuvfXW3HfffXttL5VK+fu///t88IMfTJK8/e1vP9JDgynT2dmZSy+9ND/96U/z8pe/XOQCOMq9//3vT5J8/OMfz+rVqye2b9++Pddcc02S5B3veMdBfWsRAOXzwQ9+MH/8x3+ctrY2kYvCeuihh/LlL385g4ODez33yCOP5A1veEOGhoZy3nnn5fTTTz+gY1aUSqXSVA+06K6++up84QtfyMKFC3PGGWekra0t27dvz8MPP5wnnngiSfKf/tN/yqc+9SkX6+aY9frXvz633XZbKioq8oY3vGG/kevKK6/MlVdeeWQHB1PsIx/5SL7xjW9MPF65cmWSsdPB9nyb0YoVK3LLLbeUZXxwoN71rnflk5/8ZKqrq3PppZemsbEx3/nOd7Jz586cf/75ufPOO/2fFhzzVq9ePRFvk+Sxxx7Ltm3bsnDhwixYsGBi+2233ZZ58+aVY4hwyG6//fb8xm/8RpLk7LPPzkte8pJ97jdr1ixfgMYx7Z//+Z9z8cUXp7GxMcuXL8/ChQuza9eurF+/PqtXr87u3bvzohe9KN/85jezePHiAzrmtPvWxanw7//9v09ra2vuueeerF69Otu3b091dXUWLlyY3/md38nb3vY2X1vPMa+zszPJ2ErFv/mbv9nvfkuXLhW6OOY99thjE3Hr2e6///6Jn12bjmPBzTffnPPPPz9//ud/nnvuuSfDw8M54YQTcv311+fd7363r6GnELq7u/f5O3vjxo3ZuHHjxOOhoaEjOSyYUnv+Fk/GrrG4v4twL1myROjimPaSl7wkf/RHf5Tvf//7efjhh3P//fdneHg4HR0dufTSS/P6178+v/u7v3tQX4ZmRRcAAAAAheAaXQAAAAAUgtAFAAAAQCEIXQAAAAAUgtAFAAAAQCEIXQAAAAAUgtAFAAAAQCEIXQAAAAAUgtAFAAAAQCEIXQAAAAAUgtAFAAAAQCEIXQAAAAAUgtAFAAAAQCH8/3SbPpCh6Z22AAAAAElFTkSuQmCC\n",
"text/plain": [
"\n", "$$a^{(l)} = g^{(l)}\\left( a^{(l-1)} \\Theta^{(l)} + \\beta^{(l)} \\right). $$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "![Rys. 11.6. Wielowarstwowa sieć neuronowa - feedforward](nn2.png \"Rys. 11.6. Wielowarstwowa sieć neuronowa - feedforward\")" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "* Funkcje $g^{(l)}$ to tzw. **funkcje aktywacji**.
\n", "Dla $i = 0$ przyjmujemy $a^{(0)} = \\mathrm{x}$ (wektor wierszowy cech) oraz $g^{(0)}(x) = x$ (identyczność)." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "* Parametry $\\Theta$ to wagi na połączeniach miedzy neuronami dwóch warstw.
\n", "Rozmiar macierzy $\\Theta^{(l)}$, czyli macierzy wag na połączeniach warstw $a^{(l-1)}$ i $a^{(l)}$, to $\\dim(a^{(l-1)}) \\times \\dim(a^{(l)})$." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "* Parametry $\\beta$ zastępują tutaj dodawanie kolumny z jedynkami do macierzy cech.
Macierz $\\beta^{(l)}$ ma rozmiar równy liczbie neuronów w odpowiedniej warstwie, czyli $1 \\times \\dim(a^{(l)})$." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "* **Klasyfikacja**: dla ostatniej warstwy $L$ (o rozmiarze równym liczbie klas) przyjmuje się $g^{(L)}(x) = \\mathop{\\mathrm{softmax}}(x)$.\n", "* **Regresja**: pojedynczy neuron wyjściowy jak na obrazku. Funkcją aktywacji może wtedy być np. funkcja identycznościowa." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "* Pozostałe funkcje aktywacji najcześciej mają postać sigmoidy, np. sigmoidalna, tangens hiperboliczny.\n", "* Mogą mieć też inny kształt, np. ReLU, leaky ReLU, maxout." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "### Uczenie wielowarstwowych sieci neuronowych" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Mając algorytm SGD oraz gradienty wszystkich wag, moglibyśmy trenować każdą sieć." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "* Niech:\n", "$$\\Theta = (\\Theta^{(1)},\\Theta^{(2)},\\Theta^{(3)},\\beta^{(1)},\\beta^{(2)},\\beta^{(3)})$$\n", "\n", "* Funkcja sieci neuronowej z grafiki:\n", "\n", "$$\\small h_\\Theta(x) = \\tanh(\\tanh(\\tanh(x\\Theta^{(1)}+\\beta^{(1)})\\Theta^{(2)} + \\beta^{(2)})\\Theta^{(3)} + \\beta^{(3)})$$\n", "* Funkcja kosztu dla regresji:\n", "$$J(\\Theta) = \\dfrac{1}{2m} \\sum_{i=1}^{m} (h_\\Theta(x^{(i)})- y^{(i)})^2 $$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "Jak obliczymy gradienty?\n", "\n", "$$\\nabla_{\\Theta^{(l)}} J(\\Theta) = ? \\quad \\nabla_{\\beta^{(l)}} J(\\Theta) = ?$$\n", "\n", "* Postać funkcji kosztu zależna od wybranej architektury sieci oraz funkcji aktywacji." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "$$\\small J(\\Theta) = \\frac{1}{2}(a^{(L)} - y)^2 $$\n", "$$\\small \\dfrac{\\partial}{\\partial a^{(L)}} J(\\Theta) = a^{(L)} - y$$\n", "\n", "$$\\small \\tanh^{\\prime}(x) = 1 - \\tanh^2(x)$$" ] } ], "metadata": { "celltoolbar": "Slideshow", "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.6" }, "livereveal": { "start_slideshow_at": "selected", "theme": "white" } }, "nbformat": 4, "nbformat_minor": 4 }