993 lines
42 KiB
Plaintext
993 lines
42 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"\n",
|
|
"<div class=\"alert alert-block alert-info\">\n",
|
|
"<h1> Modelowanie języka</h1>\n",
|
|
"<h2> 7. <i>Zanurzenia słów</i> [wykład]</h2> \n",
|
|
"<h3> Filip Graliński (2022)</h3>\n",
|
|
"</div>\n",
|
|
"\n",
|
|
"\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Zanurzenia słów\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"W praktyce stosowalność słowosieci okazała się zaskakująco\n",
|
|
"ograniczona. Większy przełom w przetwarzaniu języka naturalnego przyniosły\n",
|
|
"wielowymiarowe reprezentacje słów, inaczej: zanurzenia słów.\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### „Wymiary” słów\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Moglibyśmy zanurzyć (ang. *embed*) w wielowymiarowej przestrzeni, tzn. zdefiniować odwzorowanie\n",
|
|
"$E \\colon V \\rightarrow \\mathcal{R}^m$ dla pewnego $m$ i określić taki sposób estymowania\n",
|
|
"prawdopodobieństw $P(u|v)$, by dla par $E(v)$ i $E(v')$ oraz $E(u)$ i $E(u')$ znajdujących się w pobliżu\n",
|
|
"(według jakiejś metryki odległości, na przykład zwykłej odległości euklidesowej):\n",
|
|
"\n",
|
|
"$$P(u|v) \\approx P(u'|v').$$\n",
|
|
"\n",
|
|
"$E(u)$ nazywamy zanurzeniem (embeddingiem) słowa.\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Wymiary określone z góry?\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Można by sobie wyobrazić, że $m$ wymiarów mogłoby być z góry\n",
|
|
"określonych przez lingwistę. Wymiary te byłyby związane z typowymi\n",
|
|
"„osiami” rozpatrywanymi w językoznawstwie, na przykład:\n",
|
|
"\n",
|
|
"- czy słowo jest wulgarne, pospolite, potoczne, neutralne czy książkowe?\n",
|
|
"- czy słowo jest archaiczne, wychodzące z użycia czy jest neologizmem?\n",
|
|
"- czy słowo dotyczy kobiet, czy mężczyzn (w sensie rodzaju gramatycznego i/lub\n",
|
|
" socjolingwistycznym)?\n",
|
|
"- czy słowo jest w liczbie pojedynczej czy mnogiej?\n",
|
|
"- czy słowo jest rzeczownikiem czy czasownikiem?\n",
|
|
"- czy słowo jest rdzennym słowem czy zapożyczeniem?\n",
|
|
"- czy słowo jest nazwą czy słowem pospolitym?\n",
|
|
"- czy słowo opisuje konkretną rzecz czy pojęcie abstrakcyjne?\n",
|
|
"- …\n",
|
|
"\n",
|
|
"W praktyce okazało się jednak, że lepiej, żeby komputer uczył się sam\n",
|
|
"możliwych wymiarów — z góry określamy tylko $m$ (liczbę wymiarów).\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Bigramowy model języka oparty na zanurzeniach\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Zbudujemy teraz najprostszy model język oparty na zanurzeniach. Będzie to właściwie najprostszy\n",
|
|
"**neuronowy model języka**, jako że zbudowany model można traktować jako prostą sieć neuronową.\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Słownik\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"W typowym neuronowym modelu języka rozmiar słownika musi być z góry\n",
|
|
"ograniczony. Zazwyczaj jest to liczba rzędu kilkudziesięciu wyrazów —\n",
|
|
"po prostu będziemy rozpatrywać $|V|$ najczęstszych wyrazów, pozostałe zamienimy\n",
|
|
"na specjalny token `<unk>` reprezentujący nieznany (*unknown*) wyraz.\n",
|
|
"\n",
|
|
"Aby utworzyć taki słownik użyjemy gotowej klasy `Vocab` z pakietu torchtext:\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"/media/kuba/ssdsam/anaconda3/envs/lmzajecia/lib/python3.10/site-packages/torch/_masked/__init__.py:223: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at /opt/conda/conda-bld/pytorch_1646755897462/work/torch/csrc/utils/tensor_numpy.cpp:68.)\n",
|
|
" example_input = torch.tensor([[-3, -2, -1], [0, 1, 2]])\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"16"
|
|
]
|
|
},
|
|
"execution_count": 2,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"from itertools import islice\n",
|
|
"import regex as re\n",
|
|
"import sys\n",
|
|
"from torchtext.vocab import build_vocab_from_iterator\n",
|
|
"\n",
|
|
"\n",
|
|
"def get_words_from_line(line):\n",
|
|
" line = line.rstrip()\n",
|
|
" yield '<s>'\n",
|
|
" for m in re.finditer(r'[\\p{L}0-9\\*]+|\\p{P}+', line):\n",
|
|
" yield m.group(0).lower()\n",
|
|
" yield '</s>'\n",
|
|
"\n",
|
|
"\n",
|
|
"def get_word_lines_from_file(file_name):\n",
|
|
" with open(file_name, 'r') as fh:\n",
|
|
" for line in fh:\n",
|
|
" yield get_words_from_line(line)\n",
|
|
"\n",
|
|
"vocab_size = 20000\n",
|
|
"\n",
|
|
"vocab = build_vocab_from_iterator(\n",
|
|
" get_word_lines_from_file('opensubtitlesA.pl.txt'),\n",
|
|
" max_tokens = vocab_size,\n",
|
|
" specials = ['<unk>'])\n",
|
|
"\n",
|
|
"vocab['jest']"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"['<unk>', '</s>', '<s>', 'w', 'policyjny']"
|
|
]
|
|
},
|
|
"execution_count": 3,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"vocab.lookup_tokens([0, 1, 2, 10, 12345])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Definicja sieci\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Naszą prostą sieć neuronową zaimplementujemy używając frameworku PyTorch.\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"ename": "NameError",
|
|
"evalue": "name 'out' is not defined",
|
|
"output_type": "error",
|
|
"traceback": [
|
|
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
|
|
"\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)",
|
|
"Input \u001b[0;32mIn [4]\u001b[0m, in \u001b[0;36m<cell line: 22>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 20\u001b[0m vocab\u001b[38;5;241m.\u001b[39mset_default_index(vocab[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m<unk>\u001b[39m\u001b[38;5;124m'\u001b[39m])\n\u001b[1;32m 21\u001b[0m ixs \u001b[38;5;241m=\u001b[39m torch\u001b[38;5;241m.\u001b[39mtensor(vocab\u001b[38;5;241m.\u001b[39mforward([\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mpies\u001b[39m\u001b[38;5;124m'\u001b[39m]))\n\u001b[0;32m---> 22\u001b[0m \u001b[43mout\u001b[49m[\u001b[38;5;241m0\u001b[39m][vocab[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mjest\u001b[39m\u001b[38;5;124m'\u001b[39m]]\n",
|
|
"\u001b[0;31mNameError\u001b[0m: name 'out' is not defined"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from torch import nn\n",
|
|
"import torch\n",
|
|
"\n",
|
|
"embed_size = 100\n",
|
|
"\n",
|
|
"class SimpleBigramNeuralLanguageModel(nn.Module):\n",
|
|
" def __init__(self, vocabulary_size, embedding_size):\n",
|
|
" super(SimpleBigramNeuralLanguageModel, self).__init__()\n",
|
|
" self.model = nn.Sequential(\n",
|
|
" nn.Embedding(vocabulary_size, embedding_size),\n",
|
|
" nn.Linear(embedding_size, vocabulary_size),\n",
|
|
" nn.Softmax()\n",
|
|
" )\n",
|
|
"\n",
|
|
" def forward(self, x):\n",
|
|
" return self.model(x)\n",
|
|
"\n",
|
|
"model = SimpleBigramNeuralLanguageModel(vocab_size, embed_size)\n",
|
|
"\n",
|
|
"vocab.set_default_index(vocab['<unk>'])\n",
|
|
"ixs = torch.tensor(vocab.forward(['pies']))\n",
|
|
"out[0][vocab['jest']]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Teraz wyuczmy model. Wpierw tylko potasujmy nasz plik:\n",
|
|
"\n",
|
|
" shuf < opensubtitlesA.pl.txt > opensubtitlesA.pl.shuf.txt\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"!shuf < opensubtitlesA.pl.txt > opensubtitlesA.pl.shuf.txt"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from torch.utils.data import IterableDataset\n",
|
|
"import itertools\n",
|
|
"\n",
|
|
"def look_ahead_iterator(gen):\n",
|
|
" prev = None\n",
|
|
" for item in gen:\n",
|
|
" if prev is not None:\n",
|
|
" yield (prev, item)\n",
|
|
" prev = item\n",
|
|
"\n",
|
|
"class Bigrams(IterableDataset):\n",
|
|
" def __init__(self, text_file, vocabulary_size):\n",
|
|
" self.vocab = build_vocab_from_iterator(\n",
|
|
" get_word_lines_from_file(text_file),\n",
|
|
" max_tokens = vocabulary_size,\n",
|
|
" specials = ['<unk>'])\n",
|
|
" self.vocab.set_default_index(self.vocab['<unk>'])\n",
|
|
" self.vocabulary_size = vocabulary_size\n",
|
|
" self.text_file = text_file\n",
|
|
"\n",
|
|
" def __iter__(self):\n",
|
|
" return look_ahead_iterator(\n",
|
|
" (self.vocab[t] for t in itertools.chain.from_iterable(get_word_lines_from_file(self.text_file))))\n",
|
|
"\n",
|
|
"train_dataset = Bigrams('opensubtitlesA.pl.shuf.txt', vocab_size)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"(2, 72)"
|
|
]
|
|
},
|
|
"execution_count": 8,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"from torch.utils.data import DataLoader\n",
|
|
"\n",
|
|
"next(iter(train_dataset))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 9,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"[tensor([ 2, 72, 615, 11, 92]), tensor([ 72, 615, 11, 92, 4])]"
|
|
]
|
|
},
|
|
"execution_count": 9,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"from torch.utils.data import DataLoader\n",
|
|
"\n",
|
|
"next(iter(DataLoader(train_dataset, batch_size=5)))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 13,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"/media/kuba/ssdsam/anaconda3/envs/lmzajecia/lib/python3.10/site-packages/torch/nn/modules/container.py:141: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n",
|
|
" input = module(input)\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"0 tensor(10.2158, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"100 tensor(6.9743, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"200 tensor(6.2186, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"300 tensor(5.6430, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"400 tensor(5.3539, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"500 tensor(5.0689, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"600 tensor(4.9418, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"700 tensor(4.8142, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"800 tensor(4.6436, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"900 tensor(4.6770, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"1000 tensor(4.6069, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"1100 tensor(4.5514, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"1200 tensor(4.5288, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"1300 tensor(4.4578, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"1400 tensor(4.5290, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"1500 tensor(4.5229, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"1600 tensor(4.4973, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"1700 tensor(4.3793, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"1800 tensor(4.5056, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"1900 tensor(4.3709, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"2000 tensor(4.3841, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"2100 tensor(4.4515, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"2200 tensor(4.3367, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"2300 tensor(4.4187, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"2400 tensor(4.3672, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"2500 tensor(4.3117, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"2600 tensor(4.2908, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"2700 tensor(4.3188, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"2800 tensor(4.2870, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"2900 tensor(4.2855, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"3000 tensor(4.2927, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"3100 tensor(4.3358, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"3200 tensor(4.2719, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"3300 tensor(4.2606, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"3400 tensor(4.2953, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"3500 tensor(4.3175, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"3600 tensor(4.2448, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"3700 tensor(4.2430, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"3800 tensor(4.2586, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"3900 tensor(4.2905, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"4000 tensor(4.2455, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"4100 tensor(4.2214, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"4200 tensor(4.2325, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"4300 tensor(4.3036, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"4400 tensor(4.2335, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"4500 tensor(4.2377, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"4600 tensor(4.2109, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"4700 tensor(4.2942, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"4800 tensor(4.2234, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"4900 tensor(4.1918, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"5000 tensor(4.3084, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"5100 tensor(4.1666, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"5200 tensor(4.2307, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"5300 tensor(4.2050, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"5400 tensor(4.1853, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"5500 tensor(4.1917, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"5600 tensor(4.1453, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"5700 tensor(4.2423, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"5800 tensor(4.1972, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"5900 tensor(4.2143, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"6000 tensor(4.2172, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"6100 tensor(4.2463, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"6200 tensor(4.1756, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"6300 tensor(4.1223, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"6400 tensor(4.1852, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"6500 tensor(4.1559, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"6600 tensor(4.1833, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"6700 tensor(4.2090, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"6800 tensor(4.1896, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"6900 tensor(4.2057, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"7000 tensor(4.1523, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"7100 tensor(4.2645, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"7200 tensor(4.1974, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"7300 tensor(4.2031, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"7400 tensor(4.1613, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"7500 tensor(4.2018, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"7600 tensor(4.2197, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"7700 tensor(4.1976, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"7800 tensor(4.1650, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"7900 tensor(4.1380, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"8000 tensor(4.1014, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"8100 tensor(4.2058, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"8200 tensor(4.1514, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"8300 tensor(4.1187, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"8400 tensor(4.2438, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"8500 tensor(4.2094, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"8600 tensor(4.2077, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"8700 tensor(4.0819, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"8800 tensor(4.1766, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"8900 tensor(4.1805, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"9000 tensor(4.1847, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"9100 tensor(4.1929, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"9200 tensor(4.1434, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"9300 tensor(4.1678, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"9400 tensor(4.1699, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"9500 tensor(4.0885, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"9600 tensor(4.1544, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"9700 tensor(4.1828, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"9800 tensor(4.1314, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"9900 tensor(4.1473, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"10000 tensor(4.0948, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"10100 tensor(4.1396, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"10200 tensor(4.1999, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"10300 tensor(4.1027, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"10400 tensor(4.2049, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"10500 tensor(4.1470, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"10600 tensor(4.0974, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"10700 tensor(4.1239, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"10800 tensor(4.1381, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"10900 tensor(4.0569, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"11000 tensor(4.1138, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"11100 tensor(4.2053, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"11200 tensor(4.1404, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"11300 tensor(4.0741, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"11400 tensor(4.0090, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"11500 tensor(4.1568, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"11600 tensor(4.1498, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"11700 tensor(4.1052, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"11800 tensor(4.0600, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"11900 tensor(4.1274, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"12000 tensor(4.1346, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"12100 tensor(4.1024, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"12200 tensor(4.0966, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"12300 tensor(4.1036, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"12400 tensor(4.0127, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"12500 tensor(4.0575, device='cuda:0', grad_fn=<NllLossBackward0>)\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"12600 tensor(4.1542, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"12700 tensor(4.1810, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"12800 tensor(4.1948, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"12900 tensor(4.1085, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"13000 tensor(4.1283, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"13100 tensor(4.1548, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"13200 tensor(4.1015, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"13300 tensor(4.1342, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"13400 tensor(4.0724, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"13500 tensor(4.1006, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"13600 tensor(4.0998, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"13700 tensor(4.1021, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"13800 tensor(4.1175, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"13900 tensor(4.1017, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"14000 tensor(4.1877, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"14100 tensor(4.1664, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"14200 tensor(4.1582, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"14300 tensor(4.1526, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"14400 tensor(4.1208, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"14500 tensor(4.0752, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"14600 tensor(4.1907, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"14700 tensor(4.0496, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"14800 tensor(4.1371, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"14900 tensor(4.1215, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"15000 tensor(4.1059, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"15100 tensor(4.0888, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"15200 tensor(4.1359, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"15300 tensor(4.1328, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"15400 tensor(4.1044, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"15500 tensor(4.1167, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"15600 tensor(4.0449, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"15700 tensor(4.1159, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"15800 tensor(4.1082, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"15900 tensor(4.1653, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"16000 tensor(4.1111, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"16100 tensor(4.0870, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"16200 tensor(4.1085, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"16300 tensor(4.1216, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"16400 tensor(4.1307, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"16500 tensor(4.0872, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"16600 tensor(4.0754, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"16700 tensor(4.0067, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"16800 tensor(4.0413, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"16900 tensor(4.1242, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"17000 tensor(4.1169, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"17100 tensor(4.0942, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"17200 tensor(4.1518, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"17300 tensor(4.0968, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"17400 tensor(4.0476, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"17500 tensor(4.0230, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"17600 tensor(4.1268, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"17700 tensor(4.0388, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"17800 tensor(4.1741, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"17900 tensor(4.1147, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"18000 tensor(4.2020, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"18100 tensor(4.0304, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"18200 tensor(4.1171, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"18300 tensor(4.0945, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"18400 tensor(4.1019, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"18500 tensor(4.1301, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"18600 tensor(4.0979, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"18700 tensor(4.0755, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"18800 tensor(4.0760, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"18900 tensor(4.0553, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"19000 tensor(4.1530, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"19100 tensor(4.1403, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"19200 tensor(4.1449, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"19300 tensor(4.0105, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"19400 tensor(4.0742, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"19500 tensor(4.0666, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"19600 tensor(4.1549, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"19700 tensor(4.0930, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"19800 tensor(4.1271, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"19900 tensor(4.1169, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"20000 tensor(4.1053, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"20100 tensor(4.1070, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"20200 tensor(4.0848, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"20300 tensor(4.1330, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"20400 tensor(3.9828, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"20500 tensor(4.1411, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"20600 tensor(4.0537, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"20700 tensor(4.1171, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"20800 tensor(4.0510, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"20900 tensor(4.1230, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"21000 tensor(4.1241, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"21100 tensor(4.1600, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"21200 tensor(4.0699, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"21300 tensor(4.0870, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"21400 tensor(4.0774, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"21500 tensor(4.1492, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"21600 tensor(4.0883, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"21700 tensor(4.0358, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"21800 tensor(4.0569, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"21900 tensor(4.0832, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"22000 tensor(4.0827, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"22100 tensor(4.0534, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"22200 tensor(4.0173, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"22300 tensor(4.0549, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"22400 tensor(4.0613, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"22500 tensor(4.1058, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"22600 tensor(4.1230, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"22700 tensor(4.1114, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"22800 tensor(4.0541, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"22900 tensor(4.0732, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"23000 tensor(4.0983, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"23100 tensor(4.0547, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"23200 tensor(4.1198, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"23300 tensor(4.0687, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"23400 tensor(4.0676, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"23500 tensor(4.0834, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"23600 tensor(4.0996, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"23700 tensor(4.0791, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"23800 tensor(4.0700, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"23900 tensor(4.0388, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"24000 tensor(4.0625, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"24100 tensor(4.1095, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"24200 tensor(4.1589, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"24300 tensor(4.1565, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"24400 tensor(4.1396, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"24500 tensor(4.1642, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"24600 tensor(4.0868, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"24700 tensor(4.0770, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"24800 tensor(4.0577, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"24900 tensor(4.0662, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"25000 tensor(4.0877, device='cuda:0', grad_fn=<NllLossBackward0>)\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"25100 tensor(4.0505, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"25200 tensor(4.1542, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"25300 tensor(4.0740, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"25400 tensor(4.0893, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"25500 tensor(4.0370, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"25600 tensor(4.1480, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"25700 tensor(4.1070, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"25800 tensor(4.0381, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"25900 tensor(4.0800, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"26000 tensor(4.0842, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"26100 tensor(4.1127, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"26200 tensor(4.1184, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"26300 tensor(4.0885, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"26400 tensor(4.1423, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"26500 tensor(4.1359, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"26600 tensor(4.0986, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"26700 tensor(4.0580, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"26800 tensor(4.0806, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"26900 tensor(4.0169, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"27000 tensor(4.1111, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"27100 tensor(4.1417, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"27200 tensor(4.1497, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"27300 tensor(4.1093, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"27400 tensor(4.0306, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"27500 tensor(4.1214, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"27600 tensor(4.0745, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"27700 tensor(4.0559, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"27800 tensor(4.0286, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"27900 tensor(4.1266, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"28000 tensor(3.9690, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"28100 tensor(4.1141, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"28200 tensor(4.0565, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"28300 tensor(4.0682, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"28400 tensor(4.0646, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"28500 tensor(4.0386, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"28600 tensor(4.0903, device='cuda:0', grad_fn=<NllLossBackward0>)\n",
|
|
"28700 tensor(4.1060, device='cuda:0', grad_fn=<NllLossBackward0>)\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
" device = 'cuda'\n",
|
|
" model = SimpleBigramNeuralLanguageModel(vocab_size, embed_size).to(device)\n",
|
|
" data = DataLoader(train_dataset, batch_size=5000)\n",
|
|
" optimizer = torch.optim.Adam(model.parameters())\n",
|
|
" criterion = torch.nn.NLLLoss()\n",
|
|
" \n",
|
|
" model.train()\n",
|
|
" step = 0\n",
|
|
" for x, y in data:\n",
|
|
" x = x.to(device)\n",
|
|
" y = y.to(device)\n",
|
|
" optimizer.zero_grad()\n",
|
|
" ypredicted = model(x)\n",
|
|
" loss = criterion(torch.log(ypredicted), y)\n",
|
|
" if step % 100 == 0:\n",
|
|
" print(step, loss)\n",
|
|
" step += 1\n",
|
|
" loss.backward()\n",
|
|
" optimizer.step()\n",
|
|
" \n",
|
|
" torch.save(model.state_dict(), 'model1.bin')\n",
|
|
"\n",
|
|
"#Policzmy najbardziej prawdopodobne kontynuację dla zadanego słowa:\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 14,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"SimpleBigramNeuralLanguageModel(\n",
|
|
" (model): Sequential(\n",
|
|
" (0): Embedding(20000, 100)\n",
|
|
" (1): Linear(in_features=100, out_features=20000, bias=True)\n",
|
|
" (2): Softmax(dim=None)\n",
|
|
" )\n",
|
|
")"
|
|
]
|
|
},
|
|
"execution_count": 14,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 15,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"[('mnie', 26, 0.16004179418087006),\n",
|
|
" ('ciebie', 73, 0.13592898845672607),\n",
|
|
" ('<unk>', 0, 0.12769868969917297),\n",
|
|
" ('nas', 83, 0.04033529385924339),\n",
|
|
" ('niego', 172, 0.033195145428180695),\n",
|
|
" ('niej', 247, 0.021507620811462402),\n",
|
|
" ('was', 162, 0.017743170261383057),\n",
|
|
" ('siebie', 181, 0.01618184894323349),\n",
|
|
" ('nich', 222, 0.01589815877377987),\n",
|
|
" ('pana', 156, 0.014923062175512314)]"
|
|
]
|
|
},
|
|
"execution_count": 15,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"device = 'cuda'\n",
|
|
"model = SimpleBigramNeuralLanguageModel(vocab_size, embed_size).to(device)\n",
|
|
"model.load_state_dict(torch.load('model1.bin'))\n",
|
|
"model.eval()\n",
|
|
"\n",
|
|
"ixs = torch.tensor(vocab.forward(['dla'])).to(device)\n",
|
|
"\n",
|
|
"out = model(ixs)\n",
|
|
"top = torch.topk(out[0], 10)\n",
|
|
"top_indices = top.indices.tolist()\n",
|
|
"top_probs = top.values.tolist()\n",
|
|
"top_words = vocab.lookup_tokens(top_indices)\n",
|
|
"list(zip(top_words, top_indices, top_probs))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Teraz zbadajmy najbardziej podobne zanurzenia dla zadanego słowa:\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 16,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"[('.', 3, 0.3327740728855133),\n",
|
|
" ('z', 14, 0.191472589969635),\n",
|
|
" (',', 4, 0.18250100314617157),\n",
|
|
" ('w', 10, 0.06395534425973892),\n",
|
|
" ('?', 6, 0.059775471687316895),\n",
|
|
" ('i', 11, 0.019332991912961006),\n",
|
|
" ('ze', 60, 0.016418060287833214),\n",
|
|
" ('<unk>', 0, 0.014098692685365677),\n",
|
|
" ('na', 12, 0.01183203887194395),\n",
|
|
" ('...', 15, 0.010537521913647652)]"
|
|
]
|
|
},
|
|
"execution_count": 16,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"vocab = train_dataset.vocab\n",
|
|
"ixs = torch.tensor(vocab.forward(['kłopot'])).to(device)\n",
|
|
"\n",
|
|
"out = model(ixs)\n",
|
|
"top = torch.topk(out[0], 10)\n",
|
|
"top_indices = top.indices.tolist()\n",
|
|
"top_probs = top.values.tolist()\n",
|
|
"top_words = vocab.lookup_tokens(top_indices)\n",
|
|
"list(zip(top_words, top_indices, top_probs))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 17,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"[('poszedł', 1088, 1.0),\n",
|
|
" ('wsiąść', 9766, 0.46510031819343567),\n",
|
|
" ('pojedzie', 6485, 0.4598822593688965),\n",
|
|
" ('wyjeżdża', 6459, 0.4378735423088074),\n",
|
|
" ('szedłem', 8969, 0.4232063889503479),\n",
|
|
" ('zadzwoniłem', 4889, 0.41752171516418457),\n",
|
|
" ('dotrzemy', 6098, 0.40929487347602844),\n",
|
|
" ('spóźnić', 9923, 0.4015277922153473),\n",
|
|
" ('pójdę', 635, 0.3992091119289398),\n",
|
|
" ('wrócimy', 2070, 0.39785560965538025)]"
|
|
]
|
|
},
|
|
"execution_count": 17,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"cos = nn.CosineSimilarity(dim=1, eps=1e-6)\n",
|
|
"\n",
|
|
"embeddings = model.model[0].weight\n",
|
|
"\n",
|
|
"vec = embeddings[vocab['poszedł']]\n",
|
|
"\n",
|
|
"similarities = cos(vec, embeddings)\n",
|
|
"\n",
|
|
"top = torch.topk(similarities, 10)\n",
|
|
"\n",
|
|
"top_indices = top.indices.tolist()\n",
|
|
"top_probs = top.values.tolist()\n",
|
|
"top_words = vocab.lookup_tokens(top_indices)\n",
|
|
"list(zip(top_words, top_indices, top_probs))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Zapis przy użyciu wzoru matematycznego\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Powyżej zaprogramowaną sieć neuronową można opisać następującym wzorem:\n",
|
|
"\n",
|
|
"$$\\vec{y} = \\operatorname{softmax}(CE(w_{i-1}),$$\n",
|
|
"\n",
|
|
"gdzie:\n",
|
|
"\n",
|
|
"- $w_{i-1}$ to pierwszy wyraz w bigramie (poprzedzający wyraz),\n",
|
|
"- $E(w)$ to zanurzenie (embedding) wyrazy $w$ — wektor o rozmiarze $m$,\n",
|
|
"- $C$ to macierz o rozmiarze $|V| \\times m$, która rzutuje wektor zanurzenia w wektor o rozmiarze słownika,\n",
|
|
"- $\\vec{y}$ to wyjściowy wektor prawdopodobieństw o rozmiarze $|V|$.\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"##### Hiperparametry\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Zauważmy, że nasz model ma dwa hiperparametry:\n",
|
|
"\n",
|
|
"- $m$ — rozmiar zanurzenia,\n",
|
|
"- $|V|$ — rozmiar słownika, jeśli zakładamy, że możemy sterować\n",
|
|
" rozmiarem słownika (np. przez obcinanie słownika do zadanej liczby\n",
|
|
" najczęstszych wyrazów i zamiany pozostałych na specjalny token, powiedzmy, `<UNK>`.\n",
|
|
"\n",
|
|
"Oczywiście możemy próbować manipulować wartościami $m$ i $|V|$ w celu\n",
|
|
"polepszenia wyników naszego modelu.\n",
|
|
"\n",
|
|
"**Pytanie**: dlaczego nie ma sensu wartość $m \\approx |V|$ ? dlaczego nie ma sensu wartość $m = 1$?\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Diagram sieci\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Jako że mnożenie przez macierz ($C$) oznacza po prostu zastosowanie\n",
|
|
"warstwy liniowej, naszą sieć możemy interpretować jako jednowarstwową\n",
|
|
"sieć neuronową, co można zilustrować za pomocą następującego diagramu:\n",
|
|
"\n",
|
|
"![img](./07_Zanurzenia_slow/bigram1.drawio.png \"Diagram prostego bigramowego neuronowego modelu języka\")\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Zanurzenie jako mnożenie przez macierz\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Uzyskanie zanurzenia ($E(w)$) zazwyczaj realizowane jest na zasadzie\n",
|
|
"odpytania (<sub>look</sub>-up\\_). Co ciekawe, zanurzenie można intepretować jako\n",
|
|
"mnożenie przez macierz zanurzeń (embeddingów) $E$ o rozmiarze $m \\times |V|$ — jeśli słowo będziemy na wejściu kodowali przy użyciu\n",
|
|
"wektora z gorącą jedynką (<sub>one</sub>-hot encoding\\_), tzn. słowo $w$ zostanie\n",
|
|
"podane na wejściu jako wektor $\\vec{1_V}(w) = [0,\\ldots,0,1,0\\ldots,0]$ o rozmiarze $|V|$\n",
|
|
"złożony z samych zer z wyjątkiem jedynki na pozycji odpowiadającej indeksowi wyrazu $w$ w słowniku $V$.\n",
|
|
"\n",
|
|
"Wówczas wzór przyjmie postać:\n",
|
|
"\n",
|
|
"$$\\vec{y} = \\operatorname{softmax}(CE\\vec{1_V}(w_{i-1})),$$\n",
|
|
"\n",
|
|
"gdzie $E$ będzie tym razem macierzą $m \\times |V|$.\n",
|
|
"\n",
|
|
"**Pytanie**: czy $\\vec{1_V}(w)$ intepretujemy jako wektor wierszowy czy kolumnowy?\n",
|
|
"\n",
|
|
"W postaci diagramu można tę interpretację zilustrować w następujący sposób:\n",
|
|
"\n",
|
|
"![img](./07_Zanurzenia_slow/bigram2.drawio.png \"Diagram prostego bigramowego neuronowego modelu języka z wejściem w postaci one-hot\")\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.10.4"
|
|
},
|
|
"org": null
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 1
|
|
}
|