Anna Nowak 3833bbc547 poprawki

2021-05-16 12:24:44 +02:00

40 KiB

Raw Blame History

Parsing semantyczny z wykorzystaniem technik uczenia maszynowego

Wprowadzenie

Problem wykrywania slotów i ich wartości w wypowiedziach użytkownika można sformułować jako zadanie polegające na przewidywaniu dla poszczególnych słów etykiet wskazujących na to czy i do jakiego slotu dane słowo należy.

chciałbym zarezerwować stolik na jutro**/day** na godzinę dwunastą**/hour** czterdzieści**/hour** pięć**/hour** na pięć**/size** osób

Granice slotów oznacza się korzystając z wybranego schematu etykietowania.

Schemat IOB

Prefix	Znaczenie
I	wnętrze slotu (inside)
O	poza slotem (outside)
B	początek slotu (beginning)

chciałbym zarezerwować stolik na jutro**/B-day** na godzinę dwunastą**/B-hour** czterdzieści**/I-hour** pięć**/I-hour** na pięć**/B-size** osób

Schemat IOBES

Prefix	Znaczenie
I	wnętrze slotu (inside)
O	poza slotem (outside)
B	początek slotu (beginning)
E	koniec slotu (ending)
S	pojedyncze słowo (single)

chciałbym zarezerwować stolik na jutro**/S-day** na godzinę dwunastą**/B-hour** czterdzieści**/I-hour** pięć**/E-hour** na pięć**/S-size** osób

Jeżeli dla tak sformułowanego zadania przygotujemy zbiór danych złożony z wypowiedzi użytkownika z oznaczonymi slotami (tzw. _zbiór uczący), to możemy zastosować techniki (nadzorowanego) uczenia maszynowego w celu zbudowania modelu annotującego wypowiedzi użytkownika etykietami slotów.

Do zbudowania takiego modelu można wykorzystać między innymi:

warunkowe pola losowe (Lafferty i in.; 2001),
rekurencyjne sieci neuronowe, np. sieci LSTM (Hochreiter i Schmidhuber; 1997),
transformery (Vaswani i in., 2017).

Przykład

Skorzystamy ze zbioru danych przygotowanego przez Schustera (2019).

!mkdir -p l07
%cd l07
!curl -L -C -  https://fb.me/multilingual_task_oriented_data  -o data.zip
%cd ..

C:\Users\Ania\Desktop\System_Dialogowy_Janet\l07

A subdirectory or file -p already exists.
Error occurred while processing: -p.
A subdirectory or file l07 already exists.
Error occurred while processing: l07.

C:\Users\Ania\Desktop\System_Dialogowy_Janet

** Resuming transfer from byte position 8923190
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100    49  100    49    0     0     56      0 --:--:-- --:--:-- --:--:--   742

Zbiór ten gromadzi wypowiedzi w trzech językach opisane slotami dla dwunastu ram należących do trzech dziedzin Alarm, Reminder oraz Weather. Dane wczytamy korzystając z biblioteki conllu.

!pip3 install conllu
import codecs
from conllu import parse_incr
fields = ['id', 'form', 'frame', 'slot']

def nolabel2o(line, i):
    return 'O' if line[i] == 'NoLabel' else line[i]

with open('Janet_test.conllu', encoding='utf-8') as trainfile:
    trainset = list(parse_incr(trainfile, fields=fields, field_parsers={'slot': nolabel2o}))
with open('Janet_test.conllu', encoding='utf-8') as testfile:
    testset = list(parse_incr(testfile, fields=fields, field_parsers={'slot': nolabel2o}))

Requirement already satisfied: conllu in c:\programdata\anaconda3\lib\site-packages (4.4)

Zobaczmy kilka przykładowych wypowiedzi z tego zbioru.

!pip3 install tabulate
from tabulate import tabulate
tabulate(trainset[0], tablefmt='html')

Requirement already satisfied: tabulate in c:\programdata\anaconda3\lib\site-packages (0.8.9)

hej

greeting

Na potrzeby prezentacji procesu uczenia w jupyterowym notatniku zawęzimy zbiór danych do początkowych przykładów.

Budując model skorzystamy z architektury opartej o rekurencyjne sieci neuronowe zaimplementowanej w bibliotece flair (Akbik i in. 2018).

!pip3 install flair

Requirement already satisfied: flair in c:\programdata\anaconda3\lib\site-packages (0.8.0.post1)
Requirement already satisfied: huggingface-hub in c:\programdata\anaconda3\lib\site-packages (from flair) (0.0.8)
Requirement already satisfied: mpld3==0.3 in c:\programdata\anaconda3\lib\site-packages (from flair) (0.3)
Requirement already satisfied: langdetect in c:\programdata\anaconda3\lib\site-packages (from flair) (1.0.9)
Requirement already satisfied: hyperopt>=0.1.1 in c:\programdata\anaconda3\lib\site-packages (from flair) (0.2.5)
Requirement already satisfied: tabulate in c:\programdata\anaconda3\lib\site-packages (from flair) (0.8.9)
Requirement already satisfied: torch<=1.7.1,>=1.5.0 in c:\programdata\anaconda3\lib\site-packages (from flair) (1.7.1)
Requirement already satisfied: matplotlib>=2.2.3 in c:\programdata\anaconda3\lib\site-packages (from flair) (3.3.2)
Requirement already satisfied: sentencepiece==0.1.95 in c:\programdata\anaconda3\lib\site-packages (from flair) (0.1.95)
Requirement already satisfied: numpy<1.20.0 in c:\programdata\anaconda3\lib\site-packages (from flair) (1.19.2)
Requirement already satisfied: sqlitedict>=1.6.0 in c:\programdata\anaconda3\lib\site-packages (from flair) (1.7.0)
Requirement already satisfied: lxml in c:\programdata\anaconda3\lib\site-packages (from flair) (4.6.1)
Requirement already satisfied: python-dateutil>=2.6.1 in c:\programdata\anaconda3\lib\site-packages (from flair) (2.8.1)
Requirement already satisfied: gensim<=3.8.3,>=3.4.0 in c:\programdata\anaconda3\lib\site-packages (from flair) (3.8.3)
Requirement already satisfied: deprecated>=1.2.4 in c:\programdata\anaconda3\lib\site-packages (from flair) (1.2.12)
Requirement already satisfied: gdown==3.12.2 in c:\programdata\anaconda3\lib\site-packages (from flair) (3.12.2)
Requirement already satisfied: janome in c:\programdata\anaconda3\lib\site-packages (from flair) (0.4.1)
Requirement already satisfied: konoha<5.0.0,>=4.0.0 in c:\programdata\anaconda3\lib\site-packages (from flair) (4.6.4)
Requirement already satisfied: bpemb>=0.3.2 in c:\programdata\anaconda3\lib\site-packages (from flair) (0.3.3)
Requirement already satisfied: tqdm>=4.26.0 in c:\programdata\anaconda3\lib\site-packages (from flair) (4.50.2)
Requirement already satisfied: segtok>=1.5.7 in c:\programdata\anaconda3\lib\site-packages (from flair) (1.5.10)
Requirement already satisfied: transformers>=4.0.0 in c:\programdata\anaconda3\lib\site-packages (from flair) (4.6.0)
Requirement already satisfied: regex in c:\programdata\anaconda3\lib\site-packages (from flair) (2020.10.15)
Requirement already satisfied: scikit-learn>=0.21.3 in c:\programdata\anaconda3\lib\site-packages (from flair) (0.23.2)
Requirement already satisfied: ftfy in c:\programdata\anaconda3\lib\site-packages (from flair) (6.0.1)
Requirement already satisfied: filelock in c:\programdata\anaconda3\lib\site-packages (from huggingface-hub->flair) (3.0.12)
Requirement already satisfied: requests in c:\programdata\anaconda3\lib\site-packages (from huggingface-hub->flair) (2.24.0)
Requirement already satisfied: six in c:\programdata\anaconda3\lib\site-packages (from langdetect->flair) (1.15.0)
Requirement already satisfied: future in c:\programdata\anaconda3\lib\site-packages (from hyperopt>=0.1.1->flair) (0.18.2)
Requirement already satisfied: scipy in c:\programdata\anaconda3\lib\site-packages (from hyperopt>=0.1.1->flair) (1.5.2)
Requirement already satisfied: cloudpickle in c:\programdata\anaconda3\lib\site-packages (from hyperopt>=0.1.1->flair) (1.6.0)
Requirement already satisfied: networkx>=2.2 in c:\programdata\anaconda3\lib\site-packages (from hyperopt>=0.1.1->flair) (2.5)
Requirement already satisfied: typing-extensions in c:\programdata\anaconda3\lib\site-packages (from torch<=1.7.1,>=1.5.0->flair) (3.7.4.3)
Requirement already satisfied: certifi>=2020.06.20 in c:\programdata\anaconda3\lib\site-packages (from matplotlib>=2.2.3->flair) (2020.6.20)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in c:\programdata\anaconda3\lib\site-packages (from matplotlib>=2.2.3->flair) (2.4.7)
Requirement already satisfied: pillow>=6.2.0 in c:\programdata\anaconda3\lib\site-packages (from matplotlib>=2.2.3->flair) (8.0.1)
Requirement already satisfied: kiwisolver>=1.0.1 in c:\programdata\anaconda3\lib\site-packages (from matplotlib>=2.2.3->flair) (1.3.0)
Requirement already satisfied: cycler>=0.10 in c:\programdata\anaconda3\lib\site-packages (from matplotlib>=2.2.3->flair) (0.10.0)
Requirement already satisfied: Cython==0.29.14 in c:\programdata\anaconda3\lib\site-packages (from gensim<=3.8.3,>=3.4.0->flair) (0.29.14)
Requirement already satisfied: smart-open>=1.8.1 in c:\programdata\anaconda3\lib\site-packages (from gensim<=3.8.3,>=3.4.0->flair) (5.0.0)
Requirement already satisfied: wrapt<2,>=1.10 in c:\users\ania\appdata\roaming\python\python38\site-packages (from deprecated>=1.2.4->flair) (1.12.1)
Requirement already satisfied: importlib-metadata<4.0.0,>=3.7.0 in c:\programdata\anaconda3\lib\site-packages (from konoha<5.0.0,>=4.0.0->flair) (3.10.1)
Requirement already satisfied: overrides<4.0.0,>=3.0.0 in c:\programdata\anaconda3\lib\site-packages (from konoha<5.0.0,>=4.0.0->flair) (3.1.0)
Requirement already satisfied: packaging in c:\programdata\anaconda3\lib\site-packages (from transformers>=4.0.0->flair) (20.4)
Requirement already satisfied: tokenizers<0.11,>=0.10.1 in c:\programdata\anaconda3\lib\site-packages (from transformers>=4.0.0->flair) (0.10.2)
Requirement already satisfied: sacremoses in c:\programdata\anaconda3\lib\site-packages (from transformers>=4.0.0->flair) (0.0.45)
Requirement already satisfied: joblib>=0.11 in c:\programdata\anaconda3\lib\site-packages (from scikit-learn>=0.21.3->flair) (0.17.0)
Requirement already satisfied: threadpoolctl>=2.0.0 in c:\programdata\anaconda3\lib\site-packages (from scikit-learn>=0.21.3->flair) (2.1.0)
Requirement already satisfied: wcwidth in c:\programdata\anaconda3\lib\site-packages (from ftfy->flair) (0.2.5)
Requirement already satisfied: idna<3,>=2.5 in c:\programdata\anaconda3\lib\site-packages (from requests->huggingface-hub->flair) (2.10)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:\programdata\anaconda3\lib\site-packages (from requests->huggingface-hub->flair) (1.25.11)
Requirement already satisfied: chardet<4,>=3.0.2 in c:\programdata\anaconda3\lib\site-packages (from requests->huggingface-hub->flair) (3.0.4)
Requirement already satisfied: decorator>=4.3.0 in c:\programdata\anaconda3\lib\site-packages (from networkx>=2.2->hyperopt>=0.1.1->flair) (4.4.2)
Requirement already satisfied: zipp>=0.5 in c:\programdata\anaconda3\lib\site-packages (from importlib-metadata<4.0.0,>=3.7.0->konoha<5.0.0,>=4.0.0->flair) (3.4.0)
Requirement already satisfied: click in c:\programdata\anaconda3\lib\site-packages (from sacremoses->transformers>=4.0.0->flair) (7.1.2)

from flair.data import Corpus, Sentence, Token
from flair.datasets import SentenceDataset
from flair.embeddings import StackedEmbeddings
from flair.embeddings import WordEmbeddings
from flair.embeddings import CharacterEmbeddings
from flair.embeddings import FlairEmbeddings
from flair.models import SequenceTagger
from flair.trainers import ModelTrainer

!pip3 install torch
# determinizacja obliczeń
import random
import torch
random.seed(42)
torch.manual_seed(42)

if torch.cuda.is_available():
    torch.cuda.manual_seed(0)
    torch.cuda.manual_seed_all(0)
    torch.backends.cudnn.enabled = False
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True

Requirement already satisfied: torch in c:\programdata\anaconda3\lib\site-packages (1.7.1)
Requirement already satisfied: typing-extensions in c:\programdata\anaconda3\lib\site-packages (from torch) (3.7.4.3)
Requirement already satisfied: numpy in c:\programdata\anaconda3\lib\site-packages (from torch) (1.19.2)

Dane skonwertujemy do formatu wykorzystywanego przez flair, korzystając z następującej funkcji.

def conllu2flair(sentences, label=None):
    fsentences = []

    for sentence in sentences:
        fsentence = Sentence()

        for token in sentence:
            ftoken = Token(token['form'])

            if label:
                ftoken.add_tag(label, token[label])

            fsentence.add_token(ftoken)

        fsentences.append(fsentence)

    return SentenceDataset(fsentences)

corpus = Corpus(train=conllu2flair(trainset, 'slot'), test=conllu2flair(testset, 'slot'))
print(corpus)
tag_dictionary = corpus.make_tag_dictionary(tag_type='slot')
print(tag_dictionary)

Corpus: 36 train + 4 dev + 40 test sentences
Dictionary with 13 tags: <unk>, O, B-appoinment/doctor, I-appoinment/doctor, B-datetime, I-datetime, B-login/id, B-appointment/type, I-appointment/type, B-prescription/type, B-login/password, <START>, <STOP>

Nasz model będzie wykorzystywał wektorowe reprezentacje słów (zob. Word Embeddings).

embedding_types = [
    WordEmbeddings('pl'),
    FlairEmbeddings('pl-forward'),
    FlairEmbeddings('pl-backward'),
    CharacterEmbeddings(),
]

embeddings = StackedEmbeddings(embeddings=embedding_types)
tagger = SequenceTagger(hidden_size=256, embeddings=embeddings,
                        tag_dictionary=tag_dictionary,
                        tag_type='slot', use_crf=True)

Zobaczmy jak wygląda architektura sieci neuronowej, która będzie odpowiedzialna za przewidywanie slotów w wypowiedziach.

print(tagger)

SequenceTagger(
  (embeddings): StackedEmbeddings(
    (list_embedding_0): WordEmbeddings('pl')
    (list_embedding_1): FlairEmbeddings(
      (lm): LanguageModel(
        (drop): Dropout(p=0.25, inplace=False)
        (encoder): Embedding(1602, 100)
        (rnn): LSTM(100, 2048)
        (decoder): Linear(in_features=2048, out_features=1602, bias=True)
      )
    )
    (list_embedding_2): FlairEmbeddings(
      (lm): LanguageModel(
        (drop): Dropout(p=0.25, inplace=False)
        (encoder): Embedding(1602, 100)
        (rnn): LSTM(100, 2048)
        (decoder): Linear(in_features=2048, out_features=1602, bias=True)
      )
    )
    (list_embedding_3): CharacterEmbeddings(
      (char_embedding): Embedding(275, 25)
      (char_rnn): LSTM(25, 25, bidirectional=True)
    )
  )
  (word_dropout): WordDropout(p=0.05)
  (locked_dropout): LockedDropout(p=0.5)
  (embedding2nn): Linear(in_features=4446, out_features=4446, bias=True)
  (rnn): LSTM(4446, 256, batch_first=True, bidirectional=True)
  (linear): Linear(in_features=512, out_features=13, bias=True)
  (beta): 1.0
  (weights): None
  (weight_tensor) None
)

Wykonamy dziesięć iteracji (epok) uczenia a wynikowy model zapiszemy w katalogu slot-model.

trainer = ModelTrainer(tagger, corpus)
trainer.train('slot-model',
              learning_rate=0.1,
              mini_batch_size=32,
              max_epochs=10,
              train_with_dev=False)

2021-05-16 11:40:14,273 ----------------------------------------------------------------------------------------------------
2021-05-16 11:40:14,274 Model: "SequenceTagger(
  (embeddings): StackedEmbeddings(
    (list_embedding_0): WordEmbeddings('pl')
    (list_embedding_1): FlairEmbeddings(
      (lm): LanguageModel(
        (drop): Dropout(p=0.25, inplace=False)
        (encoder): Embedding(1602, 100)
        (rnn): LSTM(100, 2048)
        (decoder): Linear(in_features=2048, out_features=1602, bias=True)
      )
    )
    (list_embedding_2): FlairEmbeddings(
      (lm): LanguageModel(
        (drop): Dropout(p=0.25, inplace=False)
        (encoder): Embedding(1602, 100)
        (rnn): LSTM(100, 2048)
        (decoder): Linear(in_features=2048, out_features=1602, bias=True)
      )
    )
    (list_embedding_3): CharacterEmbeddings(
      (char_embedding): Embedding(275, 25)
      (char_rnn): LSTM(25, 25, bidirectional=True)
    )
  )
  (word_dropout): WordDropout(p=0.05)
  (locked_dropout): LockedDropout(p=0.5)
  (embedding2nn): Linear(in_features=4446, out_features=4446, bias=True)
  (rnn): LSTM(4446, 256, batch_first=True, bidirectional=True)
  (linear): Linear(in_features=512, out_features=13, bias=True)
  (beta): 1.0
  (weights): None
  (weight_tensor) None
)"
2021-05-16 11:40:14,275 ----------------------------------------------------------------------------------------------------
2021-05-16 11:40:14,277 Corpus: "Corpus: 36 train + 4 dev + 40 test sentences"
2021-05-16 11:40:14,277 ----------------------------------------------------------------------------------------------------
2021-05-16 11:40:14,278 Parameters:
2021-05-16 11:40:14,279  - learning_rate: "0.1"
2021-05-16 11:40:14,280  - mini_batch_size: "32"
2021-05-16 11:40:14,280  - patience: "3"
2021-05-16 11:40:14,281  - anneal_factor: "0.5"
2021-05-16 11:40:14,282  - max_epochs: "10"
2021-05-16 11:40:14,283  - shuffle: "True"
2021-05-16 11:40:14,285  - train_with_dev: "False"
2021-05-16 11:40:14,286  - batch_growth_annealing: "False"
2021-05-16 11:40:14,287 ----------------------------------------------------------------------------------------------------
2021-05-16 11:40:14,288 Model training base path: "slot-model"
2021-05-16 11:40:14,288 ----------------------------------------------------------------------------------------------------
2021-05-16 11:40:14,289 Device: cpu
2021-05-16 11:40:14,290 ----------------------------------------------------------------------------------------------------
2021-05-16 11:40:14,292 Embeddings storage mode: cpu
2021-05-16 11:40:14,295 ----------------------------------------------------------------------------------------------------
2021-05-16 11:40:18,737 epoch 1 - iter 1/2 - loss 13.17695141 - samples/sec: 7.21 - lr: 0.100000
2021-05-16 11:40:19,989 epoch 1 - iter 2/2 - loss 11.51309586 - samples/sec: 25.57 - lr: 0.100000
2021-05-16 11:40:19,989 ----------------------------------------------------------------------------------------------------
2021-05-16 11:40:19,989 EPOCH 1 done: loss 11.5131 - lr 0.1000000
2021-05-16 11:40:20,670 DEV : loss 5.320306777954102 - score 0.0
2021-05-16 11:40:20,671 BAD EPOCHS (no improvement): 0
saving best model
2021-05-16 11:40:30,073 ----------------------------------------------------------------------------------------------------
2021-05-16 11:40:30,802 epoch 2 - iter 1/2 - loss 8.20096970 - samples/sec: 45.04 - lr: 0.100000
2021-05-16 11:40:31,005 epoch 2 - iter 2/2 - loss 5.87843704 - samples/sec: 157.40 - lr: 0.100000
2021-05-16 11:40:31,006 ----------------------------------------------------------------------------------------------------
2021-05-16 11:40:31,008 EPOCH 2 done: loss 5.8784 - lr 0.1000000
2021-05-16 11:40:31,020 DEV : loss 2.201185703277588 - score 0.0
2021-05-16 11:40:31,038 BAD EPOCHS (no improvement): 0
saving best model
2021-05-16 11:40:40,878 ----------------------------------------------------------------------------------------------------
2021-05-16 11:40:41,800 epoch 3 - iter 1/2 - loss 3.59802794 - samples/sec: 34.83 - lr: 0.100000
2021-05-16 11:40:42,230 epoch 3 - iter 2/2 - loss 7.24588382 - samples/sec: 74.64 - lr: 0.100000
2021-05-16 11:40:42,231 ----------------------------------------------------------------------------------------------------
2021-05-16 11:40:42,233 EPOCH 3 done: loss 7.2459 - lr 0.1000000
2021-05-16 11:40:42,290 DEV : loss 2.3815672397613525 - score 0.0
2021-05-16 11:40:42,295 BAD EPOCHS (no improvement): 1
2021-05-16 11:40:42,300 ----------------------------------------------------------------------------------------------------
2021-05-16 11:40:43,662 epoch 4 - iter 1/2 - loss 4.05115032 - samples/sec: 23.57 - lr: 0.100000
2021-05-16 11:40:44,013 epoch 4 - iter 2/2 - loss 3.16846037 - samples/sec: 91.53 - lr: 0.100000
2021-05-16 11:40:44,015 ----------------------------------------------------------------------------------------------------
2021-05-16 11:40:44,018 EPOCH 4 done: loss 3.1685 - lr 0.1000000
2021-05-16 11:40:44,072 DEV : loss 1.7660648822784424 - score 0.0
2021-05-16 11:40:44,075 BAD EPOCHS (no improvement): 0
saving best model
2021-05-16 11:40:53,620 ----------------------------------------------------------------------------------------------------
2021-05-16 11:40:54,419 epoch 5 - iter 1/2 - loss 3.52825356 - samples/sec: 40.10 - lr: 0.100000
2021-05-16 11:40:54,594 epoch 5 - iter 2/2 - loss 3.12245941 - samples/sec: 183.91 - lr: 0.100000
2021-05-16 11:40:54,595 ----------------------------------------------------------------------------------------------------
2021-05-16 11:40:54,596 EPOCH 5 done: loss 3.1225 - lr 0.1000000
2021-05-16 11:40:54,624 DEV : loss 1.8835055828094482 - score 0.0
2021-05-16 11:40:54,626 BAD EPOCHS (no improvement): 1
2021-05-16 11:40:54,627 ----------------------------------------------------------------------------------------------------
2021-05-16 11:40:55,393 epoch 6 - iter 1/2 - loss 2.84318709 - samples/sec: 41.88 - lr: 0.100000
2021-05-16 11:40:55,648 epoch 6 - iter 2/2 - loss 4.79819477 - samples/sec: 125.98 - lr: 0.100000
2021-05-16 11:40:55,649 ----------------------------------------------------------------------------------------------------
2021-05-16 11:40:55,650 EPOCH 6 done: loss 4.7982 - lr 0.1000000
2021-05-16 11:40:55,675 DEV : loss 1.9106686115264893 - score 0.0
2021-05-16 11:40:55,677 BAD EPOCHS (no improvement): 2
2021-05-16 11:40:55,678 ----------------------------------------------------------------------------------------------------
2021-05-16 11:40:56,467 epoch 7 - iter 1/2 - loss 3.35292196 - samples/sec: 40.66 - lr: 0.100000
2021-05-16 11:40:56,661 epoch 7 - iter 2/2 - loss 1.90253919 - samples/sec: 165.80 - lr: 0.100000
2021-05-16 11:40:56,662 ----------------------------------------------------------------------------------------------------
2021-05-16 11:40:56,663 EPOCH 7 done: loss 1.9025 - lr 0.1000000
2021-05-16 11:40:56,689 DEV : loss 1.5785303115844727 - score 0.0
2021-05-16 11:40:56,691 BAD EPOCHS (no improvement): 0
saving best model
2021-05-16 11:41:09,226 ----------------------------------------------------------------------------------------------------
2021-05-16 11:41:10,375 epoch 8 - iter 1/2 - loss 3.24992299 - samples/sec: 27.87 - lr: 0.100000
2021-05-16 11:41:10,744 epoch 8 - iter 2/2 - loss 3.30123496 - samples/sec: 87.17 - lr: 0.100000
2021-05-16 11:41:10,745 ----------------------------------------------------------------------------------------------------
2021-05-16 11:41:10,746 EPOCH 8 done: loss 3.3012 - lr 0.1000000
2021-05-16 11:41:10,798 DEV : loss 1.590420126914978 - score 0.0
2021-05-16 11:41:10,802 BAD EPOCHS (no improvement): 1
2021-05-16 11:41:10,807 ----------------------------------------------------------------------------------------------------
2021-05-16 11:41:12,175 epoch 9 - iter 1/2 - loss 2.74546242 - samples/sec: 23.41 - lr: 0.100000
2021-05-16 11:41:12,515 epoch 9 - iter 2/2 - loss 2.34704965 - samples/sec: 94.40 - lr: 0.100000
2021-05-16 11:41:12,518 ----------------------------------------------------------------------------------------------------
2021-05-16 11:41:12,520 EPOCH 9 done: loss 2.3470 - lr 0.1000000
2021-05-16 11:41:12,573 DEV : loss 1.6068150997161865 - score 0.0
2021-05-16 11:41:12,575 BAD EPOCHS (no improvement): 2
2021-05-16 11:41:12,577 ----------------------------------------------------------------------------------------------------
2021-05-16 11:41:13,690 epoch 10 - iter 1/2 - loss 2.63941884 - samples/sec: 28.79 - lr: 0.100000
2021-05-16 11:41:13,878 epoch 10 - iter 2/2 - loss 2.18226165 - samples/sec: 171.12 - lr: 0.100000
2021-05-16 11:41:13,879 ----------------------------------------------------------------------------------------------------
2021-05-16 11:41:13,880 EPOCH 10 done: loss 2.1823 - lr 0.1000000
2021-05-16 11:41:13,906 DEV : loss 1.458857536315918 - score 0.0
2021-05-16 11:41:13,907 BAD EPOCHS (no improvement): 0
saving best model
2021-05-16 11:41:33,558 ----------------------------------------------------------------------------------------------------
2021-05-16 11:41:33,559 Testing using best model ...
2021-05-16 11:41:33,560 loading file slot-model\best-model.pt
2021-05-16 11:41:45,502 0.1765	0.1667	0.1714
2021-05-16 11:41:45,503 
Results:
- F1-score (micro) 0.1714
- F1-score (macro) 0.1161

By class:
appoinment/doctor tp: 1 - fp: 9 - fn: 5 - precision: 0.1000 - recall: 0.1667 - f1-score: 0.1250
appointment/type tp: 0 - fp: 0 - fn: 2 - precision: 0.0000 - recall: 0.0000 - f1-score: 0.0000
datetime   tp: 0 - fp: 1 - fn: 3 - precision: 0.0000 - recall: 0.0000 - f1-score: 0.0000
login/id   tp: 2 - fp: 2 - fn: 1 - precision: 0.5000 - recall: 0.6667 - f1-score: 0.5714
login/password tp: 0 - fp: 0 - fn: 3 - precision: 0.0000 - recall: 0.0000 - f1-score: 0.0000
prescription/type tp: 0 - fp: 2 - fn: 1 - precision: 0.0000 - recall: 0.0000 - f1-score: 0.0000
2021-05-16 11:41:45,503 ----------------------------------------------------------------------------------------------------

{'test_score': 0.17142857142857143,
 'dev_score_history': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
 'train_loss_history': [11.51309585571289,
  5.878437042236328,
  7.245883822441101,
  3.1684603691101074,
  3.1224594116210938,
  4.798194766044617,
  1.9025391936302185,
  3.3012349605560303,
  2.347049653530121,
  2.182261645793915],
 'dev_loss_history': [5.320306777954102,
  2.201185703277588,
  2.3815672397613525,
  1.7660648822784424,
  1.8835055828094482,
  1.9106686115264893,
  1.5785303115844727,
  1.590420126914978,
  1.6068150997161865,
  1.458857536315918]}

Jakość wyuczonego modelu możemy ocenić, korzystając z zaraportowanych powyżej metryk, tj.:

_tp (true positives)

liczba słów oznaczonych w zbiorze testowym etykietą $e$, które model oznaczył tą etykietą
_fp (false positives)

liczba słów nieoznaczonych w zbiorze testowym etykietą $e$, które model oznaczył tą etykietą
_fn (false negatives)

liczba słów oznaczonych w zbiorze testowym etykietą $e$, którym model nie nadał etykiety $e$
_precision

$$\frac{tp}{tp + fp}$$
_recall

$$\frac{tp}{tp + fn}$$
$F_1$

$$\frac{2 \cdot precision \cdot recall}{precision + recall}$$
_micro $F_1$

$F_1$ w którym $tp$, $fp$ i $fn$ są liczone łącznie dla wszystkich etykiet, tj. $tp = \sum_{e}{{tp}_e}$, $fn = \sum{e}{{fn}e}$, $fp = \sum{e}{{fp}_e}$
_macro $F_1$

średnia arytmetyczna z $F_1$ obliczonych dla poszczególnych etykiet z osobna.

Wyuczony model możemy wczytać z pliku korzystając z metody load.

model = SequenceTagger.load('slot-model/final-model.pt')

2021-05-16 11:41:45,529 loading file slot-model/final-model.pt

Wczytany model możemy wykorzystać do przewidywania slotów w wypowiedziach użytkownika, korzystając z przedstawionej poniżej funkcji predict.

def predict(model, sentence):
    csentence = [{'form': word} for word in sentence]
    fsentence = conllu2flair([csentence])[0]
    model.predict(fsentence)
    return [(token, ftoken.get_tag('slot').value) for token, ftoken in zip(sentence, fsentence)]

Jak pokazuje przykład poniżej model wyuczony tylko na 100 przykładach popełnia w dosyć prostej wypowiedzi błąd etykietując słowo alarm tagiem B-weather/noun.

tabulate(predict(model, 'doktor lekarza rodzinnego najlepiej dzisiaj w godzinach popołudniowych dziś '.split()), tablefmt='html')

doktor	I-appoinment/doctor
lekarza	B-appoinment/doctor
rodzinnego	O
najlepiej	O
dzisiaj	O
w	O
godzinach	O
popołudniowych	O
dziś	O

Literatura

Sebastian Schuster, Sonal Gupta, Rushin Shah, Mike Lewis, Cross-lingual Transfer Learning for Multilingual Task Oriented Dialog. NAACL-HLT (1) 2019, pp. 3795-3805
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. In Proceedings of the Eighteenth International Conference on Machine Learning (ICML '01). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 282–289, https://repository.upenn.edu/cgi/viewcontent.cgi?article=1162&context=cis_papers
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Comput. 9, 8 (November 15, 1997), 1735–1780, https://doi.org/10.1162/neco.1997.9.8.1735
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, Attention is All you Need, NIPS 2017, pp. 5998-6008, https://arxiv.org/abs/1706.03762
Alan Akbik, Duncan Blythe, Roland Vollgraf, Contextual String Embeddings for Sequence Labeling, Proceedings of the 27th International Conference on Computational Linguistics, pp. 1638–1649, https://www.aclweb.org/anthology/C18-1139.pdf

40 KiB Raw Blame History Unescape Escape