Seminarium/prezentacja.ipynb

DST - Dialog State Tracking
Monitorowanie stanu dialogu

Czym jest DST?

Monitorowanie stanu dialogu polega na obserwowaniu jakie informacje dostarczył i czego oczekuję od systemu uzytkownik, na podstawię całego dialogu między systemem a użytkownikiem.

Najczęściej składa sie on z aktów dialogowych i wartości slotów. Akty dialogowe mówia systemowi jakie ma zadanie wykonać, a sloty jakie informacje potrzebuję od użytkownika do wykonania zadania.

Źródła:[3]

Dlaczego jest to potrzebne?

Prowadząc dialog z człowiekiem z natury wychodzi nam zapamietywanie oraz łaczenie poprzednich wypowiedzi w całość, nie potrzebujemy się zastanawiać co oznacza odpowiedź "tak" bo zrozumieć że jest to odpowidź na zadane przez nas pytanie. System dialogowy potrzebuję właśnie takiego rozumienia kontekstu danej wypowiedzi aby prawidłowo działać.

W podanym przykładowym dialogu system nie potrzebował pytać ponownie o miejsce oraz rodzaj jedzenia które szuka użytkownik gdyż zostało to już podane wcześniej systemowi.

Dobry system DST powinnien zawierać właściwości:

Incrementality: System powinnien aktualizować swój stan po każdej wypowiedzi

Context-awereness: System potrafi używać informacji z poprzednich tur dialogu

Trainability: System ulepsza się po dostarczeniu nowych danych

Portability: System powinnien działać w róznych aplikcajca w różnych dziedzinach, nawet bez danych treningowych z domeny

Problematyka tworzenia DST

Jedną z przeszkód stojących na drodzę do idealnego DST jest właściwe inerpretowanie tego co było już powiedziane a to co właśnie powiedział użytkownik. Trzeba rozpoznać każdą zmianę jaką chce dokonać użytkownik w istniejących już wartościach slotów. Jest to zwłaszcza utrudnione przez nie zawsze precyzyjne informcaję podane do systemu. Trzeba inertpretować do czego się odnosi oraz co oznacza zdanie np."w takim razie jakieś tanie miejsce" a bez kontekstu jest to nie możliwę.

Gdy użytkownik zapyta się o rzecz, która nie istnieje w bazie system musi odpowiednio zaragować. Proste odpowiedzi typu "Taka restauracja nie isteniej" mogą tylko zniechęcić użytkownika do korzystania z systemu. Trzeba dopasować odpowiedź do brakujących danych można zapytać czy użytkownich chciałby zmienić lokalizacje lub rodzaj restuaracji o którą pytał.

Problemy wynikają także z błednego odczytania co użytkownik powiedział(ASR) lub nie precyzyjnej odpowiedzi systemu(SLU). Takie błedy mimo dobrze działającego DST mogą spowodować minięcie się z oczekiwaniami użytkownika.

Rozwiązania w DST

Slot-filling

Hand-crafted

Generative approach

Discriminative approach

Źródła: [4]

Slot-filling

Proste podejście przypisywania w każdej turze określonego aktu dialogwego wraz z odpowiadającym mu określonym znaczeniem. Działo to tylko dla prostych dialogów z wcześniej zdefiniowaną ontologią.

Hand-crafted

Na podstawie gotowych danych ontologicznych i własnoręcznie zrobionych reguł przypisywania wartości slotów

Genartaive approach

Podejście polegające na tym że stany generują obserwacje, zakładają to że obserwacje na każdym etapie tury są niezależne od siebie. Umożliwia też tworzenie własnych ontlogi na podstawie historii dialogów oraz wyników jakie generuje NLU.

Discriminative approach

the dialogue state given arbitrary and possibly correlated input features.

Podejśćie polegające na tym że stan dialogu wynika z obserwacji, zakłada żę stany dialogu podane arbitralnie są możliwe połączone z wyjściem.

Recurrent neural network- Biorąc jako argument ostatni akt dialogu oraz to co napisał właśnie użytkownik, tworzy nowy act dialogu.

Dialog State Tracking Challenge 2 & 3 Data

Z powodu tego iż zbieranie danych jest bardzo kosztowne, mało kto mógł korzystać z dobrze przygotowanych danych aby rozwijać własne DST. Dlatego aby ułatwić rozwój powstał pierwszy Dialog State Tracking Challenge i jego następca Dialog State Tracking Challenge 2 & 3. DSTC 2 zawierał ponad 15 tysięcy dobrze opisanych dialogów udostępnionych aby ludzi mogli trenować swój system.

Źródła: [2] [1]

import json
f = open('log.json')
f2 = open('label.json')
data = json.load(f)
user = json.load(f2)
for i in range(len(data['turns'])):
    print(f'system: {data["turns"][i]["output"]["transcript"]}   dialog-acts-{data["turns"][i]["output"]["dialog-acts"]}')
    print(f'user:   {user["turns"][i]["transcription"]}      dialog-acts-{data["turns"][i]["output"]["dialog-acts"]}')

system: Hello , welcome to the Cambridge restaurant system? You can ask for restaurants by area , price range or food type . How may I help you?   dialog-acts-[{'slots': [], 'act': 'welcomemsg'}]
user:   expensive restaurant in the south part of town      dialog-acts-[{'slots': [], 'act': 'welcomemsg'}]
system: What kind of food would you like?   dialog-acts-[{'slots': [['slot', 'food']], 'act': 'request'}]
user:   any      dialog-acts-[{'slots': [['slot', 'food']], 'act': 'request'}]
system: the good luck chinese food takeaway is a nice restaurant in the south of town serving chinese food   dialog-acts-[{'slots': [['name', 'the good luck chinese food takeaway']], 'act': 'offer'}, {'slots': [['food', 'chinese']], 'act': 'inform'}, {'slots': [['pricerange', 'expensive']], 'act': 'inform'}, {'slots': [['area', 'south']], 'act': 'inform'}]
user:   address      dialog-acts-[{'slots': [['name', 'the good luck chinese food takeaway']], 'act': 'offer'}, {'slots': [['food', 'chinese']], 'act': 'inform'}, {'slots': [['pricerange', 'expensive']], 'act': 'inform'}, {'slots': [['area', 'south']], 'act': 'inform'}]
system: Sure , the good luck chinese food takeaway is on 82 Cherry Hinton Road Cherry Hinton   dialog-acts-[{'slots': [['name', 'the good luck chinese food takeaway']], 'act': 'offer'}, {'slots': [['addr', '82 cherry hinton road cherry hinton']], 'act': 'inform'}]
user:   type of food      dialog-acts-[{'slots': [['name', 'the good luck chinese food takeaway']], 'act': 'offer'}, {'slots': [['addr', '82 cherry hinton road cherry hinton']], 'act': 'inform'}]
system: the good luck chinese food takeaway is a nice restaurant in the south of town in the expensive price range   dialog-acts-[{'slots': [['name', 'the good luck chinese food takeaway']], 'act': 'offer'}, {'slots': [['pricerange', 'expensive']], 'act': 'inform'}, {'slots': [['area', 'south']], 'act': 'inform'}]
user:   thank you good bye      dialog-acts-[{'slots': [['name', 'the good luck chinese food takeaway']], 'act': 'offer'}, {'slots': [['pricerange', 'expensive']], 'act': 'inform'}, {'slots': [['area', 'south']], 'act': 'inform'}]

for i in range(len(data['turns'])):
    print(data["turns"][i]["input"]["live"]['asr-hyps'])
    for l in data["turns"][i]["input"]["live"]['asr-hyps']:
        print(l)

[{'asr-hyp': 'expensive restaurant in south part of town', 'score': -0.512136}, {'asr-hyp': 'a restaurant in south part of town', 'score': -2.429358}, {'asr-hyp': 'expensive expensive restaurant in south part of town', 'score': -2.429358}, {'asr-hyp': 'expensive restaurant restaurant in south part of town', 'score': -3.210609}, {'asr-hyp': 'expensive in south part of town', 'score': -3.211676}, {'asr-hyp': 'expensive restaurant south part of town', 'score': -3.347932}, {'asr-hyp': 'the expensive restaurant in south part of town', 'score': -4.268281}, {'asr-hyp': 'a expensive restaurant in south part of town', 'score': -4.34658}, {'asr-hyp': 'a restaurant restaurant in south part of town', 'score': -5.127831}, {'asr-hyp': 'expensive expensive restaurant restaurant in south part of town', 'score': -5.127831}]
{'asr-hyp': 'expensive restaurant in south part of town', 'score': -0.512136}
{'asr-hyp': 'a restaurant in south part of town', 'score': -2.429358}
{'asr-hyp': 'expensive expensive restaurant in south part of town', 'score': -2.429358}
{'asr-hyp': 'expensive restaurant restaurant in south part of town', 'score': -3.210609}
{'asr-hyp': 'expensive in south part of town', 'score': -3.211676}
{'asr-hyp': 'expensive restaurant south part of town', 'score': -3.347932}
{'asr-hyp': 'the expensive restaurant in south part of town', 'score': -4.268281}
{'asr-hyp': 'a expensive restaurant in south part of town', 'score': -4.34658}
{'asr-hyp': 'a restaurant restaurant in south part of town', 'score': -5.127831}
{'asr-hyp': 'expensive expensive restaurant restaurant in south part of town', 'score': -5.127831}
[{'asr-hyp': 'any', 'score': -0.02858}, {'asr-hyp': '', 'score': -4.777333}, {'asr-hyp': 'any me', 'score': -4.791492}, {'asr-hyp': 'any give', 'score': -4.885694}, {'asr-hyp': 'any the', 'score': -6.819719}, {'asr-hyp': 'in', 'score': -6.9403}, {'asr-hyp': 'any get', 'score': -7.371329}, {'asr-hyp': 'any a', 'score': -8.3173}, {'asr-hyp': 'and', 'score': -8.77938}, {'asr-hyp': 'any any', 'score': -8.946321}]
{'asr-hyp': 'any', 'score': -0.02858}
{'asr-hyp': '', 'score': -4.777333}
{'asr-hyp': 'any me', 'score': -4.791492}
{'asr-hyp': 'any give', 'score': -4.885694}
{'asr-hyp': 'any the', 'score': -6.819719}
{'asr-hyp': 'in', 'score': -6.9403}
{'asr-hyp': 'any get', 'score': -7.371329}
{'asr-hyp': 'any a', 'score': -8.3173}
{'asr-hyp': 'and', 'score': -8.77938}
{'asr-hyp': 'any any', 'score': -8.946321}
[{'asr-hyp': 'address', 'score': -0.455029}, {'asr-hyp': '', 'score': -2.265617}, {'asr-hyp': 'address address', 'score': -2.268945}, {'asr-hyp': 'the address', 'score': -2.274268}, {'asr-hyp': 'the', 'score': -4.084856}, {'asr-hyp': 'the address address', 'score': -4.088183}, {'asr-hyp': 'address their', 'score': -7.867221}, {'asr-hyp': 'address uh', 'score': -8.652037}, {'asr-hyp': 'and', 'score': -8.784164}, {'asr-hyp': 'i', 'score': -9.450777}]
{'asr-hyp': 'address', 'score': -0.455029}
{'asr-hyp': '', 'score': -2.265617}
{'asr-hyp': 'address address', 'score': -2.268945}
{'asr-hyp': 'the address', 'score': -2.274268}
{'asr-hyp': 'the', 'score': -4.084856}
{'asr-hyp': 'the address address', 'score': -4.088183}
{'asr-hyp': 'address their', 'score': -7.867221}
{'asr-hyp': 'address uh', 'score': -8.652037}
{'asr-hyp': 'and', 'score': -8.784164}
{'asr-hyp': 'i', 'score': -9.450777}
[{'asr-hyp': 'type of food', 'score': -0.016592}, {'asr-hyp': 'type type of food', 'score': -5.183987}, {'asr-hyp': 'the of food', 'score': -5.404256}, {'asr-hyp': 'of food', 'score': -6.505753}, {'asr-hyp': 'type food', 'score': -6.617192}, {'asr-hyp': 'type global of food', 'score': -7.340683}, {'asr-hyp': 'type budget of food', 'score': -7.360768}, {'asr-hyp': 'type the food', 'score': -7.5319}, {'asr-hyp': 'and of food', 'score': -7.617202}, {'asr-hyp': 'i of food', 'score': -7.727244}]
{'asr-hyp': 'type of food', 'score': -0.016592}
{'asr-hyp': 'type type of food', 'score': -5.183987}
{'asr-hyp': 'the of food', 'score': -5.404256}
{'asr-hyp': 'of food', 'score': -6.505753}
{'asr-hyp': 'type food', 'score': -6.617192}
{'asr-hyp': 'type global of food', 'score': -7.340683}
{'asr-hyp': 'type budget of food', 'score': -7.360768}
{'asr-hyp': 'type the food', 'score': -7.5319}
{'asr-hyp': 'and of food', 'score': -7.617202}
{'asr-hyp': 'i of food', 'score': -7.727244}
[{'asr-hyp': 'thank you goodbye', 'score': -0.30683}, {'asr-hyp': 'thank you goodbye bye', 'score': -2.107196}, {'asr-hyp': 'thank you good', 'score': -2.109648}, {'asr-hyp': 'thank you good bye', 'score': -3.910014}, {'asr-hyp': 'thank you you', 'score': -7.879262}, {'asr-hyp': 'thank goodbye', 'score': -7.957914}, {'asr-hyp': 'thank you goodbye bar', 'score': -8.884375}, {'asr-hyp': 'thank you you bye', 'score': -9.679628}, {'asr-hyp': 'thank goodbye bye', 'score': -9.758281}, {'asr-hyp': 'thank good', 'score': -9.760733}]
{'asr-hyp': 'thank you goodbye', 'score': -0.30683}
{'asr-hyp': 'thank you goodbye bye', 'score': -2.107196}
{'asr-hyp': 'thank you good', 'score': -2.109648}
{'asr-hyp': 'thank you good bye', 'score': -3.910014}
{'asr-hyp': 'thank you you', 'score': -7.879262}
{'asr-hyp': 'thank goodbye', 'score': -7.957914}
{'asr-hyp': 'thank you goodbye bar', 'score': -8.884375}
{'asr-hyp': 'thank you you bye', 'score': -9.679628}
{'asr-hyp': 'thank goodbye bye', 'score': -9.758281}
{'asr-hyp': 'thank good', 'score': -9.760733}

print(data['turns'][0])

{'output': {'transcript': 'Hello , welcome to the Cambridge restaurant system? You can ask for restaurants by area , price range or food type . How may I help you?', 'end-time': 9.93, 'start-time': 0.001283, 'dialog-acts': [{'slots': [], 'act': 'welcomemsg'}], 'aborted': False}, 'turn-index': 0, 'input': {'live': {'asr-hyps': [{'asr-hyp': 'expensive restaurant in south part of town', 'score': -0.512136}, {'asr-hyp': 'a restaurant in south part of town', 'score': -2.429358}, {'asr-hyp': 'expensive expensive restaurant in south part of town', 'score': -2.429358}, {'asr-hyp': 'expensive restaurant restaurant in south part of town', 'score': -3.210609}, {'asr-hyp': 'expensive in south part of town', 'score': -3.211676}, {'asr-hyp': 'expensive restaurant south part of town', 'score': -3.347932}, {'asr-hyp': 'the expensive restaurant in south part of town', 'score': -4.268281}, {'asr-hyp': 'a expensive restaurant in south part of town', 'score': -4.34658}, {'asr-hyp': 'a restaurant restaurant in south part of town', 'score': -5.127831}, {'asr-hyp': 'expensive expensive restaurant restaurant in south part of town', 'score': -5.127831}], 'slu-hyps': [{'slu-hyp': [{'slots': [['pricerange', 'expensive']], 'act': 'inform'}, {'slots': [['area', 'south']], 'act': 'inform'}], 'score': 0.9282257165548344}, {'slu-hyp': [{'slots': [['pricerange', 'expensive']], 'act': 'confirm'}, {'slots': [['area', 'south']], 'act': 'confirm'}], 'score': 0.02923086440308149}, {'slu-hyp': [{'slots': [['pricerange', 'expensive']], 'act': 'inform'}], 'score': 0.02236294307342127}, {'slu-hyp': [{'slots': [['area', 'south']], 'act': 'inform'}], 'score': 0.020180475968662844}]}, 'end-time': 12.19, 'start-time': 9.93, 'batch': {'cnet': [{'start': 0.0, 'end': 0.525, 'arcs': [{'score': -0.0001, 'word': '<s>'}, {'score': -5000000000.0, 'word': '!null'}]}, {'start': 0.525, 'end': 0.7875, 'arcs': [{'score': -19.41379, 'word': 'i'}, {'score': -16.88836, 'word': 'a'}, {'score': -11.6955, 'word': 'ah'}, {'score': -13.1059, 'word': 'uh'}, {'score': -0.0001, 'word': '!null'}]}, {'start': 0.7875, 'end': 1.05, 'arcs': [{'score': -11.16217, 'word': 'i'}, {'score': -9.916637, 'word': 'uh'}, {'score': -13.39644, 'word': 'ok'}, {'score': -0.0001, 'word': '!null'}]}, {'start': 1.05, 'end': 1.12375, 'arcs': [{'score': -0.115834, 'word': 'expensive'}, {'score': -9.822551, 'word': 'ok'}, {'score': -5.432811, 'word': 'the'}, {'score': -15.02797, 'word': 'extensive'}, {'score': -18.94896, 'word': 'are'}, {'score': -2.254255, 'word': 'a'}, {'score': -5000000000.0, 'word': '!null'}]}, {'start': 1.12375, 'end': 1.1975, 'arcs': [{'score': -17.54265, 'word': 'ok'}, {'score': -0.0001, 'word': '!null'}]}, {'start': 1.1975, 'end': 1.345, 'arcs': [{'score': -20.00645, 'word': 'expensive'}, {'score': -23.83931, 'word': 'extensive'}, {'score': -0.0001, 'word': '!null'}]}, {'start': 1.345, 'end': 1.4925, 'arcs': [{'score': -2.212958, 'word': 'expensive'}, {'score': -21.30504, 'word': 'extensive'}, {'score': -0.1158337, 'word': '!null'}]}, {'start': 1.4925, 'end': 1.64, 'arcs': [{'score': -19.56206, 'word': 'a'}, {'score': -15.88049, 'word': 'or'}, {'score': -19.37142, 'word': 'uh'}, {'score': -0.0001, 'word': '!null'}]}, {'start': 1.64, 'end': 1.7875, 'arcs': [{'score': -0.0001, 'word': 'restaurant'}, {'score': -14.9832, 'word': 'the'}, {'score': -11.33279, 'word': 'a'}, {'score': -9.494602, 'word': 'uh'}, {'score': -11.74803, 'word': 'or'}, {'score': -17.88491, 'word': '!null'}]}, {'start': 1.7875, 'end': 1.935, 'arcs': [{'score': -13.69902, 'word': 'restaurant'}, {'score': -0.0001, 'word': '!null'}]}, {'start': 1.935, 'end': 2.23, 'arcs': [{'score': -9.268782, 'word': 'restaurant'}, {'score': -2.381682, 'word': 'in'}, {'score': -0.09705002, 'word': '!null'}]}, {'start': 2.23, 'end': 2.27, 'arcs': [{'score': -0.1433645, 'word': 'in'}, {'score': -12.21065, 'word': 'or'}, {'score': -13.78668, 'word': 'i'}, {'score': -12.6393, 'word': 'are'}, {'score': -13.47915, 'word': 'a'}, {'score': -11.99133, 'word': "that's"}, {'score': -7.282886, 'word': 'that'}, {'score': -2.348785, 'word': 'the'}, {'score': -13.8318, 'word': 'this'}, {'score': -3.286841, 'word': '!null'}]}, {'start': 2.27, 'end': 2.31, 'arcs': [{'score': -46.60755, 'word': 'it'}, {'score': -12.64047, 'word': 'which'}, {'score': -22.6327, 'word': 'this'}, {'score': -12.27715, 'word': 'the'}, {'score': -34.75492, 'word': 'as'}, {'score': -6.923453, 'word': 'is'}, {'score': -7.139083, 'word': 'of'}, {'score': -10.67487, 'word': 'a'}, {'score': -7.923023, 'word': 'tea'}, {'score': -0.002173584, 'word': '!null'}]}, {'start': 2.31, 'end': 2.48, 'arcs': [{'score': -0.005246917, 'word': 'south'}, {'score': -37.23841, 'word': 'well'}, {'score': -10.74857, 'word': 'ok'}, {'score': -46.34189, 'word': 'gastro'}, {'score': -46.3515, 'word': 'aswell'}, {'score': -9.687854, 'word': 'child'}, {'score': -5.276869, 'word': 'shop'}, {'score': -41.20222, 'word': 'style'}, {'score': -10.09526, 'word': 'stop'}, {'score': -44.53774, 'word': 'sell'}, {'score': -5000000000.0, 'word': '!null'}]}, {'start': 2.48, 'end': 2.65, 'arcs': [{'score': -22.39381, 'word': 'is'}, {'score': -36.75235, 'word': 'of'}, {'score': -32.42216, 'word': 'how'}, {'score': -38.02597, 'word': 'in'}, {'score': -0.0001, 'word': '!null'}]}, {'start': 2.65, 'end': 2.74, 'arcs': [{'score': -0.0008488684, 'word': 'part'}, {'score': -25.69577, 'word': 'partner'}, {'score': -13.48137, 'word': 'what'}, {'score': -25.05888, 'word': 'post'}, {'score': -27.47913, 'word': 'are'}, {'score': -11.4708, 'word': 'close'}, {'score': -23.77218, 'word': 'park'}, {'score': -14.65481, 'word': 'code'}, {'score': -15.24271, 'word': 'cote'}, {'score': -15.58045, 'word': 'or'}, {'score': -12.75343, 'word': 'i'}, {'score': -15.39197, 'word': 'in'}, {'score': -18.09848, 'word': 'it'}, {'score': -8.927246, 'word': 'a'}, {'score': -20.65172, 'word': 'for'}, {'score': -31.9557, 'word': 'the'}, {'score': -7.264429, 'word': '!null'}]}, {'start': 2.74, 'end': 2.83, 'arcs': [{'score': -18.37638, 'word': 'what'}, {'score': -18.29665, 'word': 'for'}, {'score': -20.94735, 'word': 'or'}, {'score': -11.55013, 'word': 'on'}, {'score': -0.0001, 'word': '!null'}]}, {'start': 2.83, 'end': 2.94, 'arcs': [{'score': -0.000271893, 'word': 'of'}, {'score': -30.09363, 'word': 'post'}, {'score': -24.60016, 'word': "don't"}, {'score': -11.64039, 'word': 'the'}, {'score': -24.75157, 'word': 'okay'}, {'score': -8.243156, 'word': '!null'}]}, {'start': 2.94, 'end': 3.1, 'arcs': [{'score': -0.0001754378, 'word': 'town'}, {'score': -14.22256, 'word': 'cocum'}, {'score': -8.735506, 'word': 'hotel'}, {'score': -23.20746, 'word': 'care'}, {'score': -11.17799, 'word': 'tell'}, {'score': -24.11379, 'word': 'okay'}, {'score': -19.73659, 'word': 'down'}, {'score': -21.45679, 'word': 'it'}, {'score': -22.00763, 'word': '!null'}]}, {'start': 3.1, 'end': 3.18, 'arcs': [{'score': -20.51506, 'word': 'on'}, {'score': -21.36264, 'word': 'down'}, {'score': -0.0001, 'word': '!null'}]}, {'start': 3.18, 'end': 3.26, 'arcs': [{'score': -11.9611, 'word': 'a'}, {'score': -0.0001, 'word': '!null'}]}, {'start': 3.26, 'end': 4.14, 'arcs': [{'score': -0.0001, 'word': '</s>'}, {'score': -23.32922, 'word': '!null'}]}], 'asr-hyps': [{'asr-hyp': 'expensive restaurant in south part of town', 'score': -0.480987}, {'asr-hyp': 'expensive expensive restaurant in south part of town', 'score': -2.578111}, {'asr-hyp': 'a restaurant in south part of town', 'score': -2.619409}, {'asr-hyp': 'expensive restaurant the south part of town', 'score': -2.686408}, {'asr-hyp': 'expensive restaurant in in south part of town', 'score': -2.765619}, {'asr-hyp': 'expensive restaurant south part of town', 'score': -3.624464}, {'asr-hyp': 'a expensive restaurant in south part of town', 'score': -4.716533}, {'asr-hyp': 'expensive expensive restaurant the south part of town', 'score': -4.783532}, {'asr-hyp': 'a restaurant the south part of town', 'score': -4.82483}, {'asr-hyp': 'expensive expensive restaurant in in south part of town', 'score': -4.862743}]}, 'audio-file': 'pt344x_0000993_0001219.wav'}}

Bibliografia

[1] DSTC https://github.com/matthen/dstc

[2] The Second Dialog State Tracking Challenge https://aclanthology.org/W14-4337.pdf

[3] Dialogue State Tracking with Incremental Reasoning https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00384/101875/Dialogue-State-Tracking-with-Incremental-Reasoning

[4] https://arxiv.org/pdf/2101.08333.pdf

[5] https://www.cs.hhu.de/fileadmin/redaktion/Fakultaeten/Mathematisch-Naturwissenschaftliche_Fakultaet/Informatik/Dialog_Systems_and_Machine_Learning/L6.pdf