Seminarium/prezentacja.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**DST** -  Dialog State Tracking   \n",
    "Monitorowanie stanu dialogu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Czym jest DST?**\n",
    "\n",
    "Monitorowanie stanu dialogu polega na obserwowaniu jakie informacje dostarczył i czego oczekuję od systemu uzytkownik, na podstawię całego dialogu między systemem a użytkownikiem. \n",
    "\n",
    "Najczęściej składa sie on z **aktów dialogowych** i  wartości **slotów**. Akty dialogowe mówia systemowi jakie ma zadanie wykonać, a sloty jakie informacje potrzebuję od użytkownika do wykonania zadania.\n",
    "\n",
    "Źródła:[3]\n",
    "\n",
    "![](DST.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Dlaczego jest to potrzebne?**\n",
    "\n",
    "Prowadząc dialog z człowiekiem z natury wychodzi nam zapamietywanie oraz łaczenie poprzednich wypowiedzi w całość, nie potrzebujemy się zastanawiać co oznacza odpowiedź \"tak\" bo zrozumieć że jest to odpowidź na zadane przez nas pytanie. System dialogowy potrzebuję właśnie takiego rozumienia kontekstu danej wypowiedzi aby prawidłowo działać."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![Dialog](Dialog_1.png)\n",
    "\n",
    "W podanym przykładowym dialogu system nie potrzebował pytać ponownie o miejsce oraz rodzaj jedzenia które szuka użytkownik gdyż zostało to już podane wcześniej systemowi. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Dobry system DST powinnien zawierać właściwości:**\n",
    "\n",
    "Incrementality: System powinnien aktualizować swój stan po każdej wypowiedzi \n",
    "\n",
    "Context-awereness: System potrafi używać informacji z poprzednich tur dialogu \n",
    "\n",
    "Trainability: System ulepsza się po dostarczeniu nowych danych \n",
    "\n",
    "Portability: System powinnien działać w róznych aplikcajca w różnych dziedzinach, nawet bez danych treningowych z domeny"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Problematyka tworzenia DST**\n",
    "\n",
    "Jedną z przeszkód stojących na drodzę do idealnego DST jest właściwe inerpretowanie tego co było już powiedziane a to co właśnie powiedział użytkownik. Trzeba rozpoznać każdą zmianę jaką chce dokonać użytkownik w istniejących już wartościach slotów. Jest to zwłaszcza utrudnione przez nie zawsze precyzyjne informcaję podane do systemu. Trzeba inertpretować do czego się odnosi oraz co oznacza zdanie np.\"w takim razie jakieś tanie miejsce\" a bez kontekstu jest to nie możliwę.\n",
    "\n",
    "Gdy użytkownik zapyta się o rzecz, która nie istnieje w bazie system musi odpowiednio zaragować. Proste odpowiedzi typu \"Taka restauracja nie isteniej\" mogą tylko zniechęcić użytkownika do korzystania z systemu. Trzeba dopasować odpowiedź do brakujących danych można zapytać czy użytkownich chciałby zmienić lokalizacje lub rodzaj restuaracji o którą pytał.  \n",
    "\n",
    "Problemy wynikają także z błednego odczytania co użytkownik powiedział(ASR) lub nie precyzyjnej odpowiedzi systemu(SLU). Takie błedy mimo dobrze działającego DST mogą spowodować minięcie się z oczekiwaniami użytkownika."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Rozwiązania w DST**\n",
    "\n",
    "Slot-filling\n",
    "\n",
    "Hand-crafted\n",
    "\n",
    "Generative approach\n",
    "\n",
    "Discriminative approach\n",
    "\n",
    "Źródła: [4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Slot-filling**\n",
    "\n",
    "Proste podejście przypisywania w każdej turze określonego aktu dialogwego wraz z odpowiadającym mu określonym znaczeniem. Działo to tylko dla prostych dialogów z wcześniej zdefiniowaną ontologią.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Hand-crafted**\n",
    "\n",
    "Na podstawie gotowych danych ontologicznych i własnoręcznie zrobionych reguł przypisywania wartości slotów "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Genartaive approach**\n",
    "\n",
    "Podejście polegające na tym że stany generują obserwacje, zakładają to że obserwacje na każdym etapie tury są niezależne od siebie. Umożliwia też tworzenie własnych ontlogi na podstawie historii dialogów oraz wyników jakie generuje NLU."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Discriminative approach**\n",
    "\n",
    "the dialogue state given\n",
    "arbitrary and possibly correlated input features.\n",
    "\n",
    "Podejśćie polegające na tym że stan dialogu wynika z obserwacji, zakłada żę stany dialogu podane arbitralnie są możliwe połączone z wyjściem.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![s2s](S2S.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![RNN](RNN.png)\n",
    "\n",
    "Recurrent neural network- Biorąc jako argument ostatni akt dialogu oraz to co napisał właśnie użytkownik, tworzy nowy act dialogu."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Dialog State Tracking Challenge 2 & 3 Data**\n",
    "\n",
    "Z powodu tego iż zbieranie danych jest bardzo kosztowne, mało kto mógł korzystać z dobrze przygotowanych danych aby rozwijać własne DST. Dlatego aby ułatwić rozwój powstał pierwszy Dialog State Tracking Challenge i jego następca Dialog State Tracking Challenge 2 & 3.\n",
    "DSTC 2 zawierał ponad 15 tysięcy dobrze opisanych dialogów udostępnionych aby ludzi mogli trenować swój system.\n",
    "\n",
    "Źródła: [2] [1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "system: Hello , welcome to the Cambridge restaurant system? You can ask for restaurants by area , price range or food type . How may I help you?   dialog-acts-[{'slots': [], 'act': 'welcomemsg'}]\n",
      "user:   expensive restaurant in the south part of town      dialog-acts-[{'slots': [], 'act': 'welcomemsg'}]\n",
      "system: What kind of food would you like?   dialog-acts-[{'slots': [['slot', 'food']], 'act': 'request'}]\n",
      "user:   any      dialog-acts-[{'slots': [['slot', 'food']], 'act': 'request'}]\n",
      "system: the good luck chinese food takeaway is a nice restaurant in the south of town serving chinese food   dialog-acts-[{'slots': [['name', 'the good luck chinese food takeaway']], 'act': 'offer'}, {'slots': [['food', 'chinese']], 'act': 'inform'}, {'slots': [['pricerange', 'expensive']], 'act': 'inform'}, {'slots': [['area', 'south']], 'act': 'inform'}]\n",
      "user:   address      dialog-acts-[{'slots': [['name', 'the good luck chinese food takeaway']], 'act': 'offer'}, {'slots': [['food', 'chinese']], 'act': 'inform'}, {'slots': [['pricerange', 'expensive']], 'act': 'inform'}, {'slots': [['area', 'south']], 'act': 'inform'}]\n",
      "system: Sure , the good luck chinese food takeaway is on 82 Cherry Hinton Road Cherry Hinton   dialog-acts-[{'slots': [['name', 'the good luck chinese food takeaway']], 'act': 'offer'}, {'slots': [['addr', '82 cherry hinton road cherry hinton']], 'act': 'inform'}]\n",
      "user:   type of food      dialog-acts-[{'slots': [['name', 'the good luck chinese food takeaway']], 'act': 'offer'}, {'slots': [['addr', '82 cherry hinton road cherry hinton']], 'act': 'inform'}]\n",
      "system: the good luck chinese food takeaway is a nice restaurant in the south of town in the expensive price range   dialog-acts-[{'slots': [['name', 'the good luck chinese food takeaway']], 'act': 'offer'}, {'slots': [['pricerange', 'expensive']], 'act': 'inform'}, {'slots': [['area', 'south']], 'act': 'inform'}]\n",
      "user:   thank you good bye      dialog-acts-[{'slots': [['name', 'the good luck chinese food takeaway']], 'act': 'offer'}, {'slots': [['pricerange', 'expensive']], 'act': 'inform'}, {'slots': [['area', 'south']], 'act': 'inform'}]\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "f = open('log.json')\n",
    "f2 = open('label.json')\n",
    "data = json.load(f)\n",
    "user = json.load(f2)\n",
    "for i in range(len(data['turns'])):\n",
    "    print(f'system: {data[\"turns\"][i][\"output\"][\"transcript\"]}   dialog-acts-{data[\"turns\"][i][\"output\"][\"dialog-acts\"]}')\n",
    "    print(f'user:   {user[\"turns\"][i][\"transcription\"]}      dialog-acts-{data[\"turns\"][i][\"output\"][\"dialog-acts\"]}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'asr-hyp': 'expensive restaurant in south part of town', 'score': -0.512136}, {'asr-hyp': 'a restaurant in south part of town', 'score': -2.429358}, {'asr-hyp': 'expensive expensive restaurant in south part of town', 'score': -2.429358}, {'asr-hyp': 'expensive restaurant restaurant in south part of town', 'score': -3.210609}, {'asr-hyp': 'expensive in south part of town', 'score': -3.211676}, {'asr-hyp': 'expensive restaurant south part of town', 'score': -3.347932}, {'asr-hyp': 'the expensive restaurant in south part of town', 'score': -4.268281}, {'asr-hyp': 'a expensive restaurant in south part of town', 'score': -4.34658}, {'asr-hyp': 'a restaurant restaurant in south part of town', 'score': -5.127831}, {'asr-hyp': 'expensive expensive restaurant restaurant in south part of town', 'score': -5.127831}]\n",
      "{'asr-hyp': 'expensive restaurant in south part of town', 'score': -0.512136}\n",
      "{'asr-hyp': 'a restaurant in south part of town', 'score': -2.429358}\n",
      "{'asr-hyp': 'expensive expensive restaurant in south part of town', 'score': -2.429358}\n",
      "{'asr-hyp': 'expensive restaurant restaurant in south part of town', 'score': -3.210609}\n",
      "{'asr-hyp': 'expensive in south part of town', 'score': -3.211676}\n",
      "{'asr-hyp': 'expensive restaurant south part of town', 'score': -3.347932}\n",
      "{'asr-hyp': 'the expensive restaurant in south part of town', 'score': -4.268281}\n",
      "{'asr-hyp': 'a expensive restaurant in south part of town', 'score': -4.34658}\n",
      "{'asr-hyp': 'a restaurant restaurant in south part of town', 'score': -5.127831}\n",
      "{'asr-hyp': 'expensive expensive restaurant restaurant in south part of town', 'score': -5.127831}\n",
      "[{'asr-hyp': 'any', 'score': -0.02858}, {'asr-hyp': '', 'score': -4.777333}, {'asr-hyp': 'any me', 'score': -4.791492}, {'asr-hyp': 'any give', 'score': -4.885694}, {'asr-hyp': 'any the', 'score': -6.819719}, {'asr-hyp': 'in', 'score': -6.9403}, {'asr-hyp': 'any get', 'score': -7.371329}, {'asr-hyp': 'any a', 'score': -8.3173}, {'asr-hyp': 'and', 'score': -8.77938}, {'asr-hyp': 'any any', 'score': -8.946321}]\n",
      "{'asr-hyp': 'any', 'score': -0.02858}\n",
      "{'asr-hyp': '', 'score': -4.777333}\n",
      "{'asr-hyp': 'any me', 'score': -4.791492}\n",
      "{'asr-hyp': 'any give', 'score': -4.885694}\n",
      "{'asr-hyp': 'any the', 'score': -6.819719}\n",
      "{'asr-hyp': 'in', 'score': -6.9403}\n",
      "{'asr-hyp': 'any get', 'score': -7.371329}\n",
      "{'asr-hyp': 'any a', 'score': -8.3173}\n",
      "{'asr-hyp': 'and', 'score': -8.77938}\n",
      "{'asr-hyp': 'any any', 'score': -8.946321}\n",
      "[{'asr-hyp': 'address', 'score': -0.455029}, {'asr-hyp': '', 'score': -2.265617}, {'asr-hyp': 'address address', 'score': -2.268945}, {'asr-hyp': 'the address', 'score': -2.274268}, {'asr-hyp': 'the', 'score': -4.084856}, {'asr-hyp': 'the address address', 'score': -4.088183}, {'asr-hyp': 'address their', 'score': -7.867221}, {'asr-hyp': 'address uh', 'score': -8.652037}, {'asr-hyp': 'and', 'score': -8.784164}, {'asr-hyp': 'i', 'score': -9.450777}]\n",
      "{'asr-hyp': 'address', 'score': -0.455029}\n",
      "{'asr-hyp': '', 'score': -2.265617}\n",
      "{'asr-hyp': 'address address', 'score': -2.268945}\n",
      "{'asr-hyp': 'the address', 'score': -2.274268}\n",
      "{'asr-hyp': 'the', 'score': -4.084856}\n",
      "{'asr-hyp': 'the address address', 'score': -4.088183}\n",
      "{'asr-hyp': 'address their', 'score': -7.867221}\n",
      "{'asr-hyp': 'address uh', 'score': -8.652037}\n",
      "{'asr-hyp': 'and', 'score': -8.784164}\n",
      "{'asr-hyp': 'i', 'score': -9.450777}\n",
      "[{'asr-hyp': 'type of food', 'score': -0.016592}, {'asr-hyp': 'type type of food', 'score': -5.183987}, {'asr-hyp': 'the of food', 'score': -5.404256}, {'asr-hyp': 'of food', 'score': -6.505753}, {'asr-hyp': 'type food', 'score': -6.617192}, {'asr-hyp': 'type global of food', 'score': -7.340683}, {'asr-hyp': 'type budget of food', 'score': -7.360768}, {'asr-hyp': 'type the food', 'score': -7.5319}, {'asr-hyp': 'and of food', 'score': -7.617202}, {'asr-hyp': 'i of food', 'score': -7.727244}]\n",
      "{'asr-hyp': 'type of food', 'score': -0.016592}\n",
      "{'asr-hyp': 'type type of food', 'score': -5.183987}\n",
      "{'asr-hyp': 'the of food', 'score': -5.404256}\n",
      "{'asr-hyp': 'of food', 'score': -6.505753}\n",
      "{'asr-hyp': 'type food', 'score': -6.617192}\n",
      "{'asr-hyp': 'type global of food', 'score': -7.340683}\n",
      "{'asr-hyp': 'type budget of food', 'score': -7.360768}\n",
      "{'asr-hyp': 'type the food', 'score': -7.5319}\n",
      "{'asr-hyp': 'and of food', 'score': -7.617202}\n",
      "{'asr-hyp': 'i of food', 'score': -7.727244}\n",
      "[{'asr-hyp': 'thank you goodbye', 'score': -0.30683}, {'asr-hyp': 'thank you goodbye bye', 'score': -2.107196}, {'asr-hyp': 'thank you good', 'score': -2.109648}, {'asr-hyp': 'thank you good bye', 'score': -3.910014}, {'asr-hyp': 'thank you you', 'score': -7.879262}, {'asr-hyp': 'thank goodbye', 'score': -7.957914}, {'asr-hyp': 'thank you goodbye bar', 'score': -8.884375}, {'asr-hyp': 'thank you you bye', 'score': -9.679628}, {'asr-hyp': 'thank goodbye bye', 'score': -9.758281}, {'asr-hyp': 'thank good', 'score': -9.760733}]\n",
      "{'asr-hyp': 'thank you goodbye', 'score': -0.30683}\n",
      "{'asr-hyp': 'thank you goodbye bye', 'score': -2.107196}\n",
      "{'asr-hyp': 'thank you good', 'score': -2.109648}\n",
      "{'asr-hyp': 'thank you good bye', 'score': -3.910014}\n",
      "{'asr-hyp': 'thank you you', 'score': -7.879262}\n",
      "{'asr-hyp': 'thank goodbye', 'score': -7.957914}\n",
      "{'asr-hyp': 'thank you goodbye bar', 'score': -8.884375}\n",
      "{'asr-hyp': 'thank you you bye', 'score': -9.679628}\n",
      "{'asr-hyp': 'thank goodbye bye', 'score': -9.758281}\n",
      "{'asr-hyp': 'thank good', 'score': -9.760733}\n"
     ]
    }
   ],
   "source": [
    "for i in range(len(data['turns'])):\n",
    "    print(data[\"turns\"][i][\"input\"][\"live\"]['asr-hyps'])\n",
    "    for l in data[\"turns\"][i][\"input\"][\"live\"]['asr-hyps']:\n",
    "        print(l)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'output': {'transcript': 'Hello , welcome to the Cambridge restaurant system? You can ask for restaurants by area , price range or food type . How may I help you?', 'end-time': 9.93, 'start-time': 0.001283, 'dialog-acts': [{'slots': [], 'act': 'welcomemsg'}], 'aborted': False}, 'turn-index': 0, 'input': {'live': {'asr-hyps': [{'asr-hyp': 'expensive restaurant in south part of town', 'score': -0.512136}, {'asr-hyp': 'a restaurant in south part of town', 'score': -2.429358}, {'asr-hyp': 'expensive expensive restaurant in south part of town', 'score': -2.429358}, {'asr-hyp': 'expensive restaurant restaurant in south part of town', 'score': -3.210609}, {'asr-hyp': 'expensive in south part of town', 'score': -3.211676}, {'asr-hyp': 'expensive restaurant south part of town', 'score': -3.347932}, {'asr-hyp': 'the expensive restaurant in south part of town', 'score': -4.268281}, {'asr-hyp': 'a expensive restaurant in south part of town', 'score': -4.34658}, {'asr-hyp': 'a restaurant restaurant in south part of town', 'score': -5.127831}, {'asr-hyp': 'expensive expensive restaurant restaurant in south part of town', 'score': -5.127831}], 'slu-hyps': [{'slu-hyp': [{'slots': [['pricerange', 'expensive']], 'act': 'inform'}, {'slots': [['area', 'south']], 'act': 'inform'}], 'score': 0.9282257165548344}, {'slu-hyp': [{'slots': [['pricerange', 'expensive']], 'act': 'confirm'}, {'slots': [['area', 'south']], 'act': 'confirm'}], 'score': 0.02923086440308149}, {'slu-hyp': [{'slots': [['pricerange', 'expensive']], 'act': 'inform'}], 'score': 0.02236294307342127}, {'slu-hyp': [{'slots': [['area', 'south']], 'act': 'inform'}], 'score': 0.020180475968662844}]}, 'end-time': 12.19, 'start-time': 9.93, 'batch': {'cnet': [{'start': 0.0, 'end': 0.525, 'arcs': [{'score': -0.0001, 'word': '<s>'}, {'score': -5000000000.0, 'word': '!null'}]}, {'start': 0.525, 'end': 0.7875, 'arcs': [{'score': -19.41379, 'word': 'i'}, {'score': -16.88836, 'word': 'a'}, {'score': -11.6955, 'word': 'ah'}, {'score': -13.1059, 'word': 'uh'}, {'score': -0.0001, 'word': '!null'}]}, {'start': 0.7875, 'end': 1.05, 'arcs': [{'score': -11.16217, 'word': 'i'}, {'score': -9.916637, 'word': 'uh'}, {'score': -13.39644, 'word': 'ok'}, {'score': -0.0001, 'word': '!null'}]}, {'start': 1.05, 'end': 1.12375, 'arcs': [{'score': -0.115834, 'word': 'expensive'}, {'score': -9.822551, 'word': 'ok'}, {'score': -5.432811, 'word': 'the'}, {'score': -15.02797, 'word': 'extensive'}, {'score': -18.94896, 'word': 'are'}, {'score': -2.254255, 'word': 'a'}, {'score': -5000000000.0, 'word': '!null'}]}, {'start': 1.12375, 'end': 1.1975, 'arcs': [{'score': -17.54265, 'word': 'ok'}, {'score': -0.0001, 'word': '!null'}]}, {'start': 1.1975, 'end': 1.345, 'arcs': [{'score': -20.00645, 'word': 'expensive'}, {'score': -23.83931, 'word': 'extensive'}, {'score': -0.0001, 'word': '!null'}]}, {'start': 1.345, 'end': 1.4925, 'arcs': [{'score': -2.212958, 'word': 'expensive'}, {'score': -21.30504, 'word': 'extensive'}, {'score': -0.1158337, 'word': '!null'}]}, {'start': 1.4925, 'end': 1.64, 'arcs': [{'score': -19.56206, 'word': 'a'}, {'score': -15.88049, 'word': 'or'}, {'score': -19.37142, 'word': 'uh'}, {'score': -0.0001, 'word': '!null'}]}, {'start': 1.64, 'end': 1.7875, 'arcs': [{'score': -0.0001, 'word': 'restaurant'}, {'score': -14.9832, 'word': 'the'}, {'score': -11.33279, 'word': 'a'}, {'score': -9.494602, 'word': 'uh'}, {'score': -11.74803, 'word': 'or'}, {'score': -17.88491, 'word': '!null'}]}, {'start': 1.7875, 'end': 1.935, 'arcs': [{'score': -13.69902, 'word': 'restaurant'}, {'score': -0.0001, 'word': '!null'}]}, {'start': 1.935, 'end': 2.23, 'arcs': [{'score': -9.268782, 'word': 'restaurant'}, {'score': -2.381682, 'word': 'in'}, {'score': -0.09705002, 'word': '!null'}]}, {'start': 2.23, 'end': 2.27, 'arcs': [{'score': -0.1433645, 'word': 'in'}, {'score': -12.21065, 'word': 'or'}, {'score': -13.78668, 'word': 'i'}, {'score': -12.6393, 'word': 'are'}, {'score': -13.47915, 'word': 'a'}, {'score': -11.99133, 'word': \"that's\"}, {'score': -7.282886, 'word': 'that'}, {'score': -2.348785, 'word': 'th
     ]
    }
   ],
   "source": [
    "print(data['turns'][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Bibliografia**\n",
    "\n",
    "**[1]**  DSTC https://github.com/matthen/dstc\n",
    "\n",
    "**[2]**  The Second Dialog State Tracking Challenge https://aclanthology.org/W14-4337.pdf\n",
    "\n",
    "**[3]**  Dialogue State Tracking with Incremental Reasoning https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00384/101875/Dialogue-State-Tracking-with-Incremental-Reasoning\n",
    "\n",
    "**[4]**  https://arxiv.org/pdf/2101.08333.pdf\n",
    "\n",
    "**[5]** https://www.cs.hhu.de/fileadmin/redaktion/Fakultaeten/Mathematisch-Naturwissenschaftliche_Fakultaet/Informatik/Dialog_Systems_and_Machine_Learning/L6.pdf"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}