47 KiB
47 KiB
0. Instalacja i importowanie modułów
0.1. Ogólne
!pip install -r requirements.txt --user
!pip listRequirement already satisfied: tflearn==0.5 in c:\users\annad\appdata\roaming\python\python38\site-packages (from -r requirements.txt (line 1)) (0.5.0) Requirement already satisfied: tensorflow in c:\users\annad\appdata\roaming\python\python38\site-packages (from -r requirements.txt (line 2)) (2.4.1) Requirement already satisfied: pystempel==1.2 in c:\users\annad\appdata\roaming\python\python38\site-packages (from -r requirements.txt (line 3)) (1.2.0) Requirement already satisfied: six in c:\users\annad\anaconda3\lib\site-packages (from tflearn==0.5->-r requirements.txt (line 1)) (1.15.0) Requirement already satisfied: Pillow in c:\users\annad\anaconda3\lib\site-packages (from tflearn==0.5->-r requirements.txt (line 1)) (8.0.1) Requirement already satisfied: numpy in c:\users\annad\anaconda3\lib\site-packages (from tflearn==0.5->-r requirements.txt (line 1)) (1.19.2) Requirement already satisfied: flatbuffers~=1.12.0 in c:\users\annad\appdata\roaming\python\python38\site-packages (from tensorflow->-r requirements.txt (line 2)) (1.12) Requirement already satisfied: gast==0.3.3 in c:\users\annad\appdata\roaming\python\python38\site-packages (from tensorflow->-r requirements.txt (line 2)) (0.3.3) Requirement already satisfied: tensorflow-estimator<2.5.0,>=2.4.0 in c:\users\annad\appdata\roaming\python\python38\site-packages (from tensorflow->-r requirements.txt (line 2)) (2.4.0) Requirement already satisfied: grpcio~=1.32.0 in c:\users\annad\appdata\roaming\python\python38\site-packages (from tensorflow->-r requirements.txt (line 2)) (1.32.0) Requirement already satisfied: wheel~=0.35 in c:\users\annad\anaconda3\lib\site-packages (from tensorflow->-r requirements.txt (line 2)) (0.35.1) Requirement already satisfied: wrapt~=1.12.1 in c:\users\annad\appdata\roaming\python\python38\site-packages (from tensorflow->-r requirements.txt (line 2)) (1.12.1) Requirement already satisfied: termcolor~=1.1.0 in c:\users\annad\appdata\roaming\python\python38\site-packages (from tensorflow->-r requirements.txt (line 2)) (1.1.0) Requirement already satisfied: google-pasta~=0.2 in c:\users\annad\appdata\roaming\python\python38\site-packages (from tensorflow->-r requirements.txt (line 2)) (0.2.0) Requirement already satisfied: absl-py~=0.10 in c:\users\annad\appdata\roaming\python\python38\site-packages (from tensorflow->-r requirements.txt (line 2)) (0.12.0) Requirement already satisfied: tensorboard~=2.4 in c:\users\annad\appdata\roaming\python\python38\site-packages (from tensorflow->-r requirements.txt (line 2)) (2.4.1) Requirement already satisfied: astunparse~=1.6.3 in c:\users\annad\appdata\roaming\python\python38\site-packages (from tensorflow->-r requirements.txt (line 2)) (1.6.3) Requirement already satisfied: opt-einsum~=3.3.0 in c:\users\annad\appdata\roaming\python\python38\site-packages (from tensorflow->-r requirements.txt (line 2)) (3.3.0) Requirement already satisfied: typing-extensions~=3.7.4 in c:\users\annad\anaconda3\lib\site-packages (from tensorflow->-r requirements.txt (line 2)) (3.7.4.3) Requirement already satisfied: h5py~=2.10.0 in c:\users\annad\anaconda3\lib\site-packages (from tensorflow->-r requirements.txt (line 2)) (2.10.0) Requirement already satisfied: keras-preprocessing~=1.1.2 in c:\users\annad\appdata\roaming\python\python38\site-packages (from tensorflow->-r requirements.txt (line 2)) (1.1.2) Requirement already satisfied: protobuf>=3.9.2 in c:\users\annad\appdata\roaming\python\python38\site-packages (from tensorflow->-r requirements.txt (line 2)) (3.15.6) Requirement already satisfied: sortedcontainers in c:\users\annad\anaconda3\lib\site-packages (from pystempel==1.2->-r requirements.txt (line 3)) (2.2.2) Requirement already satisfied: tqdm in c:\users\annad\anaconda3\lib\site-packages (from pystempel==1.2->-r requirements.txt (line 3)) (4.50.2) Requirement already satisfied: markdown>=2.6.8 in c:\users\annad\appdata\roaming\python\python38\site-packages (from tensorboard~=2.4->tensorflow->-r requirements.txt (line 2)) (3.3.4) Requirement already satisfied: requests<3,>=2.21.0 in c:\users\annad\anaconda3\lib\site-packages (from tensorboard~=2.4->tensorflow->-r requirements.txt (line 2)) (2.24.0) Requirement already satisfied: google-auth<2,>=1.6.3 in c:\users\annad\appdata\roaming\python\python38\site-packages (from tensorboard~=2.4->tensorflow->-r requirements.txt (line 2)) (1.28.0) Requirement already satisfied: setuptools>=41.0.0 in c:\users\annad\anaconda3\lib\site-packages (from tensorboard~=2.4->tensorflow->-r requirements.txt (line 2)) (50.3.1.post20201107) Requirement already satisfied: werkzeug>=0.11.15 in c:\users\annad\anaconda3\lib\site-packages (from tensorboard~=2.4->tensorflow->-r requirements.txt (line 2)) (1.0.1) Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in c:\users\annad\appdata\roaming\python\python38\site-packages (from tensorboard~=2.4->tensorflow->-r requirements.txt (line 2)) (1.8.0) Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in c:\users\annad\appdata\roaming\python\python38\site-packages (from tensorboard~=2.4->tensorflow->-r requirements.txt (line 2)) (0.4.3) Requirement already satisfied: idna<3,>=2.5 in c:\users\annad\anaconda3\lib\site-packages (from requests<3,>=2.21.0->tensorboard~=2.4->tensorflow->-r requirements.txt (line 2)) (2.10) Requirement already satisfied: certifi>=2017.4.17 in c:\users\annad\anaconda3\lib\site-packages (from requests<3,>=2.21.0->tensorboard~=2.4->tensorflow->-r requirements.txt (line 2)) (2020.6.20) Requirement already satisfied: chardet<4,>=3.0.2 in c:\users\annad\anaconda3\lib\site-packages (from requests<3,>=2.21.0->tensorboard~=2.4->tensorflow->-r requirements.txt (line 2)) (3.0.4) Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:\users\annad\anaconda3\lib\site-packages (from requests<3,>=2.21.0->tensorboard~=2.4->tensorflow->-r requirements.txt (line 2)) (1.25.11) Requirement already satisfied: rsa<5,>=3.1.4; python_version >= "3.6" in c:\users\annad\appdata\roaming\python\python38\site-packages (from google-auth<2,>=1.6.3->tensorboard~=2.4->tensorflow->-r requirements.txt (line 2)) (4.7.2) Requirement already satisfied: pyasn1-modules>=0.2.1 in c:\users\annad\appdata\roaming\python\python38\site-packages (from google-auth<2,>=1.6.3->tensorboard~=2.4->tensorflow->-r requirements.txt (line 2)) (0.2.8) Requirement already satisfied: cachetools<5.0,>=2.0.0 in c:\users\annad\appdata\roaming\python\python38\site-packages (from google-auth<2,>=1.6.3->tensorboard~=2.4->tensorflow->-r requirements.txt (line 2)) (4.2.1) Requirement already satisfied: requests-oauthlib>=0.7.0 in c:\users\annad\appdata\roaming\python\python38\site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.4->tensorflow->-r requirements.txt (line 2)) (1.3.0) Requirement already satisfied: pyasn1>=0.1.3 in c:\users\annad\appdata\roaming\python\python38\site-packages (from rsa<5,>=3.1.4; python_version >= "3.6"->google-auth<2,>=1.6.3->tensorboard~=2.4->tensorflow->-r requirements.txt (line 2)) (0.4.8) Requirement already satisfied: oauthlib>=3.0.0 in c:\users\annad\appdata\roaming\python\python38\site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.4->tensorflow->-r requirements.txt (line 2)) (3.1.0) Package Version ---------------------------------- ------------------- absl-py 0.12.0 alabaster 0.7.12 anaconda-client 1.7.2 anaconda-navigator 1.10.0 anaconda-project 0.8.3 argh 0.26.2 argon2-cffi 20.1.0 asn1crypto 1.4.0 astroid 2.4.2 astropy 4.0.2 astunparse 1.6.3 async-generator 1.10 atomicwrites 1.4.0 attrs 20.3.0 autopep8 1.5.4 Babel 2.8.1 backcall 0.2.0 backports.functools-lru-cache 1.6.1 backports.shutil-get-terminal-size 1.0.0 backports.tempfile 1.0 backports.weakref 1.0.post1 bcrypt 3.2.0 beautifulsoup4 4.9.3 bitarray 1.6.1 bkcharts 0.2 bleach 3.2.1 bokeh 2.2.3 boto 2.49.0 Bottleneck 1.3.2 brotlipy 0.7.0 cachetools 4.2.1 certifi 2020.6.20 cffi 1.14.3 chardet 3.0.4 click 7.1.2 cloudpickle 1.6.0 clyent 1.2.2 colorama 0.4.4 comtypes 1.1.7 conda 4.9.2 conda-build 3.20.5 conda-package-handling 1.7.2 conda-verify 3.4.2 contextlib2 0.6.0.post1 cryptography 3.1.1 cycler 0.10.0 Cython 0.29.21 cytoolz 0.11.0 dask 2.30.0 decorator 4.4.2 defusedxml 0.6.0 diff-match-patch 20200713 distributed 2.30.1 docutils 0.16 entrypoints 0.3 et-xmlfile 1.0.1 fastcache 1.1.0 filelock 3.0.12 flake8 3.8.4 Flask 1.1.2 flatbuffers 1.12 fsspec 0.8.3 future 0.18.2 gast 0.3.3 gevent 20.9.0 glob2 0.7 google-auth 1.28.0 google-auth-oauthlib 0.4.3 google-pasta 0.2.0 greenlet 0.4.17 grpcio 1.32.0 h5py 2.10.0 HeapDict 1.0.1 html5lib 1.1 idna 2.10 imageio 2.9.0 imagesize 1.2.0 importlib-metadata 2.0.0 iniconfig 1.1.1 intervaltree 3.1.0 ipykernel 5.3.4 ipython 7.19.0 ipython-genutils 0.2.0 ipywidgets 7.5.1 isort 5.6.4 itsdangerous 1.1.0 jdcal 1.4.1 jedi 0.17.1 Jinja2 2.11.2 joblib 0.17.0 json5 0.9.5 jsonschema 3.2.0 jupyter 1.0.0 jupyter-client 6.1.7 jupyter-console 6.2.0 jupyter-core 4.6.3 jupyterlab 2.2.6 jupyterlab-pygments 0.1.2 jupyterlab-server 1.2.0 Keras-Preprocessing 1.1.2 keyring 21.4.0 kiwisolver 1.3.0 lazy-object-proxy 1.4.3 libarchive-c 2.9 llvmlite 0.34.0 locket 0.2.0 lxml 4.6.1 Markdown 3.3.4 MarkupSafe 1.1.1 matplotlib 3.3.2 mccabe 0.6.1 menuinst 1.4.16 mistune 0.8.4 mkl-fft 1.2.0 mkl-random 1.1.1 mkl-service 2.3.0 mock 4.0.2 more-itertools 8.6.0 mpmath 1.1.0 msgpack 1.0.0 multipledispatch 0.6.0 navigator-updater 0.2.1 nbclient 0.5.1 nbconvert 6.0.7 nbformat 5.0.8 nest-asyncio 1.4.2 networkx 2.5 nltk 3.5 nose 1.3.7 notebook 6.1.4 numba 0.51.2 numexpr 2.7.1 numpy 1.19.2 numpydoc 1.1.0 oauthlib 3.1.0 olefile 0.46 openpyxl 3.0.5 opt-einsum 3.3.0 packaging 20.4 pandas 1.1.3 pandocfilters 1.4.3 paramiko 2.7.2 parso 0.7.0 partd 1.1.0 path 15.0.0 pathlib2 2.3.5 pathtools 0.1.2 patsy 0.5.1 pep8 1.7.1 pexpect 4.8.0 pickleshare 0.7.5 Pillow 8.0.1 pip 20.2.4 pkginfo 1.6.1 pluggy 0.13.1 ply 3.11 prometheus-client 0.8.0 prompt-toolkit 3.0.8 protobuf 3.15.6 psutil 5.7.2 py 1.9.0 pyasn1 0.4.8 pyasn1-modules 0.2.8 pycodestyle 2.6.0 pycosat 0.6.3 pycparser 2.20 pycurl 7.43.0.6 pydocstyle 5.1.1 pyflakes 2.2.0 Pygments 2.7.2 pylint 2.6.0 PyNaCl 1.4.0 pyodbc 4.0.0-unsupported pyOpenSSL 19.1.0 pyparsing 2.4.7 pyreadline 2.1 pyrsistent 0.17.3 PySocks 1.7.1 pystempel 1.2.0 pytest 0.0.0 python-dateutil 2.8.1 python-jsonrpc-server 0.4.0 python-language-server 0.35.1 pytz 2020.1 PyWavelets 1.1.1 pywin32 227 pywin32-ctypes 0.2.0 pywinpty 0.5.7 PyYAML 5.3.1 pyzmq 19.0.2 QDarkStyle 2.8.1 QtAwesome 1.0.1 qtconsole 4.7.7 QtPy 1.9.0 regex 2020.10.15 requests 2.24.0 requests-oauthlib 1.3.0 rope 0.18.0 rsa 4.7.2 Rtree 0.9.4 ruamel-yaml 0.15.87 scikit-image 0.17.2 scikit-learn 0.23.2 scipy 1.5.2 seaborn 0.11.0 Send2Trash 1.5.0 setuptools 50.3.1.post20201107 simplegeneric 0.8.1 singledispatch 3.4.0.3 sip 4.19.13 six 1.15.0 snowballstemmer 2.0.0 sortedcollections 1.2.1 sortedcontainers 2.2.2 soupsieve 2.0.1 Sphinx 3.2.1 sphinxcontrib-applehelp 1.0.2 sphinxcontrib-devhelp 1.0.2 sphinxcontrib-htmlhelp 1.0.3 sphinxcontrib-jsmath 1.0.1 sphinxcontrib-qthelp 1.0.3 sphinxcontrib-serializinghtml 1.1.4 sphinxcontrib-websupport 1.2.4 spyder 4.1.5 spyder-kernels 1.9.4 SQLAlchemy 1.3.20 statsmodels 0.12.0 sympy 1.6.2 tables 3.6.1 tblib 1.7.0 tensorboard 2.4.1 tensorboard-plugin-wit 1.8.0 tensorflow 2.4.1 tensorflow-estimator 2.4.0 termcolor 1.1.0 terminado 0.9.1 testpath 0.4.4 tflearn 0.5.0 threadpoolctl 2.1.0 tifffile 2020.10.1 toml 0.10.1 toolz 0.11.1 tornado 6.0.4 tqdm 4.50.2 traitlets 5.0.5 typing-extensions 3.7.4.3 ujson 4.0.1 unicodecsv 0.14.1 urllib3 1.25.11 watchdog 0.10.3 wcwidth 0.2.5 webencodings 0.5.1 Werkzeug 1.0.1 wheel 0.35.1 widgetsnbextension 3.5.1 win-inet-pton 1.1.0 win-unicode-console 0.5 wincertstore 0.2 wrapt 1.12.1 xlrd 1.2.0 XlsxWriter 1.3.7 xlwings 0.20.8 xlwt 1.3.0 xmltodict 0.12.0 yapf 0.30.0 zict 2.0.0 zipp 3.4.0 zope.event 4.5.0 zope.interface 5.1.2
import numpy as np
import tflearn
import tensorflow
import random
import json
import nltkWARNING:tensorflow:From C:\Users\annad\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\compat\v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version. Instructions for updating: non-resource variables are not supported in the long term curses is not supported on this machine (please install/reinstall curses for an optimal experience)
0.2. Angielski Stemmer: https://www.nltk.org/_modules/nltk/stem/lancaster.html
nltk.download('punkt')
from nltk.stem.lancaster import LancasterStemmer
stemmer_en = LancasterStemmer()[nltk_data] Downloading package punkt to [nltk_data] C:\Users\annad\AppData\Roaming\nltk_data... [nltk_data] Package punkt is already up-to-date!
0.3. Polski Stemmer (Docelowy): https://pypi.org/project/pystempel/
from stempel import StempelStemmer
stemmer_pl = StempelStemmer.default() #.polimorf() #jest lepsza?Loading: 92%|██████████████████████████████████████████████████▋ | 2051427/2225192 [00:01<00:00, 1634466.61bytes/s]
1. Załadowanie plików .json z bazą słów
1.1. Docelowa baza słów polskich do nauki modelu (10 rodzajów odp - PL)
with open("intents_pl.json", encoding='utf-8') as file:
data_pl = json.load(file)
print(data_pl){'intents': [{'tag': 'greeting', 'patterns': ['Cześć', 'Elo', 'Jesteś?', 'Hej', 'Dzień dobry', 'Sup', 'Witam', 'Hejka', 'Hej!'], 'responses': ['Cześć!', 'Dobrze Cię widzieć!', 'Hej, w czym mogę pomóc?'], 'context_set': ''}, {'tag': 'goodbye', 'patterns': ['narazie', 'Do zobaczenia', 'Dowidzenia', 'Dobranoc', 'Miłego dnia'], 'responses': ['Do zobaczenia później', 'Mam nadzieję, że później pogadamy', 'Narazie!'], 'context_set': ''}, {'tag': 'age', 'patterns': ['Ile masz lat', 'Ile lat ma Janet', 'Wiek', 'Jak stara jesteś', 'jaki jest twój wiek', 'Wieko', 'urodziny'], 'responses': ['Mam kilka dni', 'Urodziłam się 17.03.2021'], 'context_set': ''}, {'tag': 'name', 'patterns': ['Jak masz na imię', 'Jak Cię zwą?', 'twoje imie?', 'Imie', 'Jak cię nazywać', 'Kim jesteś'], 'responses': ['Możesz mnie nazywać Janet!', 'Jestem Janet', 'Jestem Janet, twój ulubiony chatbot'], 'context_set': ''}, {'tag': 'goout', 'patterns': ['Czy chcesz gdzieś wyjść?', 'zrobimy coś razem?', 'pójdziemy gdzieś razem?'], 'responses': ['Może kiedy indziej', 'Odezwę się latem'], 'context_set': ''}, {'tag': 'doing', 'patterns': ['Co robisz teraz?', 'co słychać', 'jakie masz plany?', 'jak się masz?'], 'responses': ['Gram w grę', 'Słucham muzyki', 'nie twój interes', 'nie mam czasu odpowiadać', ''], 'context_set': ''}, {'tag': 'game', 'patterns': ['a w co grasz?', 'a w co?', 'grasz?', 'jaka gra'], 'responses': ['nie interesuj się', 'a co cię to obchodzi', '...', 'w coś', 'W OSRS'], 'context_set': ''}, {'tag': 'music', 'patterns': ['czego?', 'czego słuchasz?', 'jakiej muzyki?', 'a czego?'], 'responses': ['Starego vinyla z 1995 roku', 'mojego ulubionego setu rejwowego', 'czegoś tam...'], 'context_set': ''}, {'tag': 'angry', 'patterns': ['bo co?', 'dlaczego?', 'jak to?'], 'responses': ['Czas zakończyć rozmowę', 'nie mam na to siły i czasu', 'zostaw mnie w spokoju'], 'context_set': ''}, {'tag': 'why', 'patterns': ['o co chodzi?', 'czemu jesteś zła', 'poczekaj'], 'responses': ['...', 'Nie pisz do mnie więcej', 'Nie lubię Cię'], 'context_set': ''}]}
1.2. Skrócona baza słów (4 rodzaje odp - PL)
with open("intents_pl_short.json", encoding='utf-8') as file:
data_pl_short = json.load(file)
print(data_pl_short){'intents': [{'tag': 'greeting', 'patterns': ['Cześć', 'Elo', 'Jesteś?', 'Hej', 'Dzień dobry', 'Sup'], 'responses': ['Cześć!', 'Dobrze Cię widzieć!', 'Hej, w czym mogę pomóc?'], 'context_set': ''}, {'tag': 'goodbye', 'patterns': ['narazie', 'Do zobaczenia', 'Dowidzenia', 'Dobranoc', 'Miłego dnia'], 'responses': ['Do zobaczenia później', 'Mam nadzieję, że później pogadamy', 'Narazie!'], 'context_set': ''}, {'tag': 'age', 'patterns': ['Ile masz lat', 'Ile lat ma Janet', 'Wiek', 'Jak stara jesteś', 'urodziny'], 'responses': ['Mam kilka dni', 'Urodziłam się 17.03.2021'], 'context_set': ''}, {'tag': 'name', 'patterns': ['Jak masz na imię', 'Jak Cię zwą?', 'twoje imie?', 'Imie', 'Jak cię nazywać', 'Kim jesteś'], 'responses': ['Możesz mnie nazywać Janet!', 'Jestem Janet', 'Jestem Janet, twój ulubiony chatbot'], 'context_set': ''}]}
1.3. Testowa baza słów angielskich (6 rodzajów odp - EN)
with open("intents_en.json", encoding='utf-8') as file:
data_en = json.load(file)
print(data_en){'intents': [{'tag': 'greeting', 'patterns': ['Hi', 'How are you', 'Is anyone there?', 'Hello', 'Good day', 'Whats up'], 'responses': ['Hello!', 'Good to see you again!', 'Hi there, how can I help?'], 'context_set': ''}, {'tag': 'goodbye', 'patterns': ['cya', 'See you later', 'Goodbye', 'I am Leaving', 'Have a Good day'], 'responses': ['Sad to see you go :(', 'Talk to you later', 'Goodbye!'], 'context_set': ''}, {'tag': 'age', 'patterns': ['how old', 'how old is tim', 'what is your age', 'how old are you', 'age?'], 'responses': ['I am 18 years old!', '18 years young!'], 'context_set': ''}, {'tag': 'name', 'patterns': ['what is your name', 'what should I call you', 'whats your name?'], 'responses': ['You can call me Tim.', "I'm Tim!", "I'm Tim aka Tech With Tim."], 'context_set': ''}, {'tag': 'shop', 'patterns': ['Id like to buy something', 'whats on the menu', 'what do you reccommend?', 'could i get something to eat'], 'responses': ['We sell chocolate chip cookies for $2!', 'Cookies are on the menu!'], 'context_set': ''}, {'tag': 'hours', 'patterns': ['when are you guys open', 'what are your hours', 'hours of operation'], 'responses': ['We are open 7am-4pm Monday-Friday!'], 'context_set': ''}]}
2. Przygotowanie danych do nauki modelu
words = []
labels = []
docs_x = []
docs_y = []2.1 Stworzenie tablicy ze wszystkimi możliwymi inputami użytkownika (+ labele)
for intent in data_pl["intents"]: #Loop przez cały json
for pattern in intent["patterns"]: #loop przez wszystkie możliwe rodzaje przykładowego inputu użytkownika
wrds = nltk.word_tokenize(pattern) #Tokenizing every word
words.extend(wrds) #Add every single tokenized word
docs_x.append(wrds) #Add the whole tokenized sentence
docs_y.append(intent["tag"]) #Pattern x coresponds to the tag y. Potrzebne do ustalenia relacji słowa z odpowiedzią
if intent["tag"] not in labels:
labels.append(intent["tag"]) #Add the tagwords = [stemmer_pl.stem(w.lower()) for w in words if w not in "?"] #stemming -> take each word and bring it to the "root" form. Only the stemmed version of the word is important to us
words = sorted(list(set(words))) #Sorting
labels = sorted(labels) #sorting
training = []
output = []
out_empty = [0 for _ in range(len(labels))]#Podgląd zmiennych
print(f"Words:\n{words}")
print(f"labels:\n{labels}")
print(f"docs_y:\n{docs_y}")
print(f"docs_x:\n{docs_x}")Words: ['!', 'a', 'bo', 'chcieć', 'chodzić', 'co', 'coś', 'czec', 'czy', 'dlaczy', 'dnia', 'do', 'dobranoc', 'dobry', 'dowidzieć', 'dzień', 'elo', 'gra', 'grać', 'gć', 'hej', 'hejka', 'ile', 'imia', 'imie', 'jak', 'jaka', 'jaki', 'janet', 'jest', 'ki', 'lat', 'mieć', 'miłeon', 'muzy', 'na', 'narazie', 'nazywać', 'o', 'plany', 'poczekać', 'pójdziemy', 'raz', 'robić', 'się', 'star', 'supć', 'słuchać', 'słychać', 'teraa', 'to', 'twoj', 'twój', 'ty', 'urodziny', 'w', 'wiek', 'wieko', 'witać', 'wyjść', 'y', 'zobaczyć', 'zrobić', 'zwać', 'zła'] labels: ['age', 'angry', 'doing', 'game', 'goodbye', 'goout', 'greeting', 'music', 'name', 'why'] docs_y: ['greeting', 'greeting', 'greeting', 'greeting', 'greeting', 'greeting', 'greeting', 'greeting', 'greeting', 'goodbye', 'goodbye', 'goodbye', 'goodbye', 'goodbye', 'age', 'age', 'age', 'age', 'age', 'age', 'age', 'name', 'name', 'name', 'name', 'name', 'name', 'goout', 'goout', 'goout', 'doing', 'doing', 'doing', 'doing', 'game', 'game', 'game', 'game', 'music', 'music', 'music', 'music', 'angry', 'angry', 'angry', 'why', 'why', 'why'] docs_x: [['Cześć'], ['Elo'], ['Jesteś', '?'], ['Hej'], ['Dzień', 'dobry'], ['Sup'], ['Witam'], ['Hejka'], ['Hej', '!'], ['narazie'], ['Do', 'zobaczenia'], ['Dowidzenia'], ['Dobranoc'], ['Miłego', 'dnia'], ['Ile', 'masz', 'lat'], ['Ile', 'lat', 'ma', 'Janet'], ['Wiek'], ['Jak', 'stara', 'jesteś'], ['jaki', 'jest', 'twój', 'wiek'], ['Wieko'], ['urodziny'], ['Jak', 'masz', 'na', 'imię'], ['Jak', 'Cię', 'zwą', '?'], ['twoje', 'imie', '?'], ['Imie'], ['Jak', 'cię', 'nazywać'], ['Kim', 'jesteś'], ['Czy', 'chcesz', 'gdzieś', 'wyjść', '?'], ['zrobimy', 'coś', 'razem', '?'], ['pójdziemy', 'gdzieś', 'razem', '?'], ['Co', 'robisz', 'teraz', '?'], ['co', 'słychać'], ['jakie', 'masz', 'plany', '?'], ['jak', 'się', 'masz', '?'], ['a', 'w', 'co', 'grasz', '?'], ['a', 'w', 'co', '?'], ['grasz', '?'], ['jaka', 'gra'], ['czego', '?'], ['czego', 'słuchasz', '?'], ['jakiej', 'muzyki', '?'], ['a', 'czego', '?'], ['bo', 'co', '?'], ['dlaczego', '?'], ['jak', 'to', '?'], ['o', 'co', 'chodzi', '?'], ['czemu', 'jesteś', 'zła'], ['poczekaj']]
3.2. Przypisywanie słów do danej kategorii (ie. "Cześć" do Greetings)
W przypadku data_pl_short są tylko 4 rodzaje odpowiedzi. "Cześć" które zostane przypisane do labela "greeting" będzie miało formę końcowego outputu "1000" jeżeli label "greetings" jest pierwszy do wyboru.
Warto też dodać, że sieć neuronowa nie przyjmuje teksu. To jest główny powód czemu przypisujemy słowa do kategorii
for x, doc in enumerate(docs_x): #Przejście przez wszystkie słowa
bag =[]
wrds = [stemmer_pl.stem(w).lower() for w in doc] #podział wszystkich słów w danym zdaniu
for w in words:
if w in wrds:
bag.append(1) #this word exist
else:
bag.append(0) #do not exist
output_row = out_empty[:] #kopia
output_row[labels.index(docs_y[x])] = 1
training.append(bag) #dodajemy nowe wyrażenie zamienione na ciąg binarny
output.append(output_row)
training = np.array(training) #Zbiór treningowy
output = np.array(output) #Zbiór outputówlen(training) #dla pl_short mamy 44 słowa48
Loading: 100%|███████████████████████████████████████████████████████| 2225192/2225192 [00:20<00:00, 1720623.81bytes/s][Alen(output[0]) #Które można przypisać do 4 kategorii10
print(training)
print(output)[[0 0 0 ... 0 0 0] [0 0 0 ... 0 0 0] [0 0 0 ... 0 0 0] ... [0 0 0 ... 0 0 0] [0 0 0 ... 0 0 1] [0 0 0 ... 0 0 0]] [[0 0 0 0 0 0 1 0 0 0] [0 0 0 0 0 0 1 0 0 0] [0 0 0 0 0 0 1 0 0 0] [0 0 0 0 0 0 1 0 0 0] [0 0 0 0 0 0 1 0 0 0] [0 0 0 0 0 0 1 0 0 0] [0 0 0 0 0 0 1 0 0 0] [0 0 0 0 0 0 1 0 0 0] [0 0 0 0 0 0 1 0 0 0] [0 0 0 0 1 0 0 0 0 0] [0 0 0 0 1 0 0 0 0 0] [0 0 0 0 1 0 0 0 0 0] [0 0 0 0 1 0 0 0 0 0] [0 0 0 0 1 0 0 0 0 0] [1 0 0 0 0 0 0 0 0 0] [1 0 0 0 0 0 0 0 0 0] [1 0 0 0 0 0 0 0 0 0] [1 0 0 0 0 0 0 0 0 0] [1 0 0 0 0 0 0 0 0 0] [1 0 0 0 0 0 0 0 0 0] [1 0 0 0 0 0 0 0 0 0] [0 0 0 0 0 0 0 0 1 0] [0 0 0 0 0 0 0 0 1 0] [0 0 0 0 0 0 0 0 1 0] [0 0 0 0 0 0 0 0 1 0] [0 0 0 0 0 0 0 0 1 0] [0 0 0 0 0 0 0 0 1 0] [0 0 0 0 0 1 0 0 0 0] [0 0 0 0 0 1 0 0 0 0] [0 0 0 0 0 1 0 0 0 0] [0 0 1 0 0 0 0 0 0 0] [0 0 1 0 0 0 0 0 0 0] [0 0 1 0 0 0 0 0 0 0] [0 0 1 0 0 0 0 0 0 0] [0 0 0 1 0 0 0 0 0 0] [0 0 0 1 0 0 0 0 0 0] [0 0 0 1 0 0 0 0 0 0] [0 0 0 1 0 0 0 0 0 0] [0 0 0 0 0 0 0 1 0 0] [0 0 0 0 0 0 0 1 0 0] [0 0 0 0 0 0 0 1 0 0] [0 0 0 0 0 0 0 1 0 0] [0 1 0 0 0 0 0 0 0 0] [0 1 0 0 0 0 0 0 0 0] [0 1 0 0 0 0 0 0 0 0] [0 0 0 0 0 0 0 0 0 1] [0 0 0 0 0 0 0 0 0 1] [0 0 0 0 0 0 0 0 0 1]]
Loading: 100%|███████████████████████████████████████████████████████| 2225192/2225192 [00:20<00:00, 1634466.61bytes/s]
3. Model i jego ćwiczenie
training = np.array(training) #zamiana typu dla sieci neuronowej
output = np.array(output) #zamiana typu dla sieci neuronowej3.1. Stworzenie DLN i inicjacja modelu
tensorflow.compat.v1.reset_default_graph() #Reset na wszelki wypadek (w sumie nie wiem czy to jakaś super ważna linijka kodu)
net = tflearn.input_data(shape=[None, len(training[0])]) #Input layer
net = tflearn.fully_connected(net, 8) #8 neurons for hidden layer
net = tflearn.fully_connected(net, 8) #8 neurons for hidden layer
#net = tflearn.fully_connected(net, 8) #8 neurons for hidden layer
net = tflearn.fully_connected(net, len(output[0]), activation="softmax") #len(output) neurons for output layer + Softmax jako najlepsze wyjście dla tego typu danych
net = tflearn.regression(net)
model = tflearn.DNN(net)
WARNING:tensorflow:From C:\Users\annad\AppData\Roaming\Python\Python38\site-packages\tflearn\initializations.py:164: calling TruncatedNormal.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor
3.2. Trening Modelu
model.fit(training, output, n_epoch=1000, batch_size=8, show_metric=True)
#Zapis Modelu
model.save("model.tflearn")Training Step: 5999 | total loss: [1m[32m0.02785[0m[0m | time: 0.007s | Adam | epoch: 1000 | loss: 0.02785 - acc: 0.9753 -- iter: 40/48 Training Step: 6000 | total loss: [1m[32m0.02583[0m[0m | time: 0.008s | Adam | epoch: 1000 | loss: 0.02583 - acc: 0.9777 -- iter: 48/48 -- INFO:tensorflow:C:\Users\annad\Desktop\System Dialogowy Janet\model.tflearn is not in all_model_checkpoint_paths. Manually adding it.
4. Input Użytkownika
4.1 Funkcja "bag_of_words(s, words)" do stemmowania twojego zdania, i przypisania mu formy binarnej
def bag_of_words(s, words):
bag = [0 for _ in range(len(words))]
s_words = nltk.word_tokenize(s)
s_words = [stemmer_pl.stem(word.lower()) for word in s_words]
for se in s_words:
for i, w in enumerate(words):
if w == se:
bag[i] = 1
return np.array(bag)4.2 Funkcja "chat()" do rozmowy z botem
def chat():
print("Możesz rozpocząć rozmowę z Botem! (type quit to stop)")
while True: #Ciągła rozmowa
inp = input("Ty: ")
if inp.lower() == "quit": #Quit by wyjść z loopa
break
result = model.predict([bag_of_words(inp,words)]) #Predictowanie przy pomocy wyćwiczonego modelu
result_index = np.argmax(result)
tag = labels[result_index]
for tg in data_pl["intents"]: #znalezienie poprawnego tagu do zdania
if tg['tag'] == tag:
responses = tg['responses']
print(random.choice(responses)) #Wyprintuj losową odpowiedz z danego zbioru odpowiedzi5. Rozmowa z botem!
chat()Możesz rozpocząć rozmowę z Botem! (type quit to stop) Ty: elo Cześć! Ty: w co grasz W OSRS Ty: nara Hej, w czym mogę pomóc? Ty: narazie Narazie! Ty: do widzenia Mam nadzieję, że później pogadamy Ty: dowidzenia Narazie! Ty: ok Hej, w czym mogę pomóc?