AMUseBot/ai_talks/AMUseBotBackend/utils/NLU_Trainer.ipynb
2023-06-05 21:23:33 +02:00

67 KiB
Raw Permalink Blame History

# Load the Drive helper and mount
from google.colab import drive
drive.mount('/content/drive')
Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).
! git clone https://github.com/YiweiJiang2015/CookDial.git
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
fatal: could not create work tree dir 'CookDial': No such file or directory
!pwd
/content/drive/MyDrive/CookDial
%cd /content/drive/MyDrive/CookDial
/content/drive/MyDrive/CookDial
!wget -O mini.sh https://repo.anaconda.com/miniconda/Miniconda3-py39_4.9.2-Linux-x86_64.sh
!chmod +x mini.sh
!bash ./mini.sh -b -f -p /usr/local
!conda install -q -y jupyter
!conda install -q -y google-colab -c conda-forge
!python -m ipykernel install --name "py39" --user
--2022-12-19 23:48:47--  https://repo.anaconda.com/miniconda/Miniconda3-py39_4.9.2-Linux-x86_64.sh
Resolving repo.anaconda.com (repo.anaconda.com)... 104.16.131.3, 104.16.130.3, 2606:4700::6810:8303, ...
Connecting to repo.anaconda.com (repo.anaconda.com)|104.16.131.3|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 61451533 (59M) [application/x-sh]
Saving to: mini.sh

mini.sh             100%[===================>]  58.60M  62.7MB/s    in 0.9s    

2022-12-19 23:48:48 (62.7 MB/s) - mini.sh saved [61451533/61451533]

PREFIX=/usr/local
Unpacking payload ...
Collecting package metadata (current_repodata.json): - \ done
Solving environment: / - \ done

## Package Plan ##

  environment location: /usr/local

  added / updated specs:
    - _libgcc_mutex==0.1=main
    - brotlipy==0.7.0=py39h27cfd23_1003
    - ca-certificates==2020.12.8=h06a4308_0
    - certifi==2020.12.5=py39h06a4308_0
    - cffi==1.14.4=py39h261ae71_0
    - chardet==3.0.4=py39h06a4308_1003
    - conda-package-handling==1.7.2=py39h27cfd23_1
    - conda==4.9.2=py39h06a4308_0
    - cryptography==3.3.1=py39h3c74f83_0
    - idna==2.10=py_0
    - ld_impl_linux-64==2.33.1=h53a641e_7
    - libedit==3.1.20191231=h14c3975_1
    - libffi==3.3=he6710b0_2
    - libgcc-ng==9.1.0=hdf63c60_0
    - libstdcxx-ng==9.1.0=hdf63c60_0
    - ncurses==6.2=he6710b0_1
    - openssl==1.1.1i=h27cfd23_0
    - pip==20.3.1=py39h06a4308_0
    - pycosat==0.6.3=py39h27cfd23_0
    - pycparser==2.20=py_2
    - pyopenssl==20.0.0=pyhd3eb1b0_1
    - pysocks==1.7.1=py39h06a4308_0
    - python==3.9.1=hdb3f193_2
    - readline==8.0=h7b6447c_0
    - requests==2.25.0=pyhd3eb1b0_0
    - ruamel_yaml==0.15.80=py39h27cfd23_0
    - setuptools==51.0.0=py39h06a4308_2
    - six==1.15.0=py39h06a4308_0
    - sqlite==3.33.0=h62c20be_0
    - tk==8.6.10=hbc83047_0
    - tqdm==4.54.1=pyhd3eb1b0_0
    - tzdata==2020d=h14c3975_0
    - urllib3==1.25.11=py_0
    - wheel==0.36.1=pyhd3eb1b0_0
    - xz==5.2.5=h7b6447c_0
    - yaml==0.2.5=h7b6447c_0
    - zlib==1.2.11=h7b6447c_3


The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
  brotlipy           pkgs/main/linux-64::brotlipy-0.7.0-py39h27cfd23_1003
  ca-certificates    pkgs/main/linux-64::ca-certificates-2020.12.8-h06a4308_0
  certifi            pkgs/main/linux-64::certifi-2020.12.5-py39h06a4308_0
  cffi               pkgs/main/linux-64::cffi-1.14.4-py39h261ae71_0
  chardet            pkgs/main/linux-64::chardet-3.0.4-py39h06a4308_1003
  conda              pkgs/main/linux-64::conda-4.9.2-py39h06a4308_0
  conda-package-han~ pkgs/main/linux-64::conda-package-handling-1.7.2-py39h27cfd23_1
  cryptography       pkgs/main/linux-64::cryptography-3.3.1-py39h3c74f83_0
  idna               pkgs/main/noarch::idna-2.10-py_0
  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.33.1-h53a641e_7
  libedit            pkgs/main/linux-64::libedit-3.1.20191231-h14c3975_1
  libffi             pkgs/main/linux-64::libffi-3.3-he6710b0_2
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0
  ncurses            pkgs/main/linux-64::ncurses-6.2-he6710b0_1
  openssl            pkgs/main/linux-64::openssl-1.1.1i-h27cfd23_0
  pip                pkgs/main/linux-64::pip-20.3.1-py39h06a4308_0
  pycosat            pkgs/main/linux-64::pycosat-0.6.3-py39h27cfd23_0
  pycparser          pkgs/main/noarch::pycparser-2.20-py_2
  pyopenssl          pkgs/main/noarch::pyopenssl-20.0.0-pyhd3eb1b0_1
  pysocks            pkgs/main/linux-64::pysocks-1.7.1-py39h06a4308_0
  python             pkgs/main/linux-64::python-3.9.1-hdb3f193_2
  readline           pkgs/main/linux-64::readline-8.0-h7b6447c_0
  requests           pkgs/main/noarch::requests-2.25.0-pyhd3eb1b0_0
  ruamel_yaml        pkgs/main/linux-64::ruamel_yaml-0.15.80-py39h27cfd23_0
  setuptools         pkgs/main/linux-64::setuptools-51.0.0-py39h06a4308_2
  six                pkgs/main/linux-64::six-1.15.0-py39h06a4308_0
  sqlite             pkgs/main/linux-64::sqlite-3.33.0-h62c20be_0
  tk                 pkgs/main/linux-64::tk-8.6.10-hbc83047_0
  tqdm               pkgs/main/noarch::tqdm-4.54.1-pyhd3eb1b0_0
  tzdata             pkgs/main/noarch::tzdata-2020d-h14c3975_0
  urllib3            pkgs/main/noarch::urllib3-1.25.11-py_0
  wheel              pkgs/main/noarch::wheel-0.36.1-pyhd3eb1b0_0
  xz                 pkgs/main/linux-64::xz-5.2.5-h7b6447c_0
  yaml               pkgs/main/linux-64::yaml-0.2.5-h7b6447c_0
  zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3


Preparing transaction: / - \ | done
Executing transaction: - \ | / - \ | / - \ | / - \ | / - \ done
installation finished.
WARNING:
    You currently have a PYTHONPATH environment variable set. This may cause
    unexpected behavior when running the Python interpreter in Miniconda3.
    For best results, please verify that your PYTHONPATH only points to
    directories of packages that are compatible with the Python interpreter
    in Miniconda3: /usr/local
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: /usr/local

  added / updated specs:
    - jupyter


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    _openmp_mutex-5.1          |            1_gnu          21 KB
    anyio-3.5.0                |   py39h06a4308_0         167 KB
    argon2-cffi-21.3.0         |     pyhd3eb1b0_0          15 KB
    argon2-cffi-bindings-21.2.0|   py39h7f8727e_0          33 KB
    asttokens-2.0.5            |     pyhd3eb1b0_0          20 KB
    attrs-22.1.0               |   py39h06a4308_0          85 KB
    babel-2.9.1                |     pyhd3eb1b0_0         5.5 MB
    backcall-0.2.0             |     pyhd3eb1b0_0          13 KB
    beautifulsoup4-4.11.1      |   py39h06a4308_0         185 KB
    bleach-4.1.0               |     pyhd3eb1b0_0         123 KB
    ca-certificates-2022.10.11 |       h06a4308_0         124 KB
    certifi-2022.9.24          |   py39h06a4308_0         154 KB
    conda-22.11.1              |   py39h06a4308_4         927 KB
    dbus-1.13.18               |       hb2f20db_0         504 KB
    debugpy-1.5.1              |   py39h295c915_0         1.7 MB
    decorator-5.1.1            |     pyhd3eb1b0_0          12 KB
    defusedxml-0.7.1           |     pyhd3eb1b0_0          23 KB
    entrypoints-0.4            |   py39h06a4308_0          16 KB
    executing-0.8.3            |     pyhd3eb1b0_0          18 KB
    expat-2.4.9                |       h6a678d5_0         156 KB
    flit-core-3.6.0            |     pyhd3eb1b0_0          42 KB
    fontconfig-2.14.1          |       h52c9d5c_1         281 KB
    freetype-2.12.1            |       h4a9f257_0         626 KB
    giflib-5.2.1               |       h7b6447c_0          78 KB
    glib-2.69.1                |       h4ff587b_1         1.7 MB
    gst-plugins-base-1.14.0    |       h8213a91_2         4.9 MB
    gstreamer-1.14.0           |       h28cd5cc_2         3.2 MB
    icu-58.2                   |       he6710b0_3        10.5 MB
    importlib-metadata-4.11.3  |   py39h06a4308_0          40 KB
    ipykernel-6.15.2           |   py39h06a4308_0         190 KB
    ipython-8.7.0              |   py39h06a4308_0         1.0 MB
    ipython_genutils-0.2.0     |     pyhd3eb1b0_1          27 KB
    ipywidgets-7.6.5           |     pyhd3eb1b0_1         105 KB
    jedi-0.18.1                |   py39h06a4308_1         982 KB
    jinja2-3.1.2               |   py39h06a4308_0         211 KB
    jpeg-9e                    |       h7f8727e_0         240 KB
    json5-0.9.6                |     pyhd3eb1b0_0          21 KB
    jsonschema-4.16.0          |   py39h06a4308_0         129 KB
    jupyter-1.0.0              |   py39h06a4308_8           7 KB
    jupyter_client-7.4.7       |   py39h06a4308_0         209 KB
    jupyter_console-6.4.3      |     pyhd3eb1b0_0          23 KB
    jupyter_core-4.11.2        |   py39h06a4308_0          80 KB
    jupyter_server-1.18.1      |   py39h06a4308_0         356 KB
    jupyterlab-3.5.0           |   py39h06a4308_0         4.2 MB
    jupyterlab_pygments-0.1.2  |             py_0           8 KB
    jupyterlab_server-2.16.3   |   py39h06a4308_0          80 KB
    jupyterlab_widgets-1.0.0   |     pyhd3eb1b0_1         109 KB
    krb5-1.19.2                |       hac12032_0         1.2 MB
    lerc-3.0                   |       h295c915_0         196 KB
    libclang-10.0.1            |default_hb85057a_2        10.8 MB
    libdeflate-1.8             |       h7f8727e_5          51 KB
    libedit-3.1.20221030       |       h5eee18b_0         181 KB
    libevent-2.1.12            |       h8f2d780_0         425 KB
    libgcc-ng-11.2.0           |       h1234567_1         5.3 MB
    libgomp-11.2.0             |       h1234567_1         474 KB
    libllvm10-10.0.1           |       hbcb73fb_5        22.1 MB
    libpng-1.6.37              |       hbc83047_0         278 KB
    libpq-12.9                 |       h16c4e8d_3         2.1 MB
    libsodium-1.0.18           |       h7b6447c_0         244 KB
    libstdcxx-ng-11.2.0        |       h1234567_1         4.7 MB
    libtiff-4.4.0              |       hecacb30_0         471 KB
    libuuid-1.41.5             |       h5eee18b_0          27 KB
    libwebp-1.2.4              |       h11a3e52_0          79 KB
    libwebp-base-1.2.4         |       h5eee18b_0         347 KB
    libxcb-1.15                |       h7f8727e_0         505 KB
    libxkbcommon-1.0.1         |       hfa300c1_0         483 KB
    libxml2-2.9.14             |       h74e7548_0         718 KB
    libxslt-1.1.35             |       h4e12654_0         453 KB
    lxml-4.9.1                 |   py39h1edc446_0         1.3 MB
    lz4-c-1.9.4                |       h6a678d5_0         154 KB
    markupsafe-2.1.1           |   py39h7f8727e_0          21 KB
    matplotlib-inline-0.1.6    |   py39h06a4308_0          16 KB
    mistune-0.8.4              |py39h27cfd23_1000          57 KB
    nbclassic-0.4.8            |   py39h06a4308_0         5.8 MB
    nbclient-0.5.13            |   py39h06a4308_0          91 KB
    nbconvert-6.5.4            |   py39h06a4308_0         513 KB
    nbformat-5.7.0             |   py39h06a4308_0         133 KB
    ncurses-6.3                |       h5eee18b_3         781 KB
    nest-asyncio-1.5.5         |   py39h06a4308_0          16 KB
    notebook-6.5.2             |   py39h06a4308_0         511 KB
    notebook-shim-0.2.2        |   py39h06a4308_0          22 KB
    nspr-4.33                  |       h295c915_0         222 KB
    nss-3.74                   |       h0370c37_0         1.9 MB
    openssl-1.1.1s             |       h7f8727e_0         3.6 MB
    packaging-21.3             |     pyhd3eb1b0_0          36 KB
    pandocfilters-1.5.0        |     pyhd3eb1b0_0          11 KB
    parso-0.8.3                |     pyhd3eb1b0_0          70 KB
    pcre-8.45                  |       h295c915_0         207 KB
    pexpect-4.8.0              |     pyhd3eb1b0_3          53 KB
    pickleshare-0.7.5          |  pyhd3eb1b0_1003          13 KB
    pluggy-1.0.0               |   py39h06a4308_1          28 KB
    ply-3.11                   |   py39h06a4308_0          81 KB
    prometheus_client-0.14.1   |   py39h06a4308_0          90 KB
    prompt-toolkit-3.0.20      |     pyhd3eb1b0_0         259 KB
    prompt_toolkit-3.0.20      |       hd3eb1b0_0          12 KB
    psutil-5.9.0               |   py39h5eee18b_0         330 KB
    ptyprocess-0.7.0           |     pyhd3eb1b0_2          17 KB
    pure_eval-0.2.2            |     pyhd3eb1b0_0          14 KB
    pygments-2.11.2            |     pyhd3eb1b0_0         759 KB
    pyparsing-3.0.9            |   py39h06a4308_0         151 KB
    pyqt-5.15.7                |   py39h6a678d5_1         5.1 MB
    pyqt5-sip-12.11.0          |   py39h6a678d5_1          87 KB
    pyrsistent-0.18.0          |   py39heee7806_0          94 KB
    python-dateutil-2.8.2      |     pyhd3eb1b0_0         233 KB
    python-fastjsonschema-2.16.2|   py39h06a4308_0         231 KB
    pytz-2022.1                |   py39h06a4308_0         194 KB
    pyzmq-23.2.0               |   py39h6a678d5_0         445 KB
    qt-main-5.15.2             |       h327a75a_7        45.1 MB
    qt-webengine-5.15.9        |       hd2b0992_4        47.1 MB
    qtconsole-5.3.2            |   py39h06a4308_0         176 KB
    qtpy-2.2.0                 |   py39h06a4308_0          84 KB
    qtwebkit-5.212             |       h4eab89a_4        14.3 MB
    ruamel.yaml-0.17.21        |   py39h5eee18b_0         178 KB
    ruamel.yaml.clib-0.2.6     |   py39h5eee18b_1         140 KB
    send2trash-1.8.0           |     pyhd3eb1b0_1          19 KB
    sip-6.6.2                  |   py39h6a678d5_0         423 KB
    sniffio-1.2.0              |   py39h06a4308_1          15 KB
    soupsieve-2.3.2.post1      |   py39h06a4308_0          65 KB
    sqlite-3.40.0              |       h5082296_0         1.2 MB
    stack_data-0.2.0           |     pyhd3eb1b0_0          22 KB
    terminado-0.13.1           |   py39h06a4308_0          30 KB
    tinycss2-1.2.1             |   py39h06a4308_0          39 KB
    toml-0.10.2                |     pyhd3eb1b0_0          20 KB
    tomli-2.0.1                |   py39h06a4308_0          24 KB
    toolz-0.12.0               |   py39h06a4308_0         105 KB
    tornado-6.2                |   py39h5eee18b_0         590 KB
    traitlets-5.7.1            |   py39h06a4308_0         200 KB
    typing-extensions-4.4.0    |   py39h06a4308_0           8 KB
    typing_extensions-4.4.0    |   py39h06a4308_0          46 KB
    wcwidth-0.2.5              |     pyhd3eb1b0_0          26 KB
    webencodings-0.5.1         |   py39h06a4308_1          20 KB
    websocket-client-0.58.0    |   py39h06a4308_4          66 KB
    widgetsnbextension-3.5.2   |   py39h06a4308_0         645 KB
    zeromq-4.3.4               |       h2531618_0         331 KB
    zipp-3.8.0                 |   py39h06a4308_0          15 KB
    zlib-1.2.13                |       h5eee18b_0         103 KB
    zstd-1.5.2                 |       ha4553b6_0         488 KB
    ------------------------------------------------------------
                                           Total:       224.9 MB

The following NEW packages will be INSTALLED:

  _openmp_mutex      pkgs/main/linux-64::_openmp_mutex-5.1-1_gnu
  anyio              pkgs/main/linux-64::anyio-3.5.0-py39h06a4308_0
  argon2-cffi        pkgs/main/noarch::argon2-cffi-21.3.0-pyhd3eb1b0_0
  argon2-cffi-bindi~ pkgs/main/linux-64::argon2-cffi-bindings-21.2.0-py39h7f8727e_0
  asttokens          pkgs/main/noarch::asttokens-2.0.5-pyhd3eb1b0_0
  attrs              pkgs/main/linux-64::attrs-22.1.0-py39h06a4308_0
  babel              pkgs/main/noarch::babel-2.9.1-pyhd3eb1b0_0
  backcall           pkgs/main/noarch::backcall-0.2.0-pyhd3eb1b0_0
  beautifulsoup4     pkgs/main/linux-64::beautifulsoup4-4.11.1-py39h06a4308_0
  bleach             pkgs/main/noarch::bleach-4.1.0-pyhd3eb1b0_0
  dbus               pkgs/main/linux-64::dbus-1.13.18-hb2f20db_0
  debugpy            pkgs/main/linux-64::debugpy-1.5.1-py39h295c915_0
  decorator          pkgs/main/noarch::decorator-5.1.1-pyhd3eb1b0_0
  defusedxml         pkgs/main/noarch::defusedxml-0.7.1-pyhd3eb1b0_0
  entrypoints        pkgs/main/linux-64::entrypoints-0.4-py39h06a4308_0
  executing          pkgs/main/noarch::executing-0.8.3-pyhd3eb1b0_0
  expat              pkgs/main/linux-64::expat-2.4.9-h6a678d5_0
  flit-core          pkgs/main/noarch::flit-core-3.6.0-pyhd3eb1b0_0
  fontconfig         pkgs/main/linux-64::fontconfig-2.14.1-h52c9d5c_1
  freetype           pkgs/main/linux-64::freetype-2.12.1-h4a9f257_0
  giflib             pkgs/main/linux-64::giflib-5.2.1-h7b6447c_0
  glib               pkgs/main/linux-64::glib-2.69.1-h4ff587b_1
  gst-plugins-base   pkgs/main/linux-64::gst-plugins-base-1.14.0-h8213a91_2
  gstreamer          pkgs/main/linux-64::gstreamer-1.14.0-h28cd5cc_2
  icu                pkgs/main/linux-64::icu-58.2-he6710b0_3
  importlib-metadata pkgs/main/linux-64::importlib-metadata-4.11.3-py39h06a4308_0
  ipykernel          pkgs/main/linux-64::ipykernel-6.15.2-py39h06a4308_0
  ipython            pkgs/main/linux-64::ipython-8.7.0-py39h06a4308_0
  ipython_genutils   pkgs/main/noarch::ipython_genutils-0.2.0-pyhd3eb1b0_1
  ipywidgets         pkgs/main/noarch::ipywidgets-7.6.5-pyhd3eb1b0_1
  jedi               pkgs/main/linux-64::jedi-0.18.1-py39h06a4308_1
  jinja2             pkgs/main/linux-64::jinja2-3.1.2-py39h06a4308_0
  jpeg               pkgs/main/linux-64::jpeg-9e-h7f8727e_0
  json5              pkgs/main/noarch::json5-0.9.6-pyhd3eb1b0_0
  jsonschema         pkgs/main/linux-64::jsonschema-4.16.0-py39h06a4308_0
  jupyter            pkgs/main/linux-64::jupyter-1.0.0-py39h06a4308_8
  jupyter_client     pkgs/main/linux-64::jupyter_client-7.4.7-py39h06a4308_0
  jupyter_console    pkgs/main/noarch::jupyter_console-6.4.3-pyhd3eb1b0_0
  jupyter_core       pkgs/main/linux-64::jupyter_core-4.11.2-py39h06a4308_0
  jupyter_server     pkgs/main/linux-64::jupyter_server-1.18.1-py39h06a4308_0
  jupyterlab         pkgs/main/linux-64::jupyterlab-3.5.0-py39h06a4308_0
  jupyterlab_pygmen~ pkgs/main/noarch::jupyterlab_pygments-0.1.2-py_0
  jupyterlab_server  pkgs/main/linux-64::jupyterlab_server-2.16.3-py39h06a4308_0
  jupyterlab_widgets pkgs/main/noarch::jupyterlab_widgets-1.0.0-pyhd3eb1b0_1
  krb5               pkgs/main/linux-64::krb5-1.19.2-hac12032_0
  lerc               pkgs/main/linux-64::lerc-3.0-h295c915_0
  libclang           pkgs/main/linux-64::libclang-10.0.1-default_hb85057a_2
  libdeflate         pkgs/main/linux-64::libdeflate-1.8-h7f8727e_5
  libevent           pkgs/main/linux-64::libevent-2.1.12-h8f2d780_0
  libgomp            pkgs/main/linux-64::libgomp-11.2.0-h1234567_1
  libllvm10          pkgs/main/linux-64::libllvm10-10.0.1-hbcb73fb_5
  libpng             pkgs/main/linux-64::libpng-1.6.37-hbc83047_0
  libpq              pkgs/main/linux-64::libpq-12.9-h16c4e8d_3
  libsodium          pkgs/main/linux-64::libsodium-1.0.18-h7b6447c_0
  libtiff            pkgs/main/linux-64::libtiff-4.4.0-hecacb30_0
  libuuid            pkgs/main/linux-64::libuuid-1.41.5-h5eee18b_0
  libwebp            pkgs/main/linux-64::libwebp-1.2.4-h11a3e52_0
  libwebp-base       pkgs/main/linux-64::libwebp-base-1.2.4-h5eee18b_0
  libxcb             pkgs/main/linux-64::libxcb-1.15-h7f8727e_0
  libxkbcommon       pkgs/main/linux-64::libxkbcommon-1.0.1-hfa300c1_0
  libxml2            pkgs/main/linux-64::libxml2-2.9.14-h74e7548_0
  libxslt            pkgs/main/linux-64::libxslt-1.1.35-h4e12654_0
  lxml               pkgs/main/linux-64::lxml-4.9.1-py39h1edc446_0
  lz4-c              pkgs/main/linux-64::lz4-c-1.9.4-h6a678d5_0
  markupsafe         pkgs/main/linux-64::markupsafe-2.1.1-py39h7f8727e_0
  matplotlib-inline  pkgs/main/linux-64::matplotlib-inline-0.1.6-py39h06a4308_0
  mistune            pkgs/main/linux-64::mistune-0.8.4-py39h27cfd23_1000
  nbclassic          pkgs/main/linux-64::nbclassic-0.4.8-py39h06a4308_0
  nbclient           pkgs/main/linux-64::nbclient-0.5.13-py39h06a4308_0
  nbconvert          pkgs/main/linux-64::nbconvert-6.5.4-py39h06a4308_0
  nbformat           pkgs/main/linux-64::nbformat-5.7.0-py39h06a4308_0
  nest-asyncio       pkgs/main/linux-64::nest-asyncio-1.5.5-py39h06a4308_0
  notebook           pkgs/main/linux-64::notebook-6.5.2-py39h06a4308_0
  notebook-shim      pkgs/main/linux-64::notebook-shim-0.2.2-py39h06a4308_0
  nspr               pkgs/main/linux-64::nspr-4.33-h295c915_0
  nss                pkgs/main/linux-64::nss-3.74-h0370c37_0
  packaging          pkgs/main/noarch::packaging-21.3-pyhd3eb1b0_0
  pandocfilters      pkgs/main/noarch::pandocfilters-1.5.0-pyhd3eb1b0_0
  parso              pkgs/main/noarch::parso-0.8.3-pyhd3eb1b0_0
  pcre               pkgs/main/linux-64::pcre-8.45-h295c915_0
  pexpect            pkgs/main/noarch::pexpect-4.8.0-pyhd3eb1b0_3
  pickleshare        pkgs/main/noarch::pickleshare-0.7.5-pyhd3eb1b0_1003
  pluggy             pkgs/main/linux-64::pluggy-1.0.0-py39h06a4308_1
  ply                pkgs/main/linux-64::ply-3.11-py39h06a4308_0
  prometheus_client  pkgs/main/linux-64::prometheus_client-0.14.1-py39h06a4308_0
  prompt-toolkit     pkgs/main/noarch::prompt-toolkit-3.0.20-pyhd3eb1b0_0
  prompt_toolkit     pkgs/main/noarch::prompt_toolkit-3.0.20-hd3eb1b0_0
  psutil             pkgs/main/linux-64::psutil-5.9.0-py39h5eee18b_0
  ptyprocess         pkgs/main/noarch::ptyprocess-0.7.0-pyhd3eb1b0_2
  pure_eval          pkgs/main/noarch::pure_eval-0.2.2-pyhd3eb1b0_0
  pygments           pkgs/main/noarch::pygments-2.11.2-pyhd3eb1b0_0
  pyparsing          pkgs/main/linux-64::pyparsing-3.0.9-py39h06a4308_0
  pyqt               pkgs/main/linux-64::pyqt-5.15.7-py39h6a678d5_1
  pyqt5-sip          pkgs/main/linux-64::pyqt5-sip-12.11.0-py39h6a678d5_1
  pyrsistent         pkgs/main/linux-64::pyrsistent-0.18.0-py39heee7806_0
  python-dateutil    pkgs/main/noarch::python-dateutil-2.8.2-pyhd3eb1b0_0
  python-fastjsonsc~ pkgs/main/linux-64::python-fastjsonschema-2.16.2-py39h06a4308_0
  pytz               pkgs/main/linux-64::pytz-2022.1-py39h06a4308_0
  pyzmq              pkgs/main/linux-64::pyzmq-23.2.0-py39h6a678d5_0
  qt-main            pkgs/main/linux-64::qt-main-5.15.2-h327a75a_7
  qt-webengine       pkgs/main/linux-64::qt-webengine-5.15.9-hd2b0992_4
  qtconsole          pkgs/main/linux-64::qtconsole-5.3.2-py39h06a4308_0
  qtpy               pkgs/main/linux-64::qtpy-2.2.0-py39h06a4308_0
  qtwebkit           pkgs/main/linux-64::qtwebkit-5.212-h4eab89a_4
  ruamel.yaml        pkgs/main/linux-64::ruamel.yaml-0.17.21-py39h5eee18b_0
  ruamel.yaml.clib   pkgs/main/linux-64::ruamel.yaml.clib-0.2.6-py39h5eee18b_1
  send2trash         pkgs/main/noarch::send2trash-1.8.0-pyhd3eb1b0_1
  sip                pkgs/main/linux-64::sip-6.6.2-py39h6a678d5_0
  sniffio            pkgs/main/linux-64::sniffio-1.2.0-py39h06a4308_1
  soupsieve          pkgs/main/linux-64::soupsieve-2.3.2.post1-py39h06a4308_0
  stack_data         pkgs/main/noarch::stack_data-0.2.0-pyhd3eb1b0_0
  terminado          pkgs/main/linux-64::terminado-0.13.1-py39h06a4308_0
  tinycss2           pkgs/main/linux-64::tinycss2-1.2.1-py39h06a4308_0
  toml               pkgs/main/noarch::toml-0.10.2-pyhd3eb1b0_0
  tomli              pkgs/main/linux-64::tomli-2.0.1-py39h06a4308_0
  toolz              pkgs/main/linux-64::toolz-0.12.0-py39h06a4308_0
  tornado            pkgs/main/linux-64::tornado-6.2-py39h5eee18b_0
  traitlets          pkgs/main/linux-64::traitlets-5.7.1-py39h06a4308_0
  typing-extensions  pkgs/main/linux-64::typing-extensions-4.4.0-py39h06a4308_0
  typing_extensions  pkgs/main/linux-64::typing_extensions-4.4.0-py39h06a4308_0
  wcwidth            pkgs/main/noarch::wcwidth-0.2.5-pyhd3eb1b0_0
  webencodings       pkgs/main/linux-64::webencodings-0.5.1-py39h06a4308_1
  websocket-client   pkgs/main/linux-64::websocket-client-0.58.0-py39h06a4308_4
  widgetsnbextension pkgs/main/linux-64::widgetsnbextension-3.5.2-py39h06a4308_0
  zeromq             pkgs/main/linux-64::zeromq-4.3.4-h2531618_0
  zipp               pkgs/main/linux-64::zipp-3.8.0-py39h06a4308_0
  zstd               pkgs/main/linux-64::zstd-1.5.2-ha4553b6_0

The following packages will be UPDATED:

  ca-certificates                      2020.12.8-h06a4308_0 --> 2022.10.11-h06a4308_0
  certifi                          2020.12.5-py39h06a4308_0 --> 2022.9.24-py39h06a4308_0
  conda                                4.9.2-py39h06a4308_0 --> 22.11.1-py39h06a4308_4
  libedit                           3.1.20191231-h14c3975_1 --> 3.1.20221030-h5eee18b_0
  libgcc-ng                                9.1.0-hdf63c60_0 --> 11.2.0-h1234567_1
  libstdcxx-ng                             9.1.0-hdf63c60_0 --> 11.2.0-h1234567_1
  ncurses                                    6.2-he6710b0_1 --> 6.3-h5eee18b_3
  openssl                                 1.1.1i-h27cfd23_0 --> 1.1.1s-h7f8727e_0
  sqlite                                  3.33.0-h62c20be_0 --> 3.40.0-h5082296_0
  zlib                                    1.2.11-h7b6447c_3 --> 1.2.13-h5eee18b_0


Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: /usr/local

  added / updated specs:
    - google-colab


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    aiohttp-3.8.1              |   py39hb9d737c_1         565 KB  conda-forge
    aiosignal-1.3.1            |     pyhd8ed1ab_0          12 KB  conda-forge
    async-timeout-4.0.2        |     pyhd8ed1ab_0           9 KB  conda-forge
    ca-certificates-2022.12.7  |       ha878542_0         143 KB  conda-forge
    cachetools-5.2.0           |     pyhd8ed1ab_0          13 KB  conda-forge
    certifi-2022.12.7          |     pyhd8ed1ab_0         147 KB  conda-forge
    charset-normalizer-2.1.1   |     pyhd8ed1ab_0          36 KB  conda-forge
    conda-22.11.1              |   py39hf3d152e_1         904 KB  conda-forge
    frozenlist-1.3.3           |   py39h5eee18b_0          45 KB
    google-auth-2.15.0         |     pyh1a96a4e_0          95 KB  conda-forge
    google-colab-1.0.0         |     pyh44b312d_0          77 KB  conda-forge
    libblas-3.9.0              |15_linux64_openblas          12 KB  conda-forge
    libcblas-3.9.0             |15_linux64_openblas          12 KB  conda-forge
    libgfortran-ng-12.2.0      |      h69a702a_19          22 KB  conda-forge
    libgfortran5-12.2.0        |      h337968e_19         1.8 MB  conda-forge
    liblapack-3.9.0            |15_linux64_openblas          12 KB  conda-forge
    libopenblas-0.3.20         |pthreads_h78a6416_0        10.1 MB  conda-forge
    multidict-6.0.2            |   py39hb9d737c_1          51 KB  conda-forge
    numpy-1.22.3               |   py39hc58783e_2         6.8 MB  conda-forge
    pandas-1.2.3               |   py39hde0f152_0        12.1 MB  conda-forge
    portpicker-1.5.2           |     pyhd8ed1ab_0          17 KB  conda-forge
    pyasn1-0.4.8               |             py_0          53 KB  conda-forge
    pyasn1-modules-0.2.7       |             py_0          60 KB  conda-forge
    python_abi-3.9             |           2_cp39           4 KB  conda-forge
    pyu2f-0.1.5                |     pyhd8ed1ab_0          31 KB  conda-forge
    rsa-4.9                    |     pyhd8ed1ab_0          29 KB  conda-forge
    yarl-1.7.2                 |   py39hb9d737c_2         132 KB  conda-forge
    ------------------------------------------------------------
                                           Total:        33.1 MB

The following NEW packages will be INSTALLED:

  aiohttp            conda-forge/linux-64::aiohttp-3.8.1-py39hb9d737c_1 
  aiosignal          conda-forge/noarch::aiosignal-1.3.1-pyhd8ed1ab_0 
  async-timeout      conda-forge/noarch::async-timeout-4.0.2-pyhd8ed1ab_0 
  cachetools         conda-forge/noarch::cachetools-5.2.0-pyhd8ed1ab_0 
  charset-normalizer conda-forge/noarch::charset-normalizer-2.1.1-pyhd8ed1ab_0 
  frozenlist         pkgs/main/linux-64::frozenlist-1.3.3-py39h5eee18b_0 
  google-auth        conda-forge/noarch::google-auth-2.15.0-pyh1a96a4e_0 
  google-colab       conda-forge/noarch::google-colab-1.0.0-pyh44b312d_0 
  libblas            conda-forge/linux-64::libblas-3.9.0-15_linux64_openblas 
  libcblas           conda-forge/linux-64::libcblas-3.9.0-15_linux64_openblas 
  libgfortran-ng     conda-forge/linux-64::libgfortran-ng-12.2.0-h69a702a_19 
  libgfortran5       conda-forge/linux-64::libgfortran5-12.2.0-h337968e_19 
  liblapack          conda-forge/linux-64::liblapack-3.9.0-15_linux64_openblas 
  libopenblas        conda-forge/linux-64::libopenblas-0.3.20-pthreads_h78a6416_0 
  multidict          conda-forge/linux-64::multidict-6.0.2-py39hb9d737c_1 
  numpy              conda-forge/linux-64::numpy-1.22.3-py39hc58783e_2 
  pandas             conda-forge/linux-64::pandas-1.2.3-py39hde0f152_0 
  portpicker         conda-forge/noarch::portpicker-1.5.2-pyhd8ed1ab_0 
  pyasn1             conda-forge/noarch::pyasn1-0.4.8-py_0 
  pyasn1-modules     conda-forge/noarch::pyasn1-modules-0.2.7-py_0 
  python_abi         conda-forge/linux-64::python_abi-3.9-2_cp39 
  pyu2f              conda-forge/noarch::pyu2f-0.1.5-pyhd8ed1ab_0 
  rsa                conda-forge/noarch::rsa-4.9-pyhd8ed1ab_0 
  yarl               conda-forge/linux-64::yarl-1.7.2-py39hb9d737c_2 

The following packages will be UPDATED:

  ca-certificates    pkgs/main::ca-certificates-2022.10.11~ --> conda-forge::ca-certificates-2022.12.7-ha878542_0 
  certifi            pkgs/main/linux-64::certifi-2022.9.24~ --> conda-forge/noarch::certifi-2022.12.7-pyhd8ed1ab_0 

The following packages will be SUPERSEDED by a higher-priority channel:

  conda              pkgs/main::conda-22.11.1-py39h06a4308~ --> conda-forge::conda-22.11.1-py39hf3d152e_1 


Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Installed kernelspec py39 in /root/.local/share/jupyter/kernels/py39
# Install all the dependencies:
! pip install -r ./src/requirements.txt
# Install the spacy dictionary 
! python -m spacy download en_core_web_sm
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Collecting en_core_web_sm
  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.3.1/en_core_web_sm-2.3.1.tar.gz (12.0 MB)
     |████████████████████████████████| 12.0 MB 4.9 MB/s 
[?25hCollecting overrides==3.1.0
  Downloading overrides-3.1.0.tar.gz (11 kB)
Collecting pandas==1.2.4
  Downloading pandas-1.2.4-cp39-cp39-manylinux1_x86_64.whl (9.7 MB)
     |████████████████████████████████| 9.7 MB 4.5 MB/s 
[?25hRequirement already satisfied: numpy>=1.16.5 in /usr/local/lib/python3.9/site-packages (from pandas==1.2.4->-r ./src/requirements.txt (line 1)) (1.22.3)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.9/site-packages (from pandas==1.2.4->-r ./src/requirements.txt (line 1)) (2022.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.9/site-packages (from pandas==1.2.4->-r ./src/requirements.txt (line 1)) (2.8.2)
Collecting pytorch-crf==0.7.2
  Downloading pytorch_crf-0.7.2-py3-none-any.whl (9.5 kB)
Collecting spacy==2.3.5
  Downloading spacy-2.3.5-cp39-cp39-manylinux2014_x86_64.whl (10.3 MB)
     |████████████████████████████████| 10.3 MB 64.1 MB/s 
[?25hRequirement already satisfied: setuptools in /usr/local/lib/python3.9/site-packages (from spacy==2.3.5->-r ./src/requirements.txt (line 2)) (51.0.0.post20201207)
Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.9/site-packages (from spacy==2.3.5->-r ./src/requirements.txt (line 2)) (2.25.0)
Requirement already satisfied: numpy>=1.16.5 in /usr/local/lib/python3.9/site-packages (from pandas==1.2.4->-r ./src/requirements.txt (line 1)) (1.22.3)
Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.9/site-packages (from spacy==2.3.5->-r ./src/requirements.txt (line 2)) (4.54.1)
Collecting tensorboard>=2.4
  Downloading tensorboard-2.11.0-py3-none-any.whl (6.0 MB)
     |████████████████████████████████| 6.0 MB 64.9 MB/s 
[?25hRequirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.9/site-packages (from tensorboard>=2.4->-r ./src/requirements.txt (line 8)) (0.36.1)
Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.9/site-packages (from spacy==2.3.5->-r ./src/requirements.txt (line 2)) (2.25.0)
Requirement already satisfied: setuptools in /usr/local/lib/python3.9/site-packages (from spacy==2.3.5->-r ./src/requirements.txt (line 2)) (51.0.0.post20201207)
Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.9/site-packages (from tensorboard>=2.4->-r ./src/requirements.txt (line 8)) (2.15.0)
Requirement already satisfied: numpy>=1.16.5 in /usr/local/lib/python3.9/site-packages (from pandas==1.2.4->-r ./src/requirements.txt (line 1)) (1.22.3)
Collecting tensorboardX==2.1
  Downloading tensorboardX-2.1-py2.py3-none-any.whl (308 kB)
     |████████████████████████████████| 308 kB 67.2 MB/s 
[?25hRequirement already satisfied: six in /usr/local/lib/python3.9/site-packages (from tensorboardX==2.1->-r ./src/requirements.txt (line 6)) (1.15.0)
Requirement already satisfied: numpy>=1.16.5 in /usr/local/lib/python3.9/site-packages (from pandas==1.2.4->-r ./src/requirements.txt (line 1)) (1.22.3)
Collecting torch>=1.7.1
  Downloading torch-1.13.1-cp39-cp39-manylinux1_x86_64.whl (887.4 MB)
     |██████████████████████████████  | 834.1 MB 1.5 MB/s eta 0:00:37tcmalloc: large alloc 1147494400 bytes == 0x55a05534a000 @  0x7f01f16fc615 0x559ff0b66226 0x559ff0bcd322 0x559ff0bcd46b 0x559ff0bcd4c9 0x559ff0bba883 0x559ff0b1ef88 0x559ff0bc904d 0x559ff0b1efc1 0x559ff0bc904d 0x559ff0b1f4fe 0x559ff0c612f5 0x559ff0bf1f07 0x559ff0c612f5 0x559ff0bf1f07 0x559ff0c612f5 0x559ff0bf1f07 0x559ff0bcf8f9 0x559ff0bcfcaa 0x559ff0bcfdc6 0x559ff0b6b69f 0x559ff0bf18b6 0x559ff0bc782a 0x559ff0bc8357 0x559ff0b1f4fe 0x559ff0bc782a 0x559ff0bc8357 0x559ff0b1f0f8 0x559ff0bc8603 0x559ff0b1ef88 0x559ff0bc904d
     |████████████████████████████████| 887.4 MB 13 kB/s 
[?25hRequirement already satisfied: typing-extensions in /usr/local/lib/python3.9/site-packages (from torch>=1.7.1->-r ./src/requirements.txt (line 7)) (4.4.0)
Collecting transformers>=4.6.1
  Downloading transformers-4.25.1-py3-none-any.whl (5.8 MB)
     |████████████████████████████████| 5.8 MB 58.0 MB/s 
[?25hRequirement already satisfied: numpy>=1.16.5 in /usr/local/lib/python3.9/site-packages (from pandas==1.2.4->-r ./src/requirements.txt (line 1)) (1.22.3)
Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.9/site-packages (from spacy==2.3.5->-r ./src/requirements.txt (line 2)) (2.25.0)
Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.9/site-packages (from spacy==2.3.5->-r ./src/requirements.txt (line 2)) (4.54.1)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.9/site-packages (from transformers>=4.6.1->-r ./src/requirements.txt (line 9)) (21.3)
Collecting absl-py>=0.4
  Downloading absl_py-1.3.0-py3-none-any.whl (124 kB)
     |████████████████████████████████| 124 kB 42.1 MB/s 
[?25hCollecting blis<0.8.0,>=0.4.0
  Downloading blis-0.7.9-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (10.2 MB)
     |████████████████████████████████| 10.2 MB 63.2 MB/s 
[?25hRequirement already satisfied: numpy>=1.16.5 in /usr/local/lib/python3.9/site-packages (from pandas==1.2.4->-r ./src/requirements.txt (line 1)) (1.22.3)
Collecting catalogue<1.1.0,>=0.0.7
  Downloading catalogue-1.0.2-py2.py3-none-any.whl (16 kB)
Collecting cymem<2.1.0,>=2.0.2
  Downloading cymem-2.0.7-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (35 kB)
Collecting filelock
  Downloading filelock-3.8.2-py3-none-any.whl (10 kB)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.9/site-packages (from google-auth<3,>=1.6.3->tensorboard>=2.4->-r ./src/requirements.txt (line 8)) (5.2.0)
Requirement already satisfied: six in /usr/local/lib/python3.9/site-packages (from tensorboardX==2.1->-r ./src/requirements.txt (line 6)) (1.15.0)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.9/site-packages (from google-auth<3,>=1.6.3->tensorboard>=2.4->-r ./src/requirements.txt (line 8)) (4.9)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.9/site-packages (from google-auth<3,>=1.6.3->tensorboard>=2.4->-r ./src/requirements.txt (line 8)) (0.2.7)
Collecting google-auth-oauthlib<0.5,>=0.4.1
  Downloading google_auth_oauthlib-0.4.6-py2.py3-none-any.whl (18 kB)
Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.9/site-packages (from tensorboard>=2.4->-r ./src/requirements.txt (line 8)) (2.15.0)
Collecting grpcio>=1.24.3
  Downloading grpcio-1.51.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.8 MB)
     |████████████████████████████████| 4.8 MB 69.1 MB/s 
[?25hCollecting huggingface-hub<1.0,>=0.10.0
  Downloading huggingface_hub-0.11.1-py3-none-any.whl (182 kB)
     |████████████████████████████████| 182 kB 70.7 MB/s 
[?25hRequirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.9/site-packages (from transformers>=4.6.1->-r ./src/requirements.txt (line 9)) (21.3)
Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.9/site-packages (from spacy==2.3.5->-r ./src/requirements.txt (line 2)) (2.25.0)
Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.9/site-packages (from spacy==2.3.5->-r ./src/requirements.txt (line 2)) (4.54.1)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.9/site-packages (from torch>=1.7.1->-r ./src/requirements.txt (line 7)) (4.4.0)
Collecting markdown>=2.6.8
  Downloading Markdown-3.4.1-py3-none-any.whl (93 kB)
     |████████████████████████████████| 93 kB 1.9 MB/s 
[?25hRequirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.9/site-packages (from markdown>=2.6.8->tensorboard>=2.4->-r ./src/requirements.txt (line 8)) (4.11.3)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.9/site-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard>=2.4->-r ./src/requirements.txt (line 8)) (3.8.0)
Collecting murmurhash<1.1.0,>=0.28.0
  Downloading murmurhash-1.0.9-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (21 kB)
Collecting nvidia-cublas-cu11==11.10.3.66
  Downloading nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl (317.1 MB)
     |████████████████████████████████| 317.1 MB 32 kB/s 
[?25hRequirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.9/site-packages (from tensorboard>=2.4->-r ./src/requirements.txt (line 8)) (0.36.1)
Requirement already satisfied: setuptools in /usr/local/lib/python3.9/site-packages (from spacy==2.3.5->-r ./src/requirements.txt (line 2)) (51.0.0.post20201207)
Collecting nvidia-cuda-nvrtc-cu11==11.7.99
  Downloading nvidia_cuda_nvrtc_cu11-11.7.99-2-py3-none-manylinux1_x86_64.whl (21.0 MB)
     |████████████████████████████████| 21.0 MB 1.2 MB/s 
[?25hCollecting nvidia-cuda-runtime-cu11==11.7.99
  Downloading nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl (849 kB)
     |████████████████████████████████| 849 kB 77.2 MB/s 
[?25hRequirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.9/site-packages (from tensorboard>=2.4->-r ./src/requirements.txt (line 8)) (0.36.1)
Requirement already satisfied: setuptools in /usr/local/lib/python3.9/site-packages (from spacy==2.3.5->-r ./src/requirements.txt (line 2)) (51.0.0.post20201207)
Collecting nvidia-cudnn-cu11==8.5.0.96
  Downloading nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl (557.1 MB)
     |████████████████████████████████| 557.1 MB 11 kB/s 
[?25hRequirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.9/site-packages (from packaging>=20.0->transformers>=4.6.1->-r ./src/requirements.txt (line 9)) (3.0.9)
Collecting plac<1.2.0,>=0.9.6
  Downloading plac-1.1.3-py2.py3-none-any.whl (20 kB)
Collecting preshed<3.1.0,>=3.0.2
  Downloading preshed-3.0.8-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (128 kB)
     |████████████████████████████████| 128 kB 76.2 MB/s 
[?25hCollecting protobuf<4,>=3.9.2
  Downloading protobuf-3.20.3-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB)
     |████████████████████████████████| 1.0 MB 62.5 MB/s 
[?25hRequirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.9/site-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard>=2.4->-r ./src/requirements.txt (line 8)) (0.4.8)
Requirement already satisfied: six in /usr/local/lib/python3.9/site-packages (from tensorboardX==2.1->-r ./src/requirements.txt (line 6)) (1.15.0)
Collecting pyyaml>=5.1
  Downloading PyYAML-6.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (661 kB)
     |████████████████████████████████| 661 kB 64.9 MB/s 
[?25hCollecting regex!=2019.12.17
  Downloading regex-2022.10.31-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (769 kB)
     |████████████████████████████████| 769 kB 61.3 MB/s 
[?25hRequirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.9/site-packages (from requests<3.0.0,>=2.13.0->spacy==2.3.5->-r ./src/requirements.txt (line 2)) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.9/site-packages (from requests<3.0.0,>=2.13.0->spacy==2.3.5->-r ./src/requirements.txt (line 2)) (2.10)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/site-packages (from requests<3.0.0,>=2.13.0->spacy==2.3.5->-r ./src/requirements.txt (line 2)) (2022.12.7)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.9/site-packages (from requests<3.0.0,>=2.13.0->spacy==2.3.5->-r ./src/requirements.txt (line 2)) (1.25.11)
Collecting requests-oauthlib>=0.7.0
  Downloading requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.9/site-packages (from spacy==2.3.5->-r ./src/requirements.txt (line 2)) (2.25.0)
Collecting oauthlib>=3.0.0
  Downloading oauthlib-3.2.2-py3-none-any.whl (151 kB)
     |████████████████████████████████| 151 kB 72.9 MB/s 
[?25hRequirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.9/site-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard>=2.4->-r ./src/requirements.txt (line 8)) (0.4.8)
Collecting srsly<1.1.0,>=1.0.2
  Downloading srsly-1.0.6-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (209 kB)
     |████████████████████████████████| 209 kB 75.4 MB/s 
[?25hCollecting tensorboard-data-server<0.7.0,>=0.6.0
  Downloading tensorboard_data_server-0.6.1-py3-none-manylinux2010_x86_64.whl (4.9 MB)
     |████████████████████████████████| 4.9 MB 50.9 MB/s 
[?25hCollecting tensorboard-plugin-wit>=1.6.0
  Downloading tensorboard_plugin_wit-1.8.1-py3-none-any.whl (781 kB)
     |████████████████████████████████| 781 kB 63.7 MB/s 
[?25hCollecting thinc<7.5.0,>=7.4.1
  Downloading thinc-7.4.6-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
     |████████████████████████████████| 1.1 MB 61.9 MB/s 
[?25hRequirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.9/site-packages (from spacy==2.3.5->-r ./src/requirements.txt (line 2)) (4.54.1)
Requirement already satisfied: numpy>=1.16.5 in /usr/local/lib/python3.9/site-packages (from pandas==1.2.4->-r ./src/requirements.txt (line 1)) (1.22.3)
Collecting tokenizers!=0.11.3,<0.14,>=0.11.1
  Downloading tokenizers-0.13.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.6 MB)
     |████████████████████████████████| 7.6 MB 55.7 MB/s 
[?25hCollecting wasabi<1.1.0,>=0.4.0
  Downloading wasabi-0.10.1-py3-none-any.whl (26 kB)
Collecting werkzeug>=1.0.1
  Downloading Werkzeug-2.2.2-py3-none-any.whl (232 kB)
     |████████████████████████████████| 232 kB 73.6 MB/s 
[?25hRequirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.9/site-packages (from werkzeug>=1.0.1->tensorboard>=2.4->-r ./src/requirements.txt (line 8)) (2.1.1)
Building wheels for collected packages: en-core-web-sm, overrides
  Building wheel for en-core-web-sm (setup.py) ... [?25l[?25hdone
  Created wheel for en-core-web-sm: filename=en_core_web_sm-2.3.1-py3-none-any.whl size=12047107 sha256=19466519d049944b18f99c457b2a0f157c00b3e1ea869289bc6a4431665de650
  Stored in directory: /root/.cache/pip/wheels/19/d6/1c/5484b95647df5d7afaf74abde458c66c1cd427e69e801fe826
  Building wheel for overrides (setup.py) ... [?25l[?25hdone
  Created wheel for overrides: filename=overrides-3.1.0-py3-none-any.whl size=10174 sha256=7e73fa98f72561a9ab69fbde0dd03ce1a28aded313b227e2eb71d099669a9259
  Stored in directory: /root/.cache/pip/wheels/7d/11/0e/73fdcb3d71d97e33c230900efe85923ee9d49515d050503174
Successfully built en-core-web-sm overrides
Installing collected packages: murmurhash, cymem, wasabi, srsly, preshed, plac, oauthlib, catalogue, blis, thinc, requests-oauthlib, pyyaml, nvidia-cublas-cu11, filelock, werkzeug, tokenizers, tensorboard-plugin-wit, tensorboard-data-server, spacy, regex, protobuf, nvidia-cudnn-cu11, nvidia-cuda-runtime-cu11, nvidia-cuda-nvrtc-cu11, markdown, huggingface-hub, grpcio, google-auth-oauthlib, absl-py, transformers, torch, tensorboardX, tensorboard, pytorch-crf, pandas, overrides, en-core-web-sm
  Attempting uninstall: pandas
    Found existing installation: pandas 1.2.3
    Uninstalling pandas-1.2.3:
      Successfully uninstalled pandas-1.2.3
Successfully installed absl-py-1.3.0 blis-0.7.9 catalogue-1.0.2 cymem-2.0.7 en-core-web-sm-2.3.1 filelock-3.8.2 google-auth-oauthlib-0.4.6 grpcio-1.51.1 huggingface-hub-0.11.1 markdown-3.4.1 murmurhash-1.0.9 nvidia-cublas-cu11-11.10.3.66 nvidia-cuda-nvrtc-cu11-11.7.99 nvidia-cuda-runtime-cu11-11.7.99 nvidia-cudnn-cu11-8.5.0.96 oauthlib-3.2.2 overrides-3.1.0 pandas-1.2.4 plac-1.1.3 preshed-3.0.8 protobuf-3.20.3 pytorch-crf-0.7.2 pyyaml-6.0 regex-2022.10.31 requests-oauthlib-1.3.1 spacy-2.3.5 srsly-1.0.6 tensorboard-2.11.0 tensorboard-data-server-0.6.1 tensorboard-plugin-wit-1.8.1 tensorboardX-2.1 thinc-7.4.6 tokenizers-0.13.2 torch-1.13.1 transformers-4.25.1 wasabi-0.10.1 werkzeug-2.2.2
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Collecting en_core_web_sm==2.3.1
  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.3.1/en_core_web_sm-2.3.1.tar.gz (12.0 MB)
     |████████████████████████████████| 12.0 MB 4.8 MB/s 
[?25hRequirement already satisfied: spacy<2.4.0,>=2.3.0 in /usr/local/lib/python3.9/site-packages (from en_core_web_sm==2.3.1) (2.3.5)
Requirement already satisfied: blis<0.8.0,>=0.4.0 in /usr/local/lib/python3.9/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (0.7.9)
Requirement already satisfied: thinc<7.5.0,>=7.4.1 in /usr/local/lib/python3.9/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (7.4.6)
Requirement already satisfied: wasabi<1.1.0,>=0.4.0 in /usr/local/lib/python3.9/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (0.10.1)
Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.9/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (2.0.7)
Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.9/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (4.54.1)
Requirement already satisfied: plac<1.2.0,>=0.9.6 in /usr/local/lib/python3.9/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (1.1.3)
Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.9/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (3.0.8)
Requirement already satisfied: srsly<1.1.0,>=1.0.2 in /usr/local/lib/python3.9/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (1.0.6)
Requirement already satisfied: setuptools in /usr/local/lib/python3.9/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (51.0.0.post20201207)
Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.9/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (1.0.9)
Requirement already satisfied: catalogue<1.1.0,>=0.0.7 in /usr/local/lib/python3.9/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (1.0.2)
Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.9/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (2.25.0)
Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.9/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (1.22.3)
Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.9/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (1.22.3)
Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.9/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (2.0.7)
Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.9/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (1.0.9)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/site-packages (from requests<3.0.0,>=2.13.0->spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (2022.12.7)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.9/site-packages (from requests<3.0.0,>=2.13.0->spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (1.25.11)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.9/site-packages (from requests<3.0.0,>=2.13.0->spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (2.10)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.9/site-packages (from requests<3.0.0,>=2.13.0->spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (3.0.4)
Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.9/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (3.0.8)
Requirement already satisfied: blis<0.8.0,>=0.4.0 in /usr/local/lib/python3.9/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (0.7.9)
Requirement already satisfied: wasabi<1.1.0,>=0.4.0 in /usr/local/lib/python3.9/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (0.10.1)
Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.9/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (4.54.1)
Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.9/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (2.0.7)
Requirement already satisfied: srsly<1.1.0,>=1.0.2 in /usr/local/lib/python3.9/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (1.0.6)
Requirement already satisfied: plac<1.2.0,>=0.9.6 in /usr/local/lib/python3.9/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (1.1.3)
Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.9/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (1.0.9)
Requirement already satisfied: catalogue<1.1.0,>=0.0.7 in /usr/local/lib/python3.9/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (1.0.2)
Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.9/site-packages (from spacy<2.4.0,>=2.3.0->en_core_web_sm==2.3.1) (1.22.3)
✔ Download and installation successful
You can now load the model via spacy.load('en_core_web_sm')
!python --version
Python 3.9.1
# %cd CookDial
[Errno 2] No such file or directory: 'CookDial'
/content/CookDial
# !mkdir data/processed
# !touch data/processed/preprocess.log
! python src/scripts/preprocess.py -c "./vocab_config.json" --lowercase "True"
File white list: ['000', '001', '002', '003', '004', '005', '006', '007', '008', '009', '010', '011', '012', '013', '014', '015', '016', '017', '018', '019', '020', '021', '022', '023', '024', '025', '026', '027', '028', '029', '030', '031', '032', '033', '034', '035', '036', '037', '038', '039', '040', '041', '042', '043', '044', '045', '046', '047', '048', '049', '050', '051', '052', '053', '054', '055', '056', '057', '058', '059', '060', '061', '062', '063', '064', '065', '066', '067', '068', '069', '070', '071', '072', '073', '074', '075', '076', '077', '078', '079', '080', '081', '082', '083', '084', '085', '086', '087', '088', '089', '090', '091', '092', '093', '094', '095', '096', '097', '098', '099', '100', '101', '102', '103', '104', '105', '106', '107', '108', '109', '110', '111', '112', '113', '114', '115', '116', '117', '118', '119', '120', '121', '122', '123', '124', '125', '126', '127', '128', '129', '130', '131', '132', '133', '134', '135', '136', '137', '138', '139', '140', '141', '142', '143', '144', '145', '146', '147', '148', '149', '150', '151', '152', '153', '154', '155', '156', '157', '158', '159', '160', '161', '162', '163', '164', '165', '166', '167', '168', '169', '170', '171', '172', '173', '174', '175', '176', '177', '178', '179', '180', '181', '182', '183', '184', '185', '186', '187', '188', '189', '190', '191', '192', '193', '194', '195', '196', '197', '198', '199', '200', '201', '202', '203', '204', '205', '206', '207', '208', '209', '210', '211', '212', '213', '214', '215', '216', '217', '218', '219', '220', '221', '222', '223', '224', '225', '226', '227', '228', '229', '230', '231', '232', '233', '234', '235', '236', '237', '238', '239', '240', '241', '242', '243', '244', '245', '246', '247', '248', '249', '250', '251', '252', '253', '254', '255', '256', '257', '258', '259']
File black list: {}
**********Preprocess starts***********

Collate dialogue and recipe files
Start to shift agent action frames to user utterances...
	Finish shifting agent action frames
Start to shift agent acts...
	Finish shifting agent acts to user utterances.
Start to parse intents of user utterances...
	Finish parsing intents
Start to add history (window=10) to each utterance...
	Finish adding history.
Start to tokenize utterance (and history)...
	Finish tokenization.
Build vocabularies from dialogues...
	Finish building vocabs from dialogues: ['words', 'agent_acts', 'intent', 'section']
Recipe 009 has more than 512 tokens.
Recipe 160 has more than 512 tokens.
Build vocabularies from recipes...
	Finish building vocabs from recipes.
	Vocab size: 2987
  There are multiple matches for span "pan"
  There are multiple matches for span "bag"
  There are multiple matches for span "the pan"
  There are multiple matches for span "the pan"
  There are multiple matches for span "the foil"
  There are multiple matches for span "the foil"
  There are multiple matches for span "foil"
  There are multiple matches for span "foil"
Start to integrate recipes into dialogues...
	Integrating recipes is done.
Start converting func-args to full_set pointer sequence...
/content/CookDial/src/scripts/preprocess.py:1162: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  gold_full_set_pointer_seq = np.zeros((SLOT_MATRIX_ROWS), dtype=np.int)
	Finish full_set pointer sequence conversion.
Start to shift agent action frame args to user utterances...
	Finish shifting agent action frame args
All data is merged into one file /content/CookDial/data/processed/dialog/cookdial_dialog_merged.json
**********Preprocess is finished. Elapsed time: 10.81 min**********
%cd content/CookDial/src
/content/CookDial/src
!ls
config_agent_task.json	logger		 requirements.txt  utils
config_gene_task.json	metrics		 scripts	   vocab_config.json
config_user_task.json	models		 trainers
dataloaders		modules		 train.py
__init__.py		parse_config.py  train.sh
!pip install sentencepiece
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Collecting sentencepiece
  Downloading sentencepiece-0.1.97-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
     |████████████████████████████████| 1.3 MB 4.4 MB/s 
[?25hInstalling collected packages: sentencepiece
Successfully installed sentencepiece-0.1.97
%cd ./src
# Make sure train.sh is executable
!chmod 751 train.sh
!./train.sh --config ./config_user_task.json --mode "train"
[Errno 2] No such file or directory: './src'
/content
chmod: cannot access 'train.sh': No such file or directory
/bin/bash: ./train.sh: No such file or directory