KWT-2024/lab/lab_03.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "coastal-lincoln",
   "metadata": {},
   "source": [
    "![Logo 1](https://git.wmi.amu.edu.pl/AITech/Szablon/raw/branch/master/Logotyp_AITech1.jpg)\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "<h1> Komputerowe wspomaganie tłumaczenia </h1>\n",
    "<h2> 3. <i>Terminologia</i> [laboratoria]</h2> \n",
    "<h3>Rafał Jaworski (2021)</h3>\n",
    "</div>\n",
    "\n",
    "![Logo 2](https://git.wmi.amu.edu.pl/AITech/Szablon/raw/branch/master/Logotyp_AITech2.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aggregate-listing",
   "metadata": {},
   "source": [
    "Na dzisiejszych zajęciach zajmiemy się bliżej słownikami używanymi do wspomagania tłumaczenia. Oczywiście na rynku dostępnych jest bardzo wiele słowników w formacie elektronicznym. Wiele z nich jest gotowych do użycia w SDL Trados, memoQ i innych narzędziach CAT. Zawierają one setki tysięcy lub miliony haseł i oferują natychmiastową pomoc tłumaczowi."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "israeli-excuse",
   "metadata": {},
   "source": [
    "Problem jednak w tym, iż często nie zawierają odpowiedniej terminologii specjalistycznej - używanej przez klienta zamawiającego tłumaczenie. Terminy specjalistyczne są bardzo częste w tekstach tłumaczonych ze względu na następujące zjawiska:\n",
    "- Teksty o tematyce ogólnej są tłumaczone dość rzadko (nikt nie tłumaczy pocztówek z pozdrowieniami z wakacji...)\n",
    "- Te same słowa mogą mieć zarówno znaczenie ogólne, jak i bardzo specjalistyczne (np. \"dziedziczenie\" w kontekście prawnym lub informatycznym)\n",
    "- Klient używa nazw lub słów wymyślonych przez siebie, np. na potrzeby marketingowe."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "reflected-enforcement",
   "metadata": {},
   "source": [
    "Nietrywialnymi zadaniami stają się: odnalezienie terminu specjalistycznego w tekście źródłowym oraz podanie prawidłowego tłumaczenia tego terminu na język docelowy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "statutory-florist",
   "metadata": {},
   "source": [
    "Brzmi prosto? Spróbujmy wykonać ręcznie tę drugą operację."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "danish-anchor",
   "metadata": {},
   "source": [
    "### Ćwiczenie 1: Podaj tłumaczenie terminu \"prowadnice szaf metalowych\" na język angielski. Opisz, z jakich narzędzi skorzystałaś/eś."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "diverse-sunglasses",
   "metadata": {},
   "source": [
    "Odpowiedź: Narzędzie DeepL: https://www.deepl.com/translator\n",
    "\n",
    "przetłumaczyło tekst \"prowadnice szaf metalowych\" na \"metal cabinet slides\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "limited-waterproof",
   "metadata": {},
   "source": [
    "W dalszych ćwiczeniach skupimy się jednak na odszukaniu terminu specjalistycznego w tekście. W tym celu będą potrzebne dwie operacje:\n",
    "1. Przygotowanie słownika specjalistycznego.\n",
    "2. Detekcja terminologii przy użyciu przygotowanego słownika specjalistycznego."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "literary-blues",
   "metadata": {},
   "source": [
    "Zajmijmy się najpierw krokiem nr 2 (gdyż jest prostszy). Rozważmy następujący tekst:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "8f6b6fa9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "loving-prince",
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \" For all Java programmers:\"\n",
    "text += \" This section explains how to compile and run a Swing application from the command line.\"\n",
    "text += \" For information on compiling and running a Swing application using NetBeans IDE,\"\n",
    "text += \" see Running Tutorial Examples in NetBeans IDE. The compilation instructions work for all Swing programs\"\n",
    "text += \" — applets, as well as applications. Here are the steps you need to follow:\"\n",
    "text += \" Install the latest release of the Java SE platform, if you haven't already done so.\"\n",
    "text += \" Create a program that uses Swing components. Compile the program. Run the program.\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "extreme-cycling",
   "metadata": {},
   "source": [
    "Załóżmy, że posiadamy następujący słownik:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "bound-auction",
   "metadata": {},
   "outputs": [],
   "source": [
    "dictionary = ['program', 'application', 'applet' 'compile']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "821ee3ee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\" For all Java programmers: This section explains how to compile and run a Swing application from the command line. For information on compiling and running a Swing application using NetBeans IDE, see Running Tutorial Examples in NetBeans IDE. The compilation instructions work for all Swing programs — applets, as well as applications. Here are the steps you need to follow: Install the latest release of the Java SE platform, if you haven't already done so. Create a program that uses Swing components. Compile the program. Run the program.\""
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "other-trinidad",
   "metadata": {},
   "source": [
    "### Ćwiczenie 2: Napisz program, który wypisze pozycje wszystkich wystąpień poszczególnych terminów specjalistycznych. Dla każdego terminu należy wypisać listę par (pozycja_startowa, pozycja końcowa)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "cognitive-cedar",
   "metadata": {},
   "outputs": [],
   "source": [
    "count_dictionary = {}\n",
    "\n",
    "def terminology_lookup(text, tags):\n",
    "    text = text.lower()\n",
    "    return [(tag, [[m.start(), m.end()] \n",
    "                   for m in re.finditer(tag, text)])\n",
    "            for tag in tags if tag in text]\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "9fe3b66f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('program', [[14, 21], [291, 298], [468, 475], [516, 523], [533, 540]]),\n",
       " ('application', [[80, 91], [164, 175], [322, 333]])]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "terminology_lookup(text, dictionary)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "interior-things",
   "metadata": {},
   "source": [
    "Zwykłe wyszukiwanie w tekście ma pewne wady. Na przykład, gdy szukaliśmy słowa \"program\", złapaliśmy przypadkiem słowo \"programmer\". Złapaliśmy także słowo \"programs\", co jest poprawne, ale niepoprawnie podaliśmy jego pozycję w tekście."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aggressive-plane",
   "metadata": {},
   "source": [
    "Żeby poradzić sobie z tymi problemami, musimy wykorzystać techniki przetwarzania języka naturalnego. Wypróbujmy pakiet spaCy:\n",
    "\n",
    "`pip3 install spacy`\n",
    "\n",
    "oraz\n",
    "\n",
    "`python3 -m spacy download en_core_web_sm`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "7b7b7569",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: spacy in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (3.7.4)\n",
      "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy) (3.0.12)\n",
      "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy) (1.0.5)\n",
      "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy) (1.0.10)\n",
      "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy) (2.0.8)\n",
      "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy) (3.0.9)\n",
      "Requirement already satisfied: thinc<8.3.0,>=8.2.2 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy) (8.2.3)\n",
      "Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy) (1.1.2)\n",
      "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy) (2.4.8)\n",
      "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy) (2.0.10)\n",
      "Requirement already satisfied: weasel<0.4.0,>=0.1.0 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy) (0.3.4)\n",
      "Requirement already satisfied: typer<0.10.0,>=0.3.0 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy) (0.9.4)\n",
      "Requirement already satisfied: smart-open<7.0.0,>=5.2.1 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy) (6.4.0)\n",
      "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy) (4.66.2)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy) (2.31.0)\n",
      "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy) (2.7.0)\n",
      "Requirement already satisfied: jinja2 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy) (3.1.3)\n",
      "Requirement already satisfied: setuptools in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy) (65.5.0)\n",
      "Requirement already satisfied: packaging>=20.0 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy) (24.0)\n",
      "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy) (3.3.0)\n",
      "Requirement already satisfied: numpy>=1.19.0 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy) (1.26.4)\n",
      "Requirement already satisfied: annotated-types>=0.4.0 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy) (0.6.0)\n",
      "Requirement already satisfied: pydantic-core==2.18.1 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy) (2.18.1)\n",
      "Requirement already satisfied: typing-extensions>=4.6.1 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy) (4.11.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from requests<3.0.0,>=2.13.0->spacy) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from requests<3.0.0,>=2.13.0->spacy) (3.7)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from requests<3.0.0,>=2.13.0->spacy) (2.2.1)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from requests<3.0.0,>=2.13.0->spacy) (2024.2.2)\n",
      "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from thinc<8.3.0,>=8.2.2->spacy) (0.7.11)\n",
      "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from thinc<8.3.0,>=8.2.2->spacy) (0.1.4)\n",
      "Requirement already satisfied: click<9.0.0,>=7.1.1 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from typer<0.10.0,>=0.3.0->spacy) (8.1.7)\n",
      "Requirement already satisfied: cloudpathlib<0.17.0,>=0.7.0 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from weasel<0.4.0,>=0.1.0->spacy) (0.16.0)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from jinja2->spacy) (2.1.5)\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip available: \u001b[0m\u001b[31;49m22.3\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.0\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip3 install spacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f4d06ed3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting en-core-web-sm==3.7.1\n",
      "  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.7.1/en_core_web_sm-3.7.1-py3-none-any.whl (12.8 MB)\n",
      "\u001b[2K     \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.8/12.8 MB\u001b[0m \u001b[31m27.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mm eta \u001b[36m0:00:01\u001b[0m0:01\u001b[0m:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: spacy<3.8.0,>=3.7.2 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from en-core-web-sm==3.7.1) (3.7.4)\n",
      "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (3.0.12)\n",
      "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (1.0.5)\n",
      "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (1.0.10)\n",
      "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.0.8)\n",
      "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (3.0.9)\n",
      "Requirement already satisfied: thinc<8.3.0,>=8.2.2 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (8.2.3)\n",
      "Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (1.1.2)\n",
      "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.4.8)\n",
      "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.0.10)\n",
      "Requirement already satisfied: weasel<0.4.0,>=0.1.0 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (0.3.4)\n",
      "Requirement already satisfied: typer<0.10.0,>=0.3.0 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (0.9.4)\n",
      "Requirement already satisfied: smart-open<7.0.0,>=5.2.1 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (6.4.0)\n",
      "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (4.66.2)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.31.0)\n",
      "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.7.0)\n",
      "Requirement already satisfied: jinja2 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (3.1.3)\n",
      "Requirement already satisfied: setuptools in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (65.5.0)\n",
      "Requirement already satisfied: packaging>=20.0 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (24.0)\n",
      "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (3.3.0)\n",
      "Requirement already satisfied: numpy>=1.19.0 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (1.26.4)\n",
      "Requirement already satisfied: annotated-types>=0.4.0 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (0.6.0)\n",
      "Requirement already satisfied: pydantic-core==2.18.1 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.18.1)\n",
      "Requirement already satisfied: typing-extensions>=4.6.1 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (4.11.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (3.7)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.2.1)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2024.2.2)\n",
      "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from thinc<8.3.0,>=8.2.2->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (0.7.11)\n",
      "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from thinc<8.3.0,>=8.2.2->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (0.1.4)\n",
      "Requirement already satisfied: click<9.0.0,>=7.1.1 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from typer<0.10.0,>=0.3.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (8.1.7)\n",
      "Requirement already satisfied: cloudpathlib<0.17.0,>=0.7.0 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from weasel<0.4.0,>=0.1.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (0.16.0)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /Users/potoato/.pyenv/versions/3.11.0/lib/python3.11/site-packages (from jinja2->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.1.5)\n",
      "Installing collected packages: en-core-web-sm\n",
      "Successfully installed en-core-web-sm-3.7.1\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip available: \u001b[0m\u001b[31;49m22.3\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.0\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
      "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
      "You can now load the package via spacy.load('en_core_web_sm')\n"
     ]
    }
   ],
   "source": [
    "!python3 -m spacy download en_core_web_sm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "tribal-attention",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " \n",
      "for\n",
      "all\n",
      "Java\n",
      "programmer\n",
      ":\n",
      "this\n",
      "section\n",
      "explain\n",
      "how\n",
      "to\n",
      "compile\n",
      "and\n",
      "run\n",
      "a\n",
      "swing\n",
      "application\n",
      "from\n",
      "the\n",
      "command\n",
      "line\n",
      ".\n",
      "for\n",
      "information\n",
      "on\n",
      "compile\n",
      "and\n",
      "run\n",
      "a\n",
      "swing\n",
      "application\n",
      "use\n",
      "NetBeans\n",
      "IDE\n",
      ",\n",
      "see\n",
      "run\n",
      "Tutorial\n",
      "Examples\n",
      "in\n",
      "NetBeans\n",
      "IDE\n",
      ".\n",
      "the\n",
      "compilation\n",
      "instruction\n",
      "work\n",
      "for\n",
      "all\n",
      "Swing\n",
      "program\n",
      "—\n",
      "applet\n",
      ",\n",
      "as\n",
      "well\n",
      "as\n",
      "application\n",
      ".\n",
      "here\n",
      "be\n",
      "the\n",
      "step\n",
      "you\n",
      "need\n",
      "to\n",
      "follow\n",
      ":\n",
      "install\n",
      "the\n",
      "late\n",
      "release\n",
      "of\n",
      "the\n",
      "Java\n",
      "SE\n",
      "platform\n",
      ",\n",
      "if\n",
      "you\n",
      "have\n",
      "not\n",
      "already\n",
      "do\n",
      "so\n",
      ".\n",
      "create\n",
      "a\n",
      "program\n",
      "that\n",
      "use\n",
      "swing\n",
      "component\n",
      ".\n",
      "compile\n",
      "the\n",
      "program\n",
      ".\n",
      "run\n",
      "the\n",
      "program\n",
      ".\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "doc = nlp(text)\n",
    "\n",
    "for token in doc:\n",
    "    print(token.lemma_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "regional-craft",
   "metadata": {},
   "source": [
    "Sukces! Nastąpił podział tekstu na słowa (tokenizacja) oraz sprowadzenie do formy podstawowej każdego słowa (lematyzacja)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "toxic-subsection",
   "metadata": {},
   "source": [
    "### Ćwiczenie 3: Zmodyfikuj program z ćwiczenia 2 tak, aby zwracał również odmienione słowa. Na przykład, dla słowa \"program\" powinien znaleźć również \"programs\", ustawiając pozycje w tekście odpowiednio dla słowa \"programs\". Wykorzystaj właściwość idx tokenu."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "surgical-demonstration",
   "metadata": {},
   "outputs": [],
   "source": [
    "def terminology_lookup():\n",
    "    return None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "straight-letter",
   "metadata": {},
   "source": [
    "Teraz czas zająć się problemem przygotowania słownika specjalistycznego. W tym celu napiszemy nasz własny ekstraktor terminologii. Wejściem do ekstraktora będzie tekst zawierający specjalistyczną terminologię. Wyjściem - lista terminów."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "nearby-frontier",
   "metadata": {},
   "source": [
    "Przyjmijmy następujące podejście - terminami specjalistycznymi będą najcześćiej występujące rzeczowniki w tekście. Wykonajmy krok pierwszy:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "harmful-lightning",
   "metadata": {},
   "source": [
    "### Ćwiczenie 4: Wypisz wszystkie rzeczowniki z tekstu. Wykorzystaj możliwości spaCy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "superb-butterfly",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_nouns(text):\n",
    "    doc = nlp(text)\n",
    "    return [token.text for token in doc if token.pos_ == \"NOUN\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "5e2be152",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ADP'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc[1].pos_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "11430dc5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['programmers',\n",
       " 'section',\n",
       " 'Swing',\n",
       " 'application',\n",
       " 'command',\n",
       " 'line',\n",
       " 'information',\n",
       " 'Swing',\n",
       " 'application',\n",
       " 'compilation',\n",
       " 'instructions',\n",
       " 'programs',\n",
       " 'applets',\n",
       " 'applications',\n",
       " 'steps',\n",
       " 'release',\n",
       " 'platform',\n",
       " 'program',\n",
       " 'Swing',\n",
       " 'components',\n",
       " 'program',\n",
       " 'program']"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_nouns(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "musical-creator",
   "metadata": {},
   "source": [
    "Teraz czas na podliczenie wystąpień poszczególnych rzeczowników. Uwaga - różne formy tego samego słowa zliczamy razem jako wystąpienia tego słowa (np. \"program\" i \"programs\"). Najwygodniejszą metodą podliczania jest zastosowanie tzw. tally (po polsku \"zestawienie\"). Jest to słownik, którego kluczem jest słowo w formie podstawowej, a wartością liczba wystąpień tego słowa, wliczając słowa odmienione. Przykład gotowego tally:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "acting-tolerance",
   "metadata": {},
   "outputs": [],
   "source": [
    "tally = {\"program\" : 4, \"component\" : 1}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "vanilla-estimate",
   "metadata": {},
   "source": [
    "### Ćwiczenie 5: Napisz program do ekstrakcji terminologii z tekstu według powyższych wytycznych."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "eight-redhead",
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_terms(text):\n",
    "    doc = nlp(text)\n",
    "    noun_counts = {}\n",
    "    for token in doc:\n",
    "        if token.pos_ == \"NOUN\":\n",
    "            noun_counts[token.text] = noun_counts.get(token.text, 0) + 1\n",
    "    return noun_counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "c7d46f26",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'programmers': 1,\n",
       " 'section': 1,\n",
       " 'Swing': 3,\n",
       " 'application': 2,\n",
       " 'command': 1,\n",
       " 'line': 1,\n",
       " 'information': 1,\n",
       " 'compilation': 1,\n",
       " 'instructions': 1,\n",
       " 'programs': 1,\n",
       " 'applets': 1,\n",
       " 'applications': 1,\n",
       " 'steps': 1,\n",
       " 'release': 1,\n",
       " 'platform': 1,\n",
       " 'program': 3,\n",
       " 'components': 1}"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "extract_terms(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "loaded-smell",
   "metadata": {},
   "source": [
    "### Ćwiczenie 6: Rozszerz powyższy program o ekstrację czasowników i przymiotników."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "monetary-mambo",
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_terms(text):\n",
    "    doc = nlp(text.lower())\n",
    "    noun_counts = {}\n",
    "    for token in doc:\n",
    "        if token.pos_ in [\"NOUN\", \"VERB\", \"ADJ\"]:\n",
    "            noun_counts[token.text] = noun_counts.get(token.text, 0) + 1\n",
    "    return noun_counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "4259ee3f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'explains': 1,\n",
       " 'compile': 2,\n",
       " 'run': 2,\n",
       " 'compiling': 1,\n",
       " 'running': 2,\n",
       " 'using': 1,\n",
       " 'see': 1,\n",
       " 'work': 1,\n",
       " 'need': 1,\n",
       " 'follow': 1,\n",
       " 'install': 1,\n",
       " 'done': 1,\n",
       " 'create': 1,\n",
       " 'uses': 1}"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "extract_terms(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d239c20",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "author": "Rafał Jaworski",
  "email": "rjawor@amu.edu.pl",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "lang": "pl",
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  },
  "subtitle": "3. Terminologia",
  "title": "Komputerowe wspomaganie tłumaczenia",
  "year": "2021"
 },
 "nbformat": 4,
 "nbformat_minor": 5
}