638 lines
17 KiB
Plaintext
638 lines
17 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "coastal-lincoln",
|
|
"metadata": {},
|
|
"source": [
|
|
"\n",
|
|
"<div class=\"alert alert-block alert-info\">\n",
|
|
"<h1> Komputerowe wspomaganie tłumaczenia </h1>\n",
|
|
"<h2> 3. <i>Terminologia</i> [laboratoria]</h2> \n",
|
|
"<h3>Rafał Jaworski (2021)</h3>\n",
|
|
"</div>\n",
|
|
"\n",
|
|
""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "aggregate-listing",
|
|
"metadata": {},
|
|
"source": [
|
|
"Na dzisiejszych zajęciach zajmiemy się bliżej słownikami używanymi do wspomagania tłumaczenia. Oczywiście na rynku dostępnych jest bardzo wiele słowników w formacie elektronicznym. Wiele z nich jest gotowych do użycia w SDL Trados, memoQ i innych narzędziach CAT. Zawierają one setki tysięcy lub miliony haseł i oferują natychmiastową pomoc tłumaczowi."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "israeli-excuse",
|
|
"metadata": {},
|
|
"source": [
|
|
"Problem jednak w tym, iż często nie zawierają odpowiedniej terminologii specjalistycznej - używanej przez klienta zamawiającego tłumaczenie. Terminy specjalistyczne są bardzo częste w tekstach tłumaczonych ze względu na następujące zjawiska:\n",
|
|
"- Teksty o tematyce ogólnej są tłumaczone dość rzadko (nikt nie tłumaczy pocztówek z pozdrowieniami z wakacji...)\n",
|
|
"- Te same słowa mogą mieć zarówno znaczenie ogólne, jak i bardzo specjalistyczne (np. \"dziedziczenie\" w kontekście prawnym lub informatycznym)\n",
|
|
"- Klient używa nazw lub słów wymyślonych przez siebie, np. na potrzeby marketingowe."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "reflected-enforcement",
|
|
"metadata": {},
|
|
"source": [
|
|
"Nietrywialnymi zadaniami stają się: odnalezienie terminu specjalistycznego w tekście źródłowym oraz podanie prawidłowego tłumaczenia tego terminu na język docelowy"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "statutory-florist",
|
|
"metadata": {},
|
|
"source": [
|
|
"Brzmi prosto? Spróbujmy wykonać ręcznie tę drugą operację."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "danish-anchor",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Ćwiczenie 1: Podaj tłumaczenie terminu \"prowadnice szaf metalowych\" na język angielski. Opisz, z jakich narzędzi skorzystałaś/eś."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "diverse-sunglasses",
|
|
"metadata": {},
|
|
"source": [
|
|
"Odpowiedź:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "limited-waterproof",
|
|
"metadata": {},
|
|
"source": [
|
|
"W dalszych ćwiczeniach skupimy się jednak na odszukaniu terminu specjalistycznego w tekście. W tym celu będą potrzebne dwie operacje:\n",
|
|
"1. Przygotowanie słownika specjalistycznego.\n",
|
|
"2. Detekcja terminologii przy użyciu przygotowanego słownika specjalistycznego."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "literary-blues",
|
|
"metadata": {},
|
|
"source": [
|
|
"Zajmijmy się najpierw krokiem nr 2 (gdyż jest prostszy). Rozważmy następujący tekst:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 55,
|
|
"id": "loving-prince",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"text = \" For all Java programmers:\"\n",
|
|
"text += \" This section explains how to compile and run a Swing application from the command line.\"\n",
|
|
"text += \" For information on compiling and running a Swing application using NetBeans IDE,\"\n",
|
|
"text += \" see Running Tutorial Examples in NetBeans IDE. The compilation instructions work for all Swing programs\"\n",
|
|
"text += \" — applets, as well as applications. Here are the steps you need to follow:\"\n",
|
|
"text += \" Install the latest release of the Java SE platform, if you haven't already done so.\"\n",
|
|
"text += \" Create a program that uses Swing components. Compile the program. Run the program.\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "05436dad",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "extreme-cycling",
|
|
"metadata": {},
|
|
"source": [
|
|
"Załóżmy, że posiadamy następujący słownik:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 56,
|
|
"id": "bound-auction",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"dictionary = ['program', 'application', 'applet', 'compile']"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "other-trinidad",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Ćwiczenie 2: Napisz program, który wypisze pozycje wszystkich wystąpień poszczególnych terminów specjalistycznych. Dla każdego terminu należy wypisać listę par (pozycja_startowa, pozycja końcowa)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 17,
|
|
"id": "cognitive-cedar",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"{'program': [(468, 475), (516, 523), (533, 540)],\n",
|
|
" 'application': [(80, 91), (164, 175)],\n",
|
|
" 'compile': [(56, 63), (504, 511)]}"
|
|
]
|
|
},
|
|
"execution_count": 17,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import re\n",
|
|
"\n",
|
|
"def terminology_lookup(dictionary, text):\n",
|
|
" termValues = dict()\n",
|
|
" for element in dictionary:\n",
|
|
" values = []\n",
|
|
" pattern = re.compile(r'\\b{}\\b'.format(re.escape(element)))\n",
|
|
" for match in pattern.finditer(text.lower()):\n",
|
|
" values.append((match.start(), match.end()))\n",
|
|
" \n",
|
|
" if len(values) != 0:\n",
|
|
" termValues[element] = values\n",
|
|
" \n",
|
|
" return termValues\n",
|
|
"\n",
|
|
"terminology_lookup(dictionary, text)\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "interior-things",
|
|
"metadata": {},
|
|
"source": [
|
|
"Zwykłe wyszukiwanie w tekście ma pewne wady. Na przykład, gdy szukaliśmy słowa \"program\", złapaliśmy przypadkiem słowo \"programmer\". Złapaliśmy także słowo \"programs\", co jest poprawne, ale niepoprawnie podaliśmy jego pozycję w tekście."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "aggressive-plane",
|
|
"metadata": {},
|
|
"source": [
|
|
"Żeby poradzić sobie z tymi problemami, musimy wykorzystać techniki przetwarzania języka naturalnego. Wypróbujmy pakiet spaCy:\n",
|
|
"\n",
|
|
"`pip3 install spacy`\n",
|
|
"\n",
|
|
"oraz\n",
|
|
"\n",
|
|
"`python3 -m spacy download en_core_web_sm`"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 18,
|
|
"id": "tribal-attention",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" \n",
|
|
"for\n",
|
|
"all\n",
|
|
"Java\n",
|
|
"programmer\n",
|
|
":\n",
|
|
"this\n",
|
|
"section\n",
|
|
"explain\n",
|
|
"how\n",
|
|
"to\n",
|
|
"compile\n",
|
|
"and\n",
|
|
"run\n",
|
|
"a\n",
|
|
"swing\n",
|
|
"application\n",
|
|
"from\n",
|
|
"the\n",
|
|
"command\n",
|
|
"line\n",
|
|
".\n",
|
|
"for\n",
|
|
"information\n",
|
|
"on\n",
|
|
"compile\n",
|
|
"and\n",
|
|
"run\n",
|
|
"a\n",
|
|
"swing\n",
|
|
"application\n",
|
|
"use\n",
|
|
"NetBeans\n",
|
|
"IDE\n",
|
|
",\n",
|
|
"see\n",
|
|
"run\n",
|
|
"Tutorial\n",
|
|
"Examples\n",
|
|
"in\n",
|
|
"NetBeans\n",
|
|
"IDE\n",
|
|
".\n",
|
|
"the\n",
|
|
"compilation\n",
|
|
"instruction\n",
|
|
"work\n",
|
|
"for\n",
|
|
"all\n",
|
|
"Swing\n",
|
|
"program\n",
|
|
"—\n",
|
|
"applet\n",
|
|
",\n",
|
|
"as\n",
|
|
"well\n",
|
|
"as\n",
|
|
"application\n",
|
|
".\n",
|
|
"here\n",
|
|
"be\n",
|
|
"the\n",
|
|
"step\n",
|
|
"you\n",
|
|
"need\n",
|
|
"to\n",
|
|
"follow\n",
|
|
":\n",
|
|
"install\n",
|
|
"the\n",
|
|
"late\n",
|
|
"release\n",
|
|
"of\n",
|
|
"the\n",
|
|
"Java\n",
|
|
"SE\n",
|
|
"platform\n",
|
|
",\n",
|
|
"if\n",
|
|
"you\n",
|
|
"have\n",
|
|
"not\n",
|
|
"already\n",
|
|
"do\n",
|
|
"so\n",
|
|
".\n",
|
|
"create\n",
|
|
"a\n",
|
|
"program\n",
|
|
"that\n",
|
|
"use\n",
|
|
"swing\n",
|
|
"component\n",
|
|
".\n",
|
|
"compile\n",
|
|
"the\n",
|
|
"program\n",
|
|
".\n",
|
|
"run\n",
|
|
"the\n",
|
|
"program\n",
|
|
".\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"import spacy\n",
|
|
"nlp = spacy.load(\"en_core_web_sm\")\n",
|
|
"\n",
|
|
"doc = nlp(text)\n",
|
|
"\n",
|
|
"for token in doc:\n",
|
|
" print(token.lemma_)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "regional-craft",
|
|
"metadata": {},
|
|
"source": [
|
|
"Sukces! Nastąpił podział tekstu na słowa (tokenizacja) oraz sprowadzenie do formy podstawowej każdego słowa (lematyzacja)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "toxic-subsection",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Ćwiczenie 3: Zmodyfikuj program z ćwiczenia 2 tak, aby zwracał również odmienione słowa. Na przykład, dla słowa \"program\" powinien znaleźć również \"programs\", ustawiając pozycje w tekście odpowiednio dla słowa \"programs\". Wykorzystaj właściwość idx tokenu."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 43,
|
|
"id": "surgical-demonstration",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"{'program': [(291, 299), (468, 475), (516, 523), (533, 540)],\n",
|
|
" 'application': [(80, 91), (164, 175), (322, 334)],\n",
|
|
" 'applet': [(302, 309)],\n",
|
|
" 'compile': [(56, 63), (134, 143), (504, 511)]}"
|
|
]
|
|
},
|
|
"execution_count": 43,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"def terminology_lookup(dictionary, text):\n",
|
|
" termValues = dict()\n",
|
|
" lowerText = text.lower()\n",
|
|
" nlp = spacy.load(\"en_core_web_sm\")\n",
|
|
"\n",
|
|
" splitText = nlp(lowerText)\n",
|
|
" for findingWord in dictionary:\n",
|
|
" values = []\n",
|
|
" startFromIndex = 0\n",
|
|
"\n",
|
|
" for word in splitText:\n",
|
|
" if word.lemma_ == findingWord:\n",
|
|
" textBegining = lowerText.index(word.text,startFromIndex)\n",
|
|
" textEnding = textBegining + len(word)\n",
|
|
" startFromIndex = textEnding\n",
|
|
" values.append((textBegining,textEnding))\n",
|
|
" \n",
|
|
" if len(values) != 0:\n",
|
|
" termValues[findingWord] = values\n",
|
|
" \n",
|
|
" return termValues\n",
|
|
"\n",
|
|
"terminology_lookup(dictionary, text)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "straight-letter",
|
|
"metadata": {},
|
|
"source": [
|
|
"Teraz czas zająć się problemem przygotowania słownika specjalistycznego. W tym celu napiszemy nasz własny ekstraktor terminologii. Wejściem do ekstraktora będzie tekst zawierający specjalistyczną terminologię. Wyjściem - lista terminów."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "nearby-frontier",
|
|
"metadata": {},
|
|
"source": [
|
|
"Przyjmijmy następujące podejście - terminami specjalistycznymi będą najcześćiej występujące rzeczowniki w tekście. Wykonajmy krok pierwszy:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "harmful-lightning",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Ćwiczenie 4: Wypisz wszystkie rzeczowniki z tekstu. Wykorzystaj możliwości spaCy."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 54,
|
|
"id": "superb-butterfly",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"set()"
|
|
]
|
|
},
|
|
"execution_count": 54,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import spacy\n",
|
|
"\n",
|
|
"def get_nouns(text):\n",
|
|
" nlp = spacy.load(\"en_core_web_sm\")\n",
|
|
" doc = nlp(text)\n",
|
|
" nouns = [token.text for token in doc if token.pos_ == \"NOUN\"]\n",
|
|
" return set(nouns)\n",
|
|
"\n",
|
|
"get_nouns(text)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "musical-creator",
|
|
"metadata": {},
|
|
"source": [
|
|
"Teraz czas na podliczenie wystąpień poszczególnych rzeczowników. Uwaga - różne formy tego samego słowa zliczamy razem jako wystąpienia tego słowa (np. \"program\" i \"programs\"). Najwygodniejszą metodą podliczania jest zastosowanie tzw. tally (po polsku \"zestawienie\"). Jest to słownik, którego kluczem jest słowo w formie podstawowej, a wartością liczba wystąpień tego słowa, wliczając słowa odmienione. Przykład gotowego tally:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"id": "acting-tolerance",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"tally = {\"program\" : 4, \"component\" : 1}"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "vanilla-estimate",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Ćwiczenie 5: Napisz program do ekstrakcji terminologii z tekstu według powyższych wytycznych."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 71,
|
|
"id": "eight-redhead",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"{'line': 1,\n",
|
|
" 'release': 1,\n",
|
|
" 'compilation': 1,\n",
|
|
" 'component': 1,\n",
|
|
" 'section': 1,\n",
|
|
" 'information': 1,\n",
|
|
" 'program': 4,\n",
|
|
" 'command': 1,\n",
|
|
" 'platform': 1,\n",
|
|
" 'applet': 1,\n",
|
|
" 'application': 3,\n",
|
|
" 'swing': 4,\n",
|
|
" 'instruction': 1,\n",
|
|
" 'step': 1,\n",
|
|
" 'programmer': 1}"
|
|
]
|
|
},
|
|
"execution_count": 71,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import spacy\n",
|
|
"\n",
|
|
"def get_nouns(text):\n",
|
|
" nlp = spacy.load(\"en_core_web_sm\")\n",
|
|
" doc = nlp(text)\n",
|
|
" nouns = [token.lemma_ for token in doc if token.pos_ == \"NOUN\"]\n",
|
|
" return set(nouns)\n",
|
|
"\n",
|
|
"def getElementsNumbers(dictionary, text):\n",
|
|
" termValues = dict()\n",
|
|
" lowerText = text.lower()\n",
|
|
" nlp = spacy.load(\"en_core_web_sm\")\n",
|
|
"\n",
|
|
" splitText = nlp(lowerText)\n",
|
|
" for findingWord in dictionary:\n",
|
|
" elementNumber = 0\n",
|
|
"\n",
|
|
" for word in splitText:\n",
|
|
" if word.lemma_ == findingWord:\n",
|
|
" elementNumber = elementNumber +1\n",
|
|
" \n",
|
|
" if elementNumber != 0:\n",
|
|
" termValues[findingWord] = elementNumber\n",
|
|
" \n",
|
|
" return termValues\n",
|
|
"\n",
|
|
"def extract_terms(text):\n",
|
|
" return getElementsNumbers(get_nouns(text), text)\n",
|
|
"\n",
|
|
"extract_terms(text)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "loaded-smell",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Ćwiczenie 6: Rozszerz powyższy program o ekstrację czasowników i przymiotników."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 86,
|
|
"id": "monetary-mambo",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"def get_dictonery_by_type(text, type):\n",
|
|
" nlp = spacy.load(\"en_core_web_sm\")\n",
|
|
" doc = nlp(text)\n",
|
|
" nouns = [token.lemma_ for token in doc if token.pos_ == type]\n",
|
|
" return set(nouns)\n",
|
|
"\n",
|
|
"\n",
|
|
"def extract_terms(text, type):\n",
|
|
" return getElementsNumbers(get_dictonery_by_type(text, type), text)\n",
|
|
"\n",
|
|
"\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 87,
|
|
"id": "8f7eeb73",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"{'compile': 3,\n",
|
|
" 'work': 1,\n",
|
|
" 'install': 1,\n",
|
|
" 'create': 1,\n",
|
|
" 'explain': 1,\n",
|
|
" 'run': 4,\n",
|
|
" 'see': 1,\n",
|
|
" 'need': 1,\n",
|
|
" 'do': 1,\n",
|
|
" 'follow': 1,\n",
|
|
" 'use': 2}"
|
|
]
|
|
},
|
|
"execution_count": 87,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"extract_terms(text, 'VERB')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 93,
|
|
"id": "71c14cab",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"{'late': 1}"
|
|
]
|
|
},
|
|
"execution_count": 93,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"extract_terms(text, 'ADJ')"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"author": "Rafał Jaworski",
|
|
"email": "rjawor@amu.edu.pl",
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"lang": "pl",
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.11.7"
|
|
},
|
|
"subtitle": "3. Terminologia",
|
|
"title": "Komputerowe wspomaganie tłumaczenia",
|
|
"year": "2021"
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|