diff --git a/wyk/01_Wyszukiwarki-wprowadzenie.ipynb b/wyk/01_Wyszukiwarki-wprowadzenie.ipynb
index 95aaebb..aef457f 100644
--- a/wyk/01_Wyszukiwarki-wprowadzenie.ipynb
+++ b/wyk/01_Wyszukiwarki-wprowadzenie.ipynb
@@ -15,16 +15,14 @@
]
},
{
- "cell_type": "code",
- "execution_count": null,
+ "cell_type": "markdown",
"metadata": {},
- "outputs": [],
"source": [
"# Wyszukiwarki - wprowadzenie\n",
"\n",
"## Systemy wyszukiwania informacji (information retrieval systems)\n",
"\n",
- ""
+ ""
]
},
{
@@ -36,7 +34,7 @@
"source": [
"## Wyszukiwarki\n",
"\n",
- ""
+ ""
]
},
{
@@ -108,7 +106,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "Dostępne są też \"ekstrakty\" czystego tekstu - zob. http://data.statmt.org/ngrams/raw/, np. 59 GB czystego tekstu po polsku z 2012 roku."
+ "Dostępne są też „ekstrakty” czystego tekstu — zob. http://data.statmt.org/ngrams/raw/, np. 59 GB czystego tekstu po polsku z 2012 roku."
]
},
{
@@ -303,7 +301,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "### Odpytywać \"pasożytniczo\" inną wyszukiwarkę"
+ "### Odpytywać „pasożytniczo” inną wyszukiwarkę"
]
},
{
@@ -312,7 +310,7 @@
"metadata": {},
"outputs": [],
"source": [
- "# see https://hackernoon.com/how-to-scrape-google-with-python-bo7d2tal\n",
+ "# zob. https://hackernoon.com/how-to-scrape-google-with-python-bo7d2tal\n",
"\n",
"import urllib\n",
"import requests\n",
diff --git a/wyk/02_Wyszukiwarki-roboty.ipynb b/wyk/02_Wyszukiwarki-roboty.ipynb
index 6bbbb6f..de377ca 100644
--- a/wyk/02_Wyszukiwarki-roboty.ipynb
+++ b/wyk/02_Wyszukiwarki-roboty.ipynb
@@ -328,7 +328,7 @@
"\n",
"* urllib\n",
"* request\n",
- "* Beautiful Soup (do parsowania HTML-a)"
+ "* Beautiful Soup (do parsowania dokumentów HTML)"
]
},
{
diff --git a/wyk/05_Geste_wektory.ipynb b/wyk/05_Geste_wektory.ipynb
index 085ea81..dde6f50 100644
--- a/wyk/05_Geste_wektory.ipynb
+++ b/wyk/05_Geste_wektory.ipynb
@@ -1,1645 +1,1643 @@
{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {
- "collapsed": false
- },
- "source": [
- "\n",
- "\n",
- "
Ekstrakcja informacji
\n",
- " 5. G\u0119ste reprezentacje wektorowe [wyk\u0142ad]
\n",
- " Filip Grali\u0144ski (2021)
\n",
- "\n",
- "\n",
- ""
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Zag\u0119szczamy wektory\n",
- "\n",
- "Podstawowy problem z wektorow\u0105 reprezentacj\u0105 typu tf-idf polega na tym, \u017ce wektory dokument\u00f3w (i macierz ca\u0142ej kolekcji dokument\u00f3w) s\u0105 _rzadkie_, tzn. zawieraj\u0105 du\u017co zer. W praktyce potrzebujemy bardziej \"g\u0119stej\" czy \"kompaktowej\" reprezentacji numerycznej dokument\u00f3w. \n",
- "\n",
- "## _Hashing trick_\n",
- "\n",
- "Powierzchownie problem mo\u017cemy rozwi\u0105za\u0107 przez u\u017cycie tzw. _sztuczki z haszowaniem_ (_hashing trick_). B\u0119dziemy potrzebowa\u0107 funkcji mieszaj\u0105cej (haszuj\u0105cej) $H$, kt\u00f3ra rzutuje napisy na liczby, kt\u00f3rych reprezentacja binarna sk\u0142ada si\u0119 z $b$ bit\u00f3w:\n",
- "\n",
- "$$H : \\Sigma^{*} \\rightarrow \\{0,\\dots,2^b-1\\}$$\n",
- "\n",
- "($\\Sigma^{*}$ to zbi\u00f3r wszystkich napis\u00f3w.)\n",
- "\n",
- "**Pytanie:** Czy funkcja $H$ mo\u017ce by\u0107 r\u00f3\u017cnowarto\u015bciowa?\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Jako funkcji $H$ mo\u017cemy np. u\u017cy\u0107 funkcji MurmurHash2 lub 3."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 28,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "Hash64 0x4a80abc136f926e7"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "data": {
- "text/plain": [
- "Hash64 0x6c3a641663470e2c"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "data": {
- "text/plain": [
- "Hash64 0x6c3a641663470e2c"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "data": {
- "text/plain": [
- "Hash64 0xa714568917576314"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "data": {
- "text/plain": [
- "Hash64 0x875d9e7e413747c8"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "data": {
- "text/plain": [
- "Hash64 0x13ce831936ebc69e"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "data": {
- "text/plain": [
- "Hash64 0xb04ce6229407c882"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "data": {
- "text/plain": [
- "Hash64 0x6ecd7bae29ae0450"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "import Data.Digest.Murmur64\n",
- "\n",
- "hash64 \"Komputer\"\n",
- "hash64 \"komputer\"\n",
- "hash64 \"komputer\"\n",
- "hash64 \"komputerze\"\n",
- "hash64 \"komputerek\"\n",
- "hash64 \"abrakadabra\"\n",
- "hash64 \"\"\n",
- "hash64 \" \"\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "**Pytanie:** podobne napisy maj\u0105 zupe\u0142nie r\u00f3\u017cne warto\u015bci funkcji haszuj\u0105cej, czy to dobrze, czy to \u017ale?"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Musimy tylko sparametryzowa\u0107 nasz\u0105 funkcj\u0119 rozmiarem \"odcisku\" (parametr $b$)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "3628"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "data": {
- "text/plain": [
- "25364"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "data": {
- "text/plain": [
- "2877"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "data": {
- "text/plain": [
- "50846"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "data": {
- "text/plain": [
- "12"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "{-# LANGUAGE OverloadedStrings #-}\n",
- "\n",
- "import Data.Text\n",
- "\n",
- "-- pomocnicza funkcja, kt\u00f3ra konwertuje warto\u015b\u0107 specjalnego\n",
- "-- typu Hash64 do zwyk\u0142ej liczby ca\u0142kowitej\n",
- "hashValueAsInteger :: Hash64 -> Integer\n",
- "hashValueAsInteger = toInteger . asWord64\n",
- "\n",
- "-- unpack to funkcja, kt\u00f3ra warto\u015b\u0107 typu String konwertuje do Text\n",
- "hash :: Integer -> Text -> Integer\n",
- "hash b t = hashValueAsInteger (hash64 $ unpack t) `mod` (2 ^ b)\n",
- "\n",
- "hash 16 \"komputer\"\n",
- "hash 16 \"komputerze\"\n",
- "hash 16 \"komputerem\"\n",
- "hash 16 \"abrakadabra\"\n",
- "hash 4 \"komputer\""
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "**Pytanie:** Jakie warto\u015bci $b$ b\u0119d\u0105 bezsensowne?"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Sztuczka z haszowaniem polega na tym, \u017ce zamiast numerowa\u0107 s\u0142owa korzystaj\u0105c ze s\u0142ownika, po prostu u\u017cywamy funkcji haszuj\u0105cej. W ten spos\u00f3b wektor b\u0119dzie _zawsze_ rozmiar $2^b$ - bez wzgl\u0119du na rozmiar s\u0142ownika."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Zacznijmy od przywo\u0142ania wszystkich potrzebnych definicji."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {},
- "outputs": [],
- "source": [
- "{-# LANGUAGE OverloadedStrings #-}\n",
- "{-# LANGUAGE QuasiQuotes #-}\n",
- "\n",
- "import Data.Text hiding(map, filter, zip)\n",
- "import Text.Regex.PCRE.Heavy\n",
- "\n",
- "isStopWord :: Text -> Bool\n",
- "isStopWord \"w\" = True\n",
- "isStopWord \"jest\" = True\n",
- "isStopWord \"\u017ce\" = True\n",
- "isStopWord w = w \u2248 [re|^\\p{P}+$|]\n",
- "\n",
- "\n",
- "removeStopWords :: [Text] -> [Text]\n",
- "removeStopWords = filter (not . isStopWord)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {},
- "outputs": [],
- "source": [
- "{-# LANGUAGE OverloadedStrings #-}\n",
- "{-# LANGUAGE QuasiQuotes #-}\n",
- "{-# LANGUAGE FlexibleContexts #-}\n",
- "\n",
- "import Data.Text hiding(map, filter, zip)\n",
- "import Prelude hiding(words, take)\n",
- "import Text.Regex.PCRE.Heavy\n",
- "import Data.Map as Map hiding(take, map, filter)\n",
- "import Data.Set as Set hiding(map)\n",
- "\n",
- "tokenize :: Text -> [Text]\n",
- "tokenize = map fst . scan [re|C\\+\\+|[\\p{L}0-9]+|\\p{P}|]\n",
- "\n",
- "\n",
- "mockInflectionDictionary :: Map Text Text\n",
- "mockInflectionDictionary = Map.fromList [\n",
- " (\"kota\", \"kot\"),\n",
- " (\"butach\", \"but\"),\n",
- " (\"masz\", \"mie\u0107\"),\n",
- " (\"ma\", \"mie\u0107\"),\n",
- " (\"buta\", \"but\"),\n",
- " (\"zgubi\u0142em\", \"zgubi\u0107\")]\n",
- "\n",
- "lemmatizeWord :: Map Text Text -> Text -> Text\n",
- "lemmatizeWord dict w = findWithDefault w w dict\n",
- "\n",
- "lemmatize :: Map Text Text -> [Text] -> [Text]\n",
- "lemmatize dict = map (lemmatizeWord dict)\n",
- "\n",
- "\n",
- "poorMansStemming = Data.Text.take 6\n",
- "\n",
- "normalize :: Text -> [Text]\n",
- "normalize = map poorMansStemming . removeStopWords . map toLower . lemmatize mockInflectionDictionary . tokenize\n",
- "\n",
- "getVocabulary :: [Text] -> Set Text \n",
- "getVocabulary = Set.unions . map (Set.fromList . normalize) \n",
- " \n",
- "idf :: [[Text]] -> Text -> Double\n",
- "idf coll t = log (fromIntegral n / fromIntegral df)\n",
- " where df = Prelude.length $ Prelude.filter (\\d -> t `elem` d) coll\n",
- " n = Prelude.length coll\n",
- " \n",
- "vectorizeTfIdf :: Int -> [[Text]] -> Map Int Text -> [Text] -> [Double]\n",
- "vectorizeTfIdf vecSize coll v doc = map (\\i -> count (v ! i) doc * idf coll (v ! i)) [0..(vecSize-1)]\n",
- " where count t doc = fromIntegral $ (Prelude.length . Prelude.filter (== t)) doc "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {},
- "outputs": [],
- "source": [
- "import System.IO\n",
- "import Data.List.Split as SP\n",
- "\n",
- "legendsh <- openFile \"legendy.txt\" ReadMode\n",
- "hSetEncoding legendsh utf8\n",
- "contents <- hGetContents legendsh\n",
- "ls = Prelude.lines contents\n",
- "items = map (map pack . SP.splitOn \"\\t\") ls\n",
- "\n",
- "labelsL = map Prelude.head items\n",
- "collectionL = map (!!1) items\n",
- "\n",
- "collectionLNormalized = map normalize collectionL\n",
- "voc' = getVocabulary collectionL\n",
- "\n",
- "vocLSize = Prelude.length voc'\n",
- "\n",
- "vocL :: Map Int Text\n",
- "vocL = Map.fromList $ zip [0..] $ Set.toList voc'\n",
- "\n",
- "invvocL :: Map Text Int\n",
- "invvocL = Map.fromList $ zip (Set.toList voc') [0..]\n",
- "\n",
- "lVectorized = map (vectorizeTfIdf vocLSize collectionLNormalized vocL) collectionLNormalized\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/html": [
- "Eta reduce
Found:
formatNumber x = printf \"% 7.2f\" x
Why Not:
formatNumber = printf \"% 7.2f\"
Use zipWith
Found:
map (\\ (lab, ix) -> lab <> \" \" <> similarTo simFun vs ix)\n",
- " $ zip labels [0 .. (Prelude.length vs - 1)]
Why Not:
zipWith\n",
- " (curry (\\ (lab, ix) -> lab <> \" \" <> similarTo simFun vs ix))\n",
- " labels [0 .. (Prelude.length vs - 1)]
Avoid lambda
Found:
\\ l -> pack $ printf \"% 7s\" l
Why Not:
pack . printf \"% 7s\"
Eta reduce
Found:
vectorizeWithHashingTrick b coll doc\n",
- " = Prelude.foldr ((+++) . wordVector b coll) (zero $ 2 ^ b) doc
Why Not:
vectorizeWithHashingTrick b coll\n",
- " = Prelude.foldr ((+++) . wordVector b coll) (zero $ 2 ^ b)
Fuse foldr/map
Found:
Prelude.foldr (+++) (zero 100)\n",
- " $ map (\\ (t, Just v) -> idf coll t *** toOurVector v)\n",
- " $ Prelude.filter (\\ (_, v) -> isJust v)\n",
- " $ map (\\ t -> (t, getVector model t)) d
Why Not:
foldr\n",
- " ((+++) . (\\ (t, Just v) -> idf coll t *** toOurVector v))\n",
- " (zero 100)\n",
- " (Prelude.filter (\\ (_, v) -> isJust v)\n",
- " $ map (\\ t -> (t, getVector model t)) d)
\n",
+ "
Ekstrakcja informacji
\n",
+ " 5. Gęste reprezentacje wektorowe [wykład]
\n",
+ " Filip Graliński (2021)
\n",
+ "\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Zagęszczamy wektory\n",
+ "\n",
+ "Podstawowy problem z wektorową reprezentacją typu tf-idf polega na tym, że wektory dokumentów (i macierz całej kolekcji dokumentów) są _rzadkie_, tzn. zawierają dużo zer. W praktyce potrzebujemy bardziej \"gęstej\" czy \"kompaktowej\" reprezentacji numerycznej dokumentów. \n",
+ "\n",
+ "## _Hashing trick_\n",
+ "\n",
+ "Powierzchownie problem możemy rozwiązać przez użycie tzw. _sztuczki z haszowaniem_ (_hashing trick_). Będziemy potrzebować funkcji mieszającej (haszującej) $H$, która rzutuje napisy na liczby, których reprezentacja binarna składa się z $b$ bitów:\n",
+ "\n",
+ "$$H : \\Sigma^{*} \\rightarrow \\{0,\\dots,2^b-1\\}$$\n",
+ "\n",
+ "($\\Sigma^{*}$ to zbiór wszystkich napisów.)\n",
+ "\n",
+ "**Pytanie:** Czy funkcja $H$ może być różnowartościowa?\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Jako funkcji $H$ możemy np. użyć funkcji MurmurHash2 lub 3."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 28,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "Hash64 0x4a80abc136f926e7"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
},
- "nbformat": 4,
- "nbformat_minor": 4
-}
\ No newline at end of file
+ {
+ "data": {
+ "text/plain": [
+ "Hash64 0x6c3a641663470e2c"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "Hash64 0x6c3a641663470e2c"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "Hash64 0xa714568917576314"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "Hash64 0x875d9e7e413747c8"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "Hash64 0x13ce831936ebc69e"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "Hash64 0xb04ce6229407c882"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "Hash64 0x6ecd7bae29ae0450"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "import Data.Digest.Murmur64\n",
+ "\n",
+ "hash64 \"Komputer\"\n",
+ "hash64 \"komputer\"\n",
+ "hash64 \"komputer\"\n",
+ "hash64 \"komputerze\"\n",
+ "hash64 \"komputerek\"\n",
+ "hash64 \"abrakadabra\"\n",
+ "hash64 \"\"\n",
+ "hash64 \" \"\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**Pytanie:** podobne napisy mają zupełnie różne wartości funkcji haszującej, czy to dobrze, czy to źle?"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Musimy tylko sparametryzować naszą funkcję rozmiarem „odcisku” (parametr $b$)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "3628"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "25364"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "2877"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "50846"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "12"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "{-# LANGUAGE OverloadedStrings #-}\n",
+ "\n",
+ "import Data.Text\n",
+ "\n",
+ "-- pomocnicza funkcja, która konwertuje wartość specjalnego\n",
+ "-- typu Hash64 do zwykłej liczby całkowitej\n",
+ "hashValueAsInteger :: Hash64 -> Integer\n",
+ "hashValueAsInteger = toInteger . asWord64\n",
+ "\n",
+ "-- unpack to funkcja, która wartość typu String konwertuje do Text\n",
+ "hash :: Integer -> Text -> Integer\n",
+ "hash b t = hashValueAsInteger (hash64 $ unpack t) `mod` (2 ^ b)\n",
+ "\n",
+ "hash 16 \"komputer\"\n",
+ "hash 16 \"komputerze\"\n",
+ "hash 16 \"komputerem\"\n",
+ "hash 16 \"abrakadabra\"\n",
+ "hash 4 \"komputer\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**Pytanie:** Jakie wartości $b$ będą bezsensowne?"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Sztuczka z haszowaniem polega na tym, że zamiast numerować słowa korzystając ze słownika, po prostu używamy funkcji haszującej. W ten sposób wektor będzie _zawsze_ rozmiar $2^b$ - bez względu na rozmiar słownika."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Zacznijmy od przywołania wszystkich potrzebnych definicji."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "{-# LANGUAGE OverloadedStrings #-}\n",
+ "{-# LANGUAGE QuasiQuotes #-}\n",
+ "\n",
+ "import Data.Text hiding(map, filter, zip)\n",
+ "import Text.Regex.PCRE.Heavy\n",
+ "\n",
+ "isStopWord :: Text -> Bool\n",
+ "isStopWord \"w\" = True\n",
+ "isStopWord \"jest\" = True\n",
+ "isStopWord \"że\" = True\n",
+ "isStopWord w = w ≈ [re|^\\p{P}+$|]\n",
+ "\n",
+ "\n",
+ "removeStopWords :: [Text] -> [Text]\n",
+ "removeStopWords = filter (not . isStopWord)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "{-# LANGUAGE OverloadedStrings #-}\n",
+ "{-# LANGUAGE QuasiQuotes #-}\n",
+ "{-# LANGUAGE FlexibleContexts #-}\n",
+ "\n",
+ "import Data.Text hiding(map, filter, zip)\n",
+ "import Prelude hiding(words, take)\n",
+ "import Text.Regex.PCRE.Heavy\n",
+ "import Data.Map as Map hiding(take, map, filter)\n",
+ "import Data.Set as Set hiding(map)\n",
+ "\n",
+ "tokenize :: Text -> [Text]\n",
+ "tokenize = map fst . scan [re|C\\+\\+|[\\p{L}0-9]+|\\p{P}|]\n",
+ "\n",
+ "\n",
+ "mockInflectionDictionary :: Map Text Text\n",
+ "mockInflectionDictionary = Map.fromList [\n",
+ " (\"kota\", \"kot\"),\n",
+ " (\"butach\", \"but\"),\n",
+ " (\"masz\", \"mieć\"),\n",
+ " (\"ma\", \"mieć\"),\n",
+ " (\"buta\", \"but\"),\n",
+ " (\"zgubiłem\", \"zgubić\")]\n",
+ "\n",
+ "lemmatizeWord :: Map Text Text -> Text -> Text\n",
+ "lemmatizeWord dict w = findWithDefault w w dict\n",
+ "\n",
+ "lemmatize :: Map Text Text -> [Text] -> [Text]\n",
+ "lemmatize dict = map (lemmatizeWord dict)\n",
+ "\n",
+ "\n",
+ "poorMansStemming = Data.Text.take 6\n",
+ "\n",
+ "normalize :: Text -> [Text]\n",
+ "normalize = map poorMansStemming . removeStopWords . map toLower . lemmatize mockInflectionDictionary . tokenize\n",
+ "\n",
+ "getVocabulary :: [Text] -> Set Text \n",
+ "getVocabulary = Set.unions . map (Set.fromList . normalize) \n",
+ " \n",
+ "idf :: [[Text]] -> Text -> Double\n",
+ "idf coll t = log (fromIntegral n / fromIntegral df)\n",
+ " where df = Prelude.length $ Prelude.filter (\\d -> t `elem` d) coll\n",
+ " n = Prelude.length coll\n",
+ " \n",
+ "vectorizeTfIdf :: Int -> [[Text]] -> Map Int Text -> [Text] -> [Double]\n",
+ "vectorizeTfIdf vecSize coll v doc = map (\\i -> count (v ! i) doc * idf coll (v ! i)) [0..(vecSize-1)]\n",
+ " where count t doc = fromIntegral $ (Prelude.length . Prelude.filter (== t)) doc "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import System.IO\n",
+ "import Data.List.Split as SP\n",
+ "\n",
+ "legendsh <- openFile \"legendy.txt\" ReadMode\n",
+ "hSetEncoding legendsh utf8\n",
+ "contents <- hGetContents legendsh\n",
+ "ls = Prelude.lines contents\n",
+ "items = map (map pack . SP.splitOn \"\\t\") ls\n",
+ "\n",
+ "labelsL = map Prelude.head items\n",
+ "collectionL = map (!!1) items\n",
+ "\n",
+ "collectionLNormalized = map normalize collectionL\n",
+ "voc' = getVocabulary collectionL\n",
+ "\n",
+ "vocLSize = Prelude.length voc'\n",
+ "\n",
+ "vocL :: Map Int Text\n",
+ "vocL = Map.fromList $ zip [0..] $ Set.toList voc'\n",
+ "\n",
+ "invvocL :: Map Text Int\n",
+ "invvocL = Map.fromList $ zip (Set.toList voc') [0..]\n",
+ "\n",
+ "lVectorized = map (vectorizeTfIdf vocLSize collectionLNormalized vocL) collectionLNormalized\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "Eta reduce
Found:
formatNumber x = printf \"% 7.2f\" x
Why Not:
formatNumber = printf \"% 7.2f\"
Use zipWith
Found:
map (\\ (lab, ix) -> lab <> \" \" <> similarTo simFun vs ix)\n",
+ " $ zip labels [0 .. (Prelude.length vs - 1)]
Why Not:
zipWith\n",
+ " (curry (\\ (lab, ix) -> lab <> \" \" <> similarTo simFun vs ix))\n",
+ " labels [0 .. (Prelude.length vs - 1)]
Avoid lambda
Found:
\\ l -> pack $ printf \"% 7s\" l
Why Not:
pack . printf \"% 7s\"
Eta reduce
Found:
vectorizeWithHashingTrick b coll doc\n",
+ " = Prelude.foldr ((+++) . wordVector b coll) (zero $ 2 ^ b) doc
Why Not:
vectorizeWithHashingTrick b coll\n",
+ " = Prelude.foldr ((+++) . wordVector b coll) (zero $ 2 ^ b)
Fuse foldr/map
Found:
Prelude.foldr (+++) (zero 100)\n",
+ " $ map (\\ (t, Just v) -> idf coll t *** toOurVector v)\n",
+ " $ Prelude.filter (\\ (_, v) -> isJust v)\n",
+ " $ map (\\ t -> (t, getVector model t)) d
Why Not:
foldr\n",
+ " ((+++) . (\\ (t, Just v) -> idf coll t *** toOurVector v))\n",
+ " (zero 100)\n",
+ " (Prelude.filter (\\ (_, v) -> isJust v)\n",
+ " $ map (\\ t -> (t, getVector model t)) d)
\n",
- "
Ekstrakcja informacji
\n",
- " 14. Pretrenowane modele j\u0119zyka [wyk\u0142ad]
\n",
- " Filip Grali\u0144ski (2021)
\n",
- "\n",
- "\n",
- ""
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Pretrenowanie modeli\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "System AlphaZero uczy si\u0119 graj\u0105c sam ze sob\u0105 \u2014 wystarczy 24 godziny,\n",
- "by system nauczy\u0142 si\u0119 gra\u0107 w szachy lub go na nadludzkim poziomie.\n",
- "\n",
- "**Pytanie**: Dlaczego granie samemu ze sob\u0105 nie jest dobrym sposobem\n",
- " nauczenia si\u0119 grania w szachy dla cz\u0142owieka, a dla maszyny jest?\n",
- "\n",
- "Co jest odpowiednikiem grania samemu ze sob\u0105 w \u015bwiecie przetwarzania tekstu?\n",
- "Tzn. **pretrenowanie** (*pretraining*) na du\u017cym korpusie tekstu. (Tekst jest tani!)\n",
- "\n",
- "Jest kilka sposob\u00f3w na pretrenowanie modelu, w ka\u017cdym razie sprowadza\n",
- "si\u0119 do odgadywania nast\u0119pnego b\u0105d\u017a zamaskowanego s\u0142owa.\n",
- "W ka\u017cdym razie zawsze stosujemy softmax (by\u0107 mo\u017ce ze \u201esztuczkami\u201d takimi jak\n",
- "negatywne pr\u00f3bkowanie albo hierarchiczny softmax) na pewnej **reprezentacji kontekstowej**:\n",
- "\n",
- "$$\\vec{p} = \\operatorname{softmax}(f(\\vec{c})).$$\n",
- "\n",
- "Model jest karany u\u017cywaj\u0105c funkcji log loss:\n",
- "\n",
- "$$-\\log(p_j),$$\n",
- "\n",
- "gdzie $w_j$ jest wyrazem, kt\u00f3ry pojawi\u0142 si\u0119 rzeczywi\u015bcie w korpusie.\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Przewidywanie s\u0142owa (GPT-2)\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Jeden ze sposob\u00f3w pretrenowania modelu to po prostu przewidywanie\n",
- "nast\u0119pnego s\u0142owa.\n",
- "\n",
- "Zainstalujmy najpierw bibliotek\u0119 transformers.\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [],
- "source": [
- "! pip install transformers"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 17,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "50257\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "[('\u00c2\u0142', 0.6182783842086792),\n",
- " ('\u00c8', 0.1154019758105278),\n",
- " ('\u00d1\u0123', 0.026960616931319237),\n",
- " ('_____', 0.024418892338871956),\n",
- " ('________', 0.014962316490709782),\n",
- " ('\u00c3\u0124', 0.010653386823832989),\n",
- " ('\u00e4\u00b8\u0143', 0.008340531960129738),\n",
- " ('\u00d1', 0.007557711564004421),\n",
- " ('\u00ca', 0.007046067621558905),\n",
- " ('\u00e3\u0122', 0.006875576451420784),\n",
- " ('ile', 0.006685272324830294),\n",
- " ('____', 0.006307446397840977),\n",
- " ('\u00e2\u0122\u012d', 0.006306538358330727),\n",
- " ('\u00d1\u0122', 0.006197483278810978),\n",
- " ('\u0120Belarus', 0.006108700763434172),\n",
- " ('\u00c6', 0.005720408633351326),\n",
- " ('\u0120Poland', 0.0053678699769079685),\n",
- " ('\u00e1\u00b9', 0.004606408067047596),\n",
- " ('\u00ee\u0122', 0.004161055199801922),\n",
- " ('????', 0.004056799225509167),\n",
- " ('_______', 0.0038176667876541615),\n",
- " ('\u00e4\u00b8', 0.0036082742735743523),\n",
- " ('\u00cc', 0.003221835708245635),\n",
- " ('urs', 0.003080119378864765),\n",
- " ('________________', 0.0027312245219945908),\n",
- " ('\u0120Lithuania', 0.0023860156070441008),\n",
- " ('ich', 0.0021211160346865654),\n",
- " ('iz', 0.002069818088784814),\n",
- " ('vern', 0.002001357264816761),\n",
- " ('\u00c5\u0124', 0.001717406208626926)]"
- ]
- },
- "execution_count": 17,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "import torch\n",
- "from transformers import GPT2Tokenizer, GPT2LMHeadModel\n",
- "tokenizer = GPT2Tokenizer.from_pretrained('gpt2-large')\n",
- "model = GPT2LMHeadModel.from_pretrained('gpt2-large')\n",
- "text = 'Warsaw is the capital city of'\n",
- "encoded_input = tokenizer(text, return_tensors='pt')\n",
- "output = model(**encoded_input)\n",
- "next_token_probs = torch.softmax(output[0][:, -1, :][0], dim=0)\n",
- "\n",
- "nb_of_tokens = next_token_probs.size()[0]\n",
- "print(nb_of_tokens)\n",
- "\n",
- "_, top_k_indices = torch.topk(next_token_probs, 30, sorted=True)\n",
- "\n",
- "words = tokenizer.convert_ids_to_tokens(top_k_indices)\n",
- "\n",
- "top_probs = []\n",
- "\n",
- "for ix in range(len(top_k_indices)):\n",
- " top_probs.append((words[ix], next_token_probs[top_k_indices[ix]].item()))\n",
- "\n",
- "top_probs"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Zalety tego podej\u015bcia:\n",
- "\n",
- "- prostota,\n",
- "- dobra podstawa do strojenia system\u00f3w generowania tekstu zw\u0142aszcza\n",
- " \u201eotwartego\u201d (systemy dialogowe, generowanie (fake) news\u00f3w, streszczanie tekstu),\n",
- " ale niekoniecznie t\u0142umaczenia maszynowego,\n",
- "- zaskakuj\u0105ca skuteczno\u015b\u0107 przy uczeniu *few-shot* i *zero-shot*.\n",
- "\n",
- "Wady:\n",
- "\n",
- "- asymetryczno\u015b\u0107, przetwarzanie tylko z lewej do prawej, preferencja\n",
- " dla lewego kontekstu,\n",
- "- mniejsza skuteczno\u015b\u0107 przy dostrajaniu do zada\u0144 klasyfikacji i innych zada\u0144\n",
- " niepolegaj\u0105cych na prostym generowaniu.\n",
- "\n",
- "Przyk\u0142ady modeli: GPT, GPT-2, GPT-3, DialoGPT.\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Maskowanie s\u0142\u00f3w (BERT)\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Inn\u0105 metod\u0105 jest maskowanie s\u0142\u00f3w (*Masked Language Modeling*, *MLM*).\n",
- "\n",
- "W tym podej\u015bciu losowe wybrane zast\u0119pujemy losowe s\u0142owa specjalnym\n",
- "tokenem (`[MASK]`) i ka\u017cemy modelowi odgadywa\u0107 w ten spos\u00f3b\n",
- "zamaskowane s\u0142owa (z uwzgl\u0119dnieniem r\u00f3wnie\u017c prawego kontekstu!).\n",
- "\n",
- "M\u00f3ci\u0105c \u015bci\u015ble, w jednym z pierwszych modeli tego typu (BERT)\n",
- "zastosowano schemat, w kt\u00f3rym r\u00f3wnie\u017c niezamaskowane s\u0142owa s\u0105 odgadywane (!):\n",
- "\n",
- "- wybieramy losowe 15% wyraz\u00f3w do odgadni\u0119cia\n",
- "- 80% z nich zast\u0119pujemy tokenem `[MASK]`,\n",
- "- 10% zast\u0119pujemy innym losowym wyrazem,\n",
- "- 10% pozostawiamy bez zmian.\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "/home/filipg/.local/lib/python3.9/site-packages/transformers/models/auto/modeling_auto.py:806: FutureWarning: The class `AutoModelWithLMHead` is deprecated and will be removed in a future version. Please use `AutoModelForCausalLM` for causal language models, `AutoModelForMaskedLM` for masked language models and `AutoModelForSeq2SeqLM` for encoder-decoder models.\n",
- " warnings.warn(\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "W kt\u00f3rym pa\u0144stwie le\u017cy Bombaj? W USA. (score: 0.16715531051158905)\n",
- "W kt\u00f3rym pa\u0144stwie le\u017cy Bombaj? W India. (score: 0.09912960231304169)\n",
- "W kt\u00f3rym pa\u0144stwie le\u017cy Bombaj? W Indian. (score: 0.039642028510570526)\n",
- "W kt\u00f3rym pa\u0144stwie le\u017cy Bombaj? W Nepal. (score: 0.027137665078043938)\n",
- "W kt\u00f3rym pa\u0144stwie le\u017cy Bombaj? W Pakistan. (score: 0.027065709233283997)\n",
- "W kt\u00f3rym pa\u0144stwie le\u017cy Bombaj? W Polsce. (score: 0.023737527430057526)\n",
- "W kt\u00f3rym pa\u0144stwie le\u017cy Bombaj? W .... (score: 0.02306722290813923)\n",
- "W kt\u00f3rym pa\u0144stwie le\u017cy Bombaj? W Bangladesh. (score: 0.022106658667325974)\n",
- "W kt\u00f3rym pa\u0144stwie le\u017cy Bombaj? W .... (score: 0.01628892682492733)\n",
- "W kt\u00f3rym pa\u0144stwie le\u017cy Bombaj? W Niemczech. (score: 0.014501162804663181)\n"
- ]
- }
- ],
- "source": [
- "from transformers import AutoModelWithLMHead, AutoTokenizer\n",
- "import torch\n",
- "\n",
- "tokenizer = AutoTokenizer.from_pretrained(\"xlm-roberta-large\")\n",
- "model = AutoModelWithLMHead.from_pretrained(\"xlm-roberta-large\")\n",
- "\n",
- "sequence = f'W kt\u00f3rym pa\u0144stwie le\u017cy Bombaj? W {tokenizer.mask_token}.'\n",
- "\n",
- "input_ids = tokenizer.encode(sequence, return_tensors=\"pt\")\n",
- "mask_token_index = torch.where(input_ids == tokenizer.mask_token_id)[1]\n",
- "\n",
- "token_logits = model(input_ids)[0]\n",
- "mask_token_logits = token_logits[0, mask_token_index, :]\n",
- "mask_token_logits = torch.softmax(mask_token_logits, dim=1)\n",
- "\n",
- "top_10 = torch.topk(mask_token_logits, 10, dim=1)\n",
- "top_10_tokens = zip(top_10.indices[0].tolist(), top_10.values[0].tolist())\n",
- "\n",
- "for token, score in top_10_tokens:\n",
- " print(sequence.replace(tokenizer.mask_token, tokenizer.decode([token])), f\"(score: {score})\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Przyk\u0142ady: BERT, RoBERTa (r\u00f3wnie\u017c Polish RoBERTa).\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Podej\u015bcie generatywne (koder-dekoder).\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "System ma wygenerowa\u0107 odpowied\u017a na r\u00f3\u017cne pytania (r\u00f3wnie\u017c\n",
- "odpowiadaj\u0105ce zadaniu MLM), np.:\n",
- "\n",
- "- \"translate English to German: That is good.\" => \"Das ist gut.\"\n",
- "- \"cola sentence: The course is jumping well.\" => \"not acceptable\"\n",
- "- \"summarize: state authorities dispatched emergency crews tuesday to survey the damage after an onslaught of severe weather in mississippi…\"\n",
- " => \"six people hospitalized after a storm in attala county\"\n",
- "- \"Thank you for me to your party week.\" => for inviting last \n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "['World War II ended in World War II.',\n",
- " 'World War II ended in 1945..',\n",
- " 'World War II ended in 1945.',\n",
- " 'World War II ended in 1945.',\n",
- " 'World War II ended in 1945.']"
- ]
- },
- "execution_count": 2,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "from transformers import T5Tokenizer, T5Config, T5ForConditionalGeneration\n",
- "\n",
- "T5_PATH = 't5-base'\n",
- "\n",
- "t5_tokenizer = T5Tokenizer.from_pretrained(T5_PATH)\n",
- "t5_config = T5Config.from_pretrained(T5_PATH)\n",
- "t5_mlm = T5ForConditionalGeneration.from_pretrained(T5_PATH, config=t5_config)\n",
- "\n",
- "slot = ''\n",
- "\n",
- "text = f'World War II ended in {slot}.'\n",
- "\n",
- "encoded = t5_tokenizer.encode_plus(text, add_special_tokens=True, return_tensors='pt')\n",
- "input_ids = encoded['input_ids']\n",
- "\n",
- "outputs = t5_mlm.generate(input_ids=input_ids,\n",
- " num_beams=200, num_return_sequences=5,\n",
- " max_length=5)\n",
- "\n",
- "_0_index = text.index(slot)\n",
- "_result_prefix = text[:_0_index]\n",
- "_result_suffix = text[_0_index+len(slot):]\n",
- "\n",
- "def _filter(output, end_token=''):\n",
- " _txt = t5_tokenizer.decode(output[2:], skip_special_tokens=False, clean_up_tokenization_spaces=False)\n",
- " if end_token in _txt:\n",
- " _end_token_index = _txt.index(end_token)\n",
- " return _result_prefix + _txt[:_end_token_index] + _result_suffix\n",
- " else:\n",
- " return _result_prefix + _txt + _result_suffix\n",
- "\n",
- "\n",
- "results = [_filter(out) for out in outputs]\n",
- "results"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "(Zob. [https://arxiv.org/pdf/1910.10683.pdf](https://arxiv.org/pdf/1910.10683.pdf))\n",
- "\n",
- "Przyk\u0142ad: T5, mT5\n",
- "\n"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.9.6"
- },
- "org": null,
- "author": "Filip Grali\u0144ski",
- "email": "filipg@amu.edu.pl",
- "lang": "pl",
- "subtitle": "14.Pretrenowane modele j\u0119zyka[wyk\u0142ad]",
- "title": "Ekstrakcja informacji",
- "year": "2021"
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "
Ekstrakcja informacji
\n",
+ " 14. Pretrenowane modele języka [wykład]
\n",
+ " Filip Graliński (2021)
\n",
+ "\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Pretrenowanie modeli\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "System AlphaZero uczy się grając sam ze sobą — wystarczy 24 godziny,\n",
+ "by system nauczył się grać w szachy lub go na nadludzkim poziomie.\n",
+ "\n",
+ "**Pytanie**: Dlaczego granie samemu ze sobą nie jest dobrym sposobem\n",
+ " nauczenia się grania w szachy dla człowieka, a dla maszyny jest?\n",
+ "\n",
+ "Co jest odpowiednikiem grania samemu ze sobą w świecie przetwarzania tekstu?\n",
+ "Tzn. **pretrenowanie** (*pretraining*) na dużym korpusie tekstu. (Tekst jest tani!)\n",
+ "\n",
+ "Jest kilka sposobów na pretrenowanie modelu, w każdym razie sprowadza\n",
+ "się do odgadywania następnego bądź zamaskowanego słowa.\n",
+ "W każdym razie zawsze stosujemy softmax (być może ze „sztuczkami” takimi jak\n",
+ "negatywne próbkowanie albo hierarchiczny softmax) na pewnej **reprezentacji kontekstowej**:\n",
+ "\n",
+ "$$\\vec{p} = \\operatorname{softmax}(f(\\vec{c})).$$\n",
+ "\n",
+ "Model jest karany przy użyciu funkcji log loss:\n",
+ "\n",
+ "$$-\\log(p_j),$$\n",
+ "\n",
+ "gdzie $w_j$ jest wyrazem, który pojawił się rzeczywiście w korpusie.\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Przewidywanie słowa (GPT-2)\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Jeden ze sposobów pretrenowania modelu to po prostu przewidywanie\n",
+ "następnego słowa.\n",
+ "\n",
+ "Zainstalujmy najpierw bibliotekę transformers.\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "! pip install transformers"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 17,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "50257\n"
+ ]
},
- "nbformat": 4,
- "nbformat_minor": 4
-}
\ No newline at end of file
+ {
+ "data": {
+ "text/plain": [
+ "[('Âł', 0.6182783842086792),\n",
+ " ('È', 0.1154019758105278),\n",
+ " ('Ñģ', 0.026960616931319237),\n",
+ " ('_____', 0.024418892338871956),\n",
+ " ('________', 0.014962316490709782),\n",
+ " ('ÃĤ', 0.010653386823832989),\n",
+ " ('ä¸Ń', 0.008340531960129738),\n",
+ " ('Ñ', 0.007557711564004421),\n",
+ " ('Ê', 0.007046067621558905),\n",
+ " ('ãĢ', 0.006875576451420784),\n",
+ " ('ile', 0.006685272324830294),\n",
+ " ('____', 0.006307446397840977),\n",
+ " ('âĢĭ', 0.006306538358330727),\n",
+ " ('ÑĢ', 0.006197483278810978),\n",
+ " ('ĠBelarus', 0.006108700763434172),\n",
+ " ('Æ', 0.005720408633351326),\n",
+ " ('ĠPoland', 0.0053678699769079685),\n",
+ " ('á¹', 0.004606408067047596),\n",
+ " ('îĢ', 0.004161055199801922),\n",
+ " ('????', 0.004056799225509167),\n",
+ " ('_______', 0.0038176667876541615),\n",
+ " ('ä¸', 0.0036082742735743523),\n",
+ " ('Ì', 0.003221835708245635),\n",
+ " ('urs', 0.003080119378864765),\n",
+ " ('________________', 0.0027312245219945908),\n",
+ " ('ĠLithuania', 0.0023860156070441008),\n",
+ " ('ich', 0.0021211160346865654),\n",
+ " ('iz', 0.002069818088784814),\n",
+ " ('vern', 0.002001357264816761),\n",
+ " ('ÅĤ', 0.001717406208626926)]"
+ ]
+ },
+ "execution_count": 17,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "import torch\n",
+ "from transformers import GPT2Tokenizer, GPT2LMHeadModel\n",
+ "tokenizer = GPT2Tokenizer.from_pretrained('gpt2-large')\n",
+ "model = GPT2LMHeadModel.from_pretrained('gpt2-large')\n",
+ "text = 'Warsaw is the capital city of'\n",
+ "encoded_input = tokenizer(text, return_tensors='pt')\n",
+ "output = model(**encoded_input)\n",
+ "next_token_probs = torch.softmax(output[0][:, -1, :][0], dim=0)\n",
+ "\n",
+ "nb_of_tokens = next_token_probs.size()[0]\n",
+ "print(nb_of_tokens)\n",
+ "\n",
+ "_, top_k_indices = torch.topk(next_token_probs, 30, sorted=True)\n",
+ "\n",
+ "words = tokenizer.convert_ids_to_tokens(top_k_indices)\n",
+ "\n",
+ "top_probs = []\n",
+ "\n",
+ "for ix in range(len(top_k_indices)):\n",
+ " top_probs.append((words[ix], next_token_probs[top_k_indices[ix]].item()))\n",
+ "\n",
+ "top_probs"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Zalety tego podejścia:\n",
+ "\n",
+ "- prostota,\n",
+ "- dobra podstawa do strojenia systemów generowania tekstu zwłaszcza\n",
+ " „otwartego” (systemy dialogowe, generowanie (fake) newsów, streszczanie tekstu),\n",
+ " ale niekoniecznie tłumaczenia maszynowego,\n",
+ "- zaskakująca skuteczność przy uczeniu *few-shot* i *zero-shot*.\n",
+ "\n",
+ "Wady:\n",
+ "\n",
+ "- asymetryczność, przetwarzanie tylko z lewej do prawej, preferencja\n",
+ " dla lewego kontekstu,\n",
+ "- mniejsza skuteczność przy dostrajaniu do zadań klasyfikacji i innych zadań\n",
+ " niepolegających na prostym generowaniu.\n",
+ "\n",
+ "Przykłady modeli: GPT, GPT-2, GPT-3, DialoGPT.\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Maskowanie słów (BERT)\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Inną metodą jest maskowanie słów (*Masked Language Modeling*, *MLM*).\n",
+ "\n",
+ "W tym podejściu losowe wybrane zastępujemy losowe słowa specjalnym\n",
+ "tokenem (`[MASK]`) i każemy modelowi odgadywać w ten sposób\n",
+ "zamaskowane słowa (z uwzględnieniem również prawego kontekstu!).\n",
+ "\n",
+ "Móciąc ściśle, w jednym z pierwszych modeli tego typu (BERT)\n",
+ "zastosowano schemat, w którym również niezamaskowane słowa są odgadywane (!):\n",
+ "\n",
+ "- wybieramy losowe 15% wyrazów do odgadnięcia\n",
+ "- 80% z nich zastępujemy tokenem `[MASK]`,\n",
+ "- 10% zastępujemy innym losowym wyrazem,\n",
+ "- 10% pozostawiamy bez zmian.\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/home/filipg/.local/lib/python3.9/site-packages/transformers/models/auto/modeling_auto.py:806: FutureWarning: The class `AutoModelWithLMHead` is deprecated and will be removed in a future version. Please use `AutoModelForCausalLM` for causal language models, `AutoModelForMaskedLM` for masked language models and `AutoModelForSeq2SeqLM` for encoder-decoder models.\n",
+ " warnings.warn(\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "W którym państwie leży Bombaj? W USA. (score: 0.16715531051158905)\n",
+ "W którym państwie leży Bombaj? W India. (score: 0.09912960231304169)\n",
+ "W którym państwie leży Bombaj? W Indian. (score: 0.039642028510570526)\n",
+ "W którym państwie leży Bombaj? W Nepal. (score: 0.027137665078043938)\n",
+ "W którym państwie leży Bombaj? W Pakistan. (score: 0.027065709233283997)\n",
+ "W którym państwie leży Bombaj? W Polsce. (score: 0.023737527430057526)\n",
+ "W którym państwie leży Bombaj? W .... (score: 0.02306722290813923)\n",
+ "W którym państwie leży Bombaj? W Bangladesh. (score: 0.022106658667325974)\n",
+ "W którym państwie leży Bombaj? W .... (score: 0.01628892682492733)\n",
+ "W którym państwie leży Bombaj? W Niemczech. (score: 0.014501162804663181)\n"
+ ]
+ }
+ ],
+ "source": [
+ "from transformers import AutoModelWithLMHead, AutoTokenizer\n",
+ "import torch\n",
+ "\n",
+ "tokenizer = AutoTokenizer.from_pretrained(\"xlm-roberta-large\")\n",
+ "model = AutoModelWithLMHead.from_pretrained(\"xlm-roberta-large\")\n",
+ "\n",
+ "sequence = f'W którym państwie leży Bombaj? W {tokenizer.mask_token}.'\n",
+ "\n",
+ "input_ids = tokenizer.encode(sequence, return_tensors=\"pt\")\n",
+ "mask_token_index = torch.where(input_ids == tokenizer.mask_token_id)[1]\n",
+ "\n",
+ "token_logits = model(input_ids)[0]\n",
+ "mask_token_logits = token_logits[0, mask_token_index, :]\n",
+ "mask_token_logits = torch.softmax(mask_token_logits, dim=1)\n",
+ "\n",
+ "top_10 = torch.topk(mask_token_logits, 10, dim=1)\n",
+ "top_10_tokens = zip(top_10.indices[0].tolist(), top_10.values[0].tolist())\n",
+ "\n",
+ "for token, score in top_10_tokens:\n",
+ " print(sequence.replace(tokenizer.mask_token, tokenizer.decode([token])), f\"(score: {score})\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Przykłady: BERT, RoBERTa (również Polish RoBERTa).\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Podejście generatywne (koder-dekoder).\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "System ma wygenerować odpowiedź na różne pytania (również\n",
+ "odpowiadające zadaniu MLM), np.:\n",
+ "\n",
+ "- \"translate English to German: That is good.\" => \"Das ist gut.\"\n",
+ "- \"cola sentence: The course is jumping well.\" => \"not acceptable\"\n",
+ "- \"summarize: state authorities dispatched emergency crews tuesday to survey the damage after an onslaught of severe weather in mississippi…\"\n",
+ " => \"six people hospitalized after a storm in attala county\"\n",
+ "- \"Thank you for me to your party week.\" => for inviting last \n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "['World War II ended in World War II.',\n",
+ " 'World War II ended in 1945..',\n",
+ " 'World War II ended in 1945.',\n",
+ " 'World War II ended in 1945.',\n",
+ " 'World War II ended in 1945.']"
+ ]
+ },
+ "execution_count": 2,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from transformers import T5Tokenizer, T5Config, T5ForConditionalGeneration\n",
+ "\n",
+ "T5_PATH = 't5-base'\n",
+ "\n",
+ "t5_tokenizer = T5Tokenizer.from_pretrained(T5_PATH)\n",
+ "t5_config = T5Config.from_pretrained(T5_PATH)\n",
+ "t5_mlm = T5ForConditionalGeneration.from_pretrained(T5_PATH, config=t5_config)\n",
+ "\n",
+ "slot = ''\n",
+ "\n",
+ "text = f'World War II ended in {slot}.'\n",
+ "\n",
+ "encoded = t5_tokenizer.encode_plus(text, add_special_tokens=True, return_tensors='pt')\n",
+ "input_ids = encoded['input_ids']\n",
+ "\n",
+ "outputs = t5_mlm.generate(input_ids=input_ids,\n",
+ " num_beams=200, num_return_sequences=5,\n",
+ " max_length=5)\n",
+ "\n",
+ "_0_index = text.index(slot)\n",
+ "_result_prefix = text[:_0_index]\n",
+ "_result_suffix = text[_0_index+len(slot):]\n",
+ "\n",
+ "def _filter(output, end_token=''):\n",
+ " _txt = t5_tokenizer.decode(output[2:], skip_special_tokens=False, clean_up_tokenization_spaces=False)\n",
+ " if end_token in _txt:\n",
+ " _end_token_index = _txt.index(end_token)\n",
+ " return _result_prefix + _txt[:_end_token_index] + _result_suffix\n",
+ " else:\n",
+ " return _result_prefix + _txt + _result_suffix\n",
+ "\n",
+ "\n",
+ "results = [_filter(out) for out in outputs]\n",
+ "results"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "(Zob. [https://arxiv.org/pdf/1910.10683.pdf](https://arxiv.org/pdf/1910.10683.pdf))\n",
+ "\n",
+ "Przykład: T5, mT5\n",
+ "\n"
+ ]
+ }
+ ],
+ "metadata": {
+ "author": "Filip Graliński",
+ "email": "filipg@amu.edu.pl",
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "lang": "pl",
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.6"
+ },
+ "org": null,
+ "subtitle": "14.Pretrenowane modele języka[wykład]",
+ "title": "Ekstrakcja informacji",
+ "year": "2021"
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/wyk/15_transformer.ipynb b/wyk/15_transformer.ipynb
index 19df858..f303366 100644
--- a/wyk/15_transformer.ipynb
+++ b/wyk/15_transformer.ipynb
@@ -219,7 +219,10 @@
"- $V$ = $EW^V$\n",
"\n",
"W kolejnych warstwach zamiast $E$ wykorzystywane jest wyjście z poprzedniej warstwy.\n",
- "\n"
+ "\n",
+ "## Zastosowanie w ekstrakcji informacji\n",
+ "\n",
+ "W prosty sposób możemy do sieci Transformer dołączyć głowicę realizującą etykietowanie sekwencji."
]
},
{