diff --git a/wyk/05_Geste_wektory.ipynb b/wyk/05_Geste_wektory.ipynb
new file mode 100644
index 0000000..44aa971
--- /dev/null
+++ b/wyk/05_Geste_wektory.ipynb
@@ -0,0 +1,1196 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Zagęszczamy wektory\n",
+ "\n",
+ "Podstawowy problem z wektorową reprezentacją typu tf-idf polega na tym, że wektory dokumentów (i macierz całej kolekcji dokumentów) są _rzadkie_, tzn. zawierają dużo zer. W praktyce potrzebujemy bardziej \"gęstej\" czy \"kompaktowej\" reprezentacji numerycznej dokumentów. \n",
+ "\n",
+ "## _Hashing trick_\n",
+ "\n",
+ "Powierzchownie problem możemy rozwiązać przez użycie tzw. _sztuczki z haszowaniem_ (_hashing trick_). Będziemy potrzebować funkcji mieszającej (haszującej) $H$, która rzutuje na napisy na liczby, których reprezentacja binarna składa się z $b$ bitów:\n",
+ "\n",
+ "$$H : V \\rightarrow \\{0,\\dots,2^b-1\\}$$\n",
+ "\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Jako funkcji $H$ możemy np. użyć funkcji MurmurHash3."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "Hash64 0x6c3a641663470e2c"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "Hash64 0xa714568917576314"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "Hash64 0x875d9e7e413747c8"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "Hash64 0x13ce831936ebc69e"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "import Data.Digest.Murmur64\n",
+ "\n",
+ "hash64 \"komputer\"\n",
+ "hash64 \"komputerze\"\n",
+ "hash64 \"komputerek\"\n",
+ "hash64 \"abrakadabra\"\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**Pytanie:** podobne napisy mają zupełnie różne wartości funkcji haszującej, czy to dobrze, czy to źle?"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Musimy tylko sparametryzować naszą funkcję rozmiarem \"odcisku\" (parametr $b$)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "3628"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "25364"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "2877"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "50846"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "12"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "{-# LANGUAGE OverloadedStrings #-}\n",
+ "\n",
+ "import Data.Text\n",
+ "\n",
+ "-- pomocnicza funkcja, która konwertuje wartość specjalnego\n",
+ "-- typu Hash64 do zwykłej liczby całkowitej\n",
+ "hashValueAsInteger :: Hash64 -> Integer\n",
+ "hashValueAsInteger = toInteger . asWord64\n",
+ "\n",
+ "-- unpack to funkcja, która wartość typu String konwertuje do Text\n",
+ "hash :: Integer -> Text -> Integer\n",
+ "hash b t = hashValueAsInteger (hash64 $ unpack t) `mod` (2 ^ b)\n",
+ "\n",
+ "hash 16 \"komputer\"\n",
+ "hash 16 \"komputerze\"\n",
+ "hash 16 \"komputerem\"\n",
+ "hash 16 \"abrakadabra\"\n",
+ "hash 4 \"komputer\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**Pytanie:** Jakie wartości $b$ będą bezsensowne?"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Sztuczka z haszowaniem polega na tym, że zamiast numerować słowa korzystając ze słownika, po prostu używamy funkcji haszującej. W ten sposób wektor będzie _zawsze_ rozmiar $2^b$ - bez względu na rozmiar słownika."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Zacznijmy od przywołania wszystkich potrzebnych definicji."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "{-# LANGUAGE OverloadedStrings #-}\n",
+ "{-# LANGUAGE QuasiQuotes #-}\n",
+ "\n",
+ "import Data.Text hiding(map, filter, zip)\n",
+ "import Text.Regex.PCRE.Heavy\n",
+ "\n",
+ "isStopWord :: Text -> Bool\n",
+ "isStopWord \"w\" = True\n",
+ "isStopWord \"jest\" = True\n",
+ "isStopWord \"że\" = True\n",
+ "isStopWord w = w ≈ [re|^\\p{P}+$|]\n",
+ "\n",
+ "\n",
+ "removeStopWords :: [Text] -> [Text]\n",
+ "removeStopWords = filter (not . isStopWord)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "{-# LANGUAGE OverloadedStrings #-}\n",
+ "{-# LANGUAGE QuasiQuotes #-}\n",
+ "{-# LANGUAGE FlexibleContexts #-}\n",
+ "\n",
+ "import Data.Text hiding(map, filter, zip)\n",
+ "import Prelude hiding(words, take)\n",
+ "import Text.Regex.PCRE.Heavy\n",
+ "import Data.Map as Map hiding(take, map, filter)\n",
+ "import Data.Set as Set hiding(map)\n",
+ "\n",
+ "tokenize :: Text -> [Text]\n",
+ "tokenize = map fst . scan [re|C\\+\\+|[\\p{L}0-9]+|\\p{P}|]\n",
+ "\n",
+ "\n",
+ "mockInflectionDictionary :: Map Text Text\n",
+ "mockInflectionDictionary = Map.fromList [\n",
+ " (\"kota\", \"kot\"),\n",
+ " (\"butach\", \"but\"),\n",
+ " (\"masz\", \"mieć\"),\n",
+ " (\"ma\", \"mieć\"),\n",
+ " (\"buta\", \"but\"),\n",
+ " (\"zgubiłem\", \"zgubić\")]\n",
+ "\n",
+ "lemmatizeWord :: Map Text Text -> Text -> Text\n",
+ "lemmatizeWord dict w = findWithDefault w w dict\n",
+ "\n",
+ "lemmatize :: Map Text Text -> [Text] -> [Text]\n",
+ "lemmatize dict = map (lemmatizeWord dict)\n",
+ "\n",
+ "\n",
+ "poorMansStemming = Data.Text.take 6\n",
+ "\n",
+ "normalize :: Text -> [Text]\n",
+ "normalize = map poorMansStemming . removeStopWords . map toLower . lemmatize mockInflectionDictionary . tokenize\n",
+ "\n",
+ "getVocabulary :: [Text] -> Set Text \n",
+ "getVocabulary = Set.unions . map (Set.fromList . normalize) \n",
+ " \n",
+ "idf :: [[Text]] -> Text -> Double\n",
+ "idf coll t = log (fromIntegral n / fromIntegral df)\n",
+ " where df = Prelude.length $ Prelude.filter (\\d -> t `elem` d) coll\n",
+ " n = Prelude.length coll\n",
+ " \n",
+ "vectorizeTfIdf :: Int -> [[Text]] -> Map Int Text -> [Text] -> [Double]\n",
+ "vectorizeTfIdf vecSize coll v doc = map (\\i -> count (v ! i) doc * idf coll (v ! i)) [0..(vecSize-1)]\n",
+ " where count t doc = fromIntegral $ (Prelude.length . Prelude.filter (== t)) doc "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import System.IO\n",
+ "import Data.List.Split as SP\n",
+ "\n",
+ "legendsh <- openFile \"legendy.txt\" ReadMode\n",
+ "hSetEncoding legendsh utf8\n",
+ "contents <- hGetContents legendsh\n",
+ "ls = Prelude.lines contents\n",
+ "items = map (map pack . SP.splitOn \"\\t\") ls\n",
+ "\n",
+ "labelsL = map Prelude.head items\n",
+ "collectionL = map (!!1) items\n",
+ "\n",
+ "collectionLNormalized = map normalize collectionL\n",
+ "voc' = getVocabulary collectionL\n",
+ "\n",
+ "vocLSize = Prelude.length voc'\n",
+ "\n",
+ "vocL :: Map Int Text\n",
+ "vocL = Map.fromList $ zip [0..] $ Set.toList voc'\n",
+ "\n",
+ "invvocL :: Map Text Int\n",
+ "invvocL = Map.fromList $ zip (Set.toList voc') [0..]\n",
+ "\n",
+ "lVectorized = map (vectorizeTfIdf vocLSize collectionLNormalized vocL) collectionLNormalized\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "Eta reduce
Found:
formatNumber x = printf \"% 7.2f\" x
Why Not:
formatNumber = printf \"% 7.2f\"
Use zipWith
Found:
map (\\ (lab, ix) -> lab <> \" \" <> similarTo simFun vs ix)\n",
+ " $ zip labels [0 .. (Prelude.length vs - 1)]
Why Not:
zipWith\n",
+ " (curry (\\ (lab, ix) -> lab <> \" \" <> similarTo simFun vs ix))\n",
+ " labels [0 .. (Prelude.length vs - 1)]
Avoid lambda
Found:
\\ l -> pack $ printf \"% 7s\" l
Why Not:
pack . printf \"% 7s\"