841 lines
27 KiB
Plaintext
841 lines
27 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3c8a4b52",
|
|
"metadata": {},
|
|
"source": [
|
|
"\n",
|
|
"<div class=\"alert alert-block alert-info\">\n",
|
|
"<h1> Widzenie komputerowe </h1>\n",
|
|
"<h2> 10. <i>Metody głębokiego uczenia (2)</i> [laboratoria]</h2> \n",
|
|
"<h3>Andrzej Wójtowicz (2021)</h3>\n",
|
|
"</div>\n",
|
|
"\n",
|
|
""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "783d6d64",
|
|
"metadata": {},
|
|
"source": [
|
|
"W poniższym materiale zobaczymy w jaki sposób korzystać z wytrenowanych modeli sieci neuronowych w zagadnieniach związanych z wykrywaniem wielu obiektów, szacowaniem pozy człowieka, wykrywaniem i rozpoznawaniem tekstu oraz super rozdzielczością.\n",
|
|
"\n",
|
|
"Uwaga: realizacja poniższych treści będzie wymagała pobrania ok. 700 MB danych.\n",
|
|
"\n",
|
|
"Na początku załadujmy niezbędne biblioteki:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "ef18510f",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import cv2 as cv\n",
|
|
"import numpy as np\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"%matplotlib inline"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e45afc56",
|
|
"metadata": {},
|
|
"source": [
|
|
"Pobrane pliki będziemy zapisywać w katalogu `dnn`:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "7aac31ef",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"!mkdir -p dnn"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f792eb4f",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Wykrywanie obiektów\n",
|
|
"\n",
|
|
"## SSD\n",
|
|
"\n",
|
|
"W poprzednich materiałach korzystaliśmy z [SSD](https://arxiv.org/pdf/1512.02325.pdf) do wykrywania wielu twarzy na zdjęciu. W poniższym przykładzie możemy zobaczyć użycie do wykrywania wielu obiektów - sieć została wytrenowana na zbiorze [Common Objects in Context](https://cocodataset.org/) (COCO). Użyjemy modelu dostępnego dla frameworku [Tensorflow](https://github.com/tensorflow/models/tree/master/research/object_detection) (inne modele możemy znaleźć w [Detection Model Zoo](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf1_detection_zoo.md)):"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "aa10b6fa",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"!wget -q --show-progress -O dnn/ssd_mobilenet_v2_coco_2018_03_29.tar.gz http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v2_coco_2018_03_29.tar.gz\n",
|
|
"!cd dnn && tar xzf ssd_mobilenet_v2_coco_2018_03_29.tar.gz && rm ssd_mobilenet_v2_coco_2018_03_29.tar.gz"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "99ec1efa",
|
|
"metadata": {},
|
|
"source": [
|
|
"Pobraliśmy model i generujemy konfigurację:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "eac9a8da",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"!wget -q --show-progress -O dnn/ssd_mobilenet_v2_coco_2018_03_29/tf_text_graph_ssd.py https://raw.githubusercontent.com/opencv/opencv/4.5.3/samples/dnn/tf_text_graph_ssd.py\n",
|
|
"!wget -q --show-progress -O dnn/ssd_mobilenet_v2_coco_2018_03_29/tf_text_graph_common.py https://raw.githubusercontent.com/opencv/opencv/4.5.3/samples/dnn/tf_text_graph_common.py\n",
|
|
"!cd dnn/ssd_mobilenet_v2_coco_2018_03_29 && python3 tf_text_graph_ssd.py --input frozen_inference_graph.pb --output net.pbtxt --config pipeline.config"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "232e2987",
|
|
"metadata": {},
|
|
"source": [
|
|
"Wczytujemy model:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "9b4180e5",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"model = cv.dnn.readNetFromTensorflow(\"dnn/ssd_mobilenet_v2_coco_2018_03_29/frozen_inference_graph.pb\",\n",
|
|
" \"dnn/ssd_mobilenet_v2_coco_2018_03_29/net.pbtxt\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0bbfd2a4",
|
|
"metadata": {},
|
|
"source": [
|
|
"Pobieramy i wczytujemy etykiety klas obiektów:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "17335a42",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"!wget -q -O - https://raw.githubusercontent.com/tensorflow/models/master/research/object_detection/data/mscoco_complete_label_map.pbtxt | grep display_name | grep -o '\".*\"' | tr -d '\"' > dnn/ssd_mobilenet_v2_coco_2018_03_29/coco-labels.txt"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "662e1a33",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"with open('dnn/ssd_mobilenet_v2_coco_2018_03_29/coco-labels.txt', 'r') as f_fd:\n",
|
|
" classes = f_fd.read().splitlines()\n",
|
|
" \n",
|
|
"print(len(classes), classes[:5])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "94cace8a",
|
|
"metadata": {},
|
|
"source": [
|
|
"Spróbujemy sprawdzić jakie obiekty znajdują się na poniższym zdjęciu:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "91834aba",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"image = cv.imread('img/messi5.jpg')\n",
|
|
"plt.figure(figsize=[7,7])\n",
|
|
"plt.imshow(image[:,:,::-1]);"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "43774ae3",
|
|
"metadata": {},
|
|
"source": [
|
|
"Sieć zwraca nam listę obiektów z oznaczeniem współrzędnych na zdjęciu oraz identyfikatorem obiektu (ustawiliśmy próg odcięcia na 0.5):"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "84652c91",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"height, width, _ = image.shape\n",
|
|
"\n",
|
|
"image_blob = cv.dnn.blobFromImage(image=image, scalefactor=1, size=(300, 300), mean=(0,0,0), \n",
|
|
" swapRB=True, crop=False)\n",
|
|
"\n",
|
|
"model.setInput(image_blob)\n",
|
|
"detections = model.forward()\n",
|
|
"\n",
|
|
"image_out = image.copy()\n",
|
|
"\n",
|
|
"for i in range(detections.shape[2]):\n",
|
|
" confidence = detections[0, 0, i, 2]\n",
|
|
" if confidence > 0.5:\n",
|
|
"\n",
|
|
" box = detections[0, 0, i, 3:7] * np.array([width, height, width, height])\n",
|
|
" (x1, y1, x2, y2) = box.astype('int')\n",
|
|
" \n",
|
|
" class_id = int(detections[0, 0, i, 1])\n",
|
|
"\n",
|
|
" cv.rectangle(image_out, (x1, y1), (x2, y2), (0, 255, 0), 6)\n",
|
|
" label = '{:} ({:.3f})'.format(classes[class_id], confidence)\n",
|
|
" label_size, base_line = cv.getTextSize(label, cv.FONT_HERSHEY_SIMPLEX, 0.65, 1)\n",
|
|
" cv.rectangle(image_out, (x1, y1 - label_size[1]), (x1 + label_size[0], y1 + base_line), \n",
|
|
" (255, 255, 255), cv.FILLED)\n",
|
|
" cv.putText(image_out, label, (x1, y1), cv.FONT_HERSHEY_SIMPLEX, 0.65, (0, 0, 0))\n",
|
|
" \n",
|
|
"plt.figure(figsize=[12,12])\n",
|
|
"plt.imshow(image_out[:,:,::-1]);"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3fa16e91",
|
|
"metadata": {},
|
|
"source": [
|
|
"## YOLOv4"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "27ce3522",
|
|
"metadata": {},
|
|
"source": [
|
|
"Innym popularnym modelem do wykrywania obiektów jest [You Only Look Once](https://github.com/AlexeyAB/darknet) (YOLO). Porównując YOLO do innych sieci, model ten nie analizuje poszczególnych regionów, ale patrzy na obraz całościowo, co w pewien sposób stanowi balans między szybkością a precyzją. Ze względu na tę cechę model ten dobrze nadaje się do wykrywania obiektów w czasie rzeczywistym. Model powinien dobrze sobie radzić gdy zostanie mu przedstawiona nieznana wcześniej reprezentacja obiektu (np. zacieniony) lub gdy obiekt znajduje się w otoczeniu innych nieoczekiwanych obiektów.\n",
|
|
"\n",
|
|
"YOLO jest dostępne w kilku wersjach, natomiast my sprawdzimy jak sobie radzi wersja kompaktowa:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "4c3e7fb1",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"!mkdir -p dnn/yolo_v4_tiny\n",
|
|
"!wget -q --show-progress -O dnn/yolo_v4_tiny/yolov4-tiny.weights https://github.com/AlexeyAB/darknet/releases/download/yolov4/yolov4-tiny.weights\n",
|
|
"!wget -q --show-progress -O dnn/yolo_v4_tiny/yolov4-tiny.cfg https://raw.githubusercontent.com/AlexeyAB/darknet/master/cfg/yolov4-tiny.cfg\n",
|
|
"!wget -q --show-progress -O dnn/yolo_v4_tiny/coco.names https://raw.githubusercontent.com/AlexeyAB/darknet/master/cfg/coco.names"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9497b09c",
|
|
"metadata": {},
|
|
"source": [
|
|
"Wczytujemy model:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "e8cc6a3a",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"model = cv.dnn.readNetFromDarknet(\"dnn/yolo_v4_tiny/yolov4-tiny.cfg\", \n",
|
|
" \"dnn/yolo_v4_tiny/yolov4-tiny.weights\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "df331450",
|
|
"metadata": {},
|
|
"source": [
|
|
"Wczytujemy etykiety obiektów:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "8f01d354",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"with open('dnn/yolo_v4_tiny/coco.names', 'r') as f_fd:\n",
|
|
" classes = f_fd.read().splitlines()\n",
|
|
" \n",
|
|
"print(len(classes), classes[:5])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3fc5e3fc",
|
|
"metadata": {},
|
|
"source": [
|
|
"Przetestujemy działanie na poniższym zdjęciu:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "df65dee0",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"image = cv.imread('img/pedestrians.jpg')\n",
|
|
"plt.figure(figsize=[7,7])\n",
|
|
"plt.imshow(image[:,:,::-1]);"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9fbb6325",
|
|
"metadata": {},
|
|
"source": [
|
|
"Podczas korzystania z tego modelu musimy się zmierzyć z kilkoma subtelnościami. Model wykorzystuje framework Darknet, więc musimy wskazać, że chodzi nam o predykcje pochodzące z ostatniej warstwy. Dodatkowo mamy kilka progów odcięcia do zdefiniowania, tj. miarę obiektowości (*objectness*), pewności (*confidence*) oraz tłumienia niemaksymalnego aby ograniczyć występowanie nakładających się na siebie ramek z wykrytymi obiektami (por. [`cv.dnn.NMSBoxes()`](https://docs.opencv.org/4.5.3/d6/d0f/group__dnn.html#ga9d118d70a1659af729d01b10233213ee)). Poniżej mamy wynik działania:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "d8450888",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"height, width, _ = image.shape\n",
|
|
"\n",
|
|
"image_blob = cv.dnn.blobFromImage(image=image, scalefactor=1/255, size=(416, 416), mean=(0,0,0), \n",
|
|
" swapRB=True, crop=False)\n",
|
|
"\n",
|
|
"model.setInput(image_blob)\n",
|
|
"detections = model.forward([model.getLayerNames()[i[0] - 1] for i in model.getUnconnectedOutLayers()])\n",
|
|
"\n",
|
|
"image_out = image.copy()\n",
|
|
"\n",
|
|
"class_ids = []\n",
|
|
"confidences = []\n",
|
|
"boxes = []\n",
|
|
"\n",
|
|
"for out in detections:\n",
|
|
" for detection in out:\n",
|
|
" if detection[4] > 0.5: # objectness thr.\n",
|
|
" scores = detection[5:]\n",
|
|
" class_id = np.argmax(scores)\n",
|
|
" confidence = scores[class_id]\n",
|
|
" if confidence > 0.5: # confidence thr.\n",
|
|
" center_x = int(detection[0] * width)\n",
|
|
" center_y = int(detection[1] * height)\n",
|
|
" b_width = int(detection[2] * width)\n",
|
|
" b_height = int(detection[3] * height)\n",
|
|
"\n",
|
|
" b_left = int(center_x - b_width / 2)\n",
|
|
" b_top = int(center_y - b_height / 2)\n",
|
|
" class_ids.append(class_id)\n",
|
|
" confidences.append(float(confidence))\n",
|
|
" boxes.append([b_left, b_top, b_width, b_height])\n",
|
|
"\n",
|
|
"indices = cv.dnn.NMSBoxes(boxes, confidences, score_threshold=0.5, nms_threshold=0.5)\n",
|
|
"for i in indices:\n",
|
|
" idx = i[0]\n",
|
|
" box = boxes[idx]\n",
|
|
" x1 = box[0]\n",
|
|
" y1 = box[1]\n",
|
|
" x2 = box[0] + box[2]\n",
|
|
" y2 = box[1] + box[3]\n",
|
|
" cv.rectangle(image_out, (x1, y1), (x2, y2), (0, 255, 0), 6)\n",
|
|
" label = '{:} ({:.3f})'.format(classes[class_ids[idx]], confidences[idx])\n",
|
|
" \n",
|
|
" label_size, base_line = cv.getTextSize(label, cv.FONT_HERSHEY_SIMPLEX, 0.65, 1)\n",
|
|
" cv.rectangle(image_out, (x1, y1 - label_size[1]), (x1 + label_size[0], y1 + base_line), \n",
|
|
" (255, 255, 255), cv.FILLED)\n",
|
|
" cv.putText(image_out, label, (x1, y1), cv.FONT_HERSHEY_SIMPLEX, 0.65, (0, 0, 0))\n",
|
|
" \n",
|
|
"plt.figure(figsize=[12,12])\n",
|
|
"plt.imshow(image_out[:,:,::-1]);"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "41e32b8e",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Szacowanie pozy człowieka\n",
|
|
"\n",
|
|
"Kolejnym interesującym zagadnieniem jest szacowanie pozy człowieka (ang. *human pose estimation*) na podstawie zdjęcia. Celem jest tutaj wykrycie charakterystycznych punktów orientacyjnych, które mogą potem zostać wykorzystane np. treningu sportowego, kontroli gestów, korekcji postawy, itp. W tym celu wykorzystamy [OpenPose](https://github.com/CMU-Perceptual-Computing-Lab/openpose)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "6a3fedf2",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"!mkdir -p dnn/openpose\n",
|
|
"!wget -q --show-progress -O dnn/openpose/pose_iter_160000.caffemodel http://posefs1.perception.cs.cmu.edu/Users/tsimon/Projects/coco/data/models/mpi/pose_iter_160000.caffemodel\n",
|
|
"!wget -q --show-progress -O dnn/openpose/pose_deploy_linevec_faster_4_stages.prototxt https://raw.githubusercontent.com/CMU-Perceptual-Computing-Lab/openpose/master/models/pose/mpi/pose_deploy_linevec_faster_4_stages.prototxt"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "851c965b",
|
|
"metadata": {},
|
|
"source": [
|
|
"Wczytujemy model:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "d55edcb8",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"model = cv.dnn.readNetFromCaffe(\"dnn/openpose/pose_deploy_linevec_faster_4_stages.prototxt\",\n",
|
|
" \"dnn/openpose/pose_iter_160000.caffemodel\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c2c701c3",
|
|
"metadata": {},
|
|
"source": [
|
|
"Będziemy chcieli przeanalizować poniższe zdjęcie:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "aeaed6eb",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"image = cv.imread(\"img/messi5.jpg\")\n",
|
|
"plt.figure(figsize=[7,7])\n",
|
|
"plt.imshow(image[:,:,::-1]);"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "00a42dd5",
|
|
"metadata": {},
|
|
"source": [
|
|
"Zdefinujemy poniżej połączenia pomiędzy 15 punktami orientacyjnymi:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "894acae5",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"pose_points_n = 15\n",
|
|
"pose_pairs = [[0,1], [1,2], [2,3], [3,4], [1,5], [5,6], [6,7], [1,14], [14,8], [8,9], [9,10], [14,11], [11,12], [12,13]]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5a8a5028",
|
|
"metadata": {},
|
|
"source": [
|
|
"W wyniku otrzymujemy mapy prawodpodobieństwa występowania danego punktu orientacyjnego:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "24ca95c6",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"height, width, _ = image.shape\n",
|
|
"\n",
|
|
"image_blob = cv.dnn.blobFromImage(image, 1.0/255, (368, 368), (0, 0, 0), swapRB=False, crop=False)\n",
|
|
"model.setInput(image_blob)\n",
|
|
"\n",
|
|
"output = model.forward()\n",
|
|
"\n",
|
|
"plt.figure(figsize=(20,3))\n",
|
|
"for i in range(pose_points_n):\n",
|
|
" prob_map = output[0, i, :, :]\n",
|
|
" disp_map = cv.resize(prob_map, (width, height), cv.INTER_LINEAR)\n",
|
|
" plt.subplot(2, 8, i+1)\n",
|
|
" plt.axis('off')\n",
|
|
" plt.imshow(disp_map, cmap='jet', vmin=0, vmax=1)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c8be6dc1",
|
|
"metadata": {},
|
|
"source": [
|
|
"Przeskalowujemy wyniki do rozmiarów obrazu wejściowego i przy pomocy [`cv.minMaxLoc()`](https://docs.opencv.org/4.5.3/d2/de8/group__core__array.html#gab473bf2eb6d14ff97e89b355dac20707) znajdujemy wartość maksymalną (dodatkowo sprawdzamy czy wartość prawdopodobieństwa jest odpowiednio duża):"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "a3163987",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"scale_x = width / output.shape[3]\n",
|
|
"scale_y = height / output.shape[2]\n",
|
|
"\n",
|
|
"points = []\n",
|
|
"\n",
|
|
"for i in range(pose_points_n):\n",
|
|
" prob_map = output[0, i, :, :]\n",
|
|
" \n",
|
|
" _, prob, _, point = cv.minMaxLoc(prob_map)\n",
|
|
" \n",
|
|
" x = scale_x * point[0]\n",
|
|
" y = scale_y * point[1]\n",
|
|
"\n",
|
|
" if prob > 0.1: # thr.\n",
|
|
" points.append((int(x), int(y)))\n",
|
|
" else:\n",
|
|
" points.append(None)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f1f8cac3",
|
|
"metadata": {},
|
|
"source": [
|
|
"Możemy teraz nanieść punkty na obraz i połączyć je w szkielet"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "fcbda6c6",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"image_points = image.copy()\n",
|
|
"image_skeleton = image.copy()\n",
|
|
"\n",
|
|
"for i, p in enumerate(points):\n",
|
|
" cv.circle(image_points, p, 8, (255, 255, 0), thickness=-1, lineType=cv.FILLED)\n",
|
|
" cv.putText(image_points, \"{}\".format(i), p, cv.FONT_HERSHEY_SIMPLEX, 1, (0,255,255), 2, lineType=cv.LINE_AA)\n",
|
|
"\n",
|
|
"\n",
|
|
"for pair in pose_pairs:\n",
|
|
" part_a = pair[0]\n",
|
|
" part_b = pair[1]\n",
|
|
"\n",
|
|
" if points[part_a] and points[part_b]:\n",
|
|
" cv.line(image_skeleton, points[part_a], points[part_b], (0, 255, 255), 4)\n",
|
|
" cv.circle(image_skeleton, points[part_a], 7, (255, 255, 0), thickness=-1, lineType=cv.FILLED)\n",
|
|
"\n",
|
|
"plt.figure(figsize=(20,20))\n",
|
|
"plt.subplot(121)\n",
|
|
"plt.imshow(image_points[:,:,::-1])\n",
|
|
"plt.subplot(122)\n",
|
|
"plt.imshow(image_skeleton[:,:,::-1]);"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ea3421fd",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Wykrywanie i rozpoznawanie tekstu\n",
|
|
"\n",
|
|
"W kolejnym przykładzie zobaczymy jak możemy wykryć na zdjęciu tekst przy pomocy [DB](https://github.com/MhLiao/DB) oraz rozpoznać go przy pomocy [CRNN](https://arxiv.org/pdf/1507.05717.pdf)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "5ef81ed2",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import gdown\n",
|
|
"\n",
|
|
"for url, output in [('https://drive.google.com/uc?export=dowload&id=19YWhArrNccaoSza0CfkXlA8im4-lAGsR', 'dnn/DB_TD500_resnet50.onnx'), \n",
|
|
" ('https://drive.google.com/uc?export=dowload&id=12diBsVJrS9ZEl6BNUiRp9s0xPALBS7kt', 'dnn/crnn_cs.onnx'),\n",
|
|
" ('https://drive.google.com/uc?export=dowload&id=1oKXxXKusquimp7XY1mFvj9nwLzldVgBR', 'dnn/alphabet_94.txt')]:\n",
|
|
" gdown.download(url, output, quiet=False)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "72721bc5",
|
|
"metadata": {},
|
|
"source": [
|
|
"Będziemy pracować na poniższym zdjęciu:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "86e3f889",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"image = cv.imread('img/road-sign.jpg')\n",
|
|
"\n",
|
|
"plt.figure(figsize=(5,7))\n",
|
|
"plt.imshow(image[:,:,::-1]);"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ec7d3ce4",
|
|
"metadata": {},
|
|
"source": [
|
|
"Wczytujemy obsługiwany alfabet:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "5d27f129",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"with open('dnn/alphabet_94.txt', 'r') as f_fd:\n",
|
|
" alphabet = f_fd.read().splitlines()\n",
|
|
" \n",
|
|
"print(len(alphabet), alphabet[:15])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d3373c60",
|
|
"metadata": {},
|
|
"source": [
|
|
"OpenCV posiada gotowe API dla sieci DB poprzez [`cv.dnn_TextDetectionModel_DB()`](https://docs.opencv.org/4.5.3/db/d0f/classcv_1_1dnn_1_1TextDetectionModel__DB.html):"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "b3c3bfc2",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"text_detector = cv.dnn_TextDetectionModel_DB(\"dnn/DB_TD500_resnet50.onnx\")\n",
|
|
"\n",
|
|
"text_detector.setBinaryThreshold(0.4).setPolygonThreshold(0.5)\n",
|
|
"text_detector.setInputParams(scale=1.0/255, size=(640, 640), \n",
|
|
" mean=(122.67891434, 116.66876762, 104.00698793), swapRB=True)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "31300a5f",
|
|
"metadata": {},
|
|
"source": [
|
|
"W wyniku otrzymujemy ramki, na których występuje tekst (choć jak widzimy, są też wyniki fałszywie pozytywne):"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "d14502d4",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"boxes, confs = text_detector.detect(image)\n",
|
|
"\n",
|
|
"image_out = image.copy()\n",
|
|
"\n",
|
|
"cv.polylines(image_out, boxes, True, (255, 0, 255), 4)\n",
|
|
"\n",
|
|
"plt.figure(figsize=(5,7))\n",
|
|
"plt.imshow(image_out[:,:,::-1]);"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3c3eae71",
|
|
"metadata": {},
|
|
"source": [
|
|
"W kolejnym kroku przygotowujemy model do rozpoznawania tekstu przy pomocy [`cv.dnn_TextRecognitionModel()`](https://docs.opencv.org/4.5.3/de/dee/classcv_1_1dnn_1_1TextRecognitionModel.html):"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "d6b29f6a",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"text_recognizer = cv.dnn_TextRecognitionModel(\"dnn/crnn_cs.onnx\")\n",
|
|
"text_recognizer.setDecodeType(\"CTC-greedy\")\n",
|
|
"text_recognizer.setVocabulary(alphabet)\n",
|
|
"text_recognizer.setInputParams(scale=1/127.5, size=(100, 32), mean=(127.5, 127.5, 127.5), swapRB=True)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a17f6437",
|
|
"metadata": {},
|
|
"source": [
|
|
"Każdą wykrytą ramkę rzutujemy na rozmiar 100x32 i wykrywamy tekst:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "d6909f83",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"for box in boxes:\n",
|
|
" vertices = np.asarray(box).astype(np.float32)\n",
|
|
" output_size = (100, 32)\n",
|
|
" target_vertices = np.array([\n",
|
|
" [0, output_size[1] - 1],\n",
|
|
" [0, 0],\n",
|
|
" [output_size[0] - 1, 0],\n",
|
|
" [output_size[0] - 1, output_size[1] - 1]],\n",
|
|
" dtype=\"float32\")\n",
|
|
" rotation_matrix = cv.getPerspectiveTransform(vertices, target_vertices)\n",
|
|
" cropped_roi = cv.warpPerspective(image, rotation_matrix, output_size)\n",
|
|
" \n",
|
|
" result = text_recognizer.recognize(cropped_roi)\n",
|
|
" print(result)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e0b4b3c0",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Super rozdzielczość\n",
|
|
"\n",
|
|
"Podczas zwiększania rozdzielczości brakujące piksele muszą być w jakiś sposób interpolowane. Przy niewielkich powiększeniach zwykle wystarczą nam tradycyjne metody, jednak jeśli pracujemy z obrazem w niskiej rozdzielczości i chcemy go znacząco powiększyć, to chcielibyśmy również uzyskać wysoką jakość np. poprzez uwzględnienie informacji z otoczenia pikseli. Problematyka ta dotyczy zagadnienia super rozdzielczości (ang. *super-resolution*).\n",
|
|
"\n",
|
|
"W [artykule](https://arxiv.org/pdf/1902.06068.pdf) z 2020 r. możemy znaleźć porównanie dostępnych w tamtym czasie modeli (zob. wykres na str. 15); np. możemy zobaczyć, że model [EDSR](https://github.com/Saafke/EDSR_Tensorflow) radzi sobie całkiem nieźle, aczkolwiek kosztem sporego narzutu obliczeniowego (por. również benchmarki [OpenCV](https://github.com/opencv/opencv_contrib/blob/master/modules/dnn_superres/README.md)). Przetestujemy EDSR na powiększeniu 4-krotnym:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "6b9f6be9",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"!wget -q --show-progress -O dnn/EDSR_x4.pb https://raw.githubusercontent.com/Saafke/EDSR_Tensorflow/master/models/EDSR_x4.pb"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "92f09e43",
|
|
"metadata": {},
|
|
"source": [
|
|
"Przy pomocy [`cv.dnn_superres.DnnSuperResImpl_create()`](https://docs.opencv.org/4.5.3/d8/d11/classcv_1_1dnn__superres_1_1DnnSuperResImpl.html) przygotowujemy model:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "368ca179",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"sr = cv.dnn_superres.DnnSuperResImpl_create()\n",
|
|
"sr.readModel('dnn/EDSR_x4.pb')\n",
|
|
"sr.setModel('edsr', 4)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9e0169f3",
|
|
"metadata": {},
|
|
"source": [
|
|
"Następnie zwiększy rozdzielczość zadanego obrazu (operacja może zająć trochę czasu):"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "45c89529",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"image = cv.imread('img/parrot.jpg')\n",
|
|
"\n",
|
|
"image_EDSR = sr.upsample(image)\n",
|
|
"\n",
|
|
"plt.figure(figsize=(25,25))\n",
|
|
"plt.subplot(211)\n",
|
|
"plt.imshow(image[:,:,::-1])\n",
|
|
"plt.subplot(212)\n",
|
|
"plt.imshow(image_EDSR[:,:,::-1]);"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3c157587",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Zadanie 1\n",
|
|
"\n",
|
|
"Przy pomocy biblioteki [MediaPipe](https://google.github.io/mediapipe/solutions/selfie_segmentation.html) dokonaj podmień tło w selfie `img/selfie-man.jpg` na `img/selfie-background.jpg` (możesz również odbić obraz w poziomie)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d6116f61",
|
|
"metadata": {},
|
|
"source": [
|
|
""
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"author": "Andrzej Wójtowicz",
|
|
"email": "andre@amu.edu.pl",
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"lang": "pl",
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.7.3"
|
|
},
|
|
"subtitle": "10. Metody głębokiego uczenia w widzeniu komputerowym [laboratoria]",
|
|
"title": "Widzenie komputerowe",
|
|
"year": "2021"
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|