2024-02-23 19:22:37 +01:00
{
"cells": [
{
"cell_type": "code",
2024-08-10 15:25:51 +02:00
"execution_count": 1,
2024-02-23 19:22:37 +01:00
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# For tips on running notebooks in Google Colab, see\n",
"# https://pytorch.org/tutorials/beginner/colab\n",
"%matplotlib inline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# Neural Transfer Using PyTorch\n",
"\n",
"\n",
"**Author**: [Alexis Jacq](https://alexis-jacq.github.io)\n",
" \n",
"**Edited by**: [Winston Herring](https://github.com/winston6)\n",
"\n",
"## Introduction\n",
"\n",
"This tutorial explains how to implement the [Neural-Style algorithm](https://arxiv.org/abs/1508.06576)_\n",
"developed by Leon A. Gatys, Alexander S. Ecker and Matthias Bethge.\n",
"Neural-Style, or Neural-Transfer, allows you to take an image and\n",
"reproduce it with a new artistic style. The algorithm takes three images,\n",
"an input image, a content-image, and a style-image, and changes the input\n",
"to resemble the content of the content-image and the artistic style of the style-image.\n",
"\n",
" \n",
".. figure:: /_static/img/neural-style/neuralstyle.png\n",
" :alt: content1\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Underlying Principle\n",
"\n",
"The principle is simple: we define two distances, one for the content\n",
"($D_C$) and one for the style ($D_S$).
"is between two images while
"between two images. Then, we take a third image, the input, and\n",
"transform it to minimize both its content-distance with the\n",
"content-image and its style-distance with the style-image. Now we can\n",
"import the necessary packages and begin the neural transfer.\n",
"\n",
"## Importing Packages and Selecting a Device\n",
"Below is a list of the packages needed to implement the neural transfer.\n",
"\n",
"- ``torch``, ``torch.nn``, ``numpy`` (indispensables packages for\n",
" neural networks with PyTorch)\n",
"- ``torch.optim`` (efficient gradient descents)\n",
"- ``PIL``, ``PIL.Image``, ``matplotlib.pyplot`` (load and display\n",
" images)\n",
"- ``torchvision.transforms`` (transform PIL images into tensors)\n",
"- ``torchvision.models`` (train or load pretrained models)\n",
"- ``copy`` (to deep copy the models; system package)\n",
"\n"
]
},
{
"cell_type": "code",
2024-08-10 15:25:51 +02:00
"execution_count": 2,
2024-02-23 19:22:37 +01:00
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import torch\n",
"import torch.nn as nn\n",
"import torch.nn.functional as F\n",
"import torch.optim as optim\n",
"\n",
"from PIL import Image\n",
"import matplotlib.pyplot as plt\n",
"\n",
"import torchvision.transforms as transforms\n",
"from torchvision.models import vgg19, VGG19_Weights\n",
"\n",
"import copy"
]
},
{
"cell_type": "code",
2024-08-10 15:25:51 +02:00
"execution_count": 3,
2024-02-23 19:22:37 +01:00
"metadata": {},
2024-08-10 15:25:51 +02:00
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
2024-02-23 19:22:37 +01:00
"source": [
2024-08-10 15:25:51 +02:00
"torch.cuda.is_available()"
2024-02-23 19:22:37 +01:00
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next, we need to choose which device to run the network on and import the\n",
"content and style images. Running the neural transfer algorithm on large\n",
"images takes longer and will go much faster when running on a GPU. We can\n",
"use ``torch.cuda.is_available()`` to detect if there is a GPU available.\n",
"Next, we set the ``torch.device`` for use throughout the tutorial. Also the ``.to(device)``\n",
"method is used to move tensors or modules to a desired device. \n",
"\n"
]
},
{
"cell_type": "code",
2024-08-10 15:25:51 +02:00
"execution_count": 4,
2024-02-23 19:22:37 +01:00
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
"torch.set_default_device(device)"
]
},
2024-08-10 15:25:51 +02:00
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"device(type='cuda')"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"device"
]
},
2024-02-23 19:22:37 +01:00
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Loading the Images\n",
"\n",
"Now we will import the style and content images. The original PIL images have values between 0 and 255, but when\n",
"transformed into torch tensors, their values are converted to be between\n",
"0 and 1. The images also need to be resized to have the same dimensions.\n",
"An important detail to note is that neural networks from the\n",
"torch library are trained with tensor values ranging from 0 to 1. If you\n",
"try to feed the networks with 0 to 255 tensor images, then the activated\n",
"feature maps will be unable to sense the intended content and style.\n",
"However, pretrained networks from the Caffe library are trained with 0\n",
"to 255 tensor images. \n",
"\n",
"\n",
"<div class=\"alert alert-info\"><h4>Note</h4><p>Here are links to download the images required to run the tutorial:\n",
" [picasso.jpg](https://pytorch.org/tutorials/_static/img/neural-style/picasso.jpg)_ and\n",
" [dancing.jpg](https://pytorch.org/tutorials/_static/img/neural-style/dancing.jpg)_.\n",
" Download these two images and add them to a directory\n",
" with name ``images`` in your current working directory.</p></div>\n",
"\n"
]
},
{
"cell_type": "code",
2024-08-11 20:25:35 +02:00
"execution_count": 48,
2024-02-23 19:22:37 +01:00
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# desired size of the output image\n",
"imsize = 512 if torch.cuda.is_available() else 128 # use small size if no GPU\n",
"\n",
"loader = transforms.Compose([\n",
" transforms.Resize(imsize), # scale imported image\n",
" transforms.ToTensor()]) # transform it into a torch tensor\n",
"\n",
"\n",
"def image_loader(image):\n",
" #image = Image.open(image_name)\n",
" # fake batch dimension required to fit network's input dimensions\n",
" image = loader(image).unsqueeze(0)\n",
" return image.to(device, torch.float)\n",
"\n",
"style_img = Image.open(\"images/chelm.jpg\")\n",
"content_img = Image.open(\"images/owce.jpg\")\n",
"\n",
"style_img = style_img.resize(content_img.size)\n",
"style_img = image_loader(style_img)\n",
"content_img = image_loader(content_img)\n",
"\n",
"assert style_img.size() == content_img.size(), \\\n",
" \"we need to import style and content images of the same size\"\n"
]
},
2024-08-10 15:25:51 +02:00
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"512"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"imsize"
]
},
2024-02-23 19:22:37 +01:00
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, let's create a function that displays an image by reconverting a \n",
"copy of it to PIL format and displaying the copy using \n",
"``plt.imshow``. We will try displaying the content and style images \n",
"to ensure they were imported correctly.\n",
"\n"
]
},
{
"cell_type": "code",
2024-08-11 20:25:35 +02:00
"execution_count": 8,
2024-02-23 19:22:37 +01:00
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAAFZCAYAAACym9R8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOz9e7CtWVUejD9jznftfW59+ko33U2DiAjRL5/4AyQmRrxgAfFSSiwlMQXitVLBqGhMqJQSUbGiifGaS8VUsKSNlheopH4RE5F8/ioi8RIJ+QwJQoOCdAPd9PWcs/d65xy/P8bzjDnXPg00xtg0rAmnzz5rr/Wu951zzDGeMcYzxjR3d+zHfuzHfuzHfuzHfnwUjfJw38B+7Md+7Md+7Md+7MfJsQco+7Ef+7Ef+7Ef+/FRN/YAZT/2Yz/2Yz/2Yz8+6sYeoOzHfuzHfuzHfuzHR93YA5T92I/92I/92I/9+Kgbe4CyH/uxH/uxH/uxHx91Yw9Q9mM/9mM/9mM/9uOjbuwByn7sx37sx37sx3581I09QNmP/diP/diP/diPj7qxByj7sR/78b81/sE/+Acws4f7NvZjP/bjY2zsAcp+7MfH6Hjzm9+ML//yL8fjHvc4nDp1CjfffDO+4Au+AD/2Yz+2875XvOIVeM1rXvPw3OSDjFe+8pUwM/z2b//2w30r+7Ef+/Ewjj1A2Y/9+Bgcv/Ebv4GnPe1peNOb3oSv//qvx4//+I/j677u61BKwY/8yI/svPejDaDsx37sx34AwPJw38B+7Md+/OmP7/u+78OVV16J3/qt38JVV12187v3vve9D89N7cd+7Md+fARjH0HZj/34GBxve9vb8Kmf+qmXgRMAuP766/NnM8MDDzyAn/qpn4KZwczw1V/91Xj9618PM8OrX/3qyz7/Mz/zMzAzvOENb/iQ9/CqV70KT33qU3H69Glcc801eP7zn48/+qM/+hM9z1d/9Vfj3Llz+MM//EN80Rd9Ec6dO4ebb74ZP/ETPwEg0lmf93mfh7Nnz+Jxj3scfuZnfmbn83fddRe+/du/HX/+z/95nDt3DufPn8dzn/tcvOlNb7rsu975znfiS77kS3D27Flcf/31+NZv/Vb8yq/8CswM/+k//aed977xjW/Ec57zHFx55ZU4c+YMnvnMZ+I//+f//Cd6xv3Yj/3YHXuAsh/78TE4Hve4x+F3fud38N//+3//kO/76Z/+aRweHuIv/+W/jJ/+6Z/GT//0T+Mbv/Eb8Tmf8zm45ZZbcOutt172mVtvvRVPeMIT8Jmf+Zkf9Lrf933fhxe84AV44hOfiB/6oR/Ct3zLt+B1r3sdPvuzPxt33333n+iZWmt47nOfi1tuuQU/8AM/gE/4hE/Ai1/8Yrzyla/Ec57zHDztaU/DP/yH/xBXXHEFXvCCF+C2227Lz7797W/Ha17zGnzRF30RfuiHfgh/5+/8Hbz5zW/GM5/5TPzxH/9xvu+BBx7A533e5+FXf/VX8bf/9t/G3//7fx+/8Ru/gb/7d//uZffza7/2a/jsz/5s3HvvvXjZy16GV7ziFbj77rvxeZ/3efgv/+W//ImecT/2Yz+m4fuxH/vxMTf+w3/4D15r9Vqrf+ZnfqZ/x3d8h//Kr/yKHx8fX/bes2fP+gtf+MLLXn/pS1/qh4eHfvfdd+dr733ve31ZFn/Zy16Wr73sZS/zWZW84x3v8Fqrf9/3fd/O9d785jf7siyXvX5y/Ot//a8dgP/Wb/1WvvbCF77QAfgrXvGKfO0DH/iAnz592s3Mf/ZnfzZff8tb3uIAdu7x0qVL3lrb+Z7bbrvNDw8P/eUvf3m+9o//8T92AP6a17wmX7t48aI/+clPdgD++te/3t3de+/+xCc+0Z/97Gd77z3fe+HCBX/84x/vX/AFX/Ahn3E/9mM/PvzYR1D2Yz8+BscXfMEX4A1veAO+5Eu+BG9605vwAz/wA3j2s5+Nm2++Gf/23/7bh3SNF7zgBTg6OsIv/MIv5Gs/93M/h3Vd8Tf+xt/4oJ/7pV/6JfTe8RVf8RV4//vfn38e/ehH44lPfCJe//rX/4mf6+u+7uvy56uuugpPetKTcPbsWXzFV3xFvv6kJz0JV111Fd7+9rfna4eHhygl1F1rDXfeeSfOnTuHJz3pSfjd3/3dfN9rX/ta3HzzzfiSL/mSfO3UqVP4+q//+p37+L3f+z289a1vxV//638dd955Zz7jAw88gM///M/Hr//6r6P3/id+zv3Yj/3Yk2T3Yz8+ZsfTn/50/NIv/RKOj4/xpje9Ca9+9avxT/7JP8GXf/mX4/d+7/fwKZ/yKR/y809+8pPx9Kc/Hbfeeiu+9mu/FkCkd/7CX/gL+KRP+qQP+rm3vvWtcHc88YlPfNDfbzabP9HznDp1Co961KN2XrvyyivxmMc85rI+LFdeeSU+8IEP5L977/iRH/kR/NN/+k9x2223obWWv7v22mvz53e+8514whOecNn1Tj7vW9/6VgDAC1/4wg96v/fccw+uvvrqh/h0+7Ef+3Fy7AHKfuzHx/g4ODjA05/+dDz96U/HJ3/yJ+NFL3oRfv7nfx4ve9nLPuxnX/CCF+Cbv/mb8a53vQtHR0f4zd/8Tfz4j//4h/xM7x1mhl/+5V9GrfWy3587d+5P9BwPdq0P9bq758+veMUr8J3f+Z34mq/5GnzP93wPrrnmGpRS8C3f8i1/okiHPvODP/iDeMpTnvKg7/mTPud+7Md+xNgDlP3Yj4+j8bSnPQ0A8J73vCdf+1BdYJ///OfjJS95Cf7Nv/k3uHjxIjabDb7yK7/yQ37HE57wBLg7Hv/4x+OTP/mT/3Ru/H9z/MIv/AI+93M/F//qX/2rndfvvvtuXHfddfnvxz3ucfj93/99uPvOvPzBH/zBzuee8IQnAADOnz+PZz3rWf8H73w/9uPjd+w5KPuxHx+D4/Wvf/1OBEHj3//7fw8geBoaZ8+e/aCVNddddx2e+9zn4lWvehVuvfVWPOc5z9kx6A82nve856HWiu/+7u++7B7cHXfeeedH+DT/+6PWetm9/PzP/zze/e5377z27Gc/G+9+97t3eDqXLl3Cv/yX/3LnfU996lPxhCc8Af/oH/0j3H///Zd93/ve974/xbvfj/34+Bz7CMp+7MfH4Pimb/omXLhwAV/2ZV+GJz/5yTg+PsZv/MZv4Od+7ufwCZ/wCXjRi16U733qU5+KX/3VX8UP/dAP4aabbsLjH/94POMZz8jfv+AFL8CXf/mXAwC+53u+58N+9xOe8AR87/d+L1760pfiHe94B770S78UV1xxBW677Ta8+tWvxjd8wzfg27/92//0H/pDjC/6oi/Cy1/+crzoRS/CX/yLfxFvfvObceutt+ITP/ETd973jd/4jfjxH/9x/LW/9tfwzd/8zbjxxhtx66234tSpUwBGtKmUgp/8yZ/Ec5/7XHzqp34qXvSiF+Hmm2/Gu9/9brz+9a/H+fPn8e/+3b/7M33G/diPj7nx8BUQ7cd+7Mf/qfHLv/zL/jVf8zX+5Cc/2c+dO+cHBwf+SZ/0Sf5N3/RNfscdd+y89y1veYt/9md/tp8+fdoBXFZyfHR05FdffbVfeeWVfvHixcu+62SZscYv/uIv+md91mf52bNn/ezZs/7kJz/Z/9bf+lv+P//n//yQ9/7ByozPnj172Xuf+cxn+qd+6qde9vrjHvc4/8Iv/ML896VLl/zbvu3b/MYbb/TTp0/7X/pLf8nf8IY3+DOf+Ux/5jOfufPZt7/97f6FX/iFfvr0aX/Uox7l3/Zt3+a/+Iu/6AD8N3/zN3fe+1//63/15z3veX7ttdf64eGhP+5xj/Ov+Iqv8Ne97nUf8hn3Yz/248MPc3+QOPB+7Md+7AfHuq646aab8MVf/MWXcTg+XsYP//AP41u/9Vvxrne9CzfffPPDfTv7sR8fF2PPQdmP/diPDzle85rX4H3vex9e8IIXPNy38mcyLl68uPPvS5cu4V/8i3+BJz7xiXtwsh/78Wc49hyU/diP/XjQ8cY3vhH/7b/9N3zP93wPPv3TPx3PfOYzH+5b+
"text/plain": [
"<Figure size 640x480 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAAFZCAYAAACym9R8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOz9e7RlWVUfjn/m2vuccx9Vt6qru6qr3w8alQYEbB62gIAiHQW/gmgkMdokgokDSAyiEYe8jIIYiaghEk2Gr0jiwKgxEMnPAShB+osRvxDtpoFuGrqbpqqrux637uucvfeavz/mmmutvc8+5+zzug84s0f1vXfvtdd7zfmZj7UWMTNjQQta0IIWtKAFLWgfkdnrCixoQQta0IIWtKAFVWkBUBa0oAUtaEELWtC+owVAWdCCFrSgBS1oQfuOFgBlQQta0IIWtKAF7TtaAJQFLWhBC1rQgha072gBUBa0oAUtaEELWtC+owVAWdCCFrSgBS1oQfuOFgBlQQta0IIWtKAF7TtaAJQFLWhBC1rQgha072gBUBa0oAUtaEELWtC+owVAWdCCDhDde++9+Kf/9J/ixhtvxNLSEtbW1vDMZz4Tv/zLv4zt7e25lXvXXXfhzW9+M77whS/MrQylra0tvPnNb8af//mfN0r/53/+5yAi/MEf/MF8K7agBS1oVynd6wosaEELakbvf//78b3f+73odDr4wR/8QTzhCU9Ar9fDRz/6Ufz4j/847rzzTvz6r//6XMq+66678Ja3vAXPfe5zcf3118+lDKWtrS285S1vAQA897nPnWtZC1rQgvYvLQDKghZ0AOi+++7Dy172Mlx33XX40Ic+hCuuuMK/e9WrXoV77rkH73//+/ewhgta0IIWNFtauHgWtKADQL/wC7+AjY0N/Kf/9J9K4ETppptuwr/4F//C/53nOf71v/7XeMxjHoNOp4Prr78eP/VTP4Vut1v67vrrr8eLXvQifPSjH8XTn/50LC0t4cYbb8Tv/M7v+DS/9Vu/he/93u8FADzvec8DEYGISi6YP/3TP8Wzn/1srK6u4vDhw3jhC1+IO++8s1TWy1/+chw6dAhf+tKX8OIXvxiHDh3C8ePH8brXvQ5FUQAAvvCFL+D48eMAgLe85S2+rDe/+c1j9deb3/xmEBE++9nP4h/9o3+EI0eO4Pjx43jDG94AZsYDDzyA7/qu78La2hpOnjyJd7zjHaXve70e3vjGN+KWW27BkSNHsLq6imc/+9n48Ic/3FfWo48+ih/4gR/A2toajh49ittvvx2f+tSnQET4rd/6rVLau+++G9/zPd+DY8eOYWlpCU996lPxJ3/yJ2O1bUEL+mqhBUBZ0IIOAP2P//E/cOONN+KbvumbGqV/xStegTe+8Y34hm/4BvzSL/0SnvOc5+Btb3sbXvayl/Wlveeee/A93/M9+LZv+za84x3vwCWXXIKXv/zlHmB88zd/M/75P//nAICf+qmfwu/+7u/id3/3d/G4xz0OAPC7v/u7eOELX4hDhw7h7W9/O97whjfgrrvuwrOe9ay+mJWiKHDbbbfh0ksvxS/+4i/iOc95Dt7xjnd419Tx48fxa7/2awCAl7zkJb6s7/7u756o377v+74P1lr8/M//PJ7xjGfgZ3/2Z/HOd74T3/Zt34arrroKb3/723HTTTfhda97HT7ykY/479bX1/Ef/+N/xHOf+1y8/e1vx5vf/GacOXMGt912Gz75yU/6dNZafOd3fif+y3/5L7j99tvxcz/3c/jyl7+M22+/va8ud955J77xG78Rn/70p/GTP/mTeMc73oHV1VW8+MUvxh/90R9N1L4FLegrmnhBC1rQvqYLFy4wAP6u7/quRuk/+clPMgB+xSteUXr+ute9jgHwhz70If/suuuuYwD8kY98xD97+OGHudPp8I/92I/5Z+9973sZAH/4wx8u5Xnx4kU+evQov/KVryw9P3XqFB85cqT0/Pbbb2cA/DM/8zOltE95ylP4lltu8X+fOXOGAfCb3vSmRu398Ic/zAD4ve99r3/2pje9iQHwD//wD/tneZ7z1VdfzUTEP//zP++fnzt3jpeXl/n2228vpe12u6Vyzp07x5dffjn/k3/yT/yz//bf/hsD4He+853+WVEU/C3f8i0MgH/zN3/TP//Wb/1WfuITn8g7Ozv+mbWWv+mbvokf+9jHNmrrghb01UQLC8qCFrTPaX19HQBw+PDhRun/5//8nwCA1772taXnP/ZjPwYAfbEqN998M5797Gf7v48fP46v/dqvxec///mRZf3Zn/0Zzp8/j3/wD/4BHnnkEf8vSRI84xnPqHWJ/LN/9s9Kfz/72c9uVNYk9IpXvML/niQJnvrUp4KZ8UM/9EP++dGjR/vamyQJ2u02ALGSnD17Fnme46lPfSr+5m/+xqf7wAc+gFarhVe+8pX+mTEGr3rVq0r1OHv2LD70oQ/h7//9v4+LFy/6fnr00Udx22234XOf+xy+9KUvzbz9C1rQQaZFkOyCFrTPaW1tDQBw8eLFRum/+MUvwhiDm266qfT85MmTOHr0KL74xS+Wnl977bV9eVxyySU4d+7cyLI+97nPAQC+5Vu+ZWjdlZaWlnyMybhlTULVth05cgRLS0u47LLL+p4/+uijpWe//du/jXe84x24++67kWWZf37DDTf437/4xS/iiiuuwMrKSunbat/fc889YGa84Q1vwBve8Ibauj788MO46qqrmjduQQv6CqcFQFnQgvY5ra2t4corr8Tf/d3fjfUdETVKlyRJ7XNmHvmttRaAxKGcPHmy732allnMoLLmRXXlNWnvf/7P/xkvf/nL8eIXvxg//uM/jhMnTiBJErztbW/DvffeO3Y9tJ9e97rX4bbbbqtNUwU1C1rQVzstAMqCFnQA6EUvehF+/dd/HXfccQduvfXWoWmvu+46WGvxuc99zgeyAsDp06dx/vx5XHfddWOXPwjsPOYxjwEAnDhxAs9//vPHznecsnaT/uAP/gA33ngj/vAP/7BUnze96U2ldNdddx0+/OEPY2trq2RFueeee0rpbrzxRgBAq9WaWT8taEFf6bSIQVnQgg4A/cRP/ARWV1fxile8AqdPn+57f++99+KXf/mXAQDf8R3fAQB45zvfWUrzb//tvwUAvPCFLxy7/NXVVQDA+fPnS89vu+02rK2t4a1vfWvJDaJ05syZsctSQV8tazdJrSyxVeXjH/847rjjjlK62267DVmW4Td+4zf8M2st3vWud5XSnThxAs997nPxH/7Df8CXv/zlvvIm6acFLegrnRYWlAUt6ADQYx7zGLznPe/B933f9+Fxj3tc6STZj33sY3jve9+Ll7/85QCAJz3pSbj99tvx67/+6zh//jye85zn4K/+6q/w27/923jxi1+M5z3veWOX/+QnPxlJkuDtb387Lly4gE6ng2/5lm/BiRMn8Gu/9mv4gR/4AXzDN3wDXvayl+H48eO4//778f73vx/PfOYz8e/+3b8bq6zl5WXcfPPN+P3f/318zdd8DY4dO4YnPOEJeMITnjB2vSelF73oRfjDP/xDvOQlL8ELX/hC3HfffXj3u9+Nm2++GRsbGz7di1/8Yjz96U/Hj/3Yj+Gee+7B133d1+FP/uRPcPbsWQBla9C73vUuPOtZz8ITn/hEvPKVr8SNN96I06dP44477sCDDz6IT33qU7vWvgUt6EDQnu4hWtCCFjQWffazn+VXvvKVfP3113O73ebDhw/zM5/5TP7VX/3V0vbVLMv4LW95C99www3carX4mmuu4de//vWlNMyyzfiFL3xhXznPec5z+DnPeU7p2W/8xm/wjTfeyEmS9G05/vCHP8y33XYbHzlyhJeWlvgxj3kMv/zlL+e//uu/9mluv/12Xl1d7StLtwTH9LGPfYxvueUWbrfbI7ccD9tmfObMmVLaQXV4znOew49//OP939Zafutb38rXXXcddzodfspTnsLve9/7+Pbbb+frrruu9O2ZM2f4H/7Df8iHDx/mI0eO8Mtf/nL+y7/8SwbA//W//tdS2nvvvZd/8Ad/kE+ePMmtV
"text/plain": [
"<Figure size 640x480 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"unloader = transforms.ToPILImage() # reconvert into PIL image\n",
"\n",
"plt.ion()\n",
"\n",
"def imshow(tensor, title=None):\n",
" image = tensor.cpu().clone() # we clone the tensor to not do changes on it\n",
" image = image.squeeze(0) # remove the fake batch dimension\n",
" image = unloader(image)\n",
" plt.imshow(image)\n",
" if title is not None:\n",
" plt.title(title)\n",
" plt.pause(0.001) # pause a bit so that plots are updated\n",
"\n",
"\n",
"plt.figure()\n",
"imshow(style_img, title='Style Image')\n",
"\n",
"plt.figure()\n",
"imshow(content_img, title='Content Image')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Loss Functions\n",
"Content Loss\n",
"\n",
"\n",
"The content loss is a function that represents a weighted version of the\n",
"content distance for an individual layer. The function takes the feature\n",
"maps $F_{XL}$ of a layer $L$ in a network processing input $X$ and returns the\n",
"weighted content distance $w_{CL}.D_C^L(X,C)$ between the image $X$ and the\n",
"content image $C$. The feature maps of the content image($F_{CL}$) must be\n",
"known by the function in order to calculate the content distance. We\n",
"implement this function as a torch module with a constructor that takes\n",
"$F_{CL}$ as an input. The distance $\\|F_{XL} - F_{CL}\\|^2$ is the mean square error\n",
"between the two sets of feature maps, and can be computed using ``nn.MSELoss``.\n",
"\n",
"We will add this content loss module directly after the convolution\n",
"layer(s) that are being used to compute the content distance. This way\n",
"each time the network is fed an input image the content losses will be\n",
"computed at the desired layers and because of auto grad, all the\n",
"gradients will be computed. Now, in order to make the content loss layer\n",
"transparent we must define a ``forward`` method that computes the content\n",
"loss and then returns the layer’
"parameter of the module.\n",
"\n",
"\n"
]
},
{
"cell_type": "code",
2024-08-11 20:25:35 +02:00
"execution_count": 9,
2024-02-23 19:22:37 +01:00
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"class ContentLoss(nn.Module):\n",
"\n",
" def __init__(self, target,):\n",
" super(ContentLoss, self).__init__()\n",
" # we 'detach' the target content from the tree used\n",
" # to dynamically compute the gradient: this is a stated value,\n",
" # not a variable. Otherwise the forward method of the criterion\n",
" # will throw an error.\n",
" self.target = target.detach()\n",
"\n",
" def forward(self, input):\n",
" self.loss = F.mse_loss(input, self.target)\n",
" return input"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert alert-info\"><h4>Note</h4><p>**Important detail**: although this module is named ``ContentLoss``, it\n",
" is not a true PyTorch Loss function. If you want to define your content\n",
" loss as a PyTorch Loss function, you have to create a PyTorch autograd function \n",
" to recompute/implement the gradient manually in the ``backward``\n",
" method.</p></div>\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Style Loss\n",
"\n",
"The style loss module is implemented similarly to the content loss\n",
"module. It will act as a transparent layer in a\n",
"network that computes the style loss of that layer. In order to\n",
"calculate the style loss, we need to compute the gram matrix $G_{XL}$. A gram\n",
"matrix is the result of multiplying a given matrix by its transposed\n",
"matrix. In this application the given matrix is a reshaped version of\n",
"the feature maps $F_{XL}$ of a layer $L$. $F_{XL}$ is reshaped to form $\\hat{F}_{XL}$, a $K$\\ x\\ $N$\n",
"matrix, where $K$ is the number of feature maps at layer $L$ and $N$ is the\n",
"length of any vectorized feature map $F_{XL}^k$. For example, the first line\n",
"of $\\hat{F}_{XL}$ corresponds to the first vectorized feature map $F_{XL}^1$.\n",
"\n",
"Finally, the gram matrix must be normalized by dividing each element by\n",
"the total number of elements in the matrix. This normalization is to\n",
"counteract the fact that $\\hat{F}_{XL}$ matrices with a large $N$ dimension yield\n",
"larger values in the Gram matrix. These larger values will cause the\n",
"first layers (before pooling layers) to have a larger impact during the\n",
"gradient descent. Style features tend to be in the deeper layers of the\n",
"network so this normalization step is crucial.\n",
"\n",
"\n"
]
},
{
"cell_type": "code",
2024-08-11 20:25:35 +02:00
"execution_count": 10,
2024-02-23 19:22:37 +01:00
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"def gram_matrix(input):\n",
" a, b, c, d = input.size() # a=batch size(=1)\n",
" # b=number of feature maps\n",
" # (c,d)=dimensions of a f. map (N=c*d)\n",
"\n",
" features = input.view(a * b, c * d) # resize F_XL into \\hat F_XL\n",
"\n",
" G = torch.mm(features, features.t()) # compute the gram product\n",
"\n",
" # we 'normalize' the values of the gram matrix\n",
" # by dividing by the number of element in each feature maps.\n",
" return G.div(a * b * c * d)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now the style loss module looks almost exactly like the content loss\n",
"module. The style distance is also computed using the mean square\n",
"error between $G_{XL}$ and $G_{SL}$.\n",
"\n",
"\n"
]
},
{
"cell_type": "code",
2024-08-11 20:25:35 +02:00
"execution_count": 11,
2024-02-23 19:22:37 +01:00
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"class StyleLoss(nn.Module):\n",
"\n",
" def __init__(self, target_feature):\n",
" super(StyleLoss, self).__init__()\n",
" self.target = gram_matrix(target_feature).detach()\n",
"\n",
" def forward(self, input):\n",
" G = gram_matrix(input)\n",
" self.loss = F.mse_loss(G, self.target)\n",
" return input"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Importing the Model\n",
"\n",
"Now we need to import a pretrained neural network. We will use a 19\n",
"layer VGG network like the one used in the paper.\n",
"\n",
"PyTorch’
"``Sequential`` modules: ``features`` (containing convolution and pooling layers),\n",
"and ``classifier`` (containing fully connected layers). We will use the\n",
"``features`` module because we need the output of the individual\n",
"convolution layers to measure content and style loss. Some layers have\n",
"different behavior during training than evaluation, so we must set the\n",
"network to evaluation mode using ``.eval()``.\n",
"\n",
"\n"
]
},
{
"cell_type": "code",
2024-08-11 20:25:35 +02:00
"execution_count": 12,
2024-02-23 19:22:37 +01:00
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"cnn = vgg19(weights=VGG19_Weights.DEFAULT).features.eval()"
]
},
{
"cell_type": "code",
2024-08-11 20:25:35 +02:00
"execution_count": 13,
2024-02-23 19:22:37 +01:00
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Sequential(\n",
" (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
" (1): ReLU(inplace=True)\n",
" (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
" (3): ReLU(inplace=True)\n",
" (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
" (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
" (6): ReLU(inplace=True)\n",
" (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
" (8): ReLU(inplace=True)\n",
" (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
" (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
" (11): ReLU(inplace=True)\n",
" (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
" (13): ReLU(inplace=True)\n",
" (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
" (15): ReLU(inplace=True)\n",
" (16): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
" (17): ReLU(inplace=True)\n",
" (18): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
" (19): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
" (20): ReLU(inplace=True)\n",
" (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
" (22): ReLU(inplace=True)\n",
" (23): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
" (24): ReLU(inplace=True)\n",
" (25): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
" (26): ReLU(inplace=True)\n",
" (27): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
" (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
" (29): ReLU(inplace=True)\n",
" (30): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
" (31): ReLU(inplace=True)\n",
" (32): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
" (33): ReLU(inplace=True)\n",
" (34): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
" (35): ReLU(inplace=True)\n",
" (36): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
")\n"
]
}
],
"source": [
"print(cnn)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Additionally, VGG networks are trained on images with each channel\n",
"normalized by mean=[0.485, 0.456, 0.406] and std=[0.229, 0.224, 0.225].\n",
"We will use them to normalize the image before sending it into the network.\n",
"\n",
"\n"
]
},
{
"cell_type": "code",
2024-08-11 20:25:35 +02:00
"execution_count": 14,
2024-02-23 19:22:37 +01:00
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"cnn_normalization_mean = torch.tensor([0.485, 0.456, 0.406])\n",
"cnn_normalization_std = torch.tensor([0.229, 0.224, 0.225])\n",
"\n",
"# create a module to normalize input image so we can easily put it in a\n",
"# ``nn.Sequential``\n",
"class Normalization(nn.Module):\n",
" def __init__(self, mean, std):\n",
" super(Normalization, self).__init__()\n",
" # .view the mean and std to make them [C x 1 x 1] so that they can\n",
" # directly work with image Tensor of shape [B x C x H x W].\n",
" # B is batch size. C is number of channels. H is height and W is width.\n",
" self.mean = torch.tensor(mean).view(-1, 1, 1)\n",
" self.std = torch.tensor(std).view(-1, 1, 1)\n",
"\n",
" def forward(self, img):\n",
" # normalize ``img``\n",
" return (img - self.mean) / self.std"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A ``Sequential`` module contains an ordered list of child modules. For\n",
"instance, ``vgg19.features`` contains a sequence (``Conv2d``, ``ReLU``, ``MaxPool2d``,\n",
"``Conv2d``, ``ReLU``…) aligned in the right order of depth. We need to add our\n",
"content loss and style loss layers immediately after the convolution\n",
"layer they are detecting. To do this we must create a new ``Sequential``\n",
"module that has content loss and style loss modules correctly inserted.\n",
"\n",
"\n"
]
},
{
"cell_type": "code",
2024-08-11 20:25:35 +02:00
"execution_count": 15,
2024-02-23 19:22:37 +01:00
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# desired depth layers to compute style/content losses :\n",
"content_layers_default = ['conv_4']\n",
"style_layers_default = ['conv_1', 'conv_2', 'conv_3', 'conv_4', 'conv_5']\n",
"\n",
"def get_style_model_and_losses(cnn, normalization_mean, normalization_std,\n",
" style_img, content_img,\n",
" content_layers=content_layers_default,\n",
" style_layers=style_layers_default):\n",
" # normalization module\n",
" normalization = Normalization(normalization_mean, normalization_std)\n",
"\n",
" # just in order to have an iterable access to or list of content/style\n",
" # losses\n",
" content_losses = []\n",
" style_losses = []\n",
"\n",
" # assuming that ``cnn`` is a ``nn.Sequential``, so we make a new ``nn.Sequential``\n",
" # to put in modules that are supposed to be activated sequentially\n",
" model = nn.Sequential(normalization)\n",
"\n",
" i = 0 # increment every time we see a conv\n",
" for layer in cnn.children():\n",
" if isinstance(layer, nn.Conv2d):\n",
" i += 1\n",
" name = 'conv_{}'.format(i)\n",
" elif isinstance(layer, nn.ReLU):\n",
" name = 'relu_{}'.format(i)\n",
" # The in-place version doesn't play very nicely with the ``ContentLoss``\n",
" # and ``StyleLoss`` we insert below. So we replace with out-of-place\n",
" # ones here.\n",
" layer = nn.ReLU(inplace=False)\n",
" elif isinstance(layer, nn.MaxPool2d):\n",
" name = 'pool_{}'.format(i)\n",
" elif isinstance(layer, nn.BatchNorm2d):\n",
" name = 'bn_{}'.format(i)\n",
" else:\n",
" raise RuntimeError('Unrecognized layer: {}'.format(layer.__class__.__name__))\n",
"\n",
" model.add_module(name, layer)\n",
"\n",
" if name in content_layers:\n",
" # add content loss:\n",
" target = model(content_img).detach()\n",
" content_loss = ContentLoss(target)\n",
" model.add_module(\"content_loss_{}\".format(i), content_loss)\n",
" content_losses.append(content_loss)\n",
"\n",
" if name in style_layers:\n",
" # add style loss:\n",
" target_feature = model(style_img).detach()\n",
" style_loss = StyleLoss(target_feature)\n",
" model.add_module(\"style_loss_{}\".format(i), style_loss)\n",
" style_losses.append(style_loss)\n",
"\n",
" # now we trim off the layers after the last content and style losses\n",
" for i in range(len(model) - 1, -1, -1):\n",
" if isinstance(model[i], ContentLoss) or isinstance(model[i], StyleLoss):\n",
" break\n",
"\n",
" model = model[:(i + 1)]\n",
"\n",
" return model, style_losses, content_losses"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next, we select the input image. You can use a copy of the content image\n",
"or white noise.\n",
"\n",
"\n"
]
},
{
"cell_type": "code",
2024-08-11 20:25:35 +02:00
"execution_count": 16,
2024-02-23 19:22:37 +01:00
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAAFZCAYAAACym9R8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOz9edBsWVUmjD9rn5OZ73CHqlt1ax4oKBQKEJVJFIVWtNpGP0GM1g5bi5bWaKOgQ2m0pVtBDBUnWkKkpbuNEIe2w9BWmoBu/Bk40CofKn6iAgUUVFFVVN1bt+oO732nzHP2Xr8/1l5773PyZObJ4Z2oXBW33syT++x5r/WsYe9NzMxY0pKWtKQlLWlJSzpEZA66Akta0pKWtKQlLWlJdVoClCUtaUlLWtKSlnToaAlQlrSkJS1pSUta0qGjJUBZ0pKWtKQlLWlJh46WAGVJS1rSkpa0pCUdOloClCUtaUlLWtKSlnToaAlQlrSkJS1pSUta0qGjJUBZ0pKWtKQlLWlJh46WAGVJS1rSkpa0pCUdOloClCUtaUlLWtKSlnToaAlQlrSkLzB617veBSLC3/zN3xx0VQAA29vb+PEf/3H86Z/+aav0f/qnfwoiwu/93u/tbcWWtKQlHWpaApQlLWlJe0rb29t485vf3BqgLGlJS1oSsAQoS1rSkpa0pCUt6RDSEqAsaUlPAHrVq16FY8eO4fOf/zxe/vKX49ixYzh9+jRe//rXw1ob0t1///0gIvzCL/wCfvEXfxG33norVldX8eIXvxj/+I//WMnzJS95CV7ykpc0lvWkJz0p5Hf69GkAwJvf/GYQEYgIP/7jPz5V/X/8x38cRIRPfepT+Jf/8l/i5MmTOH36NH7sx34MzIwHH3wQ3/It34ITJ07guuuuw1vf+tbK+4PBAG984xvxnOc8BydPnsT6+jq++qu/Gn/yJ38yVNbjjz+O7/qu78KJEydwxRVX4K677sJHP/pREBHe9a53VdLec889+LZv+zacOnUKKysreO5zn4v3vOc9U7VtSUtaUjMtAcqSlvQEIWst7rzzTlx11VX4hV/4Bbz4xS/GW9/6VvzX//pfh9L+xm/8Bn7pl34Jd999N97whjfgH//xH/G1X/u1OHv27FRlnj59Gr/yK78CAHjFK16B3/zN38Rv/uZv4lu/9VtnasO3f/u3wzmHn/mZn8ELXvAC/ORP/iTe9ra34eu//utx44034md/9mdx++234/Wvfz0++MEPhvc2Njbwq7/6q3jJS16Cn/3Zn8WP//iP49y5c7jzzjvxd3/3dyGdcw7f/M3fjP/xP/4H7rrrLvzUT/0UHnnkEdx1111DdfnYxz6Gr/iKr8AnPvEJ/MiP/Aje+ta3Yn19HS9/+cvxB3/wBzO1b0lLWlJCvKQlLekLin7t136NAfBf//Vfh2d33XUXA+Cf+ImfqKT9si/7Mn7Oc54Tvt93330MgFdXV/mhhx4Kzz/84Q8zAP7BH/zB8OzFL34xv/jFLx4q/6677uJbb701fD937hwD4De96U2t6v8nf/InDIB/93d/Nzx705vexAD4+77v+8Kzsiz5pptuYiLin/mZnwnPL1y4wKurq3zXXXdV0vb7/Uo5Fy5c4GuvvZa/53u+Jzz7n//zfzIAftvb3haeWWv5a7/2axkA/9qv/Vp4/nVf93X8rGc9i3d3d8Mz5xx/5Vd+JT/1qU9t1dYlLWlJo2lpQVnSkp5A9G/+zb+pfP/qr/5qfPaznx1K9/KXvxw33nhj+P785z8fL3jBC/C///f/3vM6jqN//a//dficZRme+9zngpnx6le/Ojy/4oor8MVf/MWVdmVZhm63C0CsJOfPn0dZlnjuc5+Lv/3bvw3p3v/+96PT6eB7v/d7wzNjDO6+++5KPc6fP48//uM/xj//5/8cly9fxmOPPYbHHnsMjz/+OO688058+tOfxuc///mFt39JS3oi0RKgLGlJTxBaWVkJ8SBKV155JS5cuDCU9qlPferQsy/6oi/C/fffv1fVa0W33HJL5fvJkyexsrKCq6++euh5vV2//uu/ji/5ki/BysoKrrrqKpw+fRrve9/7cOnSpZDmc5/7HK6//nqsra1V3r399tsr3++9914wM37sx34Mp0+frvx705veBAB49NFH527vkpb0RKb8oCuwpCUtaX8oy7KF5kdEYOah52nQ7aKpqQ2j2pXW7bd+67fwqle9Ci9/+cvxQz/0Q7jmmmuQZRne8pa34DOf+czU9XDOAQBe//rX484772xMUwc1S1rSkqajJUBZ0pKWNESf/vSnh5596lOfCrtzALG+NLmHPve5z1W+E9HC6zct/d7v/R6e/OQn4/d///cr9VFrh9Ktt96KP/mTP8H29nbFinLvvfdW0j35yU8GAHQ6Hbz0pS/dw5ovaUlPXFq6eJa0pCUN0bvf/e5KDMVf/dVf4cMf/jC+8Ru/MTx7ylOegnvuuQfnzp0Lzz760Y/iL/7iLyp5qaC/ePHi3lZ6DKmVJbWqfPjDH8aHPvShSro777wTRVHgv/23/xaeOefwjne8o5LummuuwUte8hL8l//yX/DII48MlZf2yZKWtKTZaGlBWdKSljREt99+O170ohfh+7//+9Hv9/G2t70NV111FX74h384pPme7/ke/Kf/9J9w55134tWvfjUeffRRvPOd78QznvEMbGxshHSrq6u444478Du/8zv4oi/6Ipw6dQrPfOYz8cxnPnPf2vNN3/RN+P3f/3284hWvwMte9jLcd999eOc734k77rgDm5ubId3LX/5yPP/5z8e/+3f/Dvfeey+e9rSn4T3veQ/Onz8PoGoNesc73oEXvehFeNaznoXv/d7vxZOf/GScPXsWH/rQh/DQQw/hox/96L61b0lL+kKkpQVlSUta0hB993d/N1772tfil3/5l/FTP/VTeMYznoE//uM/xvXXXx/SPP3pT8dv/MZv4NKlS3jd616H97znPfjN3/xNfPmXf/lQfr/6q7+KG2+8ET/4gz+If/Ev/sW+37Pzqle9Cj/90z+Nj370o/i3//bf4g//8A/xW7/1W3juc59bSZdlGd73vvfh27/92/Hrv/7r+I//8T/ihhtuCBaUlZWVkPaOO+7A3/zN3+BlL3sZ3vWud+Huu+/GO9/5Thhj8MY3vnFf27ekJX0hEnFTlNuSlrSkJyTdf//9uO222/DzP//zeP3rX3/Q1Tk09O53vxuveMUr8Od//uf4qq/6qoOuzpKW9ISgpQVlSUta0pIS2tnZqXy31uLtb387Tpw40WgdWtKSlrQ3tIxBWdKSlrSkhF772tdiZ2cHL3zhC9Hv9/H7v//7+Mu//Ev89E//NFZXVw+6ekta0hOGlgBlSUta0pIS+tqv/Vq89a1vxXvf+17s7u7i9ttvx9vf/na85jWvOeiqLWlJTyg60BiUd7zjHfj5n/95nDlzBs9+9rPx9re/Hc9//vMPqjpLWtKSlrSkJS3pkNCBxaD8zu/8Dl73utfhTW96E/72b/8Wz372s3HnnXcuj4de0pKWtKQlLWlJB2dBecELXoDnPe95+OVf/mUAchjSzTffjNe+9rX4kR/5kYOo0pKWtKQlLWlJSzokdCAxKIPBAB/5yEfwhje8ITwzxuClL33p0MmOANDv99Hv98N3vY30qquuOhTHaC9pSUta0pKWtKTJxMy4fPkybrjhBhgz3olzIADlscceg7UW1157beX5tddei3vuuWco/Vve8ha8+c1v3q/qLWlJS1rSkpa0pD2kBx98EDfddNPYNEdiF88b3vAGvO51rwvfL126hFtuuQXvec97sL6+PpS+yaqyPI/uC4t0PJcWtCUt6WCJmb+g12FddjBz+Ac03+o96nv6N01T/1xPPyqPShkgAOQ/y0eGpiEQyw8EAoPhkP4eMoFmO7p+1T5gnw8zy/s+EwbATFKCT+ucw5//3w/iv//6r+L48eOYRAcCUK6++mpkWYazZ89Wnp89exbXXXfdUPper4derzf0fH19HcePHx8aqP0GKGneadlt6tU23agypqnPNDRrndq2uSmtpm87Vk2Ms
"text/plain": [
"<Figure size 640x480 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"input_img = content_img.clone()\n",
"#input_img = torch.randn(content_img.data.size())\n",
"# if you want to use white noise by using the following code:\n",
"#\n",
"# .. code-block:: python\n",
"#\n",
"# input_img = torch.randn(content_img.data.size())\n",
"\n",
"# add the original input image to the figure:\n",
"plt.figure()\n",
"imshow(input_img, title='Input Image')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Gradient Descent\n",
"\n",
"As Leon Gatys, the author of the algorithm, suggested [here](https://discuss.pytorch.org/t/pytorch-tutorial-for-neural-transfert-of-artistic-style/336/20?u=alexis-jacq)_, we will use\n",
"L-BFGS algorithm to run our gradient descent. Unlike training a network,\n",
"we want to train the input image in order to minimize the content/style\n",
"losses. We will create a PyTorch L-BFGS optimizer ``optim.LBFGS`` and pass\n",
"our image to it as the tensor to optimize.\n",
"\n",
"\n"
]
},
{
"cell_type": "code",
2024-08-11 20:25:35 +02:00
"execution_count": 17,
2024-02-23 19:22:37 +01:00
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"def get_input_optimizer(input_img):\n",
" # this line to show that input is a parameter that requires a gradient\n",
" optimizer = optim.LBFGS([input_img])\n",
" return optimizer"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally, we must define a function that performs the neural transfer. For\n",
"each iteration of the networks, it is fed an updated input and computes\n",
"new losses. We will run the ``backward`` methods of each loss module to\n",
"dynamically compute their gradients. The optimizer requires a “closure”\n",
"function, which reevaluates the module and returns the loss.\n",
"\n",
"We still have one final constraint to address. The network may try to\n",
"optimize the input with values that exceed the 0 to 1 tensor range for\n",
"the image. We can address this by correcting the input values to be\n",
"between 0 to 1 each time the network is run.\n",
"\n",
"\n"
]
},
{
"cell_type": "code",
2024-08-11 20:25:35 +02:00
"execution_count": 18,
2024-02-23 19:22:37 +01:00
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"def run_style_transfer(cnn, normalization_mean, normalization_std,\n",
" content_img, style_img, input_img, num_steps=300,\n",
" style_weight=1000000, content_weight=1):\n",
" \"\"\"Run the style transfer.\"\"\"\n",
" print('Building the style transfer model..')\n",
" model, style_losses, content_losses = get_style_model_and_losses(cnn,\n",
" normalization_mean, normalization_std, style_img, content_img)\n",
"\n",
" # We want to optimize the input and not the model parameters so we\n",
" # update all the requires_grad fields accordingly\n",
" input_img.requires_grad_(True)\n",
" # We also put the model in evaluation mode, so that specific layers \n",
" # such as dropout or batch normalization layers behave correctly. \n",
" model.eval()\n",
" model.requires_grad_(False)\n",
"\n",
" optimizer = get_input_optimizer(input_img)\n",
"\n",
" print('Optimizing..')\n",
" run = [0]\n",
" while run[0] <= num_steps:\n",
"\n",
" def closure():\n",
" # correct the values of updated input image\n",
" with torch.no_grad():\n",
" input_img.clamp_(0, 1)\n",
"\n",
" optimizer.zero_grad()\n",
" model(input_img)\n",
" style_score = 0\n",
" content_score = 0\n",
"\n",
" for sl in style_losses:\n",
" style_score += sl.loss\n",
" for cl in content_losses:\n",
" content_score += cl.loss\n",
"\n",
" style_score *= style_weight\n",
" content_score *= content_weight\n",
"\n",
" loss = style_score + content_score\n",
" loss.backward()\n",
"\n",
" run[0] += 1\n",
" if run[0] % 50 == 0:\n",
" print(\"run {}:\".format(run))\n",
" print('Style Loss : {:4f} Content Loss: {:4f}'.format(\n",
" style_score.item(), content_score.item()))\n",
" print()\n",
"\n",
" return style_score + content_score\n",
"\n",
" optimizer.step(closure)\n",
"\n",
" # a last correction...\n",
" with torch.no_grad():\n",
" input_img.clamp_(0, 1)\n",
"\n",
" return input_img"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally, we can run the algorithm.\n",
"\n",
"\n"
]
},
{
"cell_type": "code",
2024-08-11 20:25:35 +02:00
"execution_count": 19,
2024-02-23 19:22:37 +01:00
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
2024-08-11 20:25:35 +02:00
"Building the style transfer model..\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/aneta/.local/lib/python3.10/site-packages/torch/utils/_device.py:77: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
" return func(*args, **kwargs)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
2024-02-23 19:22:37 +01:00
"Optimizing..\n",
"run [50]:\n",
2024-08-11 20:25:35 +02:00
"Style Loss : 24.956041 Content Loss: 6.911002\n",
2024-02-23 19:22:37 +01:00
"\n",
"run [100]:\n",
2024-08-11 20:25:35 +02:00
"Style Loss : 5.859673 Content Loss: 4.592432\n",
2024-02-23 19:22:37 +01:00
"\n",
"run [150]:\n",
2024-08-11 20:25:35 +02:00
"Style Loss : 2.585037 Content Loss: 3.586643\n",
2024-02-23 19:22:37 +01:00
"\n",
"run [200]:\n",
2024-08-11 20:25:35 +02:00
"Style Loss : 1.855601 Content Loss: 3.089260\n",
2024-02-23 19:22:37 +01:00
"\n",
"run [250]:\n",
2024-08-11 20:25:35 +02:00
"Style Loss : 1.304306 Content Loss: 2.969698\n",
2024-02-23 19:22:37 +01:00
"\n",
"run [300]:\n",
2024-08-11 20:25:35 +02:00
"Style Loss : 0.906892 Content Loss: 2.930521\n",
2024-02-23 19:22:37 +01:00
"\n"
]
},
{
"data": {
2024-08-11 20:25:35 +02:00
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAAFZCAYAAACym9R8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOz9ffD+W1UWjl9r7ftzDkfkQSV5MJUHsXJ0NCHJSZn6xoxUYmhFKPoFgzAVsx8+FDUqp1QaacrxWatJy4b6I3WmUlNJhxofpnwIh1Ihwa9miISAgnI+92uv3x/Xda2973MQzjENPnG/9HDO+/2+79drv/ZeD9e61tprR1UVrtf1ul7X63pdr+t1vd6NrnxXD+B6Xa/rdb2u1/W6Xtfr7tcVoFyv63W9rtf1ul7X693uugKU63W9rtf1ul7X63q9211XgHK9rtf1ul7X63pdr3e76wpQrtf1ul7X63pdr+v1bnddAcr1ul7X63pdr+t1vd7tritAuV7X63pdr+t1va7Xu911BSjX63pdr+t1va7X9Xq3u64A5Xpdr+t1va7X9bpe73bXFaBcr+t1va7X9bpe1+vd7roClOt1vW7h6xWveAU+/dM/HR/wAR+A22+/HY94xCPwjGc8A694xSv+t+77lV/5lfiu7/qu351BvpPrh3/4h/HCF74Qb3zjG+/V55/1rGfhvd/7vX9vB3W9rtf1epdfV4Byva7XLXp9x3d8Bz76oz8aL33pS/GZn/mZ+IZv+AY8+9nPxg/+4A/ioz/6o/Gd3/mdv+N7/58GKHfeeee9BijX63pdr/eM6/SuHsD1ul7X675f//2//3d8xmd8Bh796EfjZS97GX7f7/t9/bfP//zPx8d//MfjMz7jM/Dyl78cj370o9+FI71e1+t6Xa/f2XVlUK7X9boFrxe/+MV461vfim/5lm+5ACcA8JCHPATf/M3fjLe85S34qq/6qv79s571LDzykY+8x71e+MIXIiL654jAW97yFnzbt30bIgIRgWc961kXn/2Zn/kZPO1pT8MDH/hAvN/7vR8+//M/H7/1W7/V93jNa16DiMC3fuu33uN5EYEXvvCFfb8v+qIvAgA86lGP6ue95jWvuU/z8chHPhKf+ImfiB/6oR/C4x//eNxxxx34iI/4CPzQD/0QALJNH/ERH4H73e9+eNzjHoef/MmfvPj+y1/+cjzrWc/Cox/9aNzvfvfDwx72MPylv/SX8L/+1/+6x7P8jPvd7354zGMeg2/+5m++xxz6+vZv/3Y87nGPwx133IH3fd/3xdOf/nT84i/+4n16t+t1vd5TryuDcr2u1y14/et//a/xyEc+Eh//8R//dv/+xCc+EY985CPxb//tv73P9/5n/+yf4TnPeQ4+5mM+Bs997nMBAI95zGMuPvO0pz0Nj3zkI/GiF70IP/qjP4qv+Zqvwa/92q/hn/7Tf3qfnvUpn/Ip+Lmf+zm85CUvwT/4B/8AD3nIQwDgHqDr3lyvetWr8Gmf9mn4rM/6LHz6p386/t7f+3t4ylOegm/6pm/C3/ybfxOf8zmfAwB40YtehKc97Wn42Z/9WWQyRvv+7/9+/PzP/zw+8zM/Ew972MPwile8At/yLd+CV7ziFfjRH/3RBh8/+ZM/iSc/+cl4+MMfjjvvvBPHceBv/+2//XbH+xVf8RX4ki/5EjztaU/Dc57zHPzqr/4qvvZrvxZPfOIT8ZM/+ZN48IMffJ/f8Xpdr/eoq67X9bpet9T1xje+sQDUn/2zf/Ydfu6TPumTCkC9+c1vrqqqZz7zmfXBH/zB9/jcl33Zl9XdTcH973//euYzn/nbfvaTPumTLn7/OZ/zOQWg/st/+S9VVfXqV7+6ANQ/+Sf/5B73AFBf9mVf1j+/+MUvLgD16le/+h2+j69nPvOZdf/73//idx/8wR9cAOqHf/iH+3f/7t/9uwJQd9xxR/3CL/xC//6bv/mbC0D94A/+YP/urW996z2e85KXvKQA1Mte9rL+3VOe8pR6r/d6r/of/+N/9O9e+cpX1ul0upjD17zmNTXGqK/4iq+4uOdP//RP1+l0usfvr9f1ul73vK4pnut1vW6x69d//dcBAA94wAPe4ef89ze/+c2/62P43M/93IufP+/zPg8A8N3f/d2/68+6t9eHfdiH4WM/9mP75yc84QkAgP/n//l/8EEf9EH3+P3P//zP9+/uuOOO/u/f+q3fwutf/3r80T/6RwEAP/ETPwEAOI4DP/ADP4CnPvWpeMQjHtGf/5AP+RD8qT/1py7G8h3f8R2Yc+JpT3saXv/61/c/D3vYw/DYxz4WP/iDP/i79drX63r9X3tdUzzX63rdYpeBh4HKb3fdWyDzO7ke+9jHXvz8mMc8Bpl5n2tHfjevHYQAwIMe9CAAwAd+4Ae+3d//2q/9Wv/uDW94A+688078i3/xL/C6173u4vNvetObAACve93r8Ju/+Zv4kA/5kHs8++6/e+UrX4mqusc8+bpx48a9eaXrdb3eo68rQLle1+sWux70oAfh4Q9/OF7+8pe/w8+9/OUvxwd8wAfggQ98IAC83SJOgMzA/+5193v/Xj7rt7vGGPfp91XV//20pz0NP/zDP4wv+qIvwkd91Efhvd/7vTHnxJOf/GTMOe/zWOaciAh8z/d8z9t9/rWPy/W6Xu/8ugKU63W9bsHrEz/xE/EP/+E/xH/8j/8RH/dxH3ePv/+H//Af8JrXvAaf9Vmf1b97n/d5n7fba+QXfuEX7vG73w5g+HrlK1+JRz3qUf3zq171Ksw5e5fQ+7zP+wDAPZ73O3nW7/X1a7/2a3jpS1+KO++8E1/6pV/av3/lK1958bn3f//3x/3udz+86lWvusc97v67xzzmMagqPOpRj8KHfuiH/t4M/Hpdr//Lr2sNyvW6Xrfg9UVf9EW444478Fmf9Vn32Ar7hje8AX/lr/wVvNd7vVdv4QXoNN/0pjddMC//83/+z7fb0O3+97//O2yc9vVf//UXP3/t134tAHQtxgMf+EA85CEPwcte9rKLz33DN3zD230WcE8w83/qMsOxMyoA8NVf/dX3+NyTnvQkfNd3fRd++Zd/uX//qle9Ct/zPd9z8dlP+ZRPwRgDd9555z3uW1Vvd/vy9bpe1+vyujIo1+t63YLXYx/7WHzbt30bnvGMZ+AjPuIj8OxnPxuPetSj8JrXvAb/+B//Y7z+9a/HS17ykovtwU9/+tPx1//6X8cnf/In46/+1b+Kt771rfjGb/xGfOiHfmgXgvp63OMehx/4gR/A3//7fx+PeMQj8KhHPaqLSwHg1a9+NT7pkz4JT37yk/EjP/Ij+PZv/3Z82qd9Gj7yIz+yP/Oc5zwHf/fv/l085znPweMf/3i87GUvw8/93M/d410e97jHAQD+1t/6W3j605+OGzdu4ClPeUoDl9/r64EPfCCe+MQn4qu+6qtw8+ZNfMAHfAC+7/u+D69+9avv8dkXvvCF+L7v+z78sT/2x/DZn/3ZOI4DX/d1X4cP//APx0/91E/15x7zmMfgy7/8y/GCF7wAr3nNa/DUpz4VD3jAA/DqV78a3/md34nnPve5+MIv/ML/I+93va7XLXu9K7cQXa/rdb3+966Xv/zl9amf+qn18Ic/vG7cuFEPe9jD6lM/9VPrp3/6p9/u57/v+76vPvzDP7xuu+22+gN/4A/Ut3/7t7/dbcY/8zM/U0984hPrjjvuKAC95dif/a//9b/Wn//zf74e8IAH1Pu8z/vU8573vPrN3/zNi3u89a1vrWc/+9n1oAc9qB7wgAfU0572tHrd6153j23GVVV/5+/8nfqAD/iAysx3uuX4t9tm/Gf+zJ+5x2cB1Od+7ude/M5boF/84hf3737pl36pPvmTP7ke/OAH14Me9KD6C3/hL9Qv//Ivv92xvvSlL60//If/cN122231mMc8pv7RP/pH9QVf8AV1v/vd7x7P/1f/6l/Vx33cx9X973//uv/9719/8A/+wfrcz/3c+tmf/dnf9v2u1/W6Xryi6m784/W6Xtfrev021wtf+ELceeed+NVf/dVuqna9gKc+9al4x
2024-02-23 19:22:37 +01:00
"text/plain": [
"<Figure size 640x480 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"output = run_style_transfer(cnn, cnn_normalization_mean, cnn_normalization_std,\n",
" content_img, style_img, input_img)\n",
"\n",
"plt.figure()\n",
"imshow(output, title='Output Image')\n",
"\n",
"# sphinx_gallery_thumbnail_number = 4\n",
"plt.ioff()\n",
"plt.show()"
]
},
2024-08-11 20:25:35 +02:00
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### VISUALIZING LAYERS"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [],
"source": [
"def tensor_to_image(tensor):\n",
" image = tensor.clone().detach().squeeze(0)\n",
" image = transforms.ToPILImage()(image)\n",
" return image"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [],
"source": [
"import io\n",
"import base64"
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {},
"outputs": [],
"source": [
"class StyleTransferModel:\n",
" def __init__(self, content_img, style_img, num_steps=300, style_weight=1000000, content_weight=1):\n",
" self.content_img = content_img\n",
" self.style_img = style_img.resize(content_img.size)\n",
" #self.style_img = self.style_img.resize(self.content_img.size)\n",
" self.style_img = image_loader(self.style_img)\n",
" self.content_img = image_loader(self.content_img)\n",
" self.input_img = self.content_img.clone()\n",
" self.num_steps = num_steps\n",
" self.style_weight = style_weight\n",
" self.content_weight = content_weight\n",
"\n",
" self.cnn = vgg19(weights=VGG19_Weights.DEFAULT).features.to(device).eval()\n",
" self.cnn_normalization_mean = torch.tensor([0.485, 0.456, 0.406]).to(device)\n",
" self.cnn_normalization_std = torch.tensor([0.229, 0.224, 0.225]).to(device)\n",
" \n",
" def run_style_transfer(self):\n",
" print('Building the style transfer model..')\n",
" model, style_losses, content_losses = get_style_model_and_losses(\n",
" self.cnn, self.cnn_normalization_mean, self.cnn_normalization_std, \n",
" self.style_img, self.content_img)\n",
"\n",
" self.input_img.requires_grad_(True)\n",
" model.eval()\n",
" model.requires_grad_(False)\n",
" optimizer = get_input_optimizer(self.input_img)\n",
"\n",
" print('Optimizing..')\n",
" run = [0]\n",
" while run[0] <= self.num_steps:\n",
" def closure():\n",
" with torch.no_grad():\n",
" self.input_img.clamp_(0, 1)\n",
"\n",
" optimizer.zero_grad()\n",
" model(self.input_img)\n",
"\n",
" style_score = 0\n",
" content_score = 0\n",
"\n",
" for sl in style_losses:\n",
" style_score += sl.loss\n",
" for cl in content_losses:\n",
" content_score += cl.loss\n",
"\n",
" style_score *= self.style_weight\n",
" content_score *= self.content_weight\n",
"\n",
" loss = style_score + content_score\n",
" loss.backward()\n",
"\n",
" run[0] += 1\n",
" if run[0] % 50 == 0:\n",
" print(f\"run {run[0]}:\")\n",
" print(f'Style Loss : {style_score.item():4f} Content Loss: {content_score.item():4f}')\n",
" print()\n",
"\n",
" return style_score + content_score\n",
"\n",
" optimizer.step(closure)\n",
"\n",
" with torch.no_grad():\n",
" self.input_img.clamp_(0, 1)\n",
"\n",
" return self.input_img\n",
"\n",
"class StyleTransferVisualizer(StyleTransferModel):\n",
" def __init__(self, content_img, style_img):\n",
" super().__init__(content_img, style_img)\n",
" self.model_layers = self.get_model_layers()\n",
"\n",
" def get_model_layers(self):\n",
" cnn = vgg19(weights=VGG19_Weights.DEFAULT).features.to(device).eval()\n",
" model_layers = []\n",
" i = 0\n",
" for layer in cnn.children():\n",
" if isinstance(layer, torch.nn.Conv2d):\n",
" i += 1\n",
" model_layers.append((f'conv_{i}', layer))\n",
" return model_layers\n",
"\n",
" def visualize_layers(self):\n",
" layer_visualizations = []\n",
" input_img = self.content_img.clone().detach()\n",
"\n",
" for name, layer in self.model_layers:\n",
" input_img = layer(input_img)\n",
"\n",
" # Store the image before and after passing through the layer\n",
" # Store the image before and after passing through the layer\n",
" before_image = tensor_to_image(self.content_img)\n",
" after_image = tensor_to_image_grid(input_img) # Use grid visualization for after_image\n",
"\n",
" before_image_bytes = self.image_to_bytes(before_image)\n",
" after_image_bytes = self.image_to_bytes(after_image)\n",
"\n",
" layer_visualizations.append((name, before_image_bytes, after_image_bytes))\n",
"\n",
" return layer_visualizations\n",
"\n",
" def image_to_bytes(self, image):\n",
" img_io = io.BytesIO()\n",
" image.save(img_io, 'JPEG')\n",
" img_io.seek(0)\n",
" return img_io.getvalue()\n"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [],
"source": [
"def image_loader(image):\n",
" #image = Image.open(image_name)\n",
" # fake batch dimension required to fit network's input dimensions\n",
"\n",
" imsize = 512 if torch.cuda.is_available() else 128 # use small size if no GPU\n",
"\n",
" loader = transforms.Compose([\n",
" transforms.Resize(imsize), # scale imported image\n",
" transforms.ToTensor()]) # transform it into a torch tensor\n",
" image = loader(image).unsqueeze(0)\n",
" return image.to(device, torch.float)"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [],
"source": [
"style_image = Image.open(\"images/chelm.jpg\")\n",
"content_image = Image.open(\"images/owce.jpg\")"
]
},
2024-02-23 19:22:37 +01:00
{
"cell_type": "code",
2024-08-11 20:25:35 +02:00
"execution_count": 31,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Building the style transfer model..\n",
"Optimizing..\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/aneta/.local/lib/python3.10/site-packages/torch/utils/_device.py:77: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
" return func(*args, **kwargs)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"run 50:\n",
"Style Loss : 26.431093 Content Loss: 6.895632\n",
"\n",
"run 100:\n",
"Style Loss : 5.664413 Content Loss: 4.582754\n",
"\n",
"run 150:\n",
"Style Loss : 2.541039 Content Loss: 3.557797\n",
"\n",
"run 200:\n",
"Style Loss : 1.846275 Content Loss: 3.080220\n",
"\n",
"run 250:\n",
"Style Loss : 1.267077 Content Loss: 2.966606\n",
"\n",
"run 300:\n",
"Style Loss : 0.902545 Content Loss: 2.930554\n",
"\n"
]
}
],
"source": [
"style_transfer = StyleTransferModel(content_image, style_image)\n",
"output = style_transfer.run_style_transfer()\n",
"\n",
"# Convert the output tensor to an image\n",
"output_image = tensor_to_image(output)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Custom way to visualize layers"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {},
"outputs": [],
"source": [
"pretrained_model = vgg19(weights=VGG19_Weights.DEFAULT).features.eval()\n",
"\n",
"# Extract convolutional layers and their weights\n",
"conv_weights = [] # List to store convolutional layer weights\n",
"conv_layers = [] # List to store convolutional layers\n",
"total_conv_layers = 0 # Counter for total convolutional layers"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Total convolution layers: 16\n"
]
}
],
"source": [
"for module in pretrained_model.children():\n",
" if isinstance(module, nn.Conv2d):\n",
" total_conv_layers += 1\n",
" conv_weights.append(module.weight)\n",
" conv_layers.append(module)\n",
" \n",
"print(f\"Total convolution layers: {total_conv_layers}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"style_img, content_img"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {},
"outputs": [],
"source": [
"input_image = content_img.to(device)"
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {},
"outputs": [],
"source": [
"# Move the model to GPU if available\n",
"device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
"pretrained_model = pretrained_model.to(device)\n",
"\n",
"\n",
"# Extract feature maps\n",
"feature_maps = [] # List to store feature maps\n",
"layer_names = [] # List to store layer names\n",
"for layer in conv_layers:\n",
"\tinput_image = layer(input_image)\n",
"\tfeature_maps.append(input_image)\n",
"\tlayer_names.append(str(layer))\n"
]
},
{
"cell_type": "code",
"execution_count": 51,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Feature maps shape\n",
"torch.Size([1, 64, 512, 910])\n",
"torch.Size([1, 64, 512, 910])\n",
"torch.Size([1, 128, 512, 910])\n",
"torch.Size([1, 128, 512, 910])\n",
"torch.Size([1, 256, 512, 910])\n",
"torch.Size([1, 256, 512, 910])\n",
"torch.Size([1, 256, 512, 910])\n",
"torch.Size([1, 256, 512, 910])\n",
"torch.Size([1, 512, 512, 910])\n",
"torch.Size([1, 512, 512, 910])\n",
"torch.Size([1, 512, 512, 910])\n",
"torch.Size([1, 512, 512, 910])\n",
"torch.Size([1, 512, 512, 910])\n",
"torch.Size([1, 512, 512, 910])\n",
"torch.Size([1, 512, 512, 910])\n",
"torch.Size([1, 512, 512, 910])\n"
]
}
],
"source": [
"# Display feature maps shapes\n",
"print(\"\\nFeature maps shape\")\n",
"for feature_map in feature_maps:\n",
"\tprint(feature_map.shape)\n",
"\n",
"# Process and visualize feature maps\n",
"processed_feature_maps = [] # List to store processed feature maps\n",
"for feature_map in feature_maps:\n",
"\tfeature_map = feature_map.squeeze(0) # Remove the batch dimension\n",
"\tmean_feature_map = torch.sum(feature_map, 0) / feature_map.shape[0] # Compute mean across channels\n",
"\tprocessed_feature_maps.append(mean_feature_map.data.cpu().numpy())\n"
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
" Processed feature maps shape\n",
"(512, 910)\n",
"(512, 910)\n",
"(512, 910)\n",
"(512, 910)\n",
"(512, 910)\n",
"(512, 910)\n",
"(512, 910)\n",
"(512, 910)\n",
"(512, 910)\n",
"(512, 910)\n",
"(512, 910)\n",
"(512, 910)\n",
"(512, 910)\n",
"(512, 910)\n",
"(512, 910)\n",
"(512, 910)\n"
]
}
],
"source": [
"# Display processed feature maps shapes\n",
"print(\"\\n Processed feature maps shape\")\n",
"for fm in processed_feature_maps:\n",
"\tprint(fm.shape)\n",
"\n",
"# Plot the feature maps\n",
"fig = plt.figure(figsize=(30, 50))\n",
"for i in range(len(processed_feature_maps)):\n",
"\tax = fig.add_subplot(5, 4, i + 1)\n",
"\tax.imshow(processed_feature_maps[i])\n",
"\tax.axis(\"off\")\n",
"\tax.set_title(layer_names[i].split('(')[0], fontsize=30)\n"
]
},
{
"cell_type": "code",
"execution_count": 63,
2024-02-23 19:22:37 +01:00
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
2024-08-11 20:25:35 +02:00
"16"
2024-02-23 19:22:37 +01:00
]
},
2024-08-11 20:25:35 +02:00
"execution_count": 63,
2024-02-23 19:22:37 +01:00
"metadata": {},
2024-08-11 20:25:35 +02:00
"output_type": "execute_result"
2024-02-23 19:22:37 +01:00
}
],
"source": [
2024-08-11 20:25:35 +02:00
"len(processed_feature_maps)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Here proper implementation using output model from class"
]
},
{
"cell_type": "code",
"execution_count": 67,
"metadata": {},
"outputs": [],
"source": [
"# Now, we'll use the resulting image for visualization\n",
"# Load pre-trained VGG19 model\n",
"pretrained_model = vgg19(weights=VGG19_Weights.DEFAULT).features.eval().to(device)"
]
},
{
"cell_type": "code",
"execution_count": 69,
"metadata": {},
"outputs": [],
"source": [
"# Extract convolutional layers from VGG19\n",
"conv_layers = []\n",
"for module in pretrained_model.children():\n",
" if isinstance(module, nn.Conv2d):\n",
" conv_layers.append(module)\n",
"\n",
"# Pass the resulting image through the convolutional layers and capture feature maps\n",
"feature_maps = []\n",
"layer_names = []\n",
"input_image = output.clone()\n",
"\n",
"for i, layer in enumerate(conv_layers):\n",
" input_image = layer(input_image)\n",
" feature_maps.append(input_image)\n",
" layer_names.append(f\"Layer {i + 1}: {str(layer)}\")"
]
},
{
"cell_type": "code",
"execution_count": 74,
"metadata": {},
"outputs": [],
"source": [
"plt.close('all')"
]
},
{
"cell_type": "code",
"execution_count": 75,
"metadata": {},
"outputs": [],
"source": [
"# Process and visualize feature maps\n",
"processed_feature_maps = []\n",
"\n",
"for feature_map in feature_maps:\n",
" feature_map = feature_map.squeeze(0) # Remove the batch dimension\n",
" mean_feature_map = torch.mean(feature_map, dim=0).cpu().detach().numpy() # Compute mean across channels\n",
" processed_feature_maps.append(mean_feature_map)\n",
"\n",
"# Plot the feature maps\n",
"fig = plt.figure(figsize=(20, 20))\n",
"for i, fm in enumerate(processed_feature_maps):\n",
" ax = fig.add_subplot(4, 4, i + 1) # Adjust grid size as needed\n",
" ax.imshow(fm, cmap='viridis') # Display feature map as image\n",
" ax.axis(\"off\")\n",
" ax.set_title(layer_names[i], fontsize=8)\n",
"\n",
"plt.tight_layout()\n",
"#plt.show()"
]
},
{
"cell_type": "code",
"execution_count": 76,
"metadata": {},
"outputs": [],
"source": [
"plt.savefig('sheep_chelm.png')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The process extracts the feature maps produced by each convolutional layer.\n",
"\n"
2024-02-23 19:22:37 +01:00
]
2024-08-11 20:25:35 +02:00
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
2024-02-23 19:22:37 +01:00
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
