aitech-ium/IUM_11.CML.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "08cbeca2",
   "metadata": {},
   "source": [
    "# Github actions i CML"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "beeb17b2",
   "metadata": {},
   "source": [
    "## Github actions\n",
    "<img src=\"img/expcontrol/github-actions.jpeg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "752995e1",
   "metadata": {},
   "source": [
    " - https://docs.github.com/en/actions\n",
    " - System ciągłej integracji \"wbudowany\" w GitHub\n",
    " - Darmowy dla publicznych repozytoriów (z większymi niż w płatnych planach [ograniczeniami dotyczącymi zasobów](https://docs.github.com/en/actions/reference/usage-limits-billing-and-administration#usage-limits))\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b06cc01",
   "metadata": {},
   "source": [
    "https://youtu.be/cP0I9w2coGU"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b66dd41f",
   "metadata": {},
   "source": [
    "### Terminologia Github Actions\n",
    " - *Workflow* - workflow odpowiada \"Pipeline\" z Jenkinsa.\n",
    " - *Job* - workflow składa się z jednego lub kilku \"jobs\". Każdy z nich może być wykonywany równolegle na innej maszynie (patrz \"runner\")\n",
    " - *Step* - odpowiednik \"Stage\" z Jenkinsa - służu do grupowania \"Actions\"\n",
    " - *Action* - odpowiednik \"Step\" z Jenkinsa - pojedyncze polecenie do wykonania, np. dodanie komentarze do Pull requesta, wykonanie polecenia systemowego itp.\n",
    " - *Runner* - odpowiednik Jenkinsowego \"Agent\" - serwer, na którym mogą być wykonywane zadania (\"jobs\")\n",
    "   - *Github-hosted runner* - serwer utrzymywany przez Github (2-core CPU, 7 GB RAM, 14 GB SSD). Windows, Linux albo macOS\n",
    "   - *Self-hosted runner* - nasz własny serwer, z zinstalowaną aplikacją [Github actions runner](https://github.com/actions/runner)\n",
    " - *Event* - zdażenie, które odapala (\"triggers\") uruchomienie Workflow. Np. wypchnięcie zmiany do repozytorium (\"push\"), utworzenie Pull requesta. [Pełna lista](https://docs.github.com/en/actions/reference/events-that-trigger-workflows)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f1f6d0a",
   "metadata": {},
   "source": [
    "### Definicja workflow\n",
    " - Workflow definiuje się w plikach YAM(o rozszerzeniu `*.yml` albo `*.yaml`) umieszczonych w specjalnym folderze `.github/workflows/` wewnątrz repozytorium\n",
    " - Pełna składnia jest opisana [tutaj](https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions)\n",
    " - Podstawowe pola:\n",
    "   - `name` [opcjonalna] - nazwa, pod którą workflow/step będzie widoczny w UI. Domyślnie ścieżka do pliku yaml\n",
    "   - `on` - definiuje kiedy workflow ma być odpalony\n",
    "   - `jobs` - grupuje razem \"zadania\" do wykonania. Każde może być wykonane na innym \"runnerze\". Domyślnie wykonywane są równolegle (ale możemy definiować [zależności między jobami](https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idneeds), co powoduje wykonanie ich sekwencyjnie\n",
    "   - `runs-on` - parametr joba, definujący na jakiej maszynie wirtualnej ma być uruchomiony (np. `ubuntu-latest`)\n",
    "   - `uses` - umożliwia użycie gotowych akcji zdefiniowanych przez nas, albo przez innych użytkowników, np. `-uses: actions/checkout@v2` spowoduje checkout plików z repozytorium\n",
    "   - `run` - pozwala uruchomić dowolne ([dostępne/zainstalowane](https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#preinstalled-software)) polecenie, np. `python3 train.py`\n",
    "   - `env` - pozwala zdefiniować zmienne środowiskowe dostępne dla akcji lub skorzystać ze [zmiennych ustawionych przez Github](https://docs.github.com/en/actions/reference/environment-variables#default-environment-variables)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "955585a9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/tomek/AITech/repo/aitech-ium-private/IUM_11\n",
      "Initialized empty Git repository in /home/tomek/AITech/repo/aitech-ium-private/IUM_11/github-actions-hello/.git/\r\n"
     ]
    }
   ],
   "source": [
    "%cd IUM_11\n",
    "!mkdir github-actions-hello;\\\n",
    "cd github-actions-hello;\\\n",
    "git init"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f4916c1f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/tomek/AITech/repo/aitech-ium-private/IUM_11/github-actions-hello\n"
     ]
    }
   ],
   "source": [
    "%cd github-actions-hello\n",
    "!mkdir -p .github/workflows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "dde8d432",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing .github/workflows/workflow.yml\n"
     ]
    }
   ],
   "source": [
    "%%writefile .github/workflows/workflow.yml\n",
    "name: github-actions-hello\n",
    "on: [push]\n",
    "jobs:\n",
    "  hello-job:\n",
    "    runs-on: ubuntu-latest\n",
    "    steps:\n",
    "      - name: Checkout repo\n",
    "        uses: actions/checkout@v2\n",
    "      - name: Setup Python\n",
    "        uses: actions/setup-python@v2.2.2\n",
    "        with:\n",
    "            python-version: '3.7'\n",
    "      - run: python3 --version"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed780dea",
   "metadata": {},
   "source": [
    "### Zależności\n",
    "\n",
    "Maszyny wirtualne (\"runners\"), na których uruchamiane są \"joby\" mają zainstalowany zbiór narzędzi. Przykładowa lista dla [Ubuntu 20.04](https://github.com/actions/virtual-environments/blob/main/images/linux/Ubuntu2004-README.md)\n",
    "\n",
    "Brakujące zależności można zainstalować, korzystając z:\n",
    " - akcji\n",
    " - poleceń systemowych takich jak `apt install` czy `pip install` uruchomionych poprzez `run`. Patrz [przykład](https://docs.github.com/en/actions/using-github-hosted-runners/customizing-github-hosted-runners#installing-software-on-ubuntu-runners)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28b582c4",
   "metadata": {},
   "source": [
    "### Akcje\n",
    "Za pomocą polecenia `uses` możemy używać przygotowanych wcześniej akcji. Mogą one pochodzić:\n",
    " - z tego samego repozytorium co workflow ([więcej](https://docs.github.com/en/actions/learn-github-actions/finding-and-customizing-actions#referencing-an-action-in-the-same-repository-where-a-workflow-file-uses-the-action))\n",
    " - z dowolnego publicznego repozytorium Github (np. [repozytorioum iterative/setup-clm](https://github.com/iterative/setup-cml), patrz przykład poniżej\n",
    " - z [Github Marketplace](https://github.com/marketplace?type=actions)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "805622e8",
   "metadata": {},
   "source": [
    "## CML - Continous Machine Learning\n",
    "<img src=\"img/expcontrol/cml.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0b3acbf",
   "metadata": {},
   "source": [
    " - Tworzone przez [iterative.ai](iterative.ai) (tak jak DVC)\n",
    " - https://cml.dev/\n",
    " - Dokumentacja: https://dvc.org/doc/cml\n",
    " - Korzysta z Github Actions lub Gitlab CI (a także [Bitbucket Pipelines](https://github.com/iterative/cml/wiki/CML-with-Bitbucket-Cloud))\n",
    " - CML dodaje do Github Actions kilka \"akcji\":\n",
    "   - `iterative/setup-cml` - dodaje poniższe akcje\n",
    "   - `cml-send-comment` - dodaje raport CML jako komentarz do Pull Requesta na Githubie\n",
    "   - `cml-send-github-check` - dodaje raport CML do zakładki \"Checks\" Pull Requesta na Githubie\n",
    "   - `cml-publish` - umożliwia dodanie obrazka do raportu\n",
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdb54b38",
   "metadata": {},
   "source": [
    "### Przykładowy Workflow CML:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "07b1035a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/tomek/AITech/repo/aitech-ium-private/IUM_11\n",
      "Cloning into 'example_cml'...\n",
      "remote: Enumerating objects: 25, done.\u001b[K\n",
      "remote: Total 25 (delta 0), reused 0 (delta 0), pack-reused 25\u001b[K\n",
      "Receiving objects: 100% (25/25), 222.95 KiB | 920.00 KiB/s, done.\n",
      "Resolving deltas: 100% (6/6), done.\n"
     ]
    }
   ],
   "source": [
    "!git clone git@github.com:TomekZet/example_cml.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "bf27a2b3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/tomek/AITech/repo/aitech-ium-private/IUM_11/example_cml\n"
     ]
    }
   ],
   "source": [
    "%cd example_cml\n",
    "!mkdir -p .github/workflows/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "64f6e21d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting .github/workflows/cml.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile .github/workflows/cml.yaml\n",
    "name: model-training\n",
    "on: [push]\n",
    "jobs:\n",
    "  run:\n",
    "    runs-on: [ubuntu-latest]\n",
    "    steps:\n",
    "      - uses: actions/checkout@v2\n",
    "      - uses: actions/setup-python@v2\n",
    "      - uses: iterative/setup-cml@v1\n",
    "      - name: Train model\n",
    "        env:\n",
    "          REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}\n",
    "        run: |\n",
    "          pip install -r requirements.txt\n",
    "          python train.py\n",
    "\n",
    "          cat metrics.txt >> report.md\n",
    "          cml-publish confusion_matrix.png --md >> report.md\n",
    "          cml-send-comment report.md"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83e49d3b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# %load train.py\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import plot_confusion_matrix\n",
    "import matplotlib.pyplot as plt\n",
    "import json\n",
    "import os\n",
    "import numpy as np\n",
    "\n",
    "# Read in data\n",
    "X_train = np.genfromtxt(\"data/train_features.csv\")\n",
    "y_train = np.genfromtxt(\"data/train_labels.csv\")\n",
    "X_test = np.genfromtxt(\"data/test_features.csv\")\n",
    "y_test = np.genfromtxt(\"data/test_labels.csv\")\n",
    "\n",
    "\n",
    "# Fit a model\n",
    "depth = 2\n",
    "clf = RandomForestClassifier(max_depth=depth)\n",
    "clf.fit(X_train,y_train)\n",
    "\n",
    "acc = clf.score(X_test, y_test)\n",
    "print(acc)\n",
    "with open(\"metrics.txt\", 'w') as outfile:\n",
    "        outfile.write(\"Accuracy: \" + str(acc) + \"\\n\")\n",
    "\n",
    "\n",
    "# Plot it\n",
    "disp = plot_confusion_matrix(clf, X_test, y_test, normalize='true',cmap=plt.cm.Blues)\n",
    "plt.savefig('confusion_matrix.png')\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8dc5748f",
   "metadata": {},
   "source": [
    "Wprowadźmy zmianę do pliku (linijka 17: `depth= = 6`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "afeaf939",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting train.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile train.py\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import plot_confusion_matrix\n",
    "import matplotlib.pyplot as plt\n",
    "import json\n",
    "import os\n",
    "import numpy as np\n",
    "\n",
    "# Read in data\n",
    "X_train = np.genfromtxt(\"data/train_features.csv\")\n",
    "y_train = np.genfromtxt(\"data/train_labels.csv\")\n",
    "X_test = np.genfromtxt(\"data/test_features.csv\")\n",
    "y_test = np.genfromtxt(\"data/test_labels.csv\")\n",
    "\n",
    "\n",
    "# Fit a model\n",
    "depth = 6\n",
    "clf = RandomForestClassifier(max_depth=depth)\n",
    "clf.fit(X_train,y_train)\n",
    "\n",
    "acc = clf.score(X_test, y_test)\n",
    "print(acc)\n",
    "with open(\"metrics.txt\", 'w') as outfile:\n",
    "        outfile.write(\"Accuracy: \" + str(acc) + \"\\n\")\n",
    "\n",
    "\n",
    "# Plot it\n",
    "disp = plot_confusion_matrix(clf, X_test, y_test, normalize='true',cmap=plt.cm.Blues)\n",
    "plt.savefig('confusion_matrix.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e4a711a",
   "metadata": {},
   "source": [
    "Stwórzmy nowy branch \"deep_depth\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "ab019b0b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Switched to a new branch 'deep_depth'\n",
      "[deep_depth 0df0f2c] Changed depth and added cml workflow\n",
      " 2 files changed, 19 insertions(+), 2 deletions(-)\n",
      " create mode 100644 .github/workflows/cml.yaml\n",
      "Enumerating objects: 8, done.\n",
      "Counting objects: 100% (8/8), done.\n",
      "Delta compression using up to 4 threads\n",
      "Compressing objects: 100% (4/4), done.\n",
      "Writing objects: 100% (6/6), 738 bytes | 738.00 KiB/s, done.\n",
      "Total 6 (delta 2), reused 0 (delta 0)\n",
      "remote: Resolving deltas: 100% (2/2), completed with 2 local objects.\u001b[K\n",
      "remote: \n",
      "remote: Create a pull request for 'deep_depth' on GitHub by visiting:\u001b[K\n",
      "remote:      https://github.com/TomekZet/example_cml/pull/new/deep_depth\u001b[K\n",
      "remote: \n",
      "To github.com:TomekZet/example_cml.git\n",
      " * [new branch]      deep_depth -> deep_depth\n"
     ]
    }
   ],
   "source": [
    "!git checkout -b deep_depth\n",
    "!git add train.py .github/workflows/cml.yaml\n",
    "!git commit -m \"Changed depth and added cml workflow\"\n",
    "!git push origin deep_depth"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b50f46a8",
   "metadata": {},
   "source": [
    "<img src=\"IUM_11/img/github-pr.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c56c8785",
   "metadata": {},
   "source": [
    "<img src=\"IUM_11/img/github-checks.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb25c587",
   "metadata": {},
   "source": [
    "## Zadania\n",
    "1. Utwórz konto na Github (jeśli jeszcze nie masz)\n",
    "2. Stwórz publiczne repozytorium. Link do niego wklej do kolumny \"Link Github\" w arkuszu [\"Zapisy na zbiory\"](https://teams.microsoft.com/l/file/F62B5988-A797-418D-B085-52E0AF8BD55E?tenantId=73689ee1-b42f-4e25-a5f6-66d1f29bc092&fileType=xlsx&objectUrl=https%3A%2F%2Fuam.sharepoint.com%2Fsites%2F2021SL06-DIUMUI0LABInynieriauczeniamaszynowego-Grupa11%2FShared%20Documents%2FGeneral%2FZapisy%20na%20zbiory.xlsx&baseUrl=https%3A%2F%2Fuam.sharepoint.com%2Fsites%2F2021SL06-DIUMUI0LABInynieriauczeniamaszynowego-Grupa11&serviceName=teams&threadId=19:d67b0dc2ee0849eba517a2aa8507df9c@thread.tacv2&groupId=8cd6b30e-edd9-48db-85ab-259fc11d0c5b)\n",
    "2. Stwórz prosty Github workflow wykorzystujący akcje CML, który:\n",
    " - zrobi checkout Twojego repozytorium\n",
    " - ściągnie pliki trenujące. Najlepiej byłoby to zrobić za pomocą DVC, ale tym razem uprośćmy zadanie ze względu na koplikacje, które mogą się pojawić przy konfiguracji uwierzytelniania. Pliki można po prostu dodać do repozytorium albo ściągnąć przez wget jeśli są publicznie dostępne\n",
    " - dokona trenowania i ewaluacji\n",
    " - wyniki opublikuje za pomocą `cml-send-github-check` i/lub `cml-send-comment`"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}