Merge pull request 'master' (#44 ) from tzietkiewicz/aitech-ium:master into master

Reviewed-on: #44
Added example of archiving artifacts in Github Actions
2023-05-17 22:56:23 +02:00 · 2023-05-17 22:54:58 +02:00 · 2023-05-17 22:30:48 +02:00
8 changed files with 11254 additions and 448 deletions
--- a/IUM_00.Organizacyjne.ipynb
+++ b/IUM_00.Organizacyjne.ipynb
@ -180,7 +180,7 @@
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
-   "version": "3.9.12"
+   "version": "3.9.16"
  },
  "slideshow": {
   "slide_type": "slide"
--- a/IUM_03.CI-Jenkins.ipynb
+++ b/IUM_03.CI-Jenkins.ipynb
@ -509,7 +509,7 @@
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
-   "version": "3.9.12"
+   "version": "3.9.16"
  },
  "slideshow": {
   "slide_type": "slide"
--- a/IUM_06.Jenkins-2.ipynb
+++ b/IUM_06.Jenkins-2.ipynb
@ -248,7 +248,7 @@
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
-   "version": "3.9.12"
+   "version": "3.9.16"
  },
  "slideshow": {
   "slide_type": "slide"
--- a/IUM_07.Sacred.ipynb
+++ b/IUM_07.Sacred.ipynb
@ -981,13 +981,25 @@
  },
  {
   "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "total 16\r\n",
+      "-rw-r--r-- 1 tomek tomek   77 May 28  2022 config.json\r\n",
+      "-rw-r--r-- 1 tomek tomek  159 May 28  2022 cout.txt\r\n",
+      "-rw-r--r-- 1 tomek tomek    2 May 28  2022 metrics.json\r\n",
+      "-rw-r--r-- 1 tomek tomek 1659 May 28  2022 run.json\r\n"
+     ]
+    }
+   ],
   "source": [
    "!ls -l my_runs/1"
   ]
@ -1555,7 +1567,7 @@
    }
   },
   "source": [
-    "## Zadanie  [15 pkt] (do 2023-05-12)\n",
+    "## Zadanie  [15 pkt] (do 2023-06-?)\n",
    "1. \"Owiń\" wywołanie swojego eksperymentu za pomocą Sacred, w ten sposób, żeby zapisane zostały [10pkt]:\n",
    " - parametry, z którymi wywołany był trening\n",
    " - powstały plik z modelem (jako artefakt)\n",
--- a/IUM_08.MLFlow.ipynb
+++ b/IUM_08.MLFlow.ipynb
--- a/IUM_09.Python_srodowiska.ipynb
+++ b/IUM_09.Python_srodowiska.ipynb
--- a/IUM_11.CML.ipynb
+++ b/IUM_11.CML.ipynb
@ -103,7 +103,7 @@
  },
  {
   "cell_type": "code",
-   "execution_count": 16,
+   "execution_count": 4,
   "id": "f4916c1f",
   "metadata": {
    "slideshow": {
@ -128,8 +128,12 @@
  {
   "cell_type": "code",
   "execution_count": 19,
-   "id": "33abf69d",
-   "metadata": {},
+   "id": "88ce689f",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
   "outputs": [
    {
     "name": "stdout",
@ -193,8 +197,12 @@
  {
   "cell_type": "code",
   "execution_count": 22,
-   "id": "780f41d6",
-   "metadata": {},
+   "id": "ff1e011e",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
   "outputs": [
    {
     "name": "stdout",
@ -216,8 +224,12 @@
  },
  {
   "cell_type": "markdown",
-   "id": "b56a1b02",
-   "metadata": {},
+   "id": "3e237076",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
   "source": [
    "### Zakładka actions na stronie repozytorium:\n",
    "https://github.com/TomekZet/ium-ga-hello/actions"
@ -226,8 +238,12 @@
  {
   "cell_type": "code",
   "execution_count": 27,
-   "id": "e3be3eb3",
-   "metadata": {},
+   "id": "32701383",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "fragment"
+    }
+   },
   "outputs": [
    {
     "name": "stdout",
@ -247,8 +263,12 @@
  },
  {
   "cell_type": "markdown",
-   "id": "b40a7f62",
-   "metadata": {},
+   "id": "1c01acb5",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
   "source": [
    "### Ręczne wywoływanie\n",
    "Workflow można również wywołać ręcznie, podając parametry.\n",
@ -258,10 +278,10 @@
  {
   "cell_type": "code",
   "execution_count": 42,
-   "id": "132d8d16",
+   "id": "a7250bf7",
   "metadata": {
    "slideshow": {
-     "slide_type": "slide"
+     "slide_type": "fragment"
    }
   },
   "outputs": [
@ -301,8 +321,12 @@
  {
   "cell_type": "code",
   "execution_count": 43,
-   "id": "c4e71172",
-   "metadata": {},
+   "id": "36ddaac0",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
   "outputs": [
    {
     "name": "stdout",
@ -364,8 +388,12 @@
  },
  {
   "cell_type": "markdown",
-   "id": "a0628b12",
-   "metadata": {},
+   "id": "a764cc0d",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
   "source": [
    "### Akcje wykonywane w kontenerze Docker\n",
    "Akcja może być wywołana w kontenerze Docker (pobranym z Docker hub albo zbudowanym z Dockerfile)\n",
@ -375,8 +403,12 @@
  {
   "cell_type": "code",
   "execution_count": 59,
-   "id": "b8222161",
-   "metadata": {},
+   "id": "ff4dab8c",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
   "outputs": [
    {
     "name": "stdout",
@ -407,9 +439,13 @@
  },
  {
   "cell_type": "code",
-   "execution_count": 47,
-   "id": "80883b9e",
-   "metadata": {},
+   "execution_count": 80,
+   "id": "f1aaff7c",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
   "outputs": [
    {
     "name": "stdout",
@ -429,21 +465,29 @@
    "# Copies your code file from your action repository to the filesystem path `/` of the container\n",
    "COPY entrypoint.sh /entrypoint.sh\n",
    "\n",
+    "VOLUME /github/workspace/\n",
+    "\n",
+    "WORKDIR /github/workspace/\n",
+    "\n",
    "# Code file to execute when the docker container starts up (`entrypoint.sh`)\n",
    "ENTRYPOINT [\"/entrypoint.sh\"]"
   ]
  },
  {
   "cell_type": "code",
-   "execution_count": 50,
-   "id": "f98fee9a",
-   "metadata": {},
+   "execution_count": 84,
+   "id": "7f778025",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
-      "Writing entrypoint.sh\n"
+      "Overwriting entrypoint.sh\n"
     ]
    }
   ],
@ -451,7 +495,9 @@
    "%%writefile entrypoint.sh\n",
    "#!/bin/sh -l\n",
    "\n",
-    "figlet \"Hello $1\"\n",
+    "figlet \"Hello $1\" | tee figlet.txt\n",
+    "echo \"Entrypoint invoked in: $PWD\"\n",
+    "readlink -f figlet.txt\n",
    "time=$(date)\n",
    "echo \"time=$time\" >> $GITHUB_OUTPUT"
   ]
@ -459,8 +505,12 @@
  {
   "cell_type": "code",
   "execution_count": 60,
-   "id": "352de060",
-   "metadata": {},
+   "id": "911975de",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "fragment"
+    }
+   },
   "outputs": [],
   "source": [
    "!chmod +x entrypoint.sh"
@ -468,8 +518,8 @@
  },
  {
   "cell_type": "code",
-   "execution_count": 57,
-   "id": "d73fb6c4",
+   "execution_count": 62,
+   "id": "483e0498",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
@ -501,6 +551,7 @@
    "          - name: Checkout repo\n",
    "            uses: actions/checkout@v2\n",
    "          - name: Use docker action\n",
+    "            id: hello\n",
    "            uses: ./\n",
    "            with:\n",
    "                who-to-greet: \"${{ github.event.inputs.input_text }}\"\n",
@ -513,26 +564,29 @@
  },
  {
   "cell_type": "code",
-   "execution_count": 61,
-   "id": "94d8eec0",
-   "metadata": {},
+   "execution_count": 63,
+   "id": "bc24dff3",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
-      "[main 97c7272] Fix path\n",
-      " 1 file changed, 0 insertions(+), 0 deletions(-)\n",
-      " mode change 100644 => 100755 entrypoint.sh\n",
-      "Enumerating objects: 3, done.\n",
-      "Counting objects: 100% (3/3), done.\n",
+      "[main 22a5094] Fix path\n",
+      " 1 file changed, 1 insertion(+)\n",
+      "Enumerating objects: 9, done.\n",
+      "Counting objects: 100% (9/9), done.\n",
      "Delta compression using up to 4 threads\n",
-      "Compressing objects: 100% (2/2), done.\n",
-      "Writing objects: 100% (2/2), 232 bytes | 232.00 KiB/s, done.\n",
-      "Total 2 (delta 1), reused 0 (delta 0), pack-reused 0\n",
+      "Compressing objects: 100% (5/5), done.\n",
+      "Writing objects: 100% (5/5), 570 bytes | 570.00 KiB/s, done.\n",
+      "Total 5 (delta 1), reused 0 (delta 0), pack-reused 0\n",
      "remote: Resolving deltas: 100% (1/1), completed with 1 local object.\u001b[K\n",
      "To github.com:TomekZet/ium-ga-hello.git\n",
-      "   0f9de38..97c7272  main -> main\n"
+      "   97c7272..22a5094  main -> main\n"
     ]
    }
   ],
@ -542,13 +596,128 @@
    "!git push"
   ]
  },
+  {
+   "cell_type": "markdown",
+   "id": "12af9d1b",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "### Archiwizowanie artefaktów\n",
+    "https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts\n",
+    "\n",
+    "Do archiwizowania artefaktów służy akcja \"upload-artifact\":\n",
+    "\n",
+    "```yaml\n",
+    "      - name: Archive artifacts\n",
+    "        uses: actions/upload-artifact@v3\n",
+    "        with:\n",
+    "          name: figlet-output\n",
+    "          path: figlet.txt\n",
+    "```"
+   ]
+  },
  {
   "cell_type": "code",
-   "execution_count": null,
-   "id": "9c396330",
-   "metadata": {},
-   "outputs": [],
-   "source": []
+   "execution_count": 11,
+   "id": "245f7c8a",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Overwriting .github/workflows/docker-artifact.yml\n"
+     ]
+    }
+   ],
+   "source": [
+    "%%writefile .github/workflows/docker-artifact.yml\n",
+    "name: github-actions-hello-docker-artifact\n",
+    "on: \n",
+    "    workflow_dispatch:\n",
+    "        inputs:\n",
+    "            input_text:\n",
+    "                description: 'Who to greet'     \n",
+    "                required: true\n",
+    "                default: 'World'\n",
+    "jobs:\n",
+    "    hello-job:\n",
+    "        name: \"Do all the hard stuff\"\n",
+    "        runs-on: ubuntu-latest\n",
+    "        steps:\n",
+    "          - name: Checkout repo\n",
+    "            uses: actions/checkout@v2\n",
+    "          - name: Use docker action\n",
+    "            id: hello\n",
+    "            uses: ./\n",
+    "            with:\n",
+    "                who-to-greet: \"${{ github.event.inputs.input_text }}\"\n",
+    "          # Use the output from the `hello` step\n",
+    "          - name: Get the output time\n",
+    "            run: echo \"The time was ${{ steps.hello.outputs.time }}\"  > time.txt\n",
+    "          - name: Archive artifacts\n",
+    "            uses: actions/upload-artifact@v3\n",
+    "            with:\n",
+    "                name: figlet-output\n",
+    "                path: |\n",
+    "                    figlet.txt\n",
+    "                    time.txt\n",
+    "    publish:\n",
+    "        name: \"Publish as github comment\"\n",
+    "        runs-on: ubuntu-latest\n",
+    "        needs: hello-job\n",
+    "        steps:\n",
+    "          - uses: actions/checkout@v3\n",
+    "          #We need to download the artifact first, jobs do not share workflow files\n",
+    "          - name: get-artifact  \n",
+    "            uses: actions/download-artifact@v3\n",
+    "            with:\n",
+    "                name: figlet-output\n",
+    "          - name: display_artifact_contents\n",
+    "            run:\n",
+    "                cat time.txt ; tr ' ' '#' < figlet.txt\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "47e301f9",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[main 5a40228] Archive in one job, use in other\n",
+      " 1 file changed, 1 insertion(+)\n",
+      "Enumerating objects: 9, done.\n",
+      "Counting objects: 100% (9/9), done.\n",
+      "Delta compression using up to 4 threads\n",
+      "Compressing objects: 100% (5/5), done.\n",
+      "Writing objects: 100% (5/5), 622 bytes | 622.00 KiB/s, done.\n",
+      "Total 5 (delta 2), reused 0 (delta 0), pack-reused 0\n",
+      "remote: Resolving deltas: 100% (2/2), completed with 2 local objects.\u001b[K\n",
+      "To github.com:TomekZet/ium-ga-hello.git\n",
+      "   4df6dc0..5a40228  main -> main\n"
+     ]
+    }
+   ],
+   "source": [
+    "!git add -u\n",
+    "!git commit -m \"Archive in one job, use in other\"\n",
+    "!git push"
+   ]
  },
  {
   "cell_type": "markdown",
@ -878,19 +1047,23 @@
    }
   },
   "source": [
-    "## Zadania [15 pkt]\n",
+    "## Zadania [20 pkt] (termin: 24.05.2023)\n",
    "1. Utwórz konto na Github (jeśli jeszcze nie masz)\n",
    "2. Stwórz publiczne repozytorium. Link do niego wklej do kolumny \"Link Github\" w arkuszu [\"Zapisy na zbiory\"](https://teams.microsoft.com/l/file/F62B5988-A797-418D-B085-52E0AF8BD55E?tenantId=73689ee1-b42f-4e25-a5f6-66d1f29bc092&fileType=xlsx&objectUrl=https%3A%2F%2Fuam.sharepoint.com%2Fsites%2F2021SL06-DIUMUI0LABInynieriauczeniamaszynowego-Grupa11%2FShared%20Documents%2FGeneral%2FZapisy%20na%20zbiory.xlsx&baseUrl=https%3A%2F%2Fuam.sharepoint.com%2Fsites%2F2021SL06-DIUMUI0LABInynieriauczeniamaszynowego-Grupa11&serviceName=teams&threadId=19:d67b0dc2ee0849eba517a2aa8507df9c@thread.tacv2&groupId=8cd6b30e-edd9-48db-85ab-259fc11d0c5b) [1 pkt]\n",
-    "2. Stwórz prosty Github workflow wykorzystujący akcje CML, który:\n",
-    " - zrobi checkout Twojego repozytorium [2 pkt]\n",
-    " - ściągnie pliki trenujące. Najlepiej byłoby to zrobić za pomocą DVC, ale tym razem uprośćmy zadanie ze względu na koplikacje, które mogą się pojawić przy konfiguracji uwierzytelniania. Pliki można po prostu dodać do repozytorium albo ściągnąć przez wget jeśli są publicznie dostępne [2 pkt]\n",
-    " - dokona trenowania i ewaluacji [8 pkt]\n",
-    " - wyniki opublikuje za pomocą `cml-send-github-check` i/lub `cml-send-comment` [2 pkt]"
+    "2. Stwórz prosty Github workflow który:\n",
+    " - zrobi checkout Twojego repozytorium [1 pkt]\n",
+    " - ściągnie pliki trenujące. Najlepiej byłoby to zrobić za pomocą DVC, ale tym razem uprośćmy zadanie ze względu na komplikacje, które mogą się pojawić przy konfiguracji uwierzytelniania. Pliki można po prostu dodać do repozytorium albo ściągnąć przez wget jeśli są publicznie dostępne [2 pkt]\n",
+    " - będzie wywoływalny przez \"Workflow dispatch\" z parametrami trenowania [2 pkt]\n",
+    " - składał się będzie z co najmniej 3 jobów:\n",
+    "      1. dokona trenowania jako osobnej akcji wykonanej w Dockerze [8 pkt]\n",
+    "      2. dokona ewaluacji modelu [6 pkt]\n",
+    "      3. zarchiwizuje plik z modelem"
   ]
  }
 ],
 "metadata": {
  "author": "Tomasz Ziętkiewicz",
+  "celltoolbar": "Slideshow",
  "email": "tomasz.zietkiewicz@amu.edu.pl",
  "kernelspec": {
   "display_name": "Python 3",
--- a/IUM_11/github-actions-hello
+++ b/IUM_11/github-actions-hello
@ -1 +1 @@
-Subproject commit 97c7272bcaccf4a578615526b71ca723bb2b6d00
+Subproject commit 5a40228be54897d47e0480318adcdbdec0e74d60
Author	SHA1	Message	Date
Tomasz Zietkiewicz	eb983c3ddd	Merge pull request 'master' (#44 ) from tzietkiewicz/aitech-ium:master into master Reviewed-on: #44	2023-05-17 22:56:23 +02:00
Tomasz	4fee50befa	Added example of archiving artifacts in Github Actions	2023-05-17 22:54:58 +02:00
Tomasz	fa535a10ac	update	2023-05-17 22:30:48 +02:00