AITech/aitech-ium
112
8
Fork 2
forked from tzietkiewicz/aitech-ium
Code Issues Pull Requests Releases Wiki Activity

master #44

Merged
tzietkiewicz merged 2 commits from tzietkiewicz/aitech-ium:master into master 2023-05-17 22:56:24 +02:00
Conversation 0 Commits 2 Files Changed 8 +11254 -448
8 changed files with 11254 additions and 448 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
IUM_00.Organizacyjne.ipynb
View File

@ -180,7 +180,7 @@
"name": "python", "name": "python",
"nbconvert_exporter": "python", "nbconvert_exporter": "python",
"pygments_lexer": "ipython3", "pygments_lexer": "ipython3",
"version": "3.9.12" "version": "3.9.16"
}, },
"slideshow": { "slideshow": {
"slide_type": "slide" "slide_type": "slide"

2
IUM_03.CI-Jenkins.ipynb
View File

@ -509,7 +509,7 @@
"name": "python", "name": "python",
"nbconvert_exporter": "python", "nbconvert_exporter": "python",
"pygments_lexer": "ipython3", "pygments_lexer": "ipython3",
"version": "3.9.12" "version": "3.9.16"
}, },
"slideshow": { "slideshow": {
"slide_type": "slide" "slide_type": "slide"

2
IUM_06.Jenkins-2.ipynb
View File

@ -248,7 +248,7 @@
"name": "python", "name": "python",
"nbconvert_exporter": "python", "nbconvert_exporter": "python",
"pygments_lexer": "ipython3", "pygments_lexer": "ipython3",
"version": "3.9.12" "version": "3.9.16"
}, },
"slideshow": { "slideshow": {
"slide_type": "slide" "slide_type": "slide"

18
IUM_07.Sacred.ipynb
View File

@ -981,13 +981,25 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": 1,
"metadata": { "metadata": {
"slideshow": { "slideshow": {
"slide_type": "slide" "slide_type": "slide"
} }
}, },
"outputs": [], "outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total 16\r\n",
"-rw-r--r-- 1 tomek tomek 77 May 28 2022 config.json\r\n",
"-rw-r--r-- 1 tomek tomek 159 May 28 2022 cout.txt\r\n",
"-rw-r--r-- 1 tomek tomek 2 May 28 2022 metrics.json\r\n",
"-rw-r--r-- 1 tomek tomek 1659 May 28 2022 run.json\r\n"
]
}
],
"source": [ "source": [
"!ls -l my_runs/1" "!ls -l my_runs/1"
] ]
@ -1555,7 +1567,7 @@
} }
}, },
"source": [ "source": [
"## Zadanie [15 pkt] (do 2023-05-12)\n", "## Zadanie [15 pkt] (do 2023-06-?)\n",
"1. \"Owiń\" wywołanie swojego eksperymentu za pomocą Sacred, w ten sposób, żeby zapisane zostały [10pkt]:\n", "1. \"Owiń\" wywołanie swojego eksperymentu za pomocą Sacred, w ten sposób, żeby zapisane zostały [10pkt]:\n",
" - parametry, z którymi wywołany był trening\n", " - parametry, z którymi wywołany był trening\n",
" - powstały plik z modelem (jako artefakt)\n", " - powstały plik z modelem (jako artefakt)\n",

8252
IUM_08.MLFlow.ipynb
View File

File diff suppressed because it is too large Load Diff

3143
IUM_09.Python_srodowiska.ipynb
View File

File diff suppressed because it is too large Load Diff

281
IUM_11.CML.ipynb
View File

@ -103,7 +103,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 16, "execution_count": 4,
"id": "f4916c1f", "id": "f4916c1f",
"metadata": { "metadata": {
"slideshow": { "slideshow": {
@ -128,8 +128,12 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 19, "execution_count": 19,
"id": "33abf69d", "id": "88ce689f",
"metadata": {}, "metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [ "outputs": [
{ {
"name": "stdout", "name": "stdout",
@ -193,8 +197,12 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 22, "execution_count": 22,
"id": "780f41d6", "id": "ff1e011e",
"metadata": {}, "metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [ "outputs": [
{ {
"name": "stdout", "name": "stdout",
@ -216,8 +224,12 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "b56a1b02", "id": "3e237076",
"metadata": {}, "metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [ "source": [
"### Zakładka actions na stronie repozytorium:\n", "### Zakładka actions na stronie repozytorium:\n",
"https://github.com/TomekZet/ium-ga-hello/actions" "https://github.com/TomekZet/ium-ga-hello/actions"
@ -226,8 +238,12 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 27, "execution_count": 27,
"id": "e3be3eb3", "id": "32701383",
"metadata": {}, "metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [ "outputs": [
{ {
"name": "stdout", "name": "stdout",
@ -247,8 +263,12 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "b40a7f62", "id": "1c01acb5",
"metadata": {}, "metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [ "source": [
"### Ręczne wywoływanie\n", "### Ręczne wywoływanie\n",
"Workflow można również wywołać ręcznie, podając parametry.\n", "Workflow można również wywołać ręcznie, podając parametry.\n",
@ -258,10 +278,10 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 42, "execution_count": 42,
"id": "132d8d16", "id": "a7250bf7",
"metadata": { "metadata": {
"slideshow": { "slideshow": {
"slide_type": "slide" "slide_type": "fragment"
} }
}, },
"outputs": [ "outputs": [
@ -301,8 +321,12 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 43, "execution_count": 43,
"id": "c4e71172", "id": "36ddaac0",
"metadata": {}, "metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [ "outputs": [
{ {
"name": "stdout", "name": "stdout",
@ -364,8 +388,12 @@
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"id": "a0628b12", "id": "a764cc0d",
"metadata": {}, "metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [ "source": [
"### Akcje wykonywane w kontenerze Docker\n", "### Akcje wykonywane w kontenerze Docker\n",
"Akcja może być wywołana w kontenerze Docker (pobranym z Docker hub albo zbudowanym z Dockerfile)\n", "Akcja może być wywołana w kontenerze Docker (pobranym z Docker hub albo zbudowanym z Dockerfile)\n",
@ -375,8 +403,12 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 59, "execution_count": 59,
"id": "b8222161", "id": "ff4dab8c",
"metadata": {}, "metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [ "outputs": [
{ {
"name": "stdout", "name": "stdout",
@ -407,9 +439,13 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 47, "execution_count": 80,
"id": "80883b9e", "id": "f1aaff7c",
"metadata": {}, "metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [ "outputs": [
{ {
"name": "stdout", "name": "stdout",
@ -429,21 +465,29 @@
"# Copies your code file from your action repository to the filesystem path `/` of the container\n", "# Copies your code file from your action repository to the filesystem path `/` of the container\n",
"COPY entrypoint.sh /entrypoint.sh\n", "COPY entrypoint.sh /entrypoint.sh\n",
"\n", "\n",
"VOLUME /github/workspace/\n",
"\n",
"WORKDIR /github/workspace/\n",
"\n",
"# Code file to execute when the docker container starts up (`entrypoint.sh`)\n", "# Code file to execute when the docker container starts up (`entrypoint.sh`)\n",
"ENTRYPOINT [\"/entrypoint.sh\"]" "ENTRYPOINT [\"/entrypoint.sh\"]"
] ]
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 50, "execution_count": 84,
"id": "f98fee9a", "id": "7f778025",
"metadata": {}, "metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [ "outputs": [
{ {
"name": "stdout", "name": "stdout",
"output_type": "stream", "output_type": "stream",
"text": [ "text": [
"Writing entrypoint.sh\n" "Overwriting entrypoint.sh\n"
] ]
} }
], ],
@ -451,7 +495,9 @@
"%%writefile entrypoint.sh\n", "%%writefile entrypoint.sh\n",
"#!/bin/sh -l\n", "#!/bin/sh -l\n",
"\n", "\n",
"figlet \"Hello $1\"\n", "figlet \"Hello $1\" | tee figlet.txt\n",
"echo \"Entrypoint invoked in: $PWD\"\n",
"readlink -f figlet.txt\n",
"time=$(date)\n", "time=$(date)\n",
"echo \"time=$time\" >> $GITHUB_OUTPUT" "echo \"time=$time\" >> $GITHUB_OUTPUT"
] ]
@ -459,8 +505,12 @@
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 60, "execution_count": 60,
"id": "352de060", "id": "911975de",
"metadata": {}, "metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [], "outputs": [],
"source": [ "source": [
"!chmod +x entrypoint.sh" "!chmod +x entrypoint.sh"
@ -468,8 +518,8 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 57, "execution_count": 62,
"id": "d73fb6c4", "id": "483e0498",
"metadata": { "metadata": {
"slideshow": { "slideshow": {
"slide_type": "slide" "slide_type": "slide"
@ -501,6 +551,7 @@
" - name: Checkout repo\n", " - name: Checkout repo\n",
" uses: actions/checkout@v2\n", " uses: actions/checkout@v2\n",
" - name: Use docker action\n", " - name: Use docker action\n",
" id: hello\n",
" uses: ./\n", " uses: ./\n",
" with:\n", " with:\n",
" who-to-greet: \"${{ github.event.inputs.input_text }}\"\n", " who-to-greet: \"${{ github.event.inputs.input_text }}\"\n",
@ -513,26 +564,29 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 61, "execution_count": 63,
"id": "94d8eec0", "id": "bc24dff3",
"metadata": {}, "metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [ "outputs": [
{ {
"name": "stdout", "name": "stdout",
"output_type": "stream", "output_type": "stream",
"text": [ "text": [
"[main 97c7272] Fix path\n", "[main 22a5094] Fix path\n",
" 1 file changed, 0 insertions(+), 0 deletions(-)\n", " 1 file changed, 1 insertion(+)\n",
" mode change 100644 => 100755 entrypoint.sh\n", "Enumerating objects: 9, done.\n",
"Enumerating objects: 3, done.\n", "Counting objects: 100% (9/9), done.\n",
"Counting objects: 100% (3/3), done.\n",
"Delta compression using up to 4 threads\n", "Delta compression using up to 4 threads\n",
"Compressing objects: 100% (2/2), done.\n", "Compressing objects: 100% (5/5), done.\n",
"Writing objects: 100% (2/2), 232 bytes | 232.00 KiB/s, done.\n", "Writing objects: 100% (5/5), 570 bytes | 570.00 KiB/s, done.\n",
"Total 2 (delta 1), reused 0 (delta 0), pack-reused 0\n", "Total 5 (delta 1), reused 0 (delta 0), pack-reused 0\n",
"remote: Resolving deltas: 100% (1/1), completed with 1 local object.\u001b[K\n", "remote: Resolving deltas: 100% (1/1), completed with 1 local object.\u001b[K\n",
"To github.com:TomekZet/ium-ga-hello.git\n", "To github.com:TomekZet/ium-ga-hello.git\n",
" 0f9de38..97c7272 main -> main\n" " 97c7272..22a5094 main -> main\n"
] ]
} }
], ],
@ -542,13 +596,128 @@
"!git push" "!git push"
] ]
}, },
{
"cell_type": "markdown",
"id": "12af9d1b",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Archiwizowanie artefaktów\n",
"https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts\n",
"\n",
"Do archiwizowania artefaktów służy akcja \"upload-artifact\":\n",
"\n",
"```yaml\n",
" - name: Archive artifacts\n",
" uses: actions/upload-artifact@v3\n",
" with:\n",
" name: figlet-output\n",
" path: figlet.txt\n",
"```"
]
},
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": 11,
"id": "9c396330", "id": "245f7c8a",
"metadata": {}, "metadata": {
"outputs": [], "slideshow": {
"source": [] "slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Overwriting .github/workflows/docker-artifact.yml\n"
]
}
],
"source": [
"%%writefile .github/workflows/docker-artifact.yml\n",
"name: github-actions-hello-docker-artifact\n",
"on: \n",
" workflow_dispatch:\n",
" inputs:\n",
" input_text:\n",
" description: 'Who to greet' \n",
" required: true\n",
" default: 'World'\n",
"jobs:\n",
" hello-job:\n",
" name: \"Do all the hard stuff\"\n",
" runs-on: ubuntu-latest\n",
" steps:\n",
" - name: Checkout repo\n",
" uses: actions/checkout@v2\n",
" - name: Use docker action\n",
" id: hello\n",
" uses: ./\n",
" with:\n",
" who-to-greet: \"${{ github.event.inputs.input_text }}\"\n",
" # Use the output from the `hello` step\n",
" - name: Get the output time\n",
" run: echo \"The time was ${{ steps.hello.outputs.time }}\" > time.txt\n",
" - name: Archive artifacts\n",
" uses: actions/upload-artifact@v3\n",
" with:\n",
" name: figlet-output\n",
" path: |\n",
" figlet.txt\n",
" time.txt\n",
" publish:\n",
" name: \"Publish as github comment\"\n",
" runs-on: ubuntu-latest\n",
" needs: hello-job\n",
" steps:\n",
" - uses: actions/checkout@v3\n",
" #We need to download the artifact first, jobs do not share workflow files\n",
" - name: get-artifact \n",
" uses: actions/download-artifact@v3\n",
" with:\n",
" name: figlet-output\n",
" - name: display_artifact_contents\n",
" run:\n",
" cat time.txt ; tr ' ' '#' < figlet.txt\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "47e301f9",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[main 5a40228] Archive in one job, use in other\n",
" 1 file changed, 1 insertion(+)\n",
"Enumerating objects: 9, done.\n",
"Counting objects: 100% (9/9), done.\n",
"Delta compression using up to 4 threads\n",
"Compressing objects: 100% (5/5), done.\n",
"Writing objects: 100% (5/5), 622 bytes | 622.00 KiB/s, done.\n",
"Total 5 (delta 2), reused 0 (delta 0), pack-reused 0\n",
"remote: Resolving deltas: 100% (2/2), completed with 2 local objects.\u001b[K\n",
"To github.com:TomekZet/ium-ga-hello.git\n",
" 4df6dc0..5a40228 main -> main\n"
]
}
],
"source": [
"!git add -u\n",
"!git commit -m \"Archive in one job, use in other\"\n",
"!git push"
]
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
@ -878,19 +1047,23 @@
} }
}, },
"source": [ "source": [
"## Zadania [15 pkt]\n", "## Zadania [20 pkt] (termin: 24.05.2023)\n",
"1. Utwórz konto na Github (jeśli jeszcze nie masz)\n", "1. Utwórz konto na Github (jeśli jeszcze nie masz)\n",
"2. Stwórz publiczne repozytorium. Link do niego wklej do kolumny \"Link Github\" w arkuszu [\"Zapisy na zbiory\"](https://teams.microsoft.com/l/file/F62B5988-A797-418D-B085-52E0AF8BD55E?tenantId=73689ee1-b42f-4e25-a5f6-66d1f29bc092&fileType=xlsx&objectUrl=https%3A%2F%2Fuam.sharepoint.com%2Fsites%2F2021SL06-DIUMUI0LABInynieriauczeniamaszynowego-Grupa11%2FShared%20Documents%2FGeneral%2FZapisy%20na%20zbiory.xlsx&baseUrl=https%3A%2F%2Fuam.sharepoint.com%2Fsites%2F2021SL06-DIUMUI0LABInynieriauczeniamaszynowego-Grupa11&serviceName=teams&threadId=19:d67b0dc2ee0849eba517a2aa8507df9c@thread.tacv2&groupId=8cd6b30e-edd9-48db-85ab-259fc11d0c5b) [1 pkt]\n", "2. Stwórz publiczne repozytorium. Link do niego wklej do kolumny \"Link Github\" w arkuszu [\"Zapisy na zbiory\"](https://teams.microsoft.com/l/file/F62B5988-A797-418D-B085-52E0AF8BD55E?tenantId=73689ee1-b42f-4e25-a5f6-66d1f29bc092&fileType=xlsx&objectUrl=https%3A%2F%2Fuam.sharepoint.com%2Fsites%2F2021SL06-DIUMUI0LABInynieriauczeniamaszynowego-Grupa11%2FShared%20Documents%2FGeneral%2FZapisy%20na%20zbiory.xlsx&baseUrl=https%3A%2F%2Fuam.sharepoint.com%2Fsites%2F2021SL06-DIUMUI0LABInynieriauczeniamaszynowego-Grupa11&serviceName=teams&threadId=19:d67b0dc2ee0849eba517a2aa8507df9c@thread.tacv2&groupId=8cd6b30e-edd9-48db-85ab-259fc11d0c5b) [1 pkt]\n",
"2. Stwórz prosty Github workflow wykorzystujący akcje CML, który:\n", "2. Stwórz prosty Github workflow który:\n",
" - zrobi checkout Twojego repozytorium [2 pkt]\n", " - zrobi checkout Twojego repozytorium [1 pkt]\n",
" - ściągnie pliki trenujące. Najlepiej byłoby to zrobić za pomocą DVC, ale tym razem uprośćmy zadanie ze względu na koplikacje, które mogą się pojawić przy konfiguracji uwierzytelniania. Pliki można po prostu dodać do repozytorium albo ściągnąć przez wget jeśli są publicznie dostępne [2 pkt]\n", " - ściągnie pliki trenujące. Najlepiej byłoby to zrobić za pomocą DVC, ale tym razem uprośćmy zadanie ze względu na komplikacje, które mogą się pojawić przy konfiguracji uwierzytelniania. Pliki można po prostu dodać do repozytorium albo ściągnąć przez wget jeśli są publicznie dostępne [2 pkt]\n",
" - dokona trenowania i ewaluacji [8 pkt]\n", " - będzie wywoływalny przez \"Workflow dispatch\" z parametrami trenowania [2 pkt]\n",
" - wyniki opublikuje za pomocą `cml-send-github-check` i/lub `cml-send-comment` [2 pkt]" " - składał się będzie z co najmniej 3 jobów:\n",
" 1. dokona trenowania jako osobnej akcji wykonanej w Dockerze [8 pkt]\n",
" 2. dokona ewaluacji modelu [6 pkt]\n",
" 3. zarchiwizuje plik z modelem"
] ]
} }
], ],
"metadata": { "metadata": {
"author": "Tomasz Ziętkiewicz", "author": "Tomasz Ziętkiewicz",
"celltoolbar": "Slideshow",
"email": "tomasz.zietkiewicz@amu.edu.pl", "email": "tomasz.zietkiewicz@amu.edu.pl",
"kernelspec": { "kernelspec": {
"display_name": "Python 3", "display_name": "Python 3",

2
IUM_11/github-actions-hello

@ -1 +1 @@
Subproject commit 97c7272bcaccf4a578615526b71ca723bb2b6d00 Subproject commit 5a40228be54897d47e0480318adcdbdec0e74d60