tzietkiewicz/aitech-ium
4
1
Fork 1
Code Issues Pull Requests Releases Wiki Activity

update
All checks were successful
sacred-mongo/pipeline/head This commit looks good
Details

Browse Source
This commit is contained in:
Tomasz 2023-05-17 22:30:48 +02:00
parent 3de0fde490
commit fa535a10ac
8 changed files with 11271 additions and 446 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
IUM_00.Organizacyjne.ipynb
View File

@ -180,7 +180,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.12"
"version": "3.9.16"
},
"slideshow": {
"slide_type": "slide"

2
IUM_03.CI-Jenkins.ipynb
View File

@ -509,7 +509,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.12"
"version": "3.9.16"
},
"slideshow": {
"slide_type": "slide"

2
IUM_06.Jenkins-2.ipynb
View File

@ -248,7 +248,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.12"
"version": "3.9.16"
},
"slideshow": {
"slide_type": "slide"

18
IUM_07.Sacred.ipynb
View File

@ -981,13 +981,25 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 1,
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total 16\r\n",
"-rw-r--r-- 1 tomek tomek 77 May 28 2022 config.json\r\n",
"-rw-r--r-- 1 tomek tomek 159 May 28 2022 cout.txt\r\n",
"-rw-r--r-- 1 tomek tomek 2 May 28 2022 metrics.json\r\n",
"-rw-r--r-- 1 tomek tomek 1659 May 28 2022 run.json\r\n"
]
}
],
"source": [
"!ls -l my_runs/1"
]
@ -1555,7 +1567,7 @@
}
},
"source": [
"## Zadanie [15 pkt] (do 2023-05-12)\n",
"## Zadanie [15 pkt] (do 2023-06-?)\n",
"1. \"Owiń\" wywołanie swojego eksperymentu za pomocą Sacred, w ten sposób, żeby zapisane zostały [10pkt]:\n",
" - parametry, z którymi wywołany był trening\n",
" - powstały plik z modelem (jako artefakt)\n",

8252
IUM_08.MLFlow.ipynb
View File

File diff suppressed because it is too large Load Diff

3143
IUM_09.Python_srodowiska.ipynb
View File

File diff suppressed because it is too large Load Diff

296
IUM_11.CML.ipynb
View File

@ -128,8 +128,12 @@
{
"cell_type": "code",
"execution_count": 19,
"id": "33abf69d",
"metadata": {},
"id": "88ce689f",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
@ -193,8 +197,12 @@
{
"cell_type": "code",
"execution_count": 22,
"id": "780f41d6",
"metadata": {},
"id": "ff1e011e",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
@ -216,8 +224,12 @@
},
{
"cell_type": "markdown",
"id": "b56a1b02",
"metadata": {},
"id": "3e237076",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Zakładka actions na stronie repozytorium:\n",
"https://github.com/TomekZet/ium-ga-hello/actions"
@ -226,8 +238,12 @@
{
"cell_type": "code",
"execution_count": 27,
"id": "e3be3eb3",
"metadata": {},
"id": "32701383",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"name": "stdout",
@ -247,8 +263,12 @@
},
{
"cell_type": "markdown",
"id": "b40a7f62",
"metadata": {},
"id": "1c01acb5",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Ręczne wywoływanie\n",
"Workflow można również wywołać ręcznie, podając parametry.\n",
@ -258,10 +278,10 @@
{
"cell_type": "code",
"execution_count": 42,
"id": "132d8d16",
"id": "a7250bf7",
"metadata": {
"slideshow": {
"slide_type": "slide"
"slide_type": "fragment"
}
},
"outputs": [
@ -301,8 +321,12 @@
{
"cell_type": "code",
"execution_count": 43,
"id": "c4e71172",
"metadata": {},
"id": "36ddaac0",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
@ -364,8 +388,12 @@
},
{
"cell_type": "markdown",
"id": "a0628b12",
"metadata": {},
"id": "a764cc0d",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Akcje wykonywane w kontenerze Docker\n",
"Akcja może być wywołana w kontenerze Docker (pobranym z Docker hub albo zbudowanym z Dockerfile)\n",
@ -375,8 +403,12 @@
{
"cell_type": "code",
"execution_count": 59,
"id": "b8222161",
"metadata": {},
"id": "ff4dab8c",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
@ -407,9 +439,13 @@
},
{
"cell_type": "code",
"execution_count": 47,
"id": "80883b9e",
"metadata": {},
"execution_count": 80,
"id": "f1aaff7c",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
@ -429,21 +465,29 @@
"# Copies your code file from your action repository to the filesystem path `/` of the container\n",
"COPY entrypoint.sh /entrypoint.sh\n",
"\n",
"VOLUME /github/workspace/\n",
"\n",
"WORKDIR /github/workspace/\n",
"\n",
"# Code file to execute when the docker container starts up (`entrypoint.sh`)\n",
"ENTRYPOINT [\"/entrypoint.sh\"]"
]
},
{
"cell_type": "code",
"execution_count": 50,
"id": "f98fee9a",
"metadata": {},
"execution_count": 84,
"id": "7f778025",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Writing entrypoint.sh\n"
"Overwriting entrypoint.sh\n"
]
}
],
@ -451,7 +495,9 @@
"%%writefile entrypoint.sh\n",
"#!/bin/sh -l\n",
"\n",
"figlet \"Hello $1\"\n",
"figlet \"Hello $1\" | tee figlet.txt\n",
"echo \"Entrypoint invoked in: $PWD\"\n",
"readlink -f figlet.txt\n",
"time=$(date)\n",
"echo \"time=$time\" >> $GITHUB_OUTPUT"
]
@ -459,8 +505,12 @@
{
"cell_type": "code",
"execution_count": 60,
"id": "352de060",
"metadata": {},
"id": "911975de",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"!chmod +x entrypoint.sh"
@ -468,8 +518,8 @@
},
{
"cell_type": "code",
"execution_count": 57,
"id": "d73fb6c4",
"execution_count": 62,
"id": "483e0498",
"metadata": {
"slideshow": {
"slide_type": "slide"
@ -501,6 +551,7 @@
" - name: Checkout repo\n",
" uses: actions/checkout@v2\n",
" - name: Use docker action\n",
" id: hello\n",
" uses: ./\n",
" with:\n",
" who-to-greet: \"${{ github.event.inputs.input_text }}\"\n",
@ -513,26 +564,29 @@
},
{
"cell_type": "code",
"execution_count": 61,
"id": "94d8eec0",
"metadata": {},
"execution_count": 63,
"id": "bc24dff3",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[main 97c7272] Fix path\n",
" 1 file changed, 0 insertions(+), 0 deletions(-)\n",
" mode change 100644 => 100755 entrypoint.sh\n",
"Enumerating objects: 3, done.\n",
"Counting objects: 100% (3/3), done.\n",
"[main 22a5094] Fix path\n",
" 1 file changed, 1 insertion(+)\n",
"Enumerating objects: 9, done.\n",
"Counting objects: 100% (9/9), done.\n",
"Delta compression using up to 4 threads\n",
"Compressing objects: 100% (2/2), done.\n",
"Writing objects: 100% (2/2), 232 bytes | 232.00 KiB/s, done.\n",
"Total 2 (delta 1), reused 0 (delta 0), pack-reused 0\n",
"Compressing objects: 100% (5/5), done.\n",
"Writing objects: 100% (5/5), 570 bytes | 570.00 KiB/s, done.\n",
"Total 5 (delta 1), reused 0 (delta 0), pack-reused 0\n",
"remote: Resolving deltas: 100% (1/1), completed with 1 local object.\u001b[K\n",
"To github.com:TomekZet/ium-ga-hello.git\n",
" 0f9de38..97c7272 main -> main\n"
" 97c7272..22a5094 main -> main\n"
]
}
],
@ -542,13 +596,147 @@
"!git push"
]
},
{
"cell_type": "markdown",
"id": "12af9d1b",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Archiwizowanie artefaktów\n",
"https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts\n",
"\n",
"Do archiwizowania artefaktów służy akcja \"upload-artifact\":\n",
"\n",
"```yaml\n",
" - name: Archive artifacts\n",
" uses: actions/upload-artifact@v3\n",
" with:\n",
" name: figlet-output\n",
" path: figlet.txt\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9c396330",
"execution_count": 1,
"id": "245f7c8a",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Writing .github/workflows/docker-artifact.yml\n"
]
}
],
"source": [
"%%writefile .github/workflows/docker-artifact.yml\n",
"name: github-actions-hello-docker-artifact\n",
"on: \n",
" workflow_dispatch:\n",
" inputs:\n",
" input_text:\n",
" description: 'Who to greet' \n",
" required: true\n",
" default: 'World'\n",
"jobs:\n",
" hello-job:\n",
" name: \"Do all the hard stuff\"\n",
" runs-on: ubuntu-latest\n",
" steps:\n",
" - name: Checkout repo\n",
" uses: actions/checkout@v2\n",
" - name: Use docker action\n",
" id: hello\n",
" uses: ./\n",
" with:\n",
" who-to-greet: \"${{ github.event.inputs.input_text }}\"\n",
" # Use the output from the `hello` step\n",
" - name: Get the output time\n",
" run: echo \"The time was ${{ steps.hello.outputs.time }}\" > time.txt\n",
" archive:\n",
" name: \"Archive the artifacts\"\n",
" runs-on: ubuntu-latest\n",
" needs: hello-job\n",
" steps:\n",
" - name: diagnose\n",
" run: pwd; ls -la\n",
" - name: Archive artifacts\n",
" uses: actions/upload-artifact@v3\n",
" with:\n",
" name: figlet-output\n",
" path: |\n",
" figlet.txt\n",
" time.txt\n"
]
},
{
"cell_type": "code",
"execution_count": 89,
"id": "47e301f9",
"metadata": {},
"outputs": [],
"source": []
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Enumerating objects: 13, done.\n",
"Counting objects: 100% (13/13), done.\n",
"Delta compression using up to 4 threads\n",
"Compressing objects: 100% (7/7), done.\n",
"Writing objects: 100% (7/7), 792 bytes | 792.00 KiB/s, done.\n",
"Total 7 (delta 3), reused 0 (delta 0), pack-reused 0\n",
"remote: Resolving deltas: 100% (3/3), completed with 3 local objects.\u001b[K\n",
"To github.com:TomekZet/ium-ga-hello.git\n",
" 3b9c623..b5a265f main -> main\n"
]
}
],
"source": [
"!git push"
]
},
{
"cell_type": "code",
"execution_count": 91,
"id": "580154ca",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[main 2fe0183] Fix\n",
" 1 file changed, 4 insertions(+)\n",
"Enumerating objects: 9, done.\n",
"Counting objects: 100% (9/9), done.\n",
"Delta compression using up to 4 threads\n",
"Compressing objects: 100% (5/5), done.\n",
"Writing objects: 100% (5/5), 564 bytes | 564.00 KiB/s, done.\n",
"Total 5 (delta 2), reused 0 (delta 0), pack-reused 0\n",
"remote: Resolving deltas: 100% (2/2), completed with 2 local objects.\u001b[K\n",
"To github.com:TomekZet/ium-ga-hello.git\n",
" b5a265f..2fe0183 main -> main\n"
]
}
],
"source": [
"!git add -u \n",
"!git commit -m \"Fix\"\n",
"!git push"
]
},
{
"cell_type": "markdown",
@ -878,19 +1066,23 @@
}
},
"source": [
"## Zadania [15 pkt]\n",
"## Zadania [20 pkt] (termin: 24.05.2023)\n",
"1. Utwórz konto na Github (jeśli jeszcze nie masz)\n",
"2. Stwórz publiczne repozytorium. Link do niego wklej do kolumny \"Link Github\" w arkuszu [\"Zapisy na zbiory\"](https://teams.microsoft.com/l/file/F62B5988-A797-418D-B085-52E0AF8BD55E?tenantId=73689ee1-b42f-4e25-a5f6-66d1f29bc092&fileType=xlsx&objectUrl=https%3A%2F%2Fuam.sharepoint.com%2Fsites%2F2021SL06-DIUMUI0LABInynieriauczeniamaszynowego-Grupa11%2FShared%20Documents%2FGeneral%2FZapisy%20na%20zbiory.xlsx&baseUrl=https%3A%2F%2Fuam.sharepoint.com%2Fsites%2F2021SL06-DIUMUI0LABInynieriauczeniamaszynowego-Grupa11&serviceName=teams&threadId=19:d67b0dc2ee0849eba517a2aa8507df9c@thread.tacv2&groupId=8cd6b30e-edd9-48db-85ab-259fc11d0c5b) [1 pkt]\n",
"2. Stwórz prosty Github workflow wykorzystujący akcje CML, który:\n",
" - zrobi checkout Twojego repozytorium [2 pkt]\n",
" - ściągnie pliki trenujące. Najlepiej byłoby to zrobić za pomocą DVC, ale tym razem uprośćmy zadanie ze względu na koplikacje, które mogą się pojawić przy konfiguracji uwierzytelniania. Pliki można po prostu dodać do repozytorium albo ściągnąć przez wget jeśli są publicznie dostępne [2 pkt]\n",
" - dokona trenowania i ewaluacji [8 pkt]\n",
" - wyniki opublikuje za pomocą `cml-send-github-check` i/lub `cml-send-comment` [2 pkt]"
"2. Stwórz prosty Github workflow który:\n",
" - zrobi checkout Twojego repozytorium [1 pkt]\n",
" - ściągnie pliki trenujące. Najlepiej byłoby to zrobić za pomocą DVC, ale tym razem uprośćmy zadanie ze względu na komplikacje, które mogą się pojawić przy konfiguracji uwierzytelniania. Pliki można po prostu dodać do repozytorium albo ściągnąć przez wget jeśli są publicznie dostępne [2 pkt]\n",
" - będzie wywoływalny przez \"Workflow dispatch\" z parametrami trenowania [2 pkt]\n",
" - składał się będzie z co najmniej 3 jobów:\n",
" 1. dokona trenowania jako osobnej akcji wykonanej w Dockerze [8 pkt]\n",
" 2. dokona ewaluacji modelu [6 pkt]\n",
" 3. zarchiwizuje plik z modelem"
]
}
],
"metadata": {
"author": "Tomasz Ziętkiewicz",
"celltoolbar": "Slideshow",
"email": "tomasz.zietkiewicz@amu.edu.pl",
"kernelspec": {
"display_name": "Python 3",

2
IUM_11/github-actions-hello

@ -1 +1 @@
Subproject commit 97c7272bcaccf4a578615526b71ca723bb2b6d00
Subproject commit 2fe01837da8bb844e05fb9bf7f12fd37a124a236