forked from tzietkiewicz/aitech-ium
Merge pull request 'master' (#20) from tzietkiewicz/aitech-ium:master into master
Reviewed-on: #20
This commit is contained in:
commit
b8bc0d05ee
87
IUM.Dla prowadzacego.ipynb
Normal file
87
IUM.Dla prowadzacego.ipynb
Normal file
@ -0,0 +1,87 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ac87c554",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Środowisko wykorzystywane na zajęciach"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "167416f3",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Certificat SSL dla strony: https://letsencrypt.org/\n",
|
||||
"Instrukcja instalacji: https://certbot.eff.org/lets-encrypt/ubuntufocal-other\n",
|
||||
"\n",
|
||||
"Instalacja certyfikatu w Jenkins: https://www.vhinandrich.com/jenkins-standalone-ssl-lets-encrypt\n",
|
||||
"\n",
|
||||
"Ścieżka do certyfikatu, którego ma używać Jenkins może być podana w dwóch miejscach (na Ubuntu 20.04):\n",
|
||||
"`/etc/default/jenkins` lub `/etc/init.d/jenkins`\n",
|
||||
"Ważne, aby w końcu w zmiennej `JENKINS_ARGS` znalazły się następujące parametry:\n",
|
||||
"```\n",
|
||||
"--httpPort=-1\n",
|
||||
"--httpsPort=8080\n",
|
||||
"--httpsKeyStore=/var/lib/jenkins/jenkins.jks\n",
|
||||
"--httpsKeyStorePassword=TUTAJ_HASŁO_PODANE_PRZY_KONWERSJI\"\n",
|
||||
"```\n",
|
||||
"albo:\n",
|
||||
"```\n",
|
||||
"--httpPort=-1\n",
|
||||
"--httpsPort=8080\n",
|
||||
"--httpsCertificate=/var/lib/jenkins/fullchain.pem\n",
|
||||
"--httpsPrivateKey=/var/lib/jenkins/privkey-rsa.pem\n",
|
||||
"```\n",
|
||||
"W zależności od tego, do jakiego formatu przekonwertowaliśmy nasz certyfikat\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Po 3 miesiącach certyfikat wygasa. Jeśli narzędzie certbot zostało poprawnie skonfigurowane,to powinno automatycznie odświeżać certyfikat. Niestety nie jest to certyfikat bezpośrednio używany przez Jenkins (ten wymaga konwersji do formatu `*.pem` albo `*.jks`). Dlatego przed wygaśnięciem certyfikatu musimy ręcznie dokonać konwersji (patrz https://www.vhinandrich.com/jenkins-standalone-ssl-lets-encrypt) albo dodać konwersję do post-hook w /etc/letsencrypt/renewal-hooks/post/ \n",
|
||||
"\n",
|
||||
"Przykładowy skrypt post (przy okazji restartuje działający na porcie 80 serwer nginx strzegący dostępu do MlFlow:\n",
|
||||
"```\n",
|
||||
"cd /home/tzietkiewicz/docker_mlflow_db && docker-compose start\n",
|
||||
"\n",
|
||||
"cd /etc/letsencrypt/live/tzietkiewicz.vm.wmi.amu.edu.pl/ && \\ \n",
|
||||
"openssl pkcs12 -inkey privkey.pem -in fullchain.pem -export -out keys.pkcs12 -password HASLO_CERTYFIKATU && \\\n",
|
||||
"keytool -importkeystore -srckeystore keys.pkcs12 -srcstoretype pkcs12 -destkeystore /var/lib/jenkins/jenkins.jks.new -srcstorepass HASLO_CERTYFIKATU -deststorepass HASLO_CERTYFIKATU && \\\n",
|
||||
"mv /var/lib/jenkins/jenkins.jks.new /var/lib/jenkins/jenkins.jks && \\\n",
|
||||
"service jenkins restart\n",
|
||||
"```\n",
|
||||
"`HASLO_CERTYFIKATU` może być dowolne - tutaj je generujemy, ale to samo musimy podać Jenkinsowi w `/etc/default/jenkins` lub `/etc/init.d/jenkins`"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "60970c71",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.1"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
@ -8,10 +8,14 @@
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"# AITech Inżynieria Uczenia Maszynowego\n",
|
||||
"![Logo 1](https://git.wmi.amu.edu.pl/AITech/Szablon/raw/branch/master/Logotyp_AITech1.jpg)\n",
|
||||
"<div class=\"alert alert-block alert-info\">\n",
|
||||
"<h1> Inżynieria uczenia maszynowego </h1>\n",
|
||||
"<h2> 0. <i>Informacje organizacyjne</i> [laboratoria]</h2> \n",
|
||||
"<h3> Tomasz Ziętkiewicz (2021)</h3>\n",
|
||||
"</div>\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## Informacje organizacyjne"
|
||||
"![Logo 2](https://git.wmi.amu.edu.pl/AITech/Szablon/raw/branch/master/Logotyp_AITech2.jpg)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -116,7 +120,7 @@
|
||||
"\n",
|
||||
"W trakcie kolejnych zajęć będą Państwo poznawać różne techniki i narzędzia wspomagające proces rozwoju modeli uczenia maszynowgo.\n",
|
||||
"\n",
|
||||
"W wyniku realizacji zadań z poszczególnych zajęć powstaną części składowe potoku uczenia maszynowego (Machine Learning Pipeline), które na końcu zostaną zintegrowane w jedną całość.\n",
|
||||
"W wyniku realizacji zadań z poszczególnych zajęć powstaną części składowe potoku uczenia maszynowego (Machine Learning Pipeline) z wykorzystaniem różnych, czasami komplementarnych a czasami alternatywnych technologii.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
" - Za wykonywanie na bieżąco zadań z poszczególnych zajęć będą przyznawane punkty.\n",
|
||||
@ -126,8 +130,6 @@
|
||||
" \n",
|
||||
" - W przypadku braku wykonania zadań w terminie nie zostaną przyznane za nie punkty.\n",
|
||||
" - W przypadku braku wystawienia przez prowadzącego punktów w terminie, uznaje się przyznanie za zadanie maksymalnej ilości punktów\n",
|
||||
" - Za wykonanie zadań cząstkowych w terminie można zdobyć 50% całkowitej liczby punktów\n",
|
||||
" - Za wykonanie finalnego projektu można zdobyć 50% całkowitej liczby punktów\n",
|
||||
" - Punkty przysługujące za każde zadanie cząstkowe będą podane przy opisie zadania\n",
|
||||
" \n",
|
||||
" Przelicznik punktów na oceny:\n",
|
||||
@ -155,12 +157,15 @@
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"author": "Tomasz Ziętkiewicz",
|
||||
"celltoolbar": "Slideshow",
|
||||
"email": "tomasz.zietkiewicz@amu.edu.pl",
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"lang": "pl",
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
@ -171,8 +176,13 @@
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.5"
|
||||
"version": "3.9.1"
|
||||
},
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
},
|
||||
"subtitle": "0.Organizacyjne[laboratoria]",
|
||||
"title": "Inżynieria uczenia maszynowego",
|
||||
"toc": {
|
||||
"base_numbering": 1,
|
||||
"nav_menu": {},
|
||||
@ -185,7 +195,8 @@
|
||||
"toc_position": {},
|
||||
"toc_section_display": false,
|
||||
"toc_window_display": false
|
||||
}
|
||||
},
|
||||
"year": "2021"
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
|
@ -8,8 +8,14 @@
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"### AITech Inżynieria Uczenia Maszynowego\n",
|
||||
"# 1. Wprowadzenie"
|
||||
"![Logo 1](https://git.wmi.amu.edu.pl/AITech/Szablon/raw/branch/master/Logotyp_AITech1.jpg)\n",
|
||||
"<div class=\"alert alert-block alert-info\">\n",
|
||||
"<h1> Inżynieria uczenia maszynowego </h1>\n",
|
||||
"<h2> 1. <i>Wprowadzenie</i> [laboratoria]</h2> \n",
|
||||
"<h3> Tomasz Ziętkiewicz (2021)</h3>\n",
|
||||
"</div>\n",
|
||||
"\n",
|
||||
"![Logo 2](https://git.wmi.amu.edu.pl/AITech/Szablon/raw/branch/master/Logotyp_AITech2.jpg)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -197,12 +203,15 @@
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"author": "Tomasz Ziętkiewicz",
|
||||
"celltoolbar": "Slideshow",
|
||||
"email": "tomasz.zietkiewicz@amu.edu.pl",
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"lang": "pl",
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
@ -213,8 +222,13 @@
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.5"
|
||||
"version": "3.9.1"
|
||||
},
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
},
|
||||
"subtitle": "1.Wprowadzenie[laboratoria]",
|
||||
"title": "Inżynieria uczenia maszynowego",
|
||||
"toc": {
|
||||
"base_numbering": 1,
|
||||
"nav_menu": {},
|
||||
@ -227,7 +241,8 @@
|
||||
"toc_position": {},
|
||||
"toc_section_display": false,
|
||||
"toc_window_display": false
|
||||
}
|
||||
},
|
||||
"year": "2021"
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
|
@ -1,5 +1,23 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"![Logo 1](https://git.wmi.amu.edu.pl/AITech/Szablon/raw/branch/master/Logotyp_AITech1.jpg)\n",
|
||||
"<div class=\"alert alert-block alert-info\">\n",
|
||||
"<h1> Inżynieria uczenia maszynowego </h1>\n",
|
||||
"<h2> 2. <i>Dane</i> [laboratoria]</h2> \n",
|
||||
"<h3> Tomasz Ziętkiewicz (2021)</h3>\n",
|
||||
"</div>\n",
|
||||
"\n",
|
||||
"![Logo 2](https://git.wmi.amu.edu.pl/AITech/Szablon/raw/branch/master/Logotyp_AITech2.jpg)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@ -1462,12 +1480,15 @@
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"author": "Tomasz Ziętkiewicz",
|
||||
"celltoolbar": "Slideshow",
|
||||
"email": "tomasz.zietkiewicz@amu.edu.pl",
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"lang": "pl",
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
@ -1478,8 +1499,13 @@
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.5"
|
||||
"version": "3.9.1"
|
||||
},
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
},
|
||||
"subtitle": "2.Dane[laboratoria]",
|
||||
"title": "Inżynieria uczenia maszynowego",
|
||||
"toc": {
|
||||
"base_numbering": 1,
|
||||
"nav_menu": {},
|
||||
@ -1492,7 +1518,8 @@
|
||||
"toc_position": {},
|
||||
"toc_section_display": false,
|
||||
"toc_window_display": false
|
||||
}
|
||||
},
|
||||
"year": "2021"
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
|
@ -2,14 +2,29 @@
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"# System ciągłej integracji na przykładzie Jenkins"
|
||||
"![Logo 1](https://git.wmi.amu.edu.pl/AITech/Szablon/raw/branch/master/Logotyp_AITech1.jpg)\n",
|
||||
"<div class=\"alert alert-block alert-info\">\n",
|
||||
"<h1> Inżynieria uczenia maszynowego </h1>\n",
|
||||
"<h2> 3. <i>System ciągłej integracji na przykładzie Jenkins</i> [laboratoria]</h2> \n",
|
||||
"<h3> Tomasz Ziętkiewicz (2021)</h3>\n",
|
||||
"</div>\n",
|
||||
"\n",
|
||||
"![Logo 2](https://git.wmi.amu.edu.pl/AITech/Szablon/raw/branch/master/Logotyp_AITech2.jpg)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"## Ciągła integracja \n",
|
||||
" - Jest to praktyka rozwoju projektów informatycznych polegająca na bardzo częstym włączaniu nowych zmian do głównej gałęzi (branch)\n",
|
||||
@ -21,7 +36,11 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"## Systemy ciągłej integracji \n",
|
||||
"- Umożliwiają utomatyczne :\n",
|
||||
@ -39,7 +58,11 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"## Systemy ciągłej integracji\n",
|
||||
" - Jenkins <br/><img style=\"height: 50px;\" src=\"img/ci/jenkins.png\"/>\n",
|
||||
@ -51,7 +74,11 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"## Jenkins \n",
|
||||
"<br/><img style=\"height: 50px;\" src=\"img/ci/jenkins.png\"/>\n",
|
||||
@ -67,7 +94,11 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"## Terminologia\n",
|
||||
" - **Job, aka. Pipleine (Projekt)** - podstawowa jednostka organizacji pracy wykonywanej przez Jenkinsa. \n",
|
||||
@ -79,7 +110,11 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
" - **Build (Zadanie)** - instancja uruchomionego projektu. Może być w trakcie wykonywania, albo zakończona z jednym z rezultatów:\n",
|
||||
" - Successful <img style=\"height: 30px;\" src=\"IUM_03/blue.png\"/>\n",
|
||||
@ -96,7 +131,11 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"## Terminologia c.d.\n",
|
||||
" - **Step (Krok?)** - każdy pipeline to sekwencja kroków do wykonania. \n",
|
||||
@ -113,7 +152,11 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"## Dokumentacja\n",
|
||||
"- https://www.jenkins.io/doc/book/pipeline/\n",
|
||||
@ -123,7 +166,11 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"## Zadania [15 pkt]\n",
|
||||
"Na dzisiejszych zajęciach przygotujemy dwa połączone ze sobą zadania:\n",
|
||||
@ -138,14 +185,22 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"## Polecenia"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"#### 1. Zaloguj się\n",
|
||||
" - zaloguj się na https://tzietkiewicz.vm.wmi.amu.edu.pl:8080 za pomocą konta wydziałowego (jak w laboratoriach WMI)"
|
||||
@ -153,7 +208,11 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"#### 2. Utwórz nowy projekt (Job) [1pkt]\n",
|
||||
"\n",
|
||||
@ -167,7 +226,11 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"#### 3. Definicja Pipeline [2pkt]\n",
|
||||
"\n",
|
||||
@ -189,7 +252,11 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"Przykładowy declarative Pipeline (https://tzietkiewicz.vm.wmi.amu.edu.pl:8080/job/hello-world/):\n",
|
||||
"\n",
|
||||
@ -228,7 +295,11 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"Przykładowy scripted Pipeline (https://tzietkiewicz.vm.wmi.amu.edu.pl:8080/job/hello-world-scripted/):\n",
|
||||
"\n",
|
||||
@ -263,7 +334,11 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"#### 4. Repozytorium [2 pkt]\n",
|
||||
"\n",
|
||||
@ -285,7 +360,11 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"#### 5. Zapisanie zbioru danych [4 pkt]\n",
|
||||
"Edytuj zawartość Jenkinsfile tak, żeby realizował on następujące zadania:\n",
|
||||
@ -301,7 +380,11 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"#### 6. Hasło do Kaggle\n",
|
||||
" - Jeśli pobierasz swój zbiór z Kaggle, to możesz mieć problem z uwierzytelnieniem.\n",
|
||||
@ -352,7 +435,11 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"#### 7. Parametry zadania [1 pkt]\n",
|
||||
"1. Dodaj do projektu `s123456-create-dataset` parametr teksotwy `CUTOFF`, który umożliwi zdefiniowanie wielkości odcięcia zbioru danych (czyli obetnie liczbę przykładów do `CUTOFF` pierwszych/losowych przykładów).\n",
|
||||
@ -366,7 +453,11 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"#### 6. Statystki [5 pkt]\n",
|
||||
"Stwórz projekt (job) s123456-dataset-stats, który zrealizuje następujące kroki:\n",
|
||||
@ -398,19 +489,32 @@
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"author": "Tomasz Ziętkiewicz",
|
||||
"celltoolbar": "Slideshow",
|
||||
"email": "tomasz.zietkiewicz@amu.edu.pl",
|
||||
"kernelspec": {
|
||||
"display_name": "Groovy",
|
||||
"language": "groovy",
|
||||
"name": "groovy"
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"lang": "pl",
|
||||
"language_info": {
|
||||
"codemirror_mode": "groovy",
|
||||
"file_extension": ".groovy",
|
||||
"mimetype": "",
|
||||
"name": "Groovy",
|
||||
"nbconverter_exporter": "",
|
||||
"version": "2.5.6"
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.1"
|
||||
},
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
},
|
||||
"subtitle": "3.CI-Jenkins[laboratoria]",
|
||||
"title": "Inżynieria uczenia maszynowego",
|
||||
"toc": {
|
||||
"base_numbering": 1,
|
||||
"nav_menu": {},
|
||||
@ -423,7 +527,8 @@
|
||||
"toc_position": {},
|
||||
"toc_section_display": false,
|
||||
"toc_window_display": false
|
||||
}
|
||||
},
|
||||
"year": "2021"
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
|
@ -8,7 +8,24 @@
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"# Konteneryzacja\n",
|
||||
"![Logo 1](https://git.wmi.amu.edu.pl/AITech/Szablon/raw/branch/master/Logotyp_AITech1.jpg)\n",
|
||||
"<div class=\"alert alert-block alert-info\">\n",
|
||||
"<h1> Inżynieria uczenia maszynowego </h1>\n",
|
||||
"<h2> 4. <i>Konteneryzacja</i> [laboratoria]</h2> \n",
|
||||
"<h3> Tomasz Ziętkiewicz (2021)</h3>\n",
|
||||
"</div>\n",
|
||||
"\n",
|
||||
"![Logo 2](https://git.wmi.amu.edu.pl/AITech/Szablon/raw/branch/master/Logotyp_AITech2.jpg)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"Plan na dziś\n",
|
||||
"\n",
|
||||
"1. Konteneryzacja i Docker\n",
|
||||
@ -562,12 +579,15 @@
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"author": "Tomasz Ziętkiewicz",
|
||||
"celltoolbar": "Slideshow",
|
||||
"email": "tomasz.zietkiewicz@amu.edu.pl",
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"lang": "pl",
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
@ -578,8 +598,13 @@
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.5"
|
||||
"version": "3.9.1"
|
||||
},
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
},
|
||||
"subtitle": "4.Konteneryzacja[laboratoria]",
|
||||
"title": "Inżynieria uczenia maszynowego",
|
||||
"toc": {
|
||||
"base_numbering": 1,
|
||||
"nav_menu": {},
|
||||
@ -592,7 +617,8 @@
|
||||
"toc_position": {},
|
||||
"toc_section_display": false,
|
||||
"toc_window_display": false
|
||||
}
|
||||
},
|
||||
"year": "2021"
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
|
@ -8,7 +8,14 @@
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"# Biblioteki Deep Learning"
|
||||
"![Logo 1](https://git.wmi.amu.edu.pl/AITech/Szablon/raw/branch/master/Logotyp_AITech1.jpg)\n",
|
||||
"<div class=\"alert alert-block alert-info\">\n",
|
||||
"<h1> Inżynieria uczenia maszynowego </h1>\n",
|
||||
"<h2> 5. <i>Biblioteki Deep Learning</i> [laboratoria]</h2> \n",
|
||||
"<h3> Tomasz Ziętkiewicz (2021)</h3>\n",
|
||||
"</div>\n",
|
||||
"\n",
|
||||
"![Logo 2](https://git.wmi.amu.edu.pl/AITech/Szablon/raw/branch/master/Logotyp_AITech2.jpg)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -305,12 +312,15 @@
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"author": "Tomasz Ziętkiewicz",
|
||||
"celltoolbar": "Slideshow",
|
||||
"email": "tomasz.zietkiewicz@amu.edu.pl",
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"lang": "pl",
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
@ -323,6 +333,11 @@
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.1"
|
||||
},
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
},
|
||||
"subtitle": "5.Biblioteki_DL[laboratoria]",
|
||||
"title": "Inżynieria uczenia maszynowego",
|
||||
"toc": {
|
||||
"base_numbering": 1,
|
||||
"nav_menu": {},
|
||||
@ -335,7 +350,8 @@
|
||||
"toc_position": {},
|
||||
"toc_section_display": false,
|
||||
"toc_window_display": false
|
||||
}
|
||||
},
|
||||
"year": "2021"
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
|
@ -8,7 +8,14 @@
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"# Jenkins II"
|
||||
"![Logo 1](https://git.wmi.amu.edu.pl/AITech/Szablon/raw/branch/master/Logotyp_AITech1.jpg)\n",
|
||||
"<div class=\"alert alert-block alert-info\">\n",
|
||||
"<h1> Inżynieria uczenia maszynowego </h1>\n",
|
||||
"<h2> 6. <i>Jenkins, część 2.</i> [laboratoria]</h2> \n",
|
||||
"<h3> Tomasz Ziętkiewicz (2021)</h3>\n",
|
||||
"</div>\n",
|
||||
"\n",
|
||||
"![Logo 2](https://git.wmi.amu.edu.pl/AITech/Szablon/raw/branch/master/Logotyp_AITech2.jpg)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -216,12 +223,15 @@
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"author": "Tomasz Ziętkiewicz",
|
||||
"celltoolbar": "Slideshow",
|
||||
"email": "tomasz.zietkiewicz@amu.edu.pl",
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"lang": "pl",
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
@ -232,8 +242,13 @@
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.5"
|
||||
"version": "3.9.1"
|
||||
},
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
},
|
||||
"subtitle": "6.Jenkins-2[laboratoria]",
|
||||
"title": "Inżynieria uczenia maszynowego",
|
||||
"toc": {
|
||||
"base_numbering": 1,
|
||||
"nav_menu": {},
|
||||
@ -246,7 +261,8 @@
|
||||
"toc_position": {},
|
||||
"toc_section_display": false,
|
||||
"toc_window_display": false
|
||||
}
|
||||
},
|
||||
"year": "2021"
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
|
@ -1,5 +1,23 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"![Logo 1](https://git.wmi.amu.edu.pl/AITech/Szablon/raw/branch/master/Logotyp_AITech1.jpg)\n",
|
||||
"<div class=\"alert alert-block alert-info\">\n",
|
||||
"<h1> Inżynieria uczenia maszynowego </h1>\n",
|
||||
"<h2> 7. <i>Sacred</i> [laboratoria]</h2> \n",
|
||||
"<h3> Tomasz Ziętkiewicz (2021)</h3>\n",
|
||||
"</div>\n",
|
||||
"\n",
|
||||
"![Logo 2](https://git.wmi.amu.edu.pl/AITech/Szablon/raw/branch/master/Logotyp_AITech2.jpg)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
@ -1493,12 +1511,15 @@
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"author": "Tomasz Ziętkiewicz",
|
||||
"celltoolbar": "Slideshow",
|
||||
"email": "tomasz.zietkiewicz@amu.edu.pl",
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"lang": "pl",
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
@ -1509,8 +1530,13 @@
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.5"
|
||||
"version": "3.9.1"
|
||||
},
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
},
|
||||
"subtitle": "7.Sacred[laboratoria]",
|
||||
"title": "Inżynieria uczenia maszynowego",
|
||||
"toc": {
|
||||
"base_numbering": 1,
|
||||
"nav_menu": {},
|
||||
@ -1523,7 +1549,8 @@
|
||||
"toc_position": {},
|
||||
"toc_section_display": false,
|
||||
"toc_window_display": false
|
||||
}
|
||||
},
|
||||
"year": "2021"
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
|
@ -8,8 +8,14 @@
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"# MLflow\n",
|
||||
"</br><img style=\"width: 50%;\" src=\"img/expcontrol/mlflow-logo-d.png\"/>\n"
|
||||
"![Logo 1](https://git.wmi.amu.edu.pl/AITech/Szablon/raw/branch/master/Logotyp_AITech1.jpg)\n",
|
||||
"<div class=\"alert alert-block alert-info\">\n",
|
||||
"<h1> Inżynieria uczenia maszynowego </h1>\n",
|
||||
"<h2> 8. <i>MLFlow</i> [laboratoria]</h2> \n",
|
||||
"<h3> Tomasz Ziętkiewicz (2021)</h3>\n",
|
||||
"</div>\n",
|
||||
"\n",
|
||||
"![Logo 2](https://git.wmi.amu.edu.pl/AITech/Szablon/raw/branch/master/Logotyp_AITech2.jpg)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -21,6 +27,8 @@
|
||||
},
|
||||
"source": [
|
||||
" ## MLflow\n",
|
||||
" </br><img style=\"width: 50%;\" src=\"img/expcontrol/mlflow-logo-d.png\"/>\n",
|
||||
"\n",
|
||||
" - https://mlflow.org/\n",
|
||||
" - Narzędzie podobne do omawianego na poprzednich zajęciach Sacred\n",
|
||||
" - Nieco inne podejście: mniej ingerencji w istniejący kod\n",
|
||||
@ -1413,12 +1421,15 @@
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"author": "Tomasz Ziętkiewicz",
|
||||
"celltoolbar": "Slideshow",
|
||||
"email": "tomasz.zietkiewicz@amu.edu.pl",
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"lang": "pl",
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
@ -1430,7 +1441,13 @@
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.1"
|
||||
}
|
||||
},
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
},
|
||||
"subtitle": "8.MLFlow[laboratoria]",
|
||||
"title": "Inżynieria uczenia maszynowego",
|
||||
"year": "2021"
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
|
@ -2,14 +2,21 @@
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "be5ab2df",
|
||||
"id": "4b9b298d",
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"# Środowiska wirtualne"
|
||||
"![Logo 1](https://git.wmi.amu.edu.pl/AITech/Szablon/raw/branch/master/Logotyp_AITech1.jpg)\n",
|
||||
"<div class=\"alert alert-block alert-info\">\n",
|
||||
"<h1> Inżynieria uczenia maszynowego </h1>\n",
|
||||
"<h2> 9. <i>Środowiska wirtualne</i> [laboratoria]</h2> \n",
|
||||
"<h3> Tomasz Ziętkiewicz (2021)</h3>\n",
|
||||
"</div>\n",
|
||||
"\n",
|
||||
"![Logo 2](https://git.wmi.amu.edu.pl/AITech/Szablon/raw/branch/master/Logotyp_AITech2.jpg)"
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -150,7 +157,11 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "90605b49",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"## pipx\n",
|
||||
" - pipx: polecenie, które instaluje moduł Pythonowy w odrębnym środowisku wirtualnym venv\n",
|
||||
@ -1715,12 +1726,15 @@
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"author": "Tomasz Ziętkiewicz",
|
||||
"celltoolbar": "Slideshow",
|
||||
"email": "tomasz.zietkiewicz@amu.edu.pl",
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"lang": "pl",
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
@ -1732,7 +1746,13 @@
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.1"
|
||||
}
|
||||
},
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
},
|
||||
"subtitle": "9.Python_srodowiska[laboratoria]",
|
||||
"title": "Inżynieria uczenia maszynowego",
|
||||
"year": "2021"
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
|
@ -1,5 +1,24 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7fe475ae",
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"![Logo 1](https://git.wmi.amu.edu.pl/AITech/Szablon/raw/branch/master/Logotyp_AITech1.jpg)\n",
|
||||
"<div class=\"alert alert-block alert-info\">\n",
|
||||
"<h1> Inżynieria uczenia maszynowego </h1>\n",
|
||||
"<h2> 10. <i>DVC</i> [laboratoria]</h2> \n",
|
||||
"<h3> Tomasz Ziętkiewicz (2021)</h3>\n",
|
||||
"</div>\n",
|
||||
"\n",
|
||||
"![Logo 2](https://git.wmi.amu.edu.pl/AITech/Szablon/raw/branch/master/Logotyp_AITech2.jpg)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0c6f27a5",
|
||||
@ -9,7 +28,6 @@
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"# DVC\n",
|
||||
"<img src=\"img/expcontrol/dvc-logo.png\">"
|
||||
]
|
||||
},
|
||||
@ -866,7 +884,11 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "c7f24f75",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!dvc push"
|
||||
@ -1343,7 +1365,11 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2f5a8590",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"## SSH remote\n",
|
||||
"Jednym z remote obsługiwanych przez DVC jest SFTP/SSH.\n",
|
||||
@ -1354,7 +1380,11 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "82a61107",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"### Lokalnie\n",
|
||||
"Będziemy potrzebować zależności :\n",
|
||||
@ -1370,7 +1400,11 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"id": "c48c5b8e",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
@ -1446,7 +1480,11 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": 27,
|
||||
"id": "e9a04876",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
@ -1465,7 +1503,11 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": 28,
|
||||
"id": "e3f27bbb",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
@ -1485,7 +1527,11 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": 32,
|
||||
"id": "5b2fa175",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
@ -1528,7 +1574,11 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1468c44c",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"### Jenkins\n",
|
||||
"\n",
|
||||
@ -1546,20 +1596,18 @@
|
||||
"```Jenkinsfile\n",
|
||||
"withCredentials(\n",
|
||||
" [sshUserPrivateKey(credentialsId: '48ac7004-216e-4260-abba-1fe5db753e18', keyFileVariable: 'IUM_SFTP_KEY', passphraseVariable: '', usernameVariable: '')]) {\n",
|
||||
" sh \"dvc init -f\"\n",
|
||||
" sh \"dvc remote add -d ium_ssh_remote ssh://ium-sftp@tzietkiewicz.vm.wmi.amu.edu.pl/ium-sftp\"\n",
|
||||
" sh \"dvc remote modify --local ium_ssh_remote keyfile $IUM_SFTP_KEY\"\n",
|
||||
" sh \"dvc pull\"}\n",
|
||||
" sh 'dvc remote add -d ium_ssh_remote ssh://ium-sftp@tzietkiewicz.vm.wmi.amu.edu.pl/ium-sftp'\n",
|
||||
" sh 'dvc remote modify --local ium_ssh_remote keyfile $IUM_SFTP_KEY'\n",
|
||||
" sh 'dvc pull'}\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"Secret text tak:\n",
|
||||
"\n",
|
||||
"```Jenkinsfile\n",
|
||||
" withCredentials([string(credentialsId: 'ium-sftp-password', variable: 'IUM_SFTP_PASS')]) {\n",
|
||||
" sh \"dvc init -f\"\n",
|
||||
" sh \"dvc remote add -d ium_ssh_remote ssh://ium-sftp@tzietkiewicz.vm.wmi.amu.edu.pl/ium-sftp\"\n",
|
||||
" sh \"dvc remote modify --local ium_ssh_remote password $IUM_SFTP_KEY\"\n",
|
||||
" sh \"dvc pull\"\n",
|
||||
" sh 'dvc remote add -d ium_ssh_remote ssh://ium-sftp@tzietkiewicz.vm.wmi.amu.edu.pl/ium-sftp'\n",
|
||||
" sh 'dvc remote modify --local ium_ssh_remote password $IUM_SFTP_KEY'\n",
|
||||
" sh 'dvc pull'\n",
|
||||
" }\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
@ -1570,12 +1618,15 @@
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"author": "Tomasz Ziętkiewicz",
|
||||
"celltoolbar": "Slideshow",
|
||||
"email": "tomasz.zietkiewicz@amu.edu.pl",
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"lang": "pl",
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
@ -1587,7 +1638,13 @@
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.1"
|
||||
}
|
||||
},
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
},
|
||||
"subtitle": "10.DVC[laboratoria]",
|
||||
"title": "Inżynieria uczenia maszynowego",
|
||||
"year": "2021"
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
|
182
IUM_11.CML.ipynb
182
IUM_11.CML.ipynb
@ -2,16 +2,31 @@
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "08cbeca2",
|
||||
"metadata": {},
|
||||
"id": "9d06fc91",
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"# Github actions i CML"
|
||||
"![Logo 1](https://git.wmi.amu.edu.pl/AITech/Szablon/raw/branch/master/Logotyp_AITech1.jpg)\n",
|
||||
"<div class=\"alert alert-block alert-info\">\n",
|
||||
"<h1> Inżynieria uczenia maszynowego </h1>\n",
|
||||
"<h2> 11. <i>Github actions i CML</i> [laboratoria]</h2> \n",
|
||||
"<h3> Tomasz Ziętkiewicz (2021)</h3>\n",
|
||||
"</div>\n",
|
||||
"\n",
|
||||
"![Logo 2](https://git.wmi.amu.edu.pl/AITech/Szablon/raw/branch/master/Logotyp_AITech2.jpg)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "beeb17b2",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"## Github actions\n",
|
||||
"<img src=\"img/expcontrol/github-actions.jpeg\">"
|
||||
@ -20,7 +35,11 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "752995e1",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
" - https://docs.github.com/en/actions\n",
|
||||
" - System ciągłej integracji \"wbudowany\" w GitHub\n",
|
||||
@ -31,7 +50,11 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2b06cc01",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"https://youtu.be/cP0I9w2coGU"
|
||||
]
|
||||
@ -39,7 +62,11 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b66dd41f",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"### Terminologia Github Actions\n",
|
||||
" - *Workflow* - workflow odpowiada \"Pipeline\" z Jenkinsa.\n",
|
||||
@ -55,7 +82,11 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "9f1f6d0a",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"### Definicja workflow\n",
|
||||
" - Workflow definiuje się w plikach YAM(o rozszerzeniu `*.yml` albo `*.yaml`) umieszczonych w specjalnym folderze `.github/workflows/` wewnątrz repozytorium\n",
|
||||
@ -74,7 +105,11 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "955585a9",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
@ -96,7 +131,11 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "f4916c1f",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
@ -115,7 +154,11 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "dde8d432",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
@ -145,7 +188,11 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ed780dea",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"### Zależności\n",
|
||||
"\n",
|
||||
@ -159,7 +206,11 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "28b582c4",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"### Akcje\n",
|
||||
"Za pomocą polecenia `uses` możemy używać przygotowanych wcześniej akcji. Mogą one pochodzić:\n",
|
||||
@ -171,7 +222,11 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "805622e8",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"## CML - Continous Machine Learning\n",
|
||||
"<img src=\"img/expcontrol/cml.png\">"
|
||||
@ -180,7 +235,11 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e0b3acbf",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
" - Tworzone przez [iterative.ai](iterative.ai) (tak jak DVC)\n",
|
||||
" - https://cml.dev/\n",
|
||||
@ -197,7 +256,11 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cdb54b38",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"### Przykładowy Workflow CML:"
|
||||
]
|
||||
@ -206,7 +269,11 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "07b1035a",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
@ -229,7 +296,11 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "bf27a2b3",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
@ -248,7 +319,11 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"id": "64f6e21d",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
@ -285,7 +360,11 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "83e49d3b",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# %load train.py\n",
|
||||
@ -323,7 +402,11 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8dc5748f",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"Wprowadźmy zmianę do pliku (linijka 17: `depth= = 6`)"
|
||||
]
|
||||
@ -332,7 +415,11 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"id": "afeaf939",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
@ -377,7 +464,11 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3e4a711a",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"Stwórzmy nowy branch \"deep_depth\":"
|
||||
]
|
||||
@ -386,7 +477,11 @@
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"id": "ab019b0b",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
@ -422,7 +517,11 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b50f46a8",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"<img src=\"IUM_11/img/github-pr.png\">"
|
||||
]
|
||||
@ -430,7 +529,11 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c56c8785",
|
||||
"metadata": {},
|
||||
"metadata": {
|
||||
"slideshow": {
|
||||
"slide_type": "slide"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"<img src=\"IUM_11/img/github-checks.png\">"
|
||||
]
|
||||
@ -438,25 +541,32 @@
|
||||