"MLflow składa się z czterech niezależnych komponentów:\n",
" - **MLflow Tracking** - pozwala śledzić zmiany parametrów, kodu, środowiska i ich wpływ na metryki. Jest to funkcjonalność bardzo zbliżona do tej, którą zapewnia Sacred\n",
" - **MLflow Projects** - umożliwia \"pakowanie\" kodu ekserymentów w taki sposób, żeby mogłby być w łatwy sposób zreprodukowane przez innych\n",
" - **MLflow Models** - ułatwia \"pakowanie\" modeli uczenia maszynowego\n",
" - **MLflow Registry** - zapewnia centralne miejsce do przechowywania i współdzielenia modeli. Zapewnia narzędzia do wersjonowania i śledzenia pochodzenia tych modeli.\n",
" \n",
"Komponenty te mogą być używane razem bądź oddzielnie."
"ls: cannot access '/tmp/mlruns': No such file or directory\n",
"WARNING:urllib3.connectionpool:Retrying (Retry(total=4, connect=4, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd9988b7130>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456\n",
"WARNING:urllib3.connectionpool:Retrying (Retry(total=3, connect=3, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd9988b7580>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456\n",
"WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd9988b7730>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456\n",
"WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=1, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd9988b78e0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456\n",
"WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=0, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd9988b7a90>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456\n",
"urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fd9988b7c40>: Failed to establish a new connection: [Errno 111] Connection refused\n",
" File \"/home/pawel/ium/venv/lib/python3.10/site-packages/urllib3/util/retry.py\", line 515, in increment\n",
" raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]\n",
"urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd9988b7c40>: Failed to establish a new connection: [Errno 111] Connection refused'))\n",
"requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd9988b7c40>: Failed to establish a new connection: [Errno 111] Connection refused'))\n",
"mlflow.exceptions.MlflowException: API request to http://localhost:5000/api/2.0/mlflow/experiments/get-by-name failed with exception HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd9988b7c40>: Failed to establish a new connection: [Errno 111] Connection refused'))\n"
"WARNING:urllib3.connectionpool:Retrying (Retry(total=4, connect=4, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa7bbd8f130>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456\n",
"WARNING:urllib3.connectionpool:Retrying (Retry(total=3, connect=3, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa7bbd8f580>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456\n",
"WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa7bbd8f730>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456\n",
"WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=1, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa7bbd8f8e0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456\n",
"WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=0, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa7bbd8fa90>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456\n",
"urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fa7bbd8fc40>: Failed to establish a new connection: [Errno 111] Connection refused\n",
" File \"/home/pawel/ium/venv/lib/python3.10/site-packages/urllib3/util/retry.py\", line 515, in increment\n",
" raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]\n",
"urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa7bbd8fc40>: Failed to establish a new connection: [Errno 111] Connection refused'))\n",
"requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa7bbd8fc40>: Failed to establish a new connection: [Errno 111] Connection refused'))\n",
"mlflow.exceptions.MlflowException: API request to http://localhost:5000/api/2.0/mlflow/experiments/get-by-name failed with exception HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa7bbd8fc40>: Failed to establish a new connection: [Errno 111] Connection refused'))\n",
"WARNING:urllib3.connectionpool:Retrying (Retry(total=4, connect=4, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8ed9947130>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456\n",
"WARNING:urllib3.connectionpool:Retrying (Retry(total=3, connect=3, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8ed9947580>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456\n",
"WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8ed9947730>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456\n",
"WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=1, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8ed99478e0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456\n",
"WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=0, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8ed9947a90>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456\n",
"urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f8ed9947c40>: Failed to establish a new connection: [Errno 111] Connection refused\n",
" File \"/home/pawel/ium/venv/lib/python3.10/site-packages/urllib3/util/retry.py\", line 515, in increment\n",
" raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]\n",
"urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8ed9947c40>: Failed to establish a new connection: [Errno 111] Connection refused'))\n",
"requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8ed9947c40>: Failed to establish a new connection: [Errno 111] Connection refused'))\n",
"mlflow.exceptions.MlflowException: API request to http://localhost:5000/api/2.0/mlflow/experiments/get-by-name failed with exception HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8ed9947c40>: Failed to establish a new connection: [Errno 111] Connection refused'))\n",
"WARNING:urllib3.connectionpool:Retrying (Retry(total=4, connect=4, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f46d5b63130>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456\n",
"WARNING:urllib3.connectionpool:Retrying (Retry(total=3, connect=3, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f46d5b63580>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456\n",
"WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f46d5b63730>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456\n",
"WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=1, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f46d5b638e0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456\n",
"WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=0, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f46d5b63a90>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456\n",
"urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f46d5b63c40>: Failed to establish a new connection: [Errno 111] Connection refused\n",
" File \"/home/pawel/ium/venv/lib/python3.10/site-packages/urllib3/util/retry.py\", line 515, in increment\n",
" raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]\n",
"urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f46d5b63c40>: Failed to establish a new connection: [Errno 111] Connection refused'))\n",
"requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f46d5b63c40>: Failed to establish a new connection: [Errno 111] Connection refused'))\n",
"mlflow.exceptions.MlflowException: API request to http://localhost:5000/api/2.0/mlflow/experiments/get-by-name failed with exception HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f46d5b63c40>: Failed to establish a new connection: [Errno 111] Connection refused'))\n",
"WARNING:urllib3.connectionpool:Retrying (Retry(total=4, connect=4, read=5, redirect=5, status=5)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fda74a13130>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/2.0/mlflow/experiments/get-by-name?experiment_name=s123456\n",
"urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fda74a13580>: Failed to establish a new connection: [Errno 111] Connection refused\n",
" - logowania metryk i parametrów można dokonać m.in. poprzez wywołania Python-owego API: `mlflow.log_param()` i `mlflow.log_metric()`. Więcej dostępnych funkcji: [link](https://mlflow.org/docs/latest/tracking.html#logging-functions)\n",
" - wywołania te muszą nastąpić po wykonaniu [`mlflow.start_run()`](https://mlflow.org/docs/latest/python_api/mlflow.html#mlflow.start_run), najlepiej wewnątrz bloku:\n",
"```python\n",
" with mlflow.start_run():\n",
" \n",
" #[...]\n",
"\n",
" mlflow.log_param(\"alpha\", alpha)\n",
" mlflow.log_param(\"l1_ratio\", l1_ratio)\n",
"```\n",
" - jest też możliwość automatycznego logwania dla wybranych bibliotek: https://mlflow.org/docs/latest/tracking.html#automatic-logging"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# MLflow Projects\n",
" - MLflow projects to zestaw konwencji i kilku narzędzi\n",
" - ułatwiają one uruchamianie eskperymentów"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Konfiguracja projektu\n",
" - W pliku `MLproject` zapisuje się konfigurację projektu ([specyfikacja](https://mlflow.org/docs/latest/projects.html))\n",
" - Zawiera ona:\n",
" - odnośnik do środowiska, w którym ma być wywołany eksperyment [szczegóły](https://mlflow.org/docs/latest/projects.html#specifying-an-environment):\n",
" - nazwa obrazu Docker\n",
" - albo ścieżka do pliku conda.yaml definiującego środowisko wykonania Conda\n",
" - parametry, z którymi można wywołać eksperyment\n",
" - python=3.6 #Te zależności będą zainstalowane za pomocą conda isntall\n",
" - pip\n",
" - pip: #Te ząś za pomocą pip install\n",
" - scikit-learn==0.23.2\n",
" - mlflow>=1.0"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Środowisko docker\n",
"- zamiast środowiska Conda możemy również podać nazwę obrazu docker, w którym ma być wywołany eksperyment.\n",
"- obraz będzie szukany lokalnie a następnie na DockerHub, lub w innym repozytorium dockera\n",
"- składnia specyfikacji ścieżki jest taka sama jak w przypadki poleceń dockera, np. docker pull [link](https://docs.docker.com/engine/reference/commandline/pull/#pull-from-a-different-registry)\n",
"- Można również podać katalogi do podmontowania wewnątrz kontenera oraz wartości zmiennych środowiskowych do ustawienia w kontenerze:\n",
" - Specyfikacja parametrów w pliku MLproject pozwala na ich walidację i używanie wartości domyślnych\n",
" - Dostępne typy:\n",
" - String\n",
" - Float - dowolna liczba (MLflow waliduje, czy podana wartość jest liczbą)\n",
" - Path - pozwala podawać ścieżki względne (przekształca je na bezwzlędne) do plików lokalnych albo do plików zdalnych (np. do s3://) - zostaną wtedy ściągnięte lokalnie\n",
" - URI - podobnie jak path, ale do rozproszonych systemów plików\n",
" parameter_name: {type: data_type, default: value} # Short syntax\n",
"\n",
" parameter_name: # Long syntax\n",
" type: data_type\n",
" default: value\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Uruchamianie projektu\n",
" - Projekt możemy uruchomić przy pomocy polecenia `mlflow run` ([dokumentacja](https://mlflow.org/docs/latest/cli.html#mlflow-run))\n",
" - Spowoduje to przygotowanie środowiska i uruchomienie eksperymentu wewnątrz środowiska\n",
" - domyślnie zostanie uruchomione polecenie zdefiniowane w \"entry point\" `main`. Żeby uruchomić inny \"entry point\", możemy użyć parametru `-e`, np:\n",
"2021/05/16 17:59:10 INFO mlflow.projects.utils: === Created directory /tmp/tmprq4mdosv for downloading remote URIs passed to arguments of type 'path' ===\n",
"2021/05/16 17:59:10 INFO mlflow.projects.backend.local: === Running command 'source /home/tomek/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-5987e03d4dbaa5faa1a697bb113be9b9bdc39b29 1>&2 && python train.py 0.42 0.1' in run with ID '1860d321ea1545ff8866e4ba199d1712' === \n",
"Elasticnet model (alpha=0.420000, l1_ratio=0.100000):\n",
" RMSE: 0.7420620899060748\n",
" MAE: 0.5722846717246247\n",
" R2: 0.21978513651550236\n",
"2021/05/16 17:59:19 INFO mlflow.projects: === Run (ID '1860d321ea1545ff8866e4ba199d1712') succeeded ===\n"
"MLflow Models to konwencja zapisu modeli, która ułatwia potem ich załadowanie i użycie"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"Rodzaje modeli (\"flavors\") wspierane przez MLflow:\n",
"\n",
" - Python Function (python_function)\n",
" - PyTorch (pytorch)\n",
" - TensorFlow (tensorflow)\n",
" - Keras (keras)\n",
" - Scikit-learn (sklearn)\n",
" - Spacy(spaCy)\n",
" - ONNX (onnx)\n",
" - R Function (crate)\n",
" - H2O (h2o)\n",
" - MLeap (mleap)\n",
" - Spark MLlib (spark)\n",
" - MXNet Gluon (gluon)\n",
" - XGBoost (xgboost)\n",
" - LightGBM (lightgbm)\n",
" - CatBoost (catboost)\n",
" - Fastai(fastai)\n",
" - Statsmodels (statsmodels)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Zapisywanie modelu\n",
"Model ML można zapisać w MLflow przy pomocy jednej z dwóch funkcji z pakietu odpowiadającego używanej przez nas bibliotece:\n",
" - `save_model()` - zapisuje model na dysku\n",
" - `log_model()` - zapisuje model razem z innymi informacjami (metrykami, parametrami). W zależności od ustawień [\"tracking_uri\"](https://mlflow.org/docs/latest/python_api/mlflow.html#mlflow.set_tracking_uri) może być to lokalny folder w `mlruns/ ` lub ścieżka na zdalnym serwerze MLflow\n",
"\n",
"```Python\n",
" mlflow.sklearn.save_model(lr, \"my_model\")\n",
"```\n",
"\n",
"```Python\n",
" mlflow.keras.save_model(lr, \"my_model\")\n",
"```\n",
"\n",
"Wywołanie tej funkcji spowoduje stworzenie katalogu \"my_model\" zawierającego:\n",
" - plik *MLmodel* zawierający informacje o sposobach, w jaki model można załadować (\"flavors\") oraz ścieżki do plików związanych z modelem, takich jak:\n",
" - *conda.yaml* - opis środowiska potrzebnego do załadowania modelu\n",
" - *model.pkl* - plik z zserializowanym modelem\n",
"- *utc_time_created* - timestamp z czasem stworzenia modelu\n",
"- *run_id* - ID uruchomienia (\"run\"), które stworzyło ten model, jeśli model był zapisany za pomocą MLflow Tracking.\n",
"- *signature* - opisa danych wejściowych i wyjściowych w formacie JSON\n",
"- *input_example* przykładowe wejście przyjmowane przez model. Można je podać poprzez parametr `input_example` funkcji [log_model](https://mlflow.org/docs/latest/python_api/mlflow.sklearn.html#mlflow.sklearn.log_model)\n",
"[2021-05-17 08:52:07 +0200] [291217] [INFO] Using worker: sync\n",
"[2021-05-17 08:52:07 +0200] [291221] [INFO] Booting worker with pid: 291221\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## MLflow Registry\n",
" - umożliwia [zapisywanie](https://mlflow.org/docs/latest/model-registry.html#adding-an-mlflow-model-to-the-model-registry) i [ładowanie](https://mlflow.org/docs/latest/model-registry.html#fetching-an-mlflow-model-from-the-model-registry) modeli z centralnego rejestru\n",
" - Modele można też serwować bezpośrednio z rejestru:\n",
"\n",
"```bash\n",
"#!/usr/bin/env sh\n",
"\n",
"# Set environment variable for the tracking URL where the Model Registry resides\n",
