2021-05-10 12:53:57 +02:00
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
2021-09-28 10:56:21 +02:00
"\n",
"<div class=\"alert alert-block alert-info\">\n",
"<h1> Inżynieria uczenia maszynowego </h1>\n",
"<h2> 8. <i>MLFlow</i> [laboratoria]</h2> \n",
2022-05-09 09:54:34 +02:00
"<h3> Tomasz Ziętkiewicz (2022)</h3>\n",
2021-09-28 10:56:21 +02:00
"</div>\n",
"\n",
""
2021-05-10 12:53:57 +02:00
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
" ## MLflow\n",
2021-09-28 10:56:21 +02:00
" </br><img style=\"width: 50%;\" src=\"img/expcontrol/mlflow-logo-d.png\"/>\n",
"\n",
2021-05-10 12:53:57 +02:00
" - https://mlflow.org/\n",
" - Narzędzie podobne do omawianego na poprzednich zajęciach Sacred\n",
" - Nieco inne podejście: mniej ingerencji w istniejący kod\n",
" - Bardziej kompleksowe rozwiązanie: 4 komponenty, pierwszy z nich ma funkcjonalność podobną do Sacred\n",
" - Działa \"z każdym\" językiem. A tak naprawdę: Python, R, Java + CLI API + REST API\n",
" - Popularna wśród pracodawców - wyniki wyszukiwania ofert pracy: 20 ofert (https://pl.indeed.com/), 36 ofert (linkedin). Sacred: 0\n",
2022-05-09 09:54:34 +02:00
" - Integracja z licznymi bibliotekami / chmurami\n",
" - Rozwiązanie OpenSource, stworzone przez firmę Databricks\n",
" - Dostępna [odpłatna wersja \"Managed\"](https://databricks.com/product/managed-mlflow) (w ordóżnieniu od \"self-hosted\")\n"
2021-05-10 12:53:57 +02:00
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Komponenty\n",
"\n",
"MLflow składa się z czterech niezależnych komponentów:\n",
" - **MLflow Tracking** - pozwala śledzić zmiany parametrów, kodu, środowiska i ich wpływ na metryki. Jest to funkcjonalność bardzo zbliżona do tej, którą zapewnia Sacred\n",
" - **MLflow Projects** - umożliwia \"pakowanie\" kodu ekserymentów w taki sposób, żeby mogłby być w łatwy sposób zreprodukowane przez innych\n",
" - **MLflow Models** - ułatwia \"pakowanie\" modeli uczenia maszynowego\n",
" - **MLflow Registry** - zapewnia centralne miejsce do przechowywania i współdzielenia modeli. Zapewnia narzędzia do wersjonowania i śledzenia pochodzenia tych modeli.\n",
" \n",
"Komponenty te mogą być używane razem bądź oddzielnie."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## MLflow Tracking - przykład\n",
"(poniższe przykłady kodu trenującego pochodzą z tutoriala MLflow: https://mlflow.org/docs/latest/tutorials-and-examples/tutorial.html)"
]
},
{
"cell_type": "code",
2021-05-17 12:52:53 +02:00
"execution_count": 1,
2021-05-10 12:53:57 +02:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [],
"source": [
"%%capture null\n",
"!pip install mlflow\n",
"!pip install sklearn"
]
},
{
"cell_type": "code",
2021-05-19 22:55:34 +02:00
"execution_count": 80,
2021-05-10 12:53:57 +02:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Overwriting IUM_08/examples/sklearn_elasticnet_wine/train.py\n"
]
}
],
"source": [
"%%writefile IUM_08/examples/sklearn_elasticnet_wine/train.py\n",
"# The data set used in this example is from http://archive.ics.uci.edu/ml/datasets/Wine+Quality\n",
"# P. Cortez, A. Cerdeira, F. Almeida, T. Matos and J. Reis.\n",
"# Modeling wine preferences by data mining from physicochemical properties. In Decision Support Systems, Elsevier, 47(4):547-553, 2009.\n",
"\n",
"import os\n",
"import warnings\n",
"import sys\n",
"\n",
"import pandas as pd\n",
"import numpy as np\n",
"from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.linear_model import ElasticNet\n",
"from urllib.parse import urlparse\n",
"import mlflow\n",
"import mlflow.sklearn\n",
"\n",
"import logging\n",
"\n",
"logging.basicConfig(level=logging.WARN)\n",
"logger = logging.getLogger(__name__)\n",
"\n",
2021-05-19 22:55:34 +02:00
"mlflow.set_tracking_uri(\"http://localhost:5000\")\n",
"mlflow.set_experiment(\"s123456\")\n",
2021-05-10 12:53:57 +02:00
"\n",
"def eval_metrics(actual, pred):\n",
" rmse = np.sqrt(mean_squared_error(actual, pred))\n",
" mae = mean_absolute_error(actual, pred)\n",
" r2 = r2_score(actual, pred)\n",
" return rmse, mae, r2\n",
"\n",
"\n",
"if __name__ == \"__main__\":\n",
" warnings.filterwarnings(\"ignore\")\n",
" np.random.seed(40)\n",
"\n",
" # Read the wine-quality csv file from the URL\n",
" csv_url = (\n",
" \"http://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-red.csv\"\n",
" )\n",
" try:\n",
" data = pd.read_csv(csv_url, sep=\";\")\n",
" except Exception as e:\n",
" logger.exception(\n",
" \"Unable to download training & test CSV, check your internet connection. Error: %s\", e\n",
" )\n",
"\n",
" # Split the data into training and test sets. (0.75, 0.25) split.\n",
" train, test = train_test_split(data)\n",
"\n",
" # The predicted column is \"quality\" which is a scalar from [3, 9]\n",
" train_x = train.drop([\"quality\"], axis=1)\n",
" test_x = test.drop([\"quality\"], axis=1)\n",
" train_y = train[[\"quality\"]]\n",
" test_y = test[[\"quality\"]]\n",
"\n",
" \n",
" alpha = float(sys.argv[1]) if len(sys.argv) > 1 else 0.5\n",
" #alpha = 0.5\n",
" l1_ratio = float(sys.argv[2]) if len(sys.argv) > 2 else 0.5\n",
" #l1_ratio = 0.5\n",
"\n",
2021-05-19 22:55:34 +02:00
" with mlflow.start_run() as run:\n",
" print(\"MLflow run experiment_id: {0}\".format(run.info.experiment_id))\n",
" print(\"MLflow run artifact_uri: {0}\".format(run.info.artifact_uri))\n",
"\n",
2021-05-10 12:53:57 +02:00
" lr = ElasticNet(alpha=alpha, l1_ratio=l1_ratio, random_state=42)\n",
" lr.fit(train_x, train_y)\n",
"\n",
" predicted_qualities = lr.predict(test_x)\n",
"\n",
" (rmse, mae, r2) = eval_metrics(test_y, predicted_qualities)\n",
"\n",
" print(\"Elasticnet model (alpha=%f, l1_ratio=%f):\" % (alpha, l1_ratio))\n",
" print(\" RMSE: %s\" % rmse)\n",
" print(\" MAE: %s\" % mae)\n",
" print(\" R2: %s\" % r2)\n",
"\n",
" mlflow.log_param(\"alpha\", alpha)\n",
" mlflow.log_param(\"l1_ratio\", l1_ratio)\n",
" mlflow.log_metric(\"rmse\", rmse)\n",
" mlflow.log_metric(\"r2\", r2)\n",
" mlflow.log_metric(\"mae\", mae)\n",
2021-05-17 12:52:53 +02:00
" \n",
" # Infer model signature to log it\n",
" signature = mlflow.models.signature.infer_signature(train_x, lr.predict(train_x))\n",
2021-05-10 12:53:57 +02:00
"\n",
" tracking_url_type_store = urlparse(mlflow.get_tracking_uri()).scheme\n",
"\n",
" # Model registry does not work with file store\n",
" if tracking_url_type_store != \"file\":\n",
"\n",
" # Register the model\n",
" # There are other ways to use the Model Registry, which depends on the use case,\n",
" # please refer to the doc for more information:\n",
" # https://mlflow.org/docs/latest/model-registry.html#api-workflow\n",
2021-05-19 22:55:34 +02:00
" mlflow.sklearn.log_model(lr, \"wines-model\", registered_model_name=\"ElasticnetWineModel\", signature=signature)\n",
2021-05-10 12:53:57 +02:00
" else:\n",
2021-05-17 12:52:53 +02:00
" mlflow.sklearn.log_model(lr, \"model\", signature=signature)"
2021-05-10 12:53:57 +02:00
]
},
{
"cell_type": "code",
2021-05-19 22:55:34 +02:00
"execution_count": 81,
2021-05-10 12:53:57 +02:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
2021-05-19 22:55:34 +02:00
"total 4\n",
"drwxrwxr-x 3 tomek tomek 4096 maj 19 21:31 1\n",
"INFO: 's123456' does not exist. Creating a new experiment\n",
"MLflow run experiment_id: 2\n",
"MLflow run artifact_uri: /tmp/mlruns/2/c15feb5df335490ba990ddd4dd977c1b/artifacts\n",
"Elasticnet model (alpha=0.500000, l1_ratio=0.500000):\n",
" RMSE: 0.7931640229276851\n",
" MAE: 0.6271946374319586\n",
" R2: 0.10862644997792614\n",
"Registered model 'ElasticnetWineModel' already exists. Creating a new version of this model...\n",
"2021/05/19 22:34:48 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation. Model name: ElasticnetWineModel, version 2\n",
"Created version '2' of model 'ElasticnetWineModel'.\n"
2021-05-10 12:53:57 +02:00
]
}
],
"source": [
2021-05-19 22:55:34 +02:00
"! ls -l /tmp/mlruns\n",
2021-05-10 12:53:57 +02:00
"### Wtyrenujmy model z domyślnymi wartościami parametrów\n",
"! cd ./IUM_08/examples/; python sklearn_elasticnet_wine/train.py"
]
},
{
"cell_type": "code",
2021-05-17 12:52:53 +02:00
"execution_count": 4,
2021-05-10 12:53:57 +02:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Elasticnet model (alpha=0.100000, l1_ratio=0.100000):\n",
" RMSE: 0.7128829045893679\n",
" MAE: 0.5462202174984664\n",
" R2: 0.2799376066653344\n",
"Elasticnet model (alpha=0.200000, l1_ratio=0.100000):\n",
" RMSE: 0.7268133518615142\n",
" MAE: 0.5586842416161892\n",
" R2: 0.251521166881557\n",
"Elasticnet model (alpha=0.300000, l1_ratio=0.100000):\n",
" RMSE: 0.7347397539240514\n",
" MAE: 0.5657315547549873\n",
" R2: 0.23510678899596094\n",
"Elasticnet model (alpha=0.400000, l1_ratio=0.100000):\n",
" RMSE: 0.7410782793160982\n",
" MAE: 0.5712718681984227\n",
" R2: 0.22185255063708875\n",
"Elasticnet model (alpha=0.500000, l1_ratio=0.100000):\n",
" RMSE: 0.7460550348172179\n",
" MAE: 0.576381895873763\n",
" R2: 0.21136606570632266\n",
"Elasticnet model (alpha=0.600000, l1_ratio=0.100000):\n",
" RMSE: 0.7510866447955419\n",
" MAE: 0.5815681289333974\n",
" R2: 0.20069264568704714\n",
"Elasticnet model (alpha=0.700000, l1_ratio=0.100000):\n",
" RMSE: 0.7560654760040749\n",
" MAE: 0.5868129921328281\n",
" R2: 0.19006056603695476\n",
"Elasticnet model (alpha=0.800000, l1_ratio=0.100000):\n",
" RMSE: 0.7609263702116827\n",
" MAE: 0.5919470003487062\n",
" R2: 0.17961256649282442\n",
"Elasticnet model (alpha=0.900000, l1_ratio=0.100000):\n",
" RMSE: 0.7656313758553691\n",
" MAE: 0.5969367233859049\n",
" R2: 0.16943586313742276\n",
"Elasticnet model (alpha=0.100000, l1_ratio=0.200000):\n",
" RMSE: 0.7201489594275661\n",
" MAE: 0.5525324524014098\n",
" R2: 0.26518433811823017\n",
"Elasticnet model (alpha=0.200000, l1_ratio=0.200000):\n",
" RMSE: 0.7336400911821402\n",
" MAE: 0.5643841279275428\n",
" R2: 0.23739466063584158\n",
"Elasticnet model (alpha=0.300000, l1_ratio=0.200000):\n",
" RMSE: 0.7397486012946922\n",
" MAE: 0.5704931175017443\n",
" R2: 0.22464242411894242\n",
"Elasticnet model (alpha=0.400000, l1_ratio=0.200000):\n",
" RMSE: 0.7468093030485085\n",
" MAE: 0.5777243300021722\n",
" R2: 0.2097706278632726\n",
"Elasticnet model (alpha=0.500000, l1_ratio=0.200000):\n",
" RMSE: 0.7543919979968401\n",
" MAE: 0.5857669727382302\n",
" R2: 0.19364204365178095\n",
"Elasticnet model (alpha=0.600000, l1_ratio=0.200000):\n",
" RMSE: 0.7622123676513404\n",
" MAE: 0.5938629318868578\n",
" R2: 0.17683724501340814\n",
"Elasticnet model (alpha=0.700000, l1_ratio=0.200000):\n",
" RMSE: 0.7700845840888665\n",
" MAE: 0.6024685725504659\n",
" R2: 0.15974600028150265\n",
"Elasticnet model (alpha=0.800000, l1_ratio=0.200000):\n",
" RMSE: 0.7778880968569085\n",
" MAE: 0.6105907461474273\n",
" R2: 0.14263059582492588\n",
"Elasticnet model (alpha=0.900000, l1_ratio=0.200000):\n",
" RMSE: 0.7855450337039626\n",
" MAE: 0.6182359127922239\n",
" R2: 0.1256689455181047\n",
"Elasticnet model (alpha=0.100000, l1_ratio=0.300000):\n",
" RMSE: 0.7260299544064643\n",
" MAE: 0.5571534327625295\n",
" R2: 0.2531337966130104\n",
"Elasticnet model (alpha=0.200000, l1_ratio=0.300000):\n",
" RMSE: 0.7357092639331829\n",
" MAE: 0.5667609266233857\n",
" R2: 0.23308686049079996\n",
"Elasticnet model (alpha=0.300000, l1_ratio=0.300000):\n",
" RMSE: 0.7443224557281489\n",
" MAE: 0.5754825491733004\n",
" R2: 0.2150247343683439\n",
"Elasticnet model (alpha=0.400000, l1_ratio=0.300000):\n",
" RMSE: 0.7545302211047864\n",
" MAE: 0.5862255018460154\n",
" R2: 0.19334652749043568\n",
"Elasticnet model (alpha=0.500000, l1_ratio=0.300000):\n",
" RMSE: 0.7657094552843393\n",
" MAE: 0.597876674089536\n",
" R2: 0.16926645189778677\n",
"Elasticnet model (alpha=0.600000, l1_ratio=0.300000):\n",
" RMSE: 0.7774287676055035\n",
" MAE: 0.6102458961382884\n",
" R2: 0.14364282001967787\n",
"Elasticnet model (alpha=0.700000, l1_ratio=0.300000):\n",
" RMSE: 0.7876149030178985\n",
" MAE: 0.6208628759605734\n",
" R2: 0.12105524358911324\n",
"Elasticnet model (alpha=0.800000, l1_ratio=0.300000):\n",
" RMSE: 0.7972426725990548\n",
" MAE: 0.6310633254738363\n",
" R2: 0.09943554388738107\n",
"Elasticnet model (alpha=0.900000, l1_ratio=0.300000):\n",
" RMSE: 0.806653553139972\n",
" MAE: 0.6407940021176486\n",
" R2: 0.07804901733081859\n",
"Elasticnet model (alpha=0.100000, l1_ratio=0.400000):\n",
" RMSE: 0.7301757756825391\n",
" MAE: 0.5603782497631705\n",
" R2: 0.24457984004307665\n",
"Elasticnet model (alpha=0.200000, l1_ratio=0.400000):\n",
" RMSE: 0.7383379454127179\n",
" MAE: 0.5696920200435643\n",
" R2: 0.22759672468382497\n",
"Elasticnet model (alpha=0.300000, l1_ratio=0.400000):\n",
" RMSE: 0.7501603725852\n",
" MAE: 0.5818749078280213\n",
" R2: 0.2026629101382652\n",
"Elasticnet model (alpha=0.400000, l1_ratio=0.400000):\n",
" RMSE: 0.7644619587468349\n",
" MAE: 0.5966303605775048\n",
" R2: 0.17197111491474282\n",
"Elasticnet model (alpha=0.500000, l1_ratio=0.400000):\n",
" RMSE: 0.7794144864140182\n",
" MAE: 0.6125287339702588\n",
" R2: 0.1392625955410326\n",
"Elasticnet model (alpha=0.600000, l1_ratio=0.400000):\n",
" RMSE: 0.7928446872861473\n",
" MAE: 0.626666444473971\n",
" R2: 0.10934405701835759\n",
"Elasticnet model (alpha=0.700000, l1_ratio=0.400000):\n",
" RMSE: 0.8064523157995205\n",
" MAE: 0.6407990295001776\n",
" R2: 0.07850896155515663\n",
"Elasticnet model (alpha=0.800000, l1_ratio=0.400000):\n",
" RMSE: 0.8200264141399415\n",
" MAE: 0.6539313398770489\n",
" R2: 0.04722706260889009\n",
"Elasticnet model (alpha=0.900000, l1_ratio=0.400000):\n",
" RMSE: 0.8317936823364004\n",
" MAE: 0.6647839366878934\n",
" R2: 0.01968654319755092\n",
"Elasticnet model (alpha=0.100000, l1_ratio=0.500000):\n",
" RMSE: 0.7308996187375898\n",
" MAE: 0.5615486628017713\n",
" R2: 0.2430813606733676\n",
"Elasticnet model (alpha=0.200000, l1_ratio=0.500000):\n",
" RMSE: 0.7415652207304311\n",
" MAE: 0.573067857646195\n",
" R2: 0.22082961765864062\n",
"Elasticnet model (alpha=0.300000, l1_ratio=0.500000):\n",
" RMSE: 0.7573787958793151\n",
" MAE: 0.5893143148791096\n",
" R2: 0.18724431943947983\n",
"Elasticnet model (alpha=0.400000, l1_ratio=0.500000):\n",
" RMSE: 0.7759342885655987\n",
" MAE: 0.6090076377075831\n",
" R2: 0.14693206734185604\n",
"Elasticnet model (alpha=0.500000, l1_ratio=0.500000):\n",
" RMSE: 0.7931640229276851\n",
" MAE: 0.6271946374319586\n",
" R2: 0.10862644997792614\n",
"Elasticnet model (alpha=0.600000, l1_ratio=0.500000):\n",
" RMSE: 0.8112953030727291\n",
" MAE: 0.645693705089251\n",
" R2: 0.06740807086129252\n",
"Elasticnet model (alpha=0.700000, l1_ratio=0.500000):\n",
" RMSE: 0.8298921852578498\n",
" MAE: 0.6629780128961713\n",
" R2: 0.024163452726365775\n",
"Elasticnet model (alpha=0.800000, l1_ratio=0.500000):\n",
" RMSE: 0.8320198635059106\n",
" MAE: 0.6657357030427604\n",
" R2: 0.019153337439844154\n",
"Elasticnet model (alpha=0.900000, l1_ratio=0.500000):\n",
" RMSE: 0.8323808561832262\n",
" MAE: 0.6669472047761406\n",
" R2: 0.0183020229672054\n",
"Elasticnet model (alpha=0.100000, l1_ratio=0.600000):\n",
" RMSE: 0.7317723392279818\n",
" MAE: 0.5627373693033669\n",
" R2: 0.24127270524006605\n",
"Elasticnet model (alpha=0.200000, l1_ratio=0.600000):\n",
" RMSE: 0.7454324777911233\n",
" MAE: 0.5772117261484206\n",
" R2: 0.21268169183406394\n",
"Elasticnet model (alpha=0.300000, l1_ratio=0.600000):\n",
" RMSE: 0.7661028672396263\n",
" MAE: 0.5984406933733759\n",
" R2: 0.16841259155853305\n",
"Elasticnet model (alpha=0.400000, l1_ratio=0.600000):\n",
" RMSE: 0.787179486885359\n",
" MAE: 0.6210967388389844\n",
" R2: 0.12202678676193257\n",
"Elasticnet model (alpha=0.500000, l1_ratio=0.600000):\n",
" RMSE: 0.809739471626647\n",
" MAE: 0.6442565454817458\n",
" R2: 0.07098152823463388\n",
"Elasticnet model (alpha=0.600000, l1_ratio=0.600000):\n",
" RMSE: 0.8317884179944764\n",
" MAE: 0.6647524814105722\n",
" R2: 0.019698951776764728\n",
"Elasticnet model (alpha=0.700000, l1_ratio=0.600000):\n",
" RMSE: 0.8321519738036909\n",
" MAE: 0.6662086037874676\n",
" R2: 0.018841829895677176\n",
"Elasticnet model (alpha=0.800000, l1_ratio=0.600000):\n",
" RMSE: 0.8326350511178233\n",
" MAE: 0.6676630843299566\n",
" R2: 0.01770234373563795\n",
"Elasticnet model (alpha=0.900000, l1_ratio=0.600000):\n",
" RMSE: 0.8332048101440411\n",
" MAE: 0.6690717294644856\n",
" R2: 0.016357542209390563\n",
"Elasticnet model (alpha=0.100000, l1_ratio=0.700000):\n",
" RMSE: 0.7327938109945942\n",
" MAE: 0.5640101718105491\n",
" R2: 0.23915303116151632\n",
"Elasticnet model (alpha=0.200000, l1_ratio=0.700000):\n",
" RMSE: 0.7499835110445395\n",
" MAE: 0.5819389930665501\n",
" R2: 0.20303883413454027\n",
"Elasticnet model (alpha=0.300000, l1_ratio=0.700000):\n",
" RMSE: 0.7747136483567111\n",
" MAE: 0.6079678532556209\n",
" R2: 0.14961391810397695\n",
"Elasticnet model (alpha=0.400000, l1_ratio=0.700000):\n",
" RMSE: 0.8004478857657858\n",
" MAE: 0.6350378679245181\n",
" R2: 0.09217977708630032\n",
"Elasticnet model (alpha=0.500000, l1_ratio=0.700000):\n",
" RMSE: 0.829586285479097\n",
" MAE: 0.6627028304266674\n",
" R2: 0.024882710417618137\n",
"Elasticnet model (alpha=0.600000, l1_ratio=0.700000):\n",
" RMSE: 0.8321502650365332\n",
" MAE: 0.6662000872414003\n",
" R2: 0.018845859373919027\n",
"Elasticnet model (alpha=0.700000, l1_ratio=0.700000):\n",
" RMSE: 0.832725785743381\n",
" MAE: 0.667898097502809\n",
" R2: 0.017488244494447747\n",
"Elasticnet model (alpha=0.800000, l1_ratio=0.700000):\n",
" RMSE: 0.8331825395236181\n",
" MAE: 0.6692175076829847\n",
" R2: 0.016410124803194592\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Elasticnet model (alpha=0.900000, l1_ratio=0.700000):\n",
" RMSE: 0.8331069437643933\n",
" MAE: 0.6697424890266508\n",
" R2: 0.016588601539516357\n",
"Elasticnet model (alpha=0.100000, l1_ratio=0.800000):\n",
" RMSE: 0.7339712501091269\n",
" MAE: 0.5654097809725043\n",
" R2: 0.23670603806205326\n",
"Elasticnet model (alpha=0.200000, l1_ratio=0.800000):\n",
" RMSE: 0.7552646505492441\n",
" MAE: 0.5873472009739388\n",
" R2: 0.19177543499093674\n",
"Elasticnet model (alpha=0.300000, l1_ratio=0.800000):\n",
" RMSE: 0.7836957692333741\n",
" MAE: 0.6176788505535867\n",
" R2: 0.12978065429593022\n",
"Elasticnet model (alpha=0.400000, l1_ratio=0.800000):\n",
" RMSE: 0.8160164529135189\n",
" MAE: 0.650349905850893\n",
" R2: 0.05652247327326554\n",
"Elasticnet model (alpha=0.500000, l1_ratio=0.800000):\n",
" RMSE: 0.8320145539945119\n",
" MAE: 0.6657081587004348\n",
" R2: 0.019165855890777572\n",
"Elasticnet model (alpha=0.600000, l1_ratio=0.800000):\n",
" RMSE: 0.8326325509502465\n",
" MAE: 0.6676500690618903\n",
" R2: 0.01770824285088779\n",
"Elasticnet model (alpha=0.700000, l1_ratio=0.800000):\n",
" RMSE: 0.8331830329685253\n",
" MAE: 0.6692142378162035\n",
" R2: 0.016408959758236752\n",
"Elasticnet model (alpha=0.800000, l1_ratio=0.800000):\n",
" RMSE: 0.8330972295348316\n",
" MAE: 0.669813814205792\n",
" R2: 0.016611535037920344\n",
"Elasticnet model (alpha=0.900000, l1_ratio=0.800000):\n",
" RMSE: 0.8330208354420413\n",
" MAE: 0.6704133670619602\n",
" R2: 0.016791878033996177\n",
"Elasticnet model (alpha=0.100000, l1_ratio=0.900000):\n",
" RMSE: 0.735314956888905\n",
" MAE: 0.566974647785579\n",
" R2: 0.23390870203034675\n",
"Elasticnet model (alpha=0.200000, l1_ratio=0.900000):\n",
" RMSE: 0.7613249071370938\n",
" MAE: 0.593613372674502\n",
" R2: 0.1787529818606436\n",
"Elasticnet model (alpha=0.300000, l1_ratio=0.900000):\n",
" RMSE: 0.7940027723712206\n",
" MAE: 0.6284316436541582\n",
" R2: 0.10674024649047587\n",
"Elasticnet model (alpha=0.400000, l1_ratio=0.900000):\n",
" RMSE: 0.831784893250733\n",
" MAE: 0.6647313794016759\n",
" R2: 0.019707259905588637\n",
"Elasticnet model (alpha=0.500000, l1_ratio=0.900000):\n",
" RMSE: 0.8323747376136406\n",
" MAE: 0.6669171677143245\n",
" R2: 0.018316455219614114\n",
"Elasticnet model (alpha=0.600000, l1_ratio=0.900000):\n",
" RMSE: 0.8332063354920289\n",
" MAE: 0.6690618761753936\n",
" R2: 0.01635394069773599\n",
"Elasticnet model (alpha=0.700000, l1_ratio=0.900000):\n",
" RMSE: 0.8331078270287657\n",
" MAE: 0.6697360518827573\n",
" R2: 0.016586516302516174\n",
"Elasticnet model (alpha=0.800000, l1_ratio=0.900000):\n",
" RMSE: 0.8330212125502486\n",
" MAE: 0.6704102143580977\n",
" R2: 0.016790987837928095\n",
"Elasticnet model (alpha=0.900000, l1_ratio=0.900000):\n",
" RMSE: 0.8329464950658837\n",
" MAE: 0.6710843636018047\n",
" R2: 0.01696735695860563\n"
]
}
],
"source": [
"### I jeszcze raz, tym razem ze zmienionymi wartościami parametrów\n",
"! cd ./IUM_08/examples/; for l in {1..9}; do for a in {1..9}; do python sklearn_elasticnet_wine/train.py 0.$a 0.$l; done; done"
]
},
{
"cell_type": "code",
2021-05-17 12:52:53 +02:00
"execution_count": 32,
2021-05-10 12:53:57 +02:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total 16\r\n",
2021-05-17 12:52:53 +02:00
"drwxrwxr-x 6 tomek tomek 4096 maj 17 08:43 375cde31bdd44a45a91fd7cee92ebcda\r\n",
"drwxrwxr-x 6 tomek tomek 4096 maj 17 10:38 b395b55b47fc43de876b67f5a4a5dae9\r\n",
"drwxrwxr-x 6 tomek tomek 4096 maj 17 09:15 b3ead42eca964113b29e7e5f8bcb7bb7\r\n",
"-rw-rw-r-- 1 tomek tomek 151 maj 17 08:43 meta.yaml\r\n"
2021-05-10 12:53:57 +02:00
]
}
],
"source": [
"### Informacje o przebieagach eksperymentu zostały zapisane w katalogu mlruns\n",
2021-05-17 12:52:53 +02:00
"! ls -l IUM_08/examples/mlruns/0 | head"
2021-05-10 12:53:57 +02:00
]
},
{
"cell_type": "code",
2021-05-17 12:52:53 +02:00
"execution_count": 33,
2021-05-10 12:53:57 +02:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
2021-05-17 12:52:53 +02:00
"total 20\r\n",
"drwxrwxr-x 3 tomek tomek 4096 maj 17 08:43 artifacts\r\n",
"-rw-rw-r-- 1 tomek tomek 423 maj 17 08:43 meta.yaml\r\n",
"drwxrwxr-x 2 tomek tomek 4096 maj 17 08:43 metrics\r\n",
"drwxrwxr-x 2 tomek tomek 4096 maj 17 08:43 params\r\n",
"drwxrwxr-x 2 tomek tomek 4096 maj 17 08:43 tags\r\n"
]
}
],
"source": [
"! ls -l IUM_08/examples/mlruns/0/375cde31bdd44a45a91fd7cee92ebcda"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[2021-05-16 17:58:43 +0200] [118029] [INFO] Starting gunicorn 20.1.0\n",
"[2021-05-16 17:58:43 +0200] [118029] [ERROR] Connection in use: ('127.0.0.1', 5000)\n",
"[2021-05-16 17:58:43 +0200] [118029] [ERROR] Retrying in 1 second.\n",
"[2021-05-16 17:58:44 +0200] [118029] [ERROR] Connection in use: ('127.0.0.1', 5000)\n",
"[2021-05-16 17:58:44 +0200] [118029] [ERROR] Retrying in 1 second.\n",
"[2021-05-16 17:58:45 +0200] [118029] [ERROR] Connection in use: ('127.0.0.1', 5000)\n",
"[2021-05-16 17:58:45 +0200] [118029] [ERROR] Retrying in 1 second.\n",
"[2021-05-16 17:58:46 +0200] [118029] [ERROR] Connection in use: ('127.0.0.1', 5000)\n",
"[2021-05-16 17:58:46 +0200] [118029] [ERROR] Retrying in 1 second.\n",
"[2021-05-16 17:58:47 +0200] [118029] [ERROR] Connection in use: ('127.0.0.1', 5000)\n",
"[2021-05-16 17:58:47 +0200] [118029] [ERROR] Retrying in 1 second.\n",
"[2021-05-16 17:58:48 +0200] [118029] [ERROR] Can't connect to ('127.0.0.1', 5000)\n",
"Running the mlflow server failed. Please see the logs above for details.\n"
2021-05-10 12:53:57 +02:00
]
}
],
"source": [
"### Możemy je obejrzeć w przeglądarce uruchamiając interfejs webowy:\n",
"### (powinniśmy to wywołać w normalnej konsoli, w jupyter będziemy mieli zablokowany kernel)\n",
"! cd IUM_08/examples/; mlflow ui"
]
},
2022-05-09 09:54:34 +02:00
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Instancja na naszym serwerze: http://tzietkiewicz.vm.wmi.amu.edu.pl:5000/#/"
]
},
2021-05-10 12:53:57 +02:00
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Wygląd interfejsu webowego\n",
"<img width=\"75%\" src=\"IUM_08/mlflowui.png\"/>"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Porównywanie wyników\n",
"<img width=\"75%\" src=\"IUM_08/compare-metrics.png\"/>"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Logowanie\n",
" - logowania metryk i parametrów można dokonać m.in. poprzez wywołania Python-owego API: `mlflow.log_param()` i `mlflow.log_metric()`. Więcej dostępnych funkcji: [link](https://mlflow.org/docs/latest/tracking.html#logging-functions)\n",
" - wywołania te muszą nastąpić po wykonaniu [`mlflow.start_run()`](https://mlflow.org/docs/latest/python_api/mlflow.html#mlflow.start_run), najlepiej wewnątrz bloku:\n",
"```python\n",
" with mlflow.start_run():\n",
" \n",
" #[...]\n",
"\n",
" mlflow.log_param(\"alpha\", alpha)\n",
" mlflow.log_param(\"l1_ratio\", l1_ratio)\n",
"```\n",
" - jest też możliwość automatycznego logwania dla wybranych bibliotek: https://mlflow.org/docs/latest/tracking.html#automatic-logging"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# MLflow Projects\n",
" - MLflow projects to zestaw konwencji i kilku narzędzi\n",
" - ułatwiają one uruchamianie eskperymentów"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Konfiguracja projektu\n",
" - W pliku `MLproject` zapisuje się konfigurację projektu ([specyfikacja](https://mlflow.org/docs/latest/projects.html))\n",
" - Zawiera ona:\n",
" - odnośnik do środowiska, w którym ma być wywołany eksperyment [szczegóły](https://mlflow.org/docs/latest/projects.html#specifying-an-environment):\n",
" - nazwa obrazu Docker\n",
" - albo ścieżka do pliku conda.yaml definiującego środowisko wykonania Conda\n",
" - parametry, z którymi można wywołać eksperyment\n",
" - polecenia służące do wywołania eksperymentu"
]
},
{
"cell_type": "code",
2021-05-17 12:52:53 +02:00
"execution_count": 9,
2021-05-10 12:53:57 +02:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Overwriting IUM_08/examples/sklearn_elasticnet_wine/MLproject\n"
]
}
],
"source": [
"%%writefile IUM_08/examples/sklearn_elasticnet_wine/MLproject\n",
"name: tutorial\n",
"\n",
"conda_env: conda.yaml #ścieżka do pliku conda.yaml z definicją środowiska\n",
" \n",
"#docker_env:\n",
"# image: mlflow-docker-example-environment\n",
"\n",
"entry_points:\n",
" main:\n",
" parameters:\n",
" alpha: {type: float, default: 0.5}\n",
" l1_ratio: {type: float, default: 0.1}\n",
" command: \"python train.py {alpha} {l1_ratio}\"\n",
" test:\n",
" parameters:\n",
" alpha: {type: cutoff, default: 0}\n",
" command: \"python test.py {cutoff}\""
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Środowisko Conda\n",
" </br><img style=\"height: 50px;\" src=\"img/environments/conda.png\"/>\n",
" - https://docs.conda.io\n",
" - Składnia plików conda.yaml definiujących środowisko: https://docs.conda.io/projects/conda/en/4.6.1/user-guide/tasks/manage-environments.html#create-env-file-manually\n",
" - Składnia YAML: [przystępnie](https://learnxinyminutes.com/docs/yaml/), [oficjalnie](https://yaml.org/spec/1.2/spec.html)"
]
},
{
"cell_type": "code",
2021-05-17 12:52:53 +02:00
"execution_count": 10,
2021-05-10 12:53:57 +02:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Overwriting IUM_08/examples/sklearn_elasticnet_wine/conda.yaml\n"
]
}
],
"source": [
"%%writefile IUM_08/examples/sklearn_elasticnet_wine/conda.yaml\n",
"name: tutorial\n",
"channels:\n",
" - defaults\n",
"dependencies:\n",
" - python=3.6 #Te zależności będą zainstalowane za pomocą conda isntall\n",
" - pip\n",
" - pip: #Te ząś za pomocą pip install\n",
" - scikit-learn==0.23.2\n",
" - mlflow>=1.0"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Środowisko docker\n",
"- zamiast środowiska Conda możemy również podać nazwę obrazu docker, w którym ma być wywołany eksperyment.\n",
"- obraz będzie szukany lokalnie a następnie na DockerHub, lub w innym repozytorium dockera\n",
"- składnia specyfikacji ścieżki jest taka sama jak w przypadki poleceń dockera, np. docker pull [link](https://docs.docker.com/engine/reference/commandline/pull/#pull-from-a-different-registry)\n",
"- Można również podać katalogi do podmontowania wewnątrz kontenera oraz wartości zmiennych środowiskowych do ustawienia w kontenerze:\n",
"```yaml\n",
"docker_env:\n",
" image: mlflow-docker-example-environment\n",
" volumes: [\"/local/path:/container/mount/path\"]\n",
" environment: [[\"NEW_ENV_VAR\", \"new_var_value\"], \"VAR_TO_COPY_FROM_HOST_ENVIRONMENT\"]\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Parametry\n",
" - Specyfikacja parametrów w pliku MLproject pozwala na ich walidację i używanie wartości domyślnych\n",
" - Dostępne typy:\n",
" - String\n",
" - Float - dowolna liczba (MLflow waliduje, czy podana wartość jest liczbą)\n",
" - Path - pozwala podawać ścieżki względne (przekształca je na bezwzlędne) do plików lokalnych albo do plików zdalnych (np. do s3://) - zostaną wtedy ściągnięte lokalnie\n",
" - URI - podobnie jak path, ale do rozproszonych systemów plików\n",
"\n",
"- [Składnia](https://mlflow.org/docs/latest/projects.html#specifying-parameters)\n",
" \n",
"```yml:\n",
" parameter_name: {type: data_type, default: value} # Short syntax\n",
"\n",
" parameter_name: # Long syntax\n",
" type: data_type\n",
" default: value\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Uruchamianie projektu\n",
" - Projekt możemy uruchomić przy pomocy polecenia `mlflow run` ([dokumentacja](https://mlflow.org/docs/latest/cli.html#mlflow-run))\n",
" - Spowoduje to przygotowanie środowiska i uruchomienie eksperymentu wewnątrz środowiska\n",
" - domyślnie zostanie uruchomione polecenie zdefiniowane w \"entry point\" `main`. Żeby uruchomić inny \"entry point\", możemy użyć parametru `-e`, np:\n",
" ```bash\n",
" mlflow run sklearn_elasticnet_wine -e test\n",
2022-05-09 09:54:34 +02:00
" ```\n",
" - Parametry do naszego polecenia możemy przekazywać przy pomocy flagi `-P`"
2021-05-10 12:53:57 +02:00
]
},
{
"cell_type": "code",
2021-05-17 12:52:53 +02:00
"execution_count": 11,
2021-05-10 12:53:57 +02:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
2021-05-17 12:52:53 +02:00
"2021/05/16 17:59:10 INFO mlflow.projects.utils: === Created directory /tmp/tmprq4mdosv for downloading remote URIs passed to arguments of type 'path' ===\n",
"2021/05/16 17:59:10 INFO mlflow.projects.backend.local: === Running command 'source /home/tomek/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-5987e03d4dbaa5faa1a697bb113be9b9bdc39b29 1>&2 && python train.py 0.42 0.1' in run with ID '1860d321ea1545ff8866e4ba199d1712' === \n",
"Elasticnet model (alpha=0.420000, l1_ratio=0.100000):\n",
" RMSE: 0.7420620899060748\n",
" MAE: 0.5722846717246247\n",
" R2: 0.21978513651550236\n",
"2021/05/16 17:59:19 INFO mlflow.projects: === Run (ID '1860d321ea1545ff8866e4ba199d1712') succeeded ===\n"
2021-05-10 12:53:57 +02:00
]
2021-05-17 12:52:53 +02:00
}
],
"source": [
"!cd IUM_08/examples/; mlflow run sklearn_elasticnet_wine -P alpha=0.42"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
2022-05-09 10:53:28 +02:00
"# Zadania [10p pkt]\n",
2021-05-17 12:52:53 +02:00
"1. Dodaj do swojego projektu logowanie parametrów i metryk za pomocą MLflow (polecenia `mlflow.log_param` i `mlflow.log_metric`\n",
"2. Dodaj plik MLProject definiujący polecenia do trenowania i testowania, ich parametry wywołania oraz środowisko (użyj zdefiniowanego wcześniej obrazu Docker)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## MLflow Models\n",
"\n",
"MLflow Models to konwencja zapisu modeli, która ułatwia potem ich załadowanie i użycie"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"Rodzaje modeli (\"flavors\") wspierane przez MLflow:\n",
"\n",
" - Python Function (python_function)\n",
" - PyTorch (pytorch)\n",
" - TensorFlow (tensorflow)\n",
" - Keras (keras)\n",
" - Scikit-learn (sklearn)\n",
" - Spacy(spaCy)\n",
" - ONNX (onnx)\n",
" - R Function (crate)\n",
" - H2O (h2o)\n",
" - MLeap (mleap)\n",
" - Spark MLlib (spark)\n",
" - MXNet Gluon (gluon)\n",
" - XGBoost (xgboost)\n",
" - LightGBM (lightgbm)\n",
" - CatBoost (catboost)\n",
" - Fastai(fastai)\n",
" - Statsmodels (statsmodels)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Zapisywanie modelu\n",
"Model ML można zapisać w MLflow przy pomocy jednej z dwóch funkcji z pakietu odpowiadającego używanej przez nas bibliotece:\n",
" - `save_model()` - zapisuje model na dysku\n",
" - `log_model()` - zapisuje model razem z innymi informacjami (metrykami, parametrami). W zależności od ustawień [\"tracking_uri\"](https://mlflow.org/docs/latest/python_api/mlflow.html#mlflow.set_tracking_uri) może być to lokalny folder w `mlruns/ ` lub ścieżka na zdalnym serwerze MLflow\n",
"\n",
"```Python\n",
" mlflow.sklearn.save_model(lr, \"my_model\")\n",
"```\n",
"\n",
"```Python\n",
" mlflow.keras.save_model(lr, \"my_model\")\n",
"```\n",
"\n",
"Wywołanie tej funkcji spowoduje stworzenie katalogu \"my_model\" zawierającego:\n",
" - plik *MLmodel* zawierający informacje o sposobach, w jaki model można załadować (\"flavors\") oraz ścieżki do plików związanych z modelem, takich jak:\n",
" - *conda.yaml* - opis środowiska potrzebnego do załadowania modelu\n",
" - *model.pkl* - plik z zserializowanym modelem\n",
"\n",
2022-05-09 09:54:34 +02:00
"Tylko plik *MLmodel* jest specjalnym plikiem MLflow - reszta zależy od konkrentego \"flavour\"\n"
2021-05-17 12:52:53 +02:00
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
2021-05-10 12:53:57 +02:00
{
"name": "stdout",
"output_type": "stream",
"text": [
2021-05-17 12:52:53 +02:00
"conda.yaml MLmodel model.pkl\r\n"
2021-05-10 12:53:57 +02:00
]
}
],
"source": [
2021-05-17 12:52:53 +02:00
"ls IUM_08/examples/my_model"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total 12\r\n",
"-rw-rw-r-- 1 tomek tomek 153 maj 17 10:38 conda.yaml\r\n",
"-rw-rw-r-- 1 tomek tomek 958 maj 17 10:38 MLmodel\r\n",
"-rw-rw-r-- 1 tomek tomek 641 maj 17 10:38 model.pkl\r\n"
]
}
],
"source": [
"! ls -l IUM_08/examples/mlruns/0/b395b55b47fc43de876b67f5a4a5dae9/artifacts/model"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [],
"source": [
"# %load IUM_08/examples/mlruns/0/b395b55b47fc43de876b67f5a4a5dae9/artifacts/model/MLmodel\n",
"artifact_path: model\n",
"flavors:\n",
" python_function:\n",
" env: conda.yaml\n",
" loader_module: mlflow.sklearn\n",
" model_path: model.pkl\n",
" python_version: 3.9.1\n",
" sklearn:\n",
" pickled_model: model.pkl\n",
" serialization_format: cloudpickle\n",
" sklearn_version: 0.24.2\n",
"run_id: b395b55b47fc43de876b67f5a4a5dae9\n",
"signature:\n",
" inputs: '[{\"name\": \"fixed acidity\", \"type\": \"double\"}, {\"name\": \"volatile acidity\",\n",
" \"type\": \"double\"}, {\"name\": \"citric acid\", \"type\": \"double\"}, {\"name\": \"residual\n",
" sugar\", \"type\": \"double\"}, {\"name\": \"chlorides\", \"type\": \"double\"}, {\"name\": \"free\n",
" sulfur dioxide\", \"type\": \"double\"}, {\"name\": \"total sulfur dioxide\", \"type\": \"double\"},\n",
" {\"name\": \"density\", \"type\": \"double\"}, {\"name\": \"pH\", \"type\": \"double\"}, {\"name\":\n",
" \"sulphates\", \"type\": \"double\"}, {\"name\": \"alcohol\", \"type\": \"double\"}]'\n",
" outputs: '[{\"type\": \"tensor\", \"tensor-spec\": {\"dtype\": \"float64\", \"shape\": [-1]}}]'\n",
"utc_time_created: '2021-05-17 08:38:41.749670'\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [],
"source": [
"# %load IUM_08/examples/my_model/conda.yaml\n",
"channels:\n",
"- defaults\n",
"- conda-forge\n",
"dependencies:\n",
"- python=3.9.1\n",
"- pip\n",
"- pip:\n",
" - mlflow\n",
" - scikit-learn==0.24.2\n",
" - cloudpickle==1.6.0\n",
"name: mlflow-env"
2021-05-10 12:53:57 +02:00
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
2021-05-17 12:52:53 +02:00
"### Dodatkowe pola w MLmodel\n",
"\n",
"\n",
"- *utc_time_created* - timestamp z czasem stworzenia modelu\n",
"- *run_id* - ID uruchomienia (\"run\"), które stworzyło ten model, jeśli model był zapisany za pomocą MLflow Tracking.\n",
"- *signature* - opisa danych wejściowych i wyjściowych w formacie JSON\n",
"- *input_example* przykładowe wejście przyjmowane przez model. Można je podać poprzez parametr `input_example` funkcji [log_model](https://mlflow.org/docs/latest/python_api/mlflow.sklearn.html#mlflow.sklearn.log_model)\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"array([5.57688397, 5.50664777, 5.52550482, 5.50431125, 5.57688397])"
]
},
"execution_count": 50,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import mlflow\n",
"import pandas as pd\n",
"model = mlflow.sklearn.load_model(\"IUM_08/examples/mlruns/0/b395b55b47fc43de876b67f5a4a5dae9/artifacts/model\")\n",
"csv_url = \"http://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-red.csv\"\n",
"data = pd.read_csv(csv_url, sep=\";\")\n",
"model.predict(data.drop([\"quality\"], axis=1).head())"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Serwowanie modeli"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Usage: mlflow models [OPTIONS] COMMAND [ARGS]...\r\n",
"\r\n",
" Deploy MLflow models locally.\r\n",
"\r\n",
" To deploy a model associated with a run on a tracking server, set the\r\n",
" MLFLOW_TRACKING_URI environment variable to the URL of the desired server.\r\n",
"\r\n",
"Options:\r\n",
" --help Show this message and exit.\r\n",
"\r\n",
"Commands:\r\n",
" build-docker **EXPERIMENTAL**: Builds a Docker image whose default...\r\n",
" predict Generate predictions in json format using a saved MLflow...\r\n",
" prepare-env **EXPERIMENTAL**: Performs any preparation necessary to...\r\n",
" serve Serve a model saved with MLflow by launching a webserver on...\r\n"
]
}
],
"source": [
"!cd IUM_08/examples/; mlflow models --help"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Usage: mlflow models serve [OPTIONS]\r\n",
"\r\n",
" Serve a model saved with MLflow by launching a webserver on the specified\r\n",
" host and port. The command supports models with the ``python_function`` or\r\n",
" ``crate`` (R Function) flavor. For information about the input data\r\n",
" formats accepted by the webserver, see the following documentation:\r\n",
" https://www.mlflow.org/docs/latest/models.html#built-in-deployment-tools.\r\n",
"\r\n",
" You can make requests to ``POST /invocations`` in pandas split- or record-\r\n",
" oriented formats.\r\n",
"\r\n",
" Example:\r\n",
"\r\n",
" .. code-block:: bash\r\n",
"\r\n",
" $ mlflow models serve -m runs:/my-run-id/model-path &\r\n",
"\r\n",
" $ curl http://127.0.0.1:5000/invocations -H 'Content-Type:\r\n",
" application/json' -d '{ \"columns\": [\"a\", \"b\", \"c\"],\r\n",
" \"data\": [[1, 2, 3], [4, 5, 6]] }'\r\n",
"\r\n",
"Options:\r\n",
" -m, --model-uri URI URI to the model. A local path, a 'runs:/' URI, or a\r\n",
" remote storage URI (e.g., an 's3://' URI). For more\r\n",
" information about supported remote URIs for model\r\n",
" artifacts, see\r\n",
" https://mlflow.org/docs/latest/tracking.html#artifact-\r\n",
" stores [required]\r\n",
"\r\n",
" -p, --port INTEGER The port to listen on (default: 5000).\r\n",
" -h, --host HOST The network address to listen on (default: 127.0.0.1).\r\n",
" Use 0.0.0.0 to bind to all addresses if you want to\r\n",
" access the tracking server from other machines.\r\n",
"\r\n",
" -w, --workers TEXT Number of gunicorn worker processes to handle requests\r\n",
" (default: 4).\r\n",
"\r\n",
" --no-conda If specified, will assume that MLmodel/MLproject is\r\n",
" running within a Conda environment with the necessary\r\n",
" dependencies for the current project instead of\r\n",
" attempting to create a new conda environment.\r\n",
"\r\n",
" --install-mlflow If specified and there is a conda environment to be\r\n",
" activated mlflow will be installed into the environment\r\n",
" after it has been activated. The version of installed\r\n",
" mlflow will be the same asthe one used to invoke this\r\n",
" command.\r\n",
"\r\n",
" --help Show this message and exit.\r\n"
]
}
],
"source": [
"!cd IUM_08/examples/; mlflow models serve --help"
]
},
{
"cell_type": "code",
"execution_count": 54,
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\"columns\":[\"fixed acidity\",\"volatile acidity\",\"citric acid\",\"residual sugar\",\"chlorides\",\"free sulfur dioxide\",\"total sulfur dioxide\",\"density\",\"pH\",\"sulphates\",\"alcohol\"],\"index\":[0],\"data\":[[7.4,0.7,0.0,1.9,0.076,11.0,34.0,0.9978,3.51,0.56,9.4]]}\n"
]
}
],
"source": [
"import pandas as pd\n",
"csv_url = \"http://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-red.csv\"\n",
"data = pd.read_csv(csv_url, sep=\";\").drop([\"quality\"], axis=1).head(1).to_json(orient='split')\n",
"print(data)"
]
},
{
"cell_type": "code",
"execution_count": 57,
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[5.576883967129615]"
]
}
],
"source": [
"!curl http://127.0.0.1:5003/invocations -H 'Content-Type: application/json' -d '{\\\n",
" \"columns\":[\\\n",
" \"fixed acidity\",\"volatile acidity\",\"citric acid\",\"residual sugar\",\"chlorides\",\"free sulfur dioxide\",\"total sulfur dioxide\",\"density\",\"pH\",\"sulphates\",\"alcohol\"],\\\n",
" \"index\":[0],\\\n",
" \"data\":[[7.4,0.7,0.0,1.9,0.076,11.0,34.0,0.9978,3.51,0.56,9.4]]}'"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"```\n",
"$ cd IUM_08/examples/\n",
"$ mlflow models serve -m my_model\n",
"2021/05/17 08:52:07 INFO mlflow.models.cli: Selected backend for flavor 'python_function'\n",
"2021/05/17 08:52:07 INFO mlflow.pyfunc.backend: === Running command 'source /home/tomek/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-503f0c7520a32f054a9d168bd099584a9439de9d 1>&2 && gunicorn --timeout=60 -b 127.0.0.1:5003 -w 1 ${GUNICORN_CMD_ARGS} -- mlflow.pyfunc.scoring_server.wsgi:app'\n",
"[2021-05-17 08:52:07 +0200] [291217] [INFO] Starting gunicorn 20.1.0\n",
"[2021-05-17 08:52:07 +0200] [291217] [INFO] Listening at: http://127.0.0.1:5003 (291217)\n",
"[2021-05-17 08:52:07 +0200] [291217] [INFO] Using worker: sync\n",
"[2021-05-17 08:52:07 +0200] [291221] [INFO] Booting worker with pid: 291221\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## MLflow Registry\n",
" - umożliwia [zapisywanie](https://mlflow.org/docs/latest/model-registry.html#adding-an-mlflow-model-to-the-model-registry) i [ładowanie](https://mlflow.org/docs/latest/model-registry.html#fetching-an-mlflow-model-from-the-model-registry) modeli z centralnego rejestru\n",
" - Modele można też serwować bezpośrednio z rejestru:\n",
"\n",
"```bash\n",
"#!/usr/bin/env sh\n",
"\n",
"# Set environment variable for the tracking URL where the Model Registry resides\n",
"export MLFLOW_TRACKING_URI=http://localhost:5000\n",
"\n",
"# Serve the production model from the model registry\n",
"mlflow models serve -m \"models:/sk-learn-random-forest-reg-model/Production\"\n",
"```\n",
"\n",
"- Żeby było to możliwe, musimy mieć uruchomiony [serwer MLflow](https://mlflow.org/docs/latest/tracking.html#tracking-server)\n",
"- Umożliwia zarządzanie wersjami modeli i oznaczanie ich różnymi fazami, np. \"Staging\", \"Production\""
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Zadania\n",
2022-05-09 09:54:34 +02:00
"1. [2 pkt] Dodaj do joba treningowego wywołania MLflow, tak, żeby przy każdym uruchomieniu stworzyć i zarchiwizować katalog z modelem. Plik MLmodel powinien zawierać pola:\n",
2021-05-17 12:52:53 +02:00
" - signature\n",
" - input_example\n",
"\n",
2022-05-09 09:54:34 +02:00
" Folder powinien również zawierać definicję środowiska - conda lub docker, umożliwiającego uruchomienie projektu.\n",
2021-05-17 12:52:53 +02:00
"\n",
"2. [6 pkt] Wybierz jedną osobę z grupy. Załóżmy, że Twoje ID to s123456 a jej s654321. Stwórz na Jenkinsie projekt `s123456-predict-s654321`, w którym:\n",
" - pobierzesz artefakt z zapisanym modelem z joba osoby s654321\n",
2022-05-09 09:54:34 +02:00
" - dokonasz na nim predykcji danych wejściowych podanych w formacie json jako parametr zadania Jenkinsowego. Domyślną wartością tego parametru niech będą przykładowe dane wejściowe z `input_example`\n",
2021-05-17 12:52:53 +02:00
" \n",
2022-05-09 09:54:34 +02:00
"3. [1 pkt] Zarejestruj swój model w MLflow registry\n",
2021-05-17 12:52:53 +02:00
"\n",
"4. [6 pkt] Stwórz na Jenkinsie projekt `s123456-predict-s654321-from-registry`, który zrealizuje to samo zadanie co `s123456-predict-s654321`, ale tym razem pobierze model z rejestru MLflow zamiast z artefaktów Jenkinsa"
2021-05-10 12:53:57 +02:00
]
2021-05-19 22:55:34 +02:00
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"Dane do konfiguracji MLflow registry\n",
"\n",
"- Podgląd w przeglądarce: http://tzietkiewicz.vm.wmi.amu.edu.pl/#/\n",
" - user: `student`\n",
2022-05-09 09:54:34 +02:00
" - hasło: IUM@2021\n",
2021-05-19 22:55:34 +02:00
"\n",
"- Tracking URI:\n",
" - Python: `mlflow.set_tracking_uri(\"http://172.17.0.1:5000\")`\n",
" - CLI: `export MLFLOW_TRACKING_URI=http://172.17.0.1:5000`\n",
" \n",
2022-05-16 12:13:09 +02:00
"- Żeby klient MLflow działający w kontenerze docker mógł zapisywać i pdczytywać artefakty, muszą Państwo podmonotwać katalog `/tmp/mlruns` i `/mlruns` (ten drugi po to, żeby po restarcie serwera katalog nie został wyczyszczony)\n",
2021-05-19 22:55:34 +02:00
"- Proszę ustawić nazwę eksperymentu na numer indeksu, dzięki temu każdy z Państwa będzie widział swoje eksperymenty oddzielnie:\n",
"`mlflow.set_experiment(\"s123456\")`"
]
2021-05-10 12:53:57 +02:00
}
],
"metadata": {
2021-09-28 10:56:21 +02:00
"author": "Tomasz Ziętkiewicz",
2021-05-10 12:53:57 +02:00
"celltoolbar": "Slideshow",
2021-09-28 10:56:21 +02:00
"email": "tomasz.zietkiewicz@amu.edu.pl",
2021-05-10 12:53:57 +02:00
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
2021-09-28 10:56:21 +02:00
"lang": "pl",
2021-05-10 12:53:57 +02:00
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
2022-04-25 11:15:16 +02:00
"version": "3.6.9"
2021-09-28 10:56:21 +02:00
},
"slideshow": {
"slide_type": "slide"
},
"subtitle": "8.MLFlow[laboratoria]",
"title": "Inżynieria uczenia maszynowego",
"year": "2021"
2021-05-10 12:53:57 +02:00
},
"nbformat": 4,
"nbformat_minor": 4
}
