2021-04-26 11:44:34 +02:00
{
"cells": [
2021-09-28 10:56:21 +02:00
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
2024-04-09 09:46:32 +02:00
"## Inżynieria uczenia maszynowego\n",
"### 17 kwietnia 2024\n",
"# 7. Sacred"
2021-09-28 10:56:21 +02:00
]
},
2021-04-26 11:44:34 +02:00
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Sacred\n",
"> Every experiment is sacred <br>\n",
"> Every experiment is great <br>\n",
"> If an experiment is wasted <br>\n",
"> God gets quite irate\n",
">\n",
" <cite>—https://github.com/IDSIA/sacred / [Sens życia według Monty Pythona](https://en.wikipedia.org/wiki/Every_Sperm_Is_Sacred) </cite>"
]
},
2023-04-12 15:14:37 +02:00
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"- https://sacred.readthedocs.io/ - dokumentacja\n",
"- https://github.com/IDSIA/sacred - Github\n",
"- Open source\n",
"- Prosty w użyciu\n",
"- Wiele webowych frontendów"
]
},
2021-04-26 11:44:34 +02:00
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"- Przeprowadzanie eksperymentów (zmiana parametrów, trenowanie, ewaluacja) uczenia maszynowego jest kosztowne i czasochłonne\n",
"- Dlatego warto przeprowadzać je w zorganizowany sposób\n",
"- I tak, żebyśmy mogli powtórzyć / odtworzyć raz uzyskane wyniki"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"> Sacred is a tool to help you:\n",
"> - configure\n",
"> - organize\n",
"> - log \n",
"> - reproduce \n",
"> experiments. \n",
"> \n",
">It is designed to do all the tedious overhead work that you need to do around your actual experiment in order to:\n",
"> - keep track of all the parameters of your experiment\n",
"> - easily run your experiment for different settings\n",
"> - save configurations for individual runs in a database\n",
"> - reproduce your results\n",
" \n",
" <cite>—https://github.com/IDSIA/sacred</cite>"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"- **ConfigScopes** A very convenient way of the local variables in a function to define the parameters your experiment uses.\n",
"- **Config Injection** You can access all parameters of your configuration from every function. They are automatically injected by name.\n",
"- **Command-line interface** You get a powerful command-line interface for each experiment that you can use to change parameters and run different variants.\n",
"- **Observers** Sacred provides Observers that log all kinds of information about your experiment, its dependencies, the configuration you used, the machine it is run on, and of course the result. These can be saved to a MongoDB, for easy access later.\n",
"- **Automatic seeding** helps controlling the randomness in your experiments, such that the results remain reproducible."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Instalacja"
]
},
{
"cell_type": "code",
2024-04-17 08:00:40 +02:00
"execution_count": 1,
2021-04-26 11:44:34 +02:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
2024-04-17 08:00:40 +02:00
"Collecting sacred\n",
" Downloading sacred-0.8.5-py2.py3-none-any.whl (107 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m107.9/107.9 KB\u001b[0m \u001b[31m2.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
"\u001b[?25hCollecting jsonpickle>=2.2.0\n",
" Downloading jsonpickle-3.0.4-py3-none-any.whl (39 kB)\n",
"Collecting colorama>=0.4\n",
" Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)\n",
"Collecting munch<5.0,>=2.5\n",
" Downloading munch-4.0.0-py2.py3-none-any.whl (9.9 kB)\n",
"Requirement already satisfied: packaging>=18.0 in ./venv/lib/python3.10/site-packages (from sacred) (24.0)\n",
"Collecting wrapt<2.0,>=1.0\n",
" Downloading wrapt-1.16.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (80 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m80.3/80.3 KB\u001b[0m \u001b[31m12.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting docopt<1.0,>=0.3\n",
" Downloading docopt-0.6.2.tar.gz (25 kB)\n",
" Preparing metadata (setup.py) ... \u001b[?25ldone\n",
"\u001b[?25hCollecting GitPython\n",
" Downloading GitPython-3.1.43-py3-none-any.whl (207 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m207.3/207.3 KB\u001b[0m \u001b[31m9.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting py-cpuinfo>=4.0\n",
" Downloading py_cpuinfo-9.0.0-py3-none-any.whl (22 kB)\n",
"Collecting gitdb<5,>=4.0.1\n",
" Downloading gitdb-4.0.11-py3-none-any.whl (62 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.7/62.7 KB\u001b[0m \u001b[31m15.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting smmap<6,>=3.0.1\n",
" Downloading smmap-5.0.1-py3-none-any.whl (24 kB)\n",
"Using legacy 'setup.py install' for docopt, since package 'wheel' is not installed.\n",
"Installing collected packages: py-cpuinfo, docopt, wrapt, smmap, munch, jsonpickle, colorama, gitdb, GitPython, sacred\n",
" Running setup.py install for docopt ... \u001b[?25ldone\n",
"\u001b[?25hSuccessfully installed GitPython-3.1.43 colorama-0.4.6 docopt-0.6.2 gitdb-4.0.11 jsonpickle-3.0.4 munch-4.0.0 py-cpuinfo-9.0.0 sacred-0.8.5 smmap-5.0.1 wrapt-1.16.0\n"
2021-04-26 11:44:34 +02:00
]
}
],
"source": [
2022-04-25 11:15:16 +02:00
"!pip3 install sacred"
2021-04-26 11:44:34 +02:00
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Funkcja main"
]
},
{
"cell_type": "code",
2024-04-17 08:00:40 +02:00
"execution_count": 11,
2021-04-26 11:44:34 +02:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
2022-04-25 11:15:16 +02:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
2024-04-17 08:00:40 +02:00
"Overwriting IUM_07/sacred_hello.py\n"
2022-04-25 11:15:16 +02:00
]
}
],
2021-04-26 11:44:34 +02:00
"source": [
2024-04-17 08:00:40 +02:00
"%%writefile IUM_07/sacred_hello.py\n",
2021-04-26 11:44:34 +02:00
"from sacred import Experiment\n",
"\n",
"ex = Experiment()\n",
"\n",
"@ex.automain\n",
"def my_main():\n",
" print('Witaj świecie!')\n"
]
},
{
"cell_type": "code",
2024-04-17 08:00:40 +02:00
"execution_count": 12,
2021-04-26 11:44:34 +02:00
"metadata": {
"slideshow": {
2022-04-25 11:15:16 +02:00
"slide_type": "fragment"
2021-04-26 11:44:34 +02:00
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
2024-04-17 08:00:40 +02:00
"WARNING - sacred_hello - No observers have been added to this run\n",
"INFO - sacred_hello - Running command 'my_main'\n",
"INFO - sacred_hello - Started\n",
"Witaj świecie!\n",
"INFO - sacred_hello - Completed after 0:00:00\n"
2021-04-26 11:44:34 +02:00
]
}
],
"source": [
2022-04-25 11:15:16 +02:00
"!python3 IUM_07/sacred_hello.py"
2021-04-26 11:44:34 +02:00
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"##### Co się dzieje w kodzie powyżej?\n",
"1. Tworzymy obiekt klasy Experiment\n",
2022-04-25 11:15:16 +02:00
"2. Dekorujemy funkcję \"my_main\" dekoratorem [automain](https://sacred.readthedocs.io/en/stable/apidoc.html#sacred.Experiment.automain)\n",
2021-04-26 11:44:34 +02:00
" Dzięki temu:\n",
" - otrzymujemy interfejs CLI, m.in. do kontrolowania poziomu logowania, przekazywania parametrów itp.\n",
" - oznaczamy funkcję \"my_main\" jako główną funkcję, która będzie wywoływana podczas wykonywania eksperymentu\n",
" - funkcja oznaczona jako główna musi być ostatnią funkcją zdefiniowaną w pliku!\n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"##### Co nam daje interejs CLI:"
]
},
{
"cell_type": "code",
2024-04-17 08:00:40 +02:00
"execution_count": 4,
2021-04-26 11:44:34 +02:00
"metadata": {
"slideshow": {
2022-04-25 11:15:16 +02:00
"slide_type": "fragment"
2021-04-26 11:44:34 +02:00
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
2024-04-17 08:00:40 +02:00
"Usage:\n",
" sacred_hello.py [(with UPDATE...)] [options]\n",
" sacred_hello.py help [COMMAND]\n",
" sacred_hello.py (-h | --help)\n",
" sacred_hello.py COMMAND [(with UPDATE...)] [options]\n",
"\n",
"\n",
"\n",
"Options:\n",
" -b VALUE --beat-interval=VALUE Set the heart-beat interval for this run. Time\n",
" between two heartbeat events is measured in\n",
" seconds.\n",
" -C VALUE --capture=VALUE Control the way stdout and stderr are captured.\n",
" The argument value must be one of [no, sys, fd]\n",
" -c VALUE --comment=VALUE Add a comment to this run.\n",
" -d --debug Set this run to debug mode. Suppress warnings\n",
" about missing observers and don't filter the\n",
" stacktrace. Also enables usage with ipython\n",
" `--pdb`.\n",
" -e --enforce_clean Fail if any version control repository is\n",
" dirty.\n",
" -F VALUE --file_storage=VALUE Add a file-storage observer to the experiment.\n",
" The value of the arguement should be the base-\n",
" directory to write the runs to\n",
" -f --force Disable warnings about suspicious changes for\n",
" this run.\n",
" -h --help Print this help message and exit.\n",
" -i VALUE --id=VALUE Set the id for this run.\n",
" -l VALUE --loglevel=VALUE Set the LogLevel. Loglevel either as 0 - 50 or\n",
" as string: DEBUG(10), INFO(20), WARNING(30),\n",
" ERROR(40), CRITICAL(50)\n",
" -m VALUE --mongo_db=VALUE Add a MongoDB Observer to the experiment. The\n",
" argument value is the database specification.\n",
" Should be in the form: `[host:port:]db_name[.c\n",
" ollection[:id]][!priority]`\n",
" -n VALUE --name=VALUE Set the name for this run.\n",
" -D --pdb Automatically enter post-mortem debugging with\n",
" pdb on failure.\n",
" -p --print-config Always print the configuration first.\n",
" -P VALUE --priority=VALUE Sets the priority for a queued up experiment.\n",
" `--priority=NUMBER` The number represent the\n",
" priority for this run.\n",
" -q --queue Only queue this run, do not start it.\n",
" -S VALUE --s3=VALUE Add a S3 File observer to the experiment. The\n",
" argument value should be\n",
" `s3://<bucket>/path/to/exp`.\n",
" -s VALUE --sql=VALUE Add a SQL Observer to the experiment. The\n",
" typical form is:\n",
" dialect://username:password@host:port/database\n",
" -t VALUE --tiny_db=VALUE Add a TinyDB Observer to the experiment. The\n",
" argument is the path to be given to the\n",
" TinyDbObserver.\n",
" -u --unobserved Ignore all observers for this run.\n",
"\n",
"\n",
"Arguments:\n",
" COMMAND Name of command to run (see below for list of commands)\n",
" UPDATE Configuration assignments of the form foo.bar=17\n",
"\n",
"\n",
"Commands:\n",
" print_config Print the updated configuration and exit.\n",
" print_dependencies Print the detected source-files and dependencies.\n",
" save_config Store the updated configuration in a file.\n",
" print_named_configs Print the available named configs and exit.\n",
" my_main \n",
"\n"
2021-04-26 11:44:34 +02:00
]
}
],
"source": [
2022-04-25 11:15:16 +02:00
"!python3 IUM_07/sacred_hello.py -h"
2021-04-26 11:44:34 +02:00
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Konfiguracje\n",
" - Konfiguracje pozwalają nam sparametryzować wywołania eksperymentu.\n",
" - Ułatwiają przekazywanie parametrów - zmienne z konfiguracji są wstrzykiwane do funkcji wywoływanych \n",
" - Mogą być automatycznie zapisywane (dzięki czemu możemy śledzić jak zmieniały się parametry i jaki miały wpływ na wyniki)\n",
" - Konfigurację można stworzyć w jeden z 3 sposobów:\n",
2022-04-25 11:15:16 +02:00
" - używając config scopes (z dekoratorem `@config`)\n",
2021-04-26 11:44:34 +02:00
" - jako słownik\n",
" - wczytując ją z pliku"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Konfiguracje - config scopes\n",
2022-04-25 11:15:16 +02:00
"Jeśli oznaczymy jakąś funkcję dekoratorem `@config`, to zostanie ona uruchoniona przed wywołaniem eksperymentu i wszystkie jej lokalne zmienne, które da się zserializować jako json, zostaną dodane do konfiguracji. Potem ich wartości zostaną wstrzyknięte do innych funkcji wywoływanych w eksperymencie. "
2021-04-26 11:44:34 +02:00
]
},
{
"cell_type": "code",
2024-04-17 08:00:40 +02:00
"execution_count": 5,
2021-04-26 11:44:34 +02:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [],
"source": [
"from sacred import Experiment\n",
"\n",
"exint = Experiment(\"sacred_scopes\", interactive=True) #Jeśli wykonujemy interaktywnie (w konsoli Pythona albo w Jupyter):\n",
"# - musimy podać nazwę eksperymentu (domyślnie jako nazwa używana jest nazwa pliku źródłowego)\n",
"# - musimy dodać parametr \"interactive=True\"\n",
2022-04-25 11:15:16 +02:00
"# - zamiast dekoratora \"@ex.automain\" używamy \"@ex.main\"\n",
2021-04-26 11:44:34 +02:00
"\n",
"@exint.config\n",
"def my_config():\n",
" recipient = \"Świecie\"\n",
" greeting = \"Witaj\"\n",
" message = \"{0} {1}!\".format(greeting, recipient)\n",
"\n",
"\n",
"@exint.main\n",
"def my_main(message):\n",
" print(message)"
]
},
{
"cell_type": "code",
2024-04-17 08:00:40 +02:00
"execution_count": 6,
2021-04-26 11:44:34 +02:00
"metadata": {
"slideshow": {
2022-04-25 11:15:16 +02:00
"slide_type": "fragment"
2021-04-26 11:44:34 +02:00
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING - sacred_scopes - No observers have been added to this run\n",
"INFO - sacred_scopes - Running command 'my_main'\n",
"INFO - sacred_scopes - Started\n",
"INFO - sacred_scopes - Completed after 0:00:00\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Witaj Świecie!\n"
]
},
{
"data": {
"text/plain": [
2024-04-17 08:00:40 +02:00
"<sacred.run.Run at 0x7ff6cf76d600>"
2021-04-26 11:44:34 +02:00
]
},
2024-04-17 08:00:40 +02:00
"execution_count": 6,
2021-04-26 11:44:34 +02:00
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"exint.run()"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"##### Możemy podejrzeć wartości zmiennych w konfiguracji:"
]
},
{
"cell_type": "code",
2024-04-17 08:00:40 +02:00
"execution_count": 7,
2021-04-26 11:44:34 +02:00
"metadata": {
"slideshow": {
2022-04-25 11:15:16 +02:00
"slide_type": "fragment"
2021-04-26 11:44:34 +02:00
}
},
"outputs": [
{
"data": {
"text/plain": [
"{'recipient': 'Świecie', 'greeting': 'Witaj', 'message': 'Witaj Świecie!'}"
]
},
2024-04-17 08:00:40 +02:00
"execution_count": 7,
2021-04-26 11:44:34 +02:00
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"my_config()"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"#### Parametry możemy podejrzeć i modyfikować z poziomu CLI\n",
" - wartości podane w CLI nadpiszą te podane w kodzie"
]
},
{
"cell_type": "code",
2024-04-17 08:00:40 +02:00
"execution_count": 13,
2021-04-26 11:44:34 +02:00
"metadata": {
"slideshow": {
2022-04-25 11:15:16 +02:00
"slide_type": "fragment"
2021-04-26 11:44:34 +02:00
}
},
2024-04-17 08:00:40 +02:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Overwriting IUM_07/sacred_scopes.py\n"
]
}
],
2021-04-26 11:44:34 +02:00
"source": [
2024-04-17 08:00:40 +02:00
"%%writefile IUM_07/sacred_scopes.py\n",
2021-04-26 11:44:34 +02:00
"from sacred import Experiment\n",
"\n",
"ex = Experiment()\n",
"\n",
"@ex.config\n",
"def my_config():\n",
" recipient = \"Świecie\"\n",
" greeting = \"Witaj\"\n",
" message = \"{0} {1}!\".format(greeting, recipient)\n",
"\n",
"@ex.automain\n",
"def my_main(message):\n",
" print(message)"
]
},
{
"cell_type": "code",
2023-04-20 12:13:19 +02:00
"execution_count": 14,
2021-04-26 11:44:34 +02:00
"metadata": {
"slideshow": {
2022-04-25 11:15:16 +02:00
"slide_type": "fragment"
2021-04-26 11:44:34 +02:00
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
2024-04-17 08:00:40 +02:00
"WARNING - sacred_scopes - No observers have been added to this run\n",
"INFO - sacred_scopes - Running command 'my_main'\n",
"INFO - sacred_scopes - Started\n",
"Witaj Przygodo!\n",
"INFO - sacred_scopes - Completed after 0:00:00\n"
2021-04-26 11:44:34 +02:00
]
}
],
"source": [
2022-04-25 11:15:16 +02:00
"!python3 IUM_07/sacred_scopes.py with 'recipient=Przygodo'"
2021-04-26 11:44:34 +02:00
]
},
{
"cell_type": "code",
2023-04-20 12:13:19 +02:00
"execution_count": 15,
2021-04-26 11:44:34 +02:00
"metadata": {
"slideshow": {
2022-04-25 11:15:16 +02:00
"slide_type": "fragment"
2021-04-26 11:44:34 +02:00
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
2024-04-17 08:00:40 +02:00
"INFO - sacred_scopes - Running command 'print_config'\n",
"INFO - sacred_scopes - Started\n",
"Configuration (\u001b[34mmodified\u001b[0m, \u001b[32madded\u001b[0m, \u001b[31mtypechanged\u001b[0m, \u001b[2mdoc\u001b[0m):\n",
" greeting = 'Witaj'\n",
" message = 'Witaj Świecie!'\n",
" recipient = 'Świecie'\n",
" seed = 166269169 \u001b[2m# the random seed for this experiment\u001b[0m\n",
"INFO - sacred_scopes - Completed after 0:00:00\n"
2021-04-26 11:44:34 +02:00
]
}
],
"source": [
2022-04-25 11:15:16 +02:00
"!python3 IUM_07/sacred_scopes.py print_config"
2021-04-26 11:44:34 +02:00
]
},
{
"cell_type": "code",
2023-04-20 12:13:19 +02:00
"execution_count": 16,
2021-04-26 11:44:34 +02:00
"metadata": {
"slideshow": {
2022-04-25 11:15:16 +02:00
"slide_type": "fragment"
2021-04-26 11:44:34 +02:00
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
2024-04-17 08:00:40 +02:00
"INFO - sacred_scopes - Running command 'print_config'\n",
"INFO - sacred_scopes - Started\n",
"Configuration (\u001b[34mmodified\u001b[0m, \u001b[32madded\u001b[0m, \u001b[31mtypechanged\u001b[0m, \u001b[2mdoc\u001b[0m):\n",
" greeting = 'Witaj'\n",
" message = 'Witaj Przygodo!'\n",
"\u001b[34m recipient = 'Przygodo'\u001b[0m\n",
" seed = 496449715 \u001b[2m# the random seed for this experiment\u001b[0m\n",
"INFO - sacred_scopes - Completed after 0:00:00\n"
2021-04-26 11:44:34 +02:00
]
}
],
"source": [
"!python IUM_07/sacred_scopes.py print_config with 'recipient=Przygodo'"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Wczytywanie konfiguracji z pliku"
]
},
{
"cell_type": "code",
2024-04-17 08:00:40 +02:00
"execution_count": 17,
2021-04-26 11:44:34 +02:00
"metadata": {
"slideshow": {
2022-04-25 11:15:16 +02:00
"slide_type": "fragment"
2021-04-26 11:44:34 +02:00
}
},
2024-04-17 08:00:40 +02:00
"outputs": [
{
"data": {
"text/plain": [
"{'recipient': 'samotności', 'greeting': 'Żegnaj'}"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
2021-04-26 11:44:34 +02:00
"source": [
"# %load IUM_07/config.json\n",
"{\n",
" \"recipient\": \"samotności\",\n",
" \"greeting\": \"Żegnaj\"\n",
"}"
]
},
{
"cell_type": "code",
2024-04-17 08:00:40 +02:00
"execution_count": 18,
2021-04-26 11:44:34 +02:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [],
"source": [
"from sacred import Experiment\n",
"\n",
"ex = Experiment(\"sacred_scopes\", interactive=True) #Jeśli wykonujemy interaktywnie (w konsoli Pythona albo w Jupyter):\n",
"# - musimy podać nazwę eksperymentu (domyślnie jako nazwa używana jest nazwa pliku źródłowego)\n",
"# - musimy dodać parametr \"interactive=True\"\n",
"# - zamiast \"automain\" używamy parametru \"main\"\n",
"\n",
"@ex.config\n",
"def my_config():\n",
" recipient = \"Świecie\"\n",
" greeting = \"Witaj\"\n",
"\n",
"ex.add_config(\"IUM_07/config.json\")\n",
"\n",
"\n",
"@ex.main\n",
"def my_main(recipient, greeting):\n",
" print(\"{0} {1}!\".format(greeting, recipient))"
]
},
{
"cell_type": "code",
2024-04-17 08:00:40 +02:00
"execution_count": 19,
2021-04-26 11:44:34 +02:00
"metadata": {
"slideshow": {
2022-04-25 11:15:16 +02:00
"slide_type": "fragment"
2021-04-26 11:44:34 +02:00
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING - sacred_scopes - No observers have been added to this run\n",
"INFO - sacred_scopes - Running command 'my_main'\n",
"INFO - sacred_scopes - Started\n",
"INFO - sacred_scopes - Completed after 0:00:00\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Żegnaj samotności!\n"
]
}
],
"source": [
"r = ex.run()"
]
},
{
"cell_type": "code",
2024-04-17 08:00:40 +02:00
"execution_count": 20,
2021-04-26 11:44:34 +02:00
"metadata": {
"slideshow": {
2022-04-25 11:15:16 +02:00
"slide_type": "fragment"
2021-04-26 11:44:34 +02:00
}
},
"outputs": [
{
"data": {
"text/plain": [
2024-04-17 08:00:40 +02:00
"{'recipient': 'samotności', 'greeting': 'Żegnaj', 'seed': 557578089}"
2021-04-26 11:44:34 +02:00
]
},
2024-04-17 08:00:40 +02:00
"execution_count": 20,
2021-04-26 11:44:34 +02:00
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"r.config"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"#### Możemy modyfikować części konfiguracji bezpośrednio przed wywołaniem"
]
},
{
"cell_type": "code",
2024-04-17 08:00:40 +02:00
"execution_count": 21,
2021-04-26 11:44:34 +02:00
"metadata": {
"slideshow": {
2022-04-25 11:15:16 +02:00
"slide_type": "fragment"
2021-04-26 11:44:34 +02:00
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING - sacred_scopes - No observers have been added to this run\n",
"INFO - sacred_scopes - Running command 'my_main'\n",
"INFO - sacred_scopes - Started\n",
"INFO - sacred_scopes - Completed after 0:00:00\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Żegnaj nudo!\n"
]
}
],
"source": [
"r = ex.run(config_updates={\"recipient\":\"nudo\"})"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
2022-04-25 11:15:16 +02:00
"#### Wstrzykiwanie zależności\n",
2021-04-26 11:44:34 +02:00
" - Oprócz funkcji głównej, wartości z konfiguracji są też wstrzykiwane do funkcji udekorowanych dekoratorem `@ex.capture`\n",
2022-04-25 11:15:16 +02:00
" - Możemy korzystać w nich ze specjalnych parametrów, np.:\n",
2021-04-26 11:44:34 +02:00
" - `_log` - daje nam dostęp do obiektu logera (więcej: [logowanie](https://sacred.readthedocs.io/en/stable/logging.html))\n",
" - `_run` - daje dostęp do obiektu reprezentującego aktualne wywołanie eksperymentu (przykład później)"
]
},
{
"cell_type": "code",
2024-04-17 08:00:40 +02:00
"execution_count": 22,
2021-04-26 11:44:34 +02:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"WARNING - sacred_scopes - No observers have been added to this run\n",
"INFO - sacred_scopes - Running command 'my_main'\n",
"INFO - sacred_scopes - Started\n",
"INFO - prepare_message - Enterred prepare_message\n",
"INFO - sacred_scopes - Completed after 0:00:00\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Witaj Świecie!\n"
]
},
{
"data": {
"text/plain": [
2024-04-17 08:00:40 +02:00
"<sacred.run.Run at 0x7ff69f8c0610>"
2021-04-26 11:44:34 +02:00
]
},
2024-04-17 08:00:40 +02:00
"execution_count": 22,
2021-04-26 11:44:34 +02:00
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from sacred import Experiment\n",
"\n",
"ex = Experiment(\"sacred_scopes\", interactive=True)\n",
"\n",
"@ex.config\n",
"def my_config():\n",
" recipient = \"Świecie\"\n",
" greeting = \"Witaj\"\n",
"\n",
"@ex.capture\n",
"def prepare_message(recipient, greeting, _log):\n",
" _log.info(\"Enterred prepare_message\")\n",
" return \"{0} {1}!\".format(greeting, recipient)\n",
"\n",
"@ex.main\n",
2021-04-26 14:18:36 +02:00
"def my_main():\n",
2021-04-26 11:44:34 +02:00
" print(prepare_message()) ## Nie musimy przekazywać wartości\n",
" \n",
"ex.run()"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Obserwowanie eksperymentów\n",
"Sacred zapisuje szereg informacji na temat każdego eksperymentu:\n",
" - czas wykonania\n",
" - konfigurację\n",
" - tekst zwrócony na stdout/stderr\n",
" - błędy, jeśli wystąpiły\n",
" - podstawowe informacje o środowisku (maszynie), na której przeprowadzono eksperyment\n",
" - użyte pliki źródłowe\n",
" - użyte zależności i ich wersje\n",
2023-04-12 15:14:37 +02:00
" - pliki otwarte za pomocą [ex.open_resource()](https://sacred.readthedocs.io/en/stable/apidoc.html#sacred.Experiment.open_resource) albo [ex.add_resource()](https://sacred.readthedocs.io/en/stable/apidoc.html#sacred.Experiment.add_resource)\n",
" - pliki dodane za pomocą [ex.add_artifact()](https://sacred.readthedocs.io/en/stable/apidoc.html#sacred.run.Run.add_artifact)"
]
},
{
"cell_type": "code",
2024-04-17 08:00:40 +02:00
"execution_count": 27,
2023-04-12 15:14:37 +02:00
"metadata": {},
2023-04-20 12:13:19 +02:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
2024-04-17 08:00:40 +02:00
"total 8\n",
"drwxr-xr-x 2 pawel pawel 4096 Apr 17 07:51 1\n",
"drwxr-xr-x 2 pawel pawel 4096 Apr 17 07:51 _sources\n"
2023-04-20 12:13:19 +02:00
]
}
],
2023-04-12 15:14:37 +02:00
"source": [
2023-04-20 12:13:19 +02:00
"!ls -l my_runs"
2021-04-26 11:44:34 +02:00
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"Obserwowane infromacje mogą zostać zapisane za pomocą jednego z [obserwatorów](https://sacred.readthedocs.io/en/stable/observers.html):\n",
" - Mongo Observer - zapisuje dane w MongoDB\n",
" - File Storage Observer - zapisuje dane lokalnie w pliku\n",
" - TinyDB Observer - korzysta z lokalnej bazy zapisanej w pliku JSON\n",
" - SQL Observer - przechowuje informacje w bazie SQL\n",
" - S3 Observer - korzysta z AWS S3\n",
" - gcs_observer - korzysta z Google Cloud Storage\n",
" - Queue Observer - rodzaj lokalnego bufora nakładanego na jeden z powyższych\n",
" - Slack Observer - używany do powiadomień wysyłanych na komunikator Slack\n",
" - Telegram Observer - używany do powiadomień wysyłanych na komunikator Telegram"
]
},
{
"cell_type": "markdown",
2021-04-26 14:18:36 +02:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
2021-04-26 11:44:34 +02:00
"source": [
"### File storage observer\n",
"- zapisuje informacje o eksperymencie w lokalnych plikach \n",
"- można go dodać tak: `ex.observers.append(FileStorageObserver('my_runs_directory'))`, gdzie `my_runs_directory` to ścieżka, gdzie będą zapisywane informacje o eksperymentach"
]
},
{
"cell_type": "code",
2024-04-17 08:00:40 +02:00
"execution_count": 24,
2021-04-26 14:18:36 +02:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
2024-04-17 08:00:40 +02:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Overwriting IUM_07/file_observer.py\n"
]
}
],
2021-04-26 11:44:34 +02:00
"source": [
2022-04-25 11:15:16 +02:00
"%%writefile IUM_07/file_observer.py\n",
2021-04-26 11:44:34 +02:00
"from sacred.observers import FileStorageObserver\n",
"from sacred import Experiment\n",
"\n",
"ex = Experiment(\"file_observer\")\n",
"\n",
"ex.observers.append(FileStorageObserver('my_runs'))\n",
"\n",
"@ex.config\n",
"def my_config():\n",
" recipient = \"Świecie\"\n",
" greeting = \"Witaj\"\n",
"\n",
"@ex.capture\n",
"def prepare_message(recipient, greeting):\n",
" return \"{0} {1}!\".format(greeting, recipient)\n",
"\n",
"@ex.automain\n",
"def my_main(recipient, greeting):\n",
" print(prepare_message()) ## Nie musimy przekazywać wartości"
]
},
{
"cell_type": "code",
2024-04-17 08:00:40 +02:00
"execution_count": 25,
2021-04-26 14:18:36 +02:00
"metadata": {
"slideshow": {
2022-04-25 11:15:16 +02:00
"slide_type": "fragment"
2021-04-26 14:18:36 +02:00
}
},
2021-04-26 11:44:34 +02:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
2024-04-17 08:00:40 +02:00
"INFO - file_observer - Running command 'my_main'\n",
"INFO - file_observer - Started run with ID \"1\"\n",
"Witaj Świecie!\n",
"INFO - file_observer - Completed after 0:00:00\n"
2021-04-26 11:44:34 +02:00
]
}
],
"source": [
2022-04-25 11:15:16 +02:00
"!python3 IUM_07/file_observer.py"
2021-04-26 11:44:34 +02:00
]
},
{
"cell_type": "markdown",
2021-04-26 14:18:36 +02:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
2021-04-26 11:44:34 +02:00
"source": [
"#### Zobaczmy jakie informacje zostały zapisane"
]
},
{
"cell_type": "code",
2024-04-17 08:00:40 +02:00
"execution_count": 28,
2021-04-26 14:18:36 +02:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
2021-04-26 11:44:34 +02:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
2024-04-17 08:00:40 +02:00
"total 8\n",
"drwxr-xr-x 2 pawel pawel 4096 Apr 17 07:51 1\n",
"drwxr-xr-x 2 pawel pawel 4096 Apr 17 07:51 _sources\n"
2021-04-26 11:44:34 +02:00
]
}
],
"source": [
"!ls -l my_runs"
]
},
{
"cell_type": "code",
2024-04-17 08:00:40 +02:00
"execution_count": 29,
2021-04-26 14:18:36 +02:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
2023-05-17 22:30:48 +02:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
2024-04-17 08:00:40 +02:00
"total 16\n",
"-rw-r--r-- 1 pawel pawel 77 Apr 17 07:51 config.json\n",
"-rw-r--r-- 1 pawel pawel 159 Apr 17 07:51 cout.txt\n",
"-rw-r--r-- 1 pawel pawel 2 Apr 17 07:51 metrics.json\n",
"-rw-r--r-- 1 pawel pawel 1715 Apr 17 07:51 run.json\n"
2023-05-17 22:30:48 +02:00
]
}
],
2021-04-26 11:44:34 +02:00
"source": [
2022-04-25 11:15:16 +02:00
"!ls -l my_runs/1"
2021-04-26 11:44:34 +02:00
]
},
{
"cell_type": "code",
2024-04-17 08:00:40 +02:00
"execution_count": 30,
2021-04-26 14:18:36 +02:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
2024-04-17 08:00:40 +02:00
"outputs": [
{
"data": {
"text/plain": [
"{'greeting': 'Witaj', 'recipient': 'Świecie', 'seed': 805857632}"
]
},
"execution_count": 30,
"metadata": {},
"output_type": "execute_result"
}
],
2021-04-26 11:44:34 +02:00
"source": [
2022-04-25 11:15:16 +02:00
"# %load my_runs/1/config.json\n",
2021-04-26 11:44:34 +02:00
"{\n",
" \"greeting\": \"Witaj\",\n",
" \"recipient\": \"\\u015awiecie\",\n",
2022-04-25 11:15:16 +02:00
" \"seed\": 805857632\n",
2021-04-26 11:44:34 +02:00
"}"
]
},
{
"cell_type": "code",
2023-04-20 12:13:19 +02:00
"execution_count": 31,
2021-04-26 14:18:36 +02:00
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
2021-04-26 11:44:34 +02:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
2024-04-17 08:00:40 +02:00
"INFO - file_observer - Running command 'my_main'\n",
"INFO - file_observer - Started run with ID \"1\"\n",
"Witaj Świecie!\n",
"INFO - file_observer - Completed after 0:00:00\n"