ium/IUM_10.DVC.ipynb

95 KiB
Raw Blame History

Logo 1

Inżynieria uczenia maszynowego

10. DVC [laboratoria]

Tomasz Ziętkiewicz (2022)

Logo 2

DVC - Data Version Control

  • dvc.org
  • "Version Control System for Machine Learning Projects" (System kontroli wersji dla projektów uczenia maszynowego)
  • Open Source
  • Umożliwia:
    • wersjonowanie danych i modeli. "Git dla danych i modeli"
    • budowanie potoków ("pipeline") definiujących jak budować/trenować/ewaluować modele. "Makefile dla uczenia maszynowego"
    • śledzenie, porównywanie metryk i parametrów
  • ściśle zintegowany z gitem
  • działa niezależnie od używanego języka/bibliotek i systemu operacyjnego
  • 5-minutowe wprowadzenie: https://www.youtube.com/watch?v=UbL7VUpv1Bs&t=197s

Instalacja i inicjalizacja

!pip3 install dvc
Collecting dvc
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/96/9d/9cca62742cb99da8002c9d6ac6c7463344deb60a219ca4f3a9778b02c67b/dvc-2.8.1-py3-none-any.whl (386kB)
    100% |████████████████████████████████| 389kB 1.4MB/s ta 0:00:01
[?25hCollecting setuptools>=34.0.0 (from dvc)
  Using cached https://files.pythonhosted.org/packages/b0/3a/88b210db68e56854d0bcf4b38e165e03be377e13907746f825790f3df5bf/setuptools-59.6.0-py3-none-any.whl
Collecting colorama>=0.3.9 (from dvc)
  Using cached https://files.pythonhosted.org/packages/44/98/5b86278fbbf250d239ae0ecb724f8572af1c91f4a11edf4d36a206189440/colorama-0.4.4-py2.py3-none-any.whl
Collecting dictdiffer>=0.8.1 (from dvc)
  Downloading https://files.pythonhosted.org/packages/47/ef/4cb333825d10317a36a1154341ba37e6e9c087bac99c1990ef07ffdb376f/dictdiffer-0.9.0-py2.py3-none-any.whl
Collecting ply>=3.9 (from dvc)
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/a3/58/35da89ee790598a0700ea49b2a66594140f44dec458c07e8e3d4979137fc/ply-3.11-py2.py3-none-any.whl (49kB)
    100% |████████████████████████████████| 51kB 1.8MB/s ta 0:00:01
[?25hCollecting nanotime>=0.5.2 (from dvc)
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/d5/54/6d5924f59cf671326e7809f4b3f70fa8df535d67e952ad0b6fea02f52faf/nanotime-0.5.2.tar.gz
Collecting fsspec[http]>=2021.10.0 (from dvc)
  Downloading https://files.pythonhosted.org/packages/f6/90/32e53b96067954c2f916667f5a11634aeabef8ed70e83133ed8037b8111b/fsspec-2022.1.0-py3-none-any.whl (133kB)
    100% |████████████████████████████████| 143kB 1.3MB/s ta 0:00:01
[?25hCollecting networkx>=2.5 (from dvc)
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/f3/b7/c7f488101c0bb5e4178f3cde416004280fd40262433496830de8a8c21613/networkx-2.5.1-py3-none-any.whl (1.6MB)
    100% |████████████████████████████████| 1.6MB 609kB/s ta 0:00:01
[?25hCollecting flufl.lock<4,>=3.2 (from dvc)
  Downloading https://files.pythonhosted.org/packages/1e/68/393c148df629f90a919de653ebb967a8bd8c83d07d2bc3150ca0faff3940/flufl.lock-3.2.tar.gz
Collecting shtab<2,>=1.3.4 (from dvc)
  Downloading https://files.pythonhosted.org/packages/37/c0/7d887589bb87db6b87b55315ac41acbbe7355074006a0968343d52ded5cd/shtab-1.5.4-py2.py3-none-any.whl
Collecting shortuuid>=0.5.0 (from dvc)
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/01/14/df1aa61e1bb75a6fff19b25d7d175b7f188dc15a70dfc80e7badd5bda1de/shortuuid-1.0.9-py3-none-any.whl
Collecting gitpython>3 (from dvc)
  Using cached https://files.pythonhosted.org/packages/55/60/f884f01eef2a7255875862ec1b12d57d74113ec6e8d9e16c4d254cd6aa3c/GitPython-3.1.20-py3-none-any.whl
Collecting tqdm<5,>=4.45.0 (from dvc)
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/8a/c4/d15f1e627fff25443ded77ea70a7b5532d6371498f9285d44d62587e209c/tqdm-4.64.0-py2.py3-none-any.whl (78kB)
    100% |████████████████████████████████| 81kB 2.0MB/s ta 0:00:01
[?25hCollecting zc.lockfile>=1.2.1 (from dvc)
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/6c/2a/268389776288f0f26c7272c70c36c96dcc0bdb88ab6216ea18e19df1fadd/zc.lockfile-2.0-py2.py3-none-any.whl
Collecting rich>=10.9.0 (from dvc)
  Downloading https://files.pythonhosted.org/packages/05/c0/844c879d659e15c4b1fbe8be89b25de7cf9b0d5e8485aadfe83ae6e7286a/rich-12.4.1-py3-none-any.whl (231kB)
    100% |████████████████████████████████| 235kB 1.5MB/s ta 0:00:01
[?25hCollecting aiohttp-retry>=2.4.5 (from dvc)
  Downloading https://files.pythonhosted.org/packages/0f/fe/58130a432d4397e174b82e57d8a11ccf5066631d03949dada4aac8b88041/aiohttp_retry-2.4.6-py3-none-any.whl
Collecting voluptuous>=0.11.7 (from dvc)
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/a7/68/927add5dfd55a0d666ffc8939ff4390b76ca3ffbc36c12369f9a034393cb/voluptuous-0.13.1-py3-none-any.whl
Collecting psutil>=5.8.0 (from dvc)
  Downloading https://files.pythonhosted.org/packages/d6/de/0999ea2562b96d7165812606b18f7169307b60cd378bc29cf3673322c7e9/psutil-5.9.1.tar.gz (479kB)
    100% |████████████████████████████████| 481kB 1.2MB/s ta 0:00:011
[?25hCollecting flatten-dict<1,>=0.4.1 (from dvc)
  Downloading https://files.pythonhosted.org/packages/43/f5/ee39c6e92acc742c052f137b47c210cd0a1b72dcd3f98495528bb4d27761/flatten_dict-0.4.2-py2.py3-none-any.whl
Collecting configobj>=5.0.6 (from dvc)
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/64/61/079eb60459c44929e684fa7d9e2fdca403f67d64dd9dbac27296be2e0fab/configobj-5.0.6.tar.gz
Collecting pygtrie>=2.3.2 (from dvc)
  Downloading https://files.pythonhosted.org/packages/a5/8b/90d0f21a27a354e808a73eb0ffb94db990ab11ad1d8b3db3e5196c882cad/pygtrie-2.4.2.tar.gz
Collecting tabulate>=0.8.7 (from dvc)
  Downloading https://files.pythonhosted.org/packages/ca/80/7c0cad11bd99985cfe7c09427ee0b4f9bd6b048bd13d4ffb32c6db237dfb/tabulate-0.8.9-py3-none-any.whl
Collecting dulwich>=0.20.23 (from dvc)
  Downloading https://files.pythonhosted.org/packages/50/88/d5f6b478086a78995a71151a956ac97d6635f0f6d80f625c96ba13cc612d/dulwich-0.20.40.tar.gz (423kB)
    100% |████████████████████████████████| 430kB 1.1MB/s ta 0:00:01
[?25hCollecting appdirs>=1.4.3 (from dvc)
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/3b/00/2344469e2084fb287c2e0b57b72910309874c3245463acd6cf5e3db69324/appdirs-1.4.4-py2.py3-none-any.whl
Collecting ruamel.yaml>=0.17.11 (from dvc)
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/9e/cb/938214ac358fbef7058343b3765c79a1b7ed0c366f7f992ce7ff38335652/ruamel.yaml-0.17.21-py3-none-any.whl (109kB)
    100% |████████████████████████████████| 112kB 2.1MB/s ta 0:00:01
[?25hCollecting python-benedict>=0.24.2 (from dvc)
  Downloading https://files.pythonhosted.org/packages/77/84/6268f9c31140f23ef290c89cb91197f5ea6cbbf102a15db13c1eea845a22/python_benedict-0.25.1-py3-none-any.whl (41kB)
    100% |████████████████████████████████| 51kB 2.2MB/s ta 0:00:01
[?25hCollecting grandalf==0.6 (from dvc)
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/54/f4/a0b6a4c6d616d0a838b2dd0bc7bf74d73e8e8cdc880bab7fdb5fdc3d0e06/grandalf-0.6-py3-none-any.whl
Collecting pyparsing==2.4.7 (from dvc)
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl (67kB)
    100% |████████████████████████████████| 71kB 2.1MB/s ta 0:00:01
[?25hCollecting dpath<3,>=2.0.2 (from dvc)
  Downloading https://files.pythonhosted.org/packages/9d/68/19b2a579121446ba80aa3574f04de6931ad549a22cd5588e43c8031de7f8/dpath-2.0.6-py3-none-any.whl
Collecting pygit2<1.7,>=1.5.0; python_version < "3.7" (from dvc)
  Downloading https://files.pythonhosted.org/packages/6b/23/a8c5b726a58282fe2cadcc63faaddd4be147c3c8e0bd38b233114adf98fd/pygit2-1.6.1.tar.gz (258kB)
    100% |████████████████████████████████| 266kB 3.5MB/s eta 0:00:01
[?25hCollecting dataclasses>=0.7; python_version < "3.7" (from dvc)
  Using cached https://files.pythonhosted.org/packages/fe/ca/75fac5856ab5cfa51bbbcefa250182e50441074fdc3f803f6e76451fab43/dataclasses-0.8-py3-none-any.whl
Collecting pyasn1>=0.4.1 (from dvc)
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/62/1e/a94a8d635fa3ce4cfc7f506003548d0a2447ae76fd5ca53932970fe3053f/pyasn1-0.4.8-py2.py3-none-any.whl (77kB)
    100% |████████████████████████████████| 81kB 2.2MB/s ta 0:00:01
[?25hCollecting pathspec<0.9.0,>=0.6.0 (from dvc)
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/29/29/a465741a3d97ea3c17d21eaad4c64205428bde56742360876c4391f930d4/pathspec-0.8.1-py2.py3-none-any.whl
Collecting contextvars>=2.1; python_version < "3.7" (from dvc)
  Downloading https://files.pythonhosted.org/packages/83/96/55b82d9f13763be9d672622e1b8106c85acb83edd7cc2fa5bc67cd9877e9/contextvars-2.4.tar.gz
Collecting distro>=1.3.0 (from dvc)
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/e1/54/d08d1ad53788515392bec14d2d6e8c410bffdc127780a9a4aa8e6854d502/distro-1.7.0-py3-none-any.whl
Collecting packaging>=19.0 (from dvc)
  Using cached https://files.pythonhosted.org/packages/05/8e/8de486cbd03baba4deef4142bd643a3e7bbe954a784dc1bb17142572d127/packaging-21.3-py3-none-any.whl
Collecting typing-extensions>=3.7.4 (from dvc)
  Using cached https://files.pythonhosted.org/packages/45/6b/44f7f8f1e110027cf88956b59f2fad776cca7e1704396d043f89effd3a0e/typing_extensions-4.1.1-py3-none-any.whl
Collecting importlib-metadata>=1.4; python_version < "3.8" (from dvc)
  Using cached https://files.pythonhosted.org/packages/a0/a1/b153a0a4caf7a7e3f15c2cd56c7702e2cf3d89b1b359d1f1c5e59d68f4ce/importlib_metadata-4.8.3-py3-none-any.whl
Collecting diskcache>=5.2.1 (from dvc)
  Downloading https://files.pythonhosted.org/packages/a1/c4/80d38cf6852ba87f8b506a91f18b3a485c668f452700689add960d7e2ecc/diskcache-5.4.0-py3-none-any.whl (44kB)
    100% |████████████████████████████████| 51kB 2.1MB/s ta 0:00:01
[?25hCollecting pydot>=1.2.4 (from dvc)
  Downloading https://files.pythonhosted.org/packages/ea/76/75b1bb82e9bad3e3d656556eaa353d8cd17c4254393b08ec9786ac8ed273/pydot-1.4.2-py2.py3-none-any.whl
Collecting funcy>=1.14 (from dvc)
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/cd/1c/895001e29f870c4625a90af00895ef9c9f4f37b0a9b967d2ed810b7be0fc/funcy-1.17-py2.py3-none-any.whl
Collecting toml>=0.10.1 (from dvc)
  Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl
Collecting requests>=2.22.0 (from dvc)
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/2d/61/08076519c80041bc0ffa1a8af0cbd3bf3e2b62af10435d269a9d0f40564d/requests-2.27.1-py2.py3-none-any.whl (63kB)
    100% |████████████████████████████████| 71kB 2.2MB/s ta 0:00:01
[?25hCollecting aiohttp; extra == "http" (from fsspec[http]>=2021.10.0->dvc)
  Downloading https://files.pythonhosted.org/packages/f6/3b/2e3b8a5b19cdceb532c61d83077a09afe1f120cb876fb771b0ce577cc0ea/aiohttp-3.8.1-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1MB)
    100% |████████████████████████████████| 1.1MB 729kB/s ta 0:00:01
[?25hCollecting decorator<5,>=4.3 (from networkx>=2.5->dvc)
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/ed/1b/72a1821152d07cf1d8b6fce298aeb06a7eb90f4d6d41acec9861e7cc6df0/decorator-4.4.2-py2.py3-none-any.whl
Collecting atpublic (from flufl.lock<4,>=3.2->dvc)
  Downloading https://files.pythonhosted.org/packages/ab/3d/3df1468805427fedcf880da42fa26353feea3a31b5a0cc71008adcfdb816/atpublic-2.3.tar.gz
Collecting gitdb<5,>=4.0.1 (from gitpython>3->dvc)
  Using cached https://files.pythonhosted.org/packages/a3/7c/5d747655049bfbf75b5fcec57c8115896cb78d6fafa84f6d3ef4c0f13a98/gitdb-4.0.9-py3-none-any.whl
Collecting importlib-resources; python_version < "3.7" (from tqdm<5,>=4.45.0->dvc)
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/24/1b/33e489669a94da3ef4562938cd306e8fa915e13939d7b8277cb5569cb405/importlib_resources-5.4.0-py3-none-any.whl
Collecting commonmark<0.10.0,>=0.9.0 (from rich>=10.9.0->dvc)
  Downloading https://files.pythonhosted.org/packages/b1/92/dfd892312d822f36c55366118b95d914e5f16de11044a27cf10a7d71bbbf/commonmark-0.9.1-py2.py3-none-any.whl (51kB)
    100% |████████████████████████████████| 51kB 1.6MB/s ta 0:00:01
[?25hCollecting pygments<3.0.0,>=2.6.0 (from rich>=10.9.0->dvc)
  Using cached https://files.pythonhosted.org/packages/5c/8e/1d9017950034297fffa336c72e693a5b51bbf85141b24a763882cf1977b5/Pygments-2.12.0-py3-none-any.whl
Collecting six<2.0,>=1.12 (from flatten-dict<1,>=0.4.1->dvc)
  Using cached https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl
Collecting urllib3>=1.24.1 (from dulwich>=0.20.23->dvc)
  Downloading https://files.pythonhosted.org/packages/ec/03/062e6444ce4baf1eac17a6a0ebfe36bb1ad05e1df0e20b110de59c278498/urllib3-1.26.9-py2.py3-none-any.whl (138kB)
    100% |████████████████████████████████| 143kB 1.7MB/s ta 0:00:01
[?25hCollecting certifi (from dulwich>=0.20.23->dvc)
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/11/dd/e015f3780f42dd9af62cf0107b44ea1298926627ecd70c17b0e484e95bcd/certifi-2022.5.18.1-py3-none-any.whl (155kB)
    100% |████████████████████████████████| 163kB 1.8MB/s ta 0:00:01
[?25hCollecting ruamel.yaml.clib>=0.2.6; platform_python_implementation == "CPython" and python_version < "3.11" (from ruamel.yaml>=0.17.11->dvc)
  Downloading https://files.pythonhosted.org/packages/2a/25/5b1dfc832ef3b83576c546d1fb3e27f136022cdd1008aab290a1e28ef220/ruamel.yaml.clib-0.2.6-cp36-cp36m-manylinux1_x86_64.whl (552kB)
    100% |████████████████████████████████| 552kB 1.2MB/s ta 0:00:01
[?25hCollecting python-slugify<7.0.0,>=6.0.1 (from python-benedict>=0.24.2->dvc)
  Downloading https://files.pythonhosted.org/packages/c1/35/74ab800f1108b95ff9b8e7672a01dbf1f357159e6d06c1f16e983674ff0c/python_slugify-6.1.2-py2.py3-none-any.whl
Collecting python-fsutil<1.0.0,>=0.6.0 (from python-benedict>=0.24.2->dvc)
  Downloading https://files.pythonhosted.org/packages/d3/67/3bceac53a29c2cf6a27ca9a940c7a059af13b5b4283d09368e9c0d7f91b4/python_fsutil-0.6.1-py3-none-any.whl
Collecting mailchecker<5.0.0,>=4.1.0 (from python-benedict>=0.24.2->dvc)
  Downloading https://files.pythonhosted.org/packages/1a/11/49e85a526dd3f2cb6bdf1dc519d64395bd56a88d9b2a5f6f3acd1f8c3f51/mailchecker-4.1.17.tar.gz (232kB)
    100% |████████████████████████████████| 235kB 2.7MB/s ta 0:00:011
[?25hCollecting pyyaml<7.0,>=6.0 (from python-benedict>=0.24.2->dvc)
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/b3/85/79b9e5b4e8d3c0ac657f4e8617713cca8408f6cdc65d2ee6554217cedff1/PyYAML-6.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (603kB)
    100% |████████████████████████████████| 604kB 1.1MB/s ta 0:00:01
[?25hCollecting xmltodict<1.0.0,>=0.12.0 (from python-benedict>=0.24.2->dvc)
  Downloading https://files.pythonhosted.org/packages/94/db/fd0326e331726f07ff7f40675cd86aa804bfd2e5016c727fa761c934990e/xmltodict-0.13.0-py2.py3-none-any.whl
Collecting phonenumbers<9.0.0,>=8.12.0 (from python-benedict>=0.24.2->dvc)
  Downloading https://files.pythonhosted.org/packages/a1/c8/23c7185f589da0edf4f9831de1e718989073aa094780f8ee34e6c29b7a2e/phonenumbers-8.12.48-py2.py3-none-any.whl (2.6MB)
    100% |████████████████████████████████| 2.6MB 434kB/s ta 0:00:011
[?25hCollecting ftfy<7.0.0,>=6.0.0 (from python-benedict>=0.24.2->dvc)
  Downloading https://files.pythonhosted.org/packages/af/da/d215a091986e5f01b80f5145cff6f22e2dc57c6b048aab2e882a07018473/ftfy-6.0.3.tar.gz (64kB)
    100% |████████████████████████████████| 71kB 2.0MB/s ta 0:00:01
[?25hCollecting python-dateutil<3.0.0,>=2.8.0 (from python-benedict>=0.24.2->dvc)
  Cache entry deserialization failed, entry ignored
  Using cached https://files.pythonhosted.org/packages/36/7a/87837f39d0296e723bb9b62bbb257d0355c7f6128853c78955f57342a56d/python_dateutil-2.8.2-py2.py3-none-any.whl
Collecting future (from grandalf==0.6->dvc)
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/45/0b/38b06fd9b92dc2b68d58b75f900e97884c45bedd2ff83203d933cf5851c9/future-0.18.2.tar.gz (829kB)
    100% |████████████████████████████████| 829kB 1.5MB/s eta 0:00:01
[?25hCollecting cffi>=1.4.0 (from pygit2<1.7,>=1.5.0; python_version < "3.7"->dvc)
  Using cached https://files.pythonhosted.org/packages/49/7b/449daf9cacfd7355cea1b4106d2be614315c29ac16567e01756167f6daab/cffi-1.15.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl
Collecting cached-property (from pygit2<1.7,>=1.5.0; python_version < "3.7"->dvc)
  Downloading https://files.pythonhosted.org/packages/48/19/f2090f7dad41e225c7f2326e4cfe6fff49e57dedb5b53636c9551f86b069/cached_property-1.5.2-py2.py3-none-any.whl
Collecting immutables>=0.9 (from contextvars>=2.1; python_version < "3.7"->dvc)
  Downloading https://files.pythonhosted.org/packages/ff/88/9c71337193c3d24c2cf3c14d5ed05eeb502f9f21fa6117edfa9b3b43bff1/immutables-0.18-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (115kB)
    100% |████████████████████████████████| 122kB 2.3MB/s ta 0:00:01
[?25hCollecting zipp>=0.5 (from importlib-metadata>=1.4; python_version < "3.8"->dvc)
  Using cached https://files.pythonhosted.org/packages/bd/df/d4a4974a3e3957fd1c1fa3082366d7fff6e428ddb55f074bf64876f8e8ad/zipp-3.6.0-py3-none-any.whl
Collecting idna<4,>=2.5; python_version >= "3" (from requests>=2.22.0->dvc)
  Downloading https://files.pythonhosted.org/packages/04/a2/d918dcd22354d8958fe113e1a3630137e0fc8b44859ade3063982eacd2a4/idna-3.3-py3-none-any.whl (61kB)
    100% |████████████████████████████████| 61kB 1.7MB/s ta 0:00:01
[?25hCollecting charset-normalizer~=2.0.0; python_version >= "3" (from requests>=2.22.0->dvc)
  Downloading https://files.pythonhosted.org/packages/06/b3/24afc8868eba069a7f03650ac750a778862dc34941a4bebeb58706715726/charset_normalizer-2.0.12-py3-none-any.whl
Collecting asynctest==0.13.0; python_version < "3.8" (from aiohttp; extra == "http"->fsspec[http]>=2021.10.0->dvc)
  Downloading https://files.pythonhosted.org/packages/e8/b6/8d17e169d577ca7678b11cd0d3ceebb0a6089a7f4a2de4b945fe4b1c86db/asynctest-0.13.0-py3-none-any.whl
Collecting yarl<2.0,>=1.0 (from aiohttp; extra == "http"->fsspec[http]>=2021.10.0->dvc)
  Downloading https://files.pythonhosted.org/packages/fa/cb/8791922f5ec97b9ebec516d062c0e113da963568ffe2c7c04d8187ab7cc3/yarl-1.7.2-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (270kB)
    100% |████████████████████████████████| 276kB 1.9MB/s ta 0:00:01
[?25hCollecting idna-ssl>=1.0; python_version < "3.7" (from aiohttp; extra == "http"->fsspec[http]>=2021.10.0->dvc)
  Downloading https://files.pythonhosted.org/packages/46/03/07c4894aae38b0de52b52586b24bf189bb83e4ddabfe2e2c8f2419eec6f4/idna-ssl-1.1.0.tar.gz
Collecting frozenlist>=1.1.1 (from aiohttp; extra == "http"->fsspec[http]>=2021.10.0->dvc)
  Downloading https://files.pythonhosted.org/packages/51/3f/f67395ff0090b9f2835838a1f61c3e840baac70fd65bae762095dead48b2/frozenlist-1.2.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (191kB)
    100% |████████████████████████████████| 194kB 1.2MB/s ta 0:00:01
[?25hCollecting async-timeout<5.0,>=4.0.0a3 (from aiohttp; extra == "http"->fsspec[http]>=2021.10.0->dvc)
  Downloading https://files.pythonhosted.org/packages/d6/c1/8991e7c5385b897b8c020cdaad718c5b087a6626d1d11a23e1ea87e325a7/async_timeout-4.0.2-py3-none-any.whl
Collecting aiosignal>=1.1.2 (from aiohttp; extra == "http"->fsspec[http]>=2021.10.0->dvc)
  Downloading https://files.pythonhosted.org/packages/3b/87/fe94898f2d44a93a35d5aa74671ed28094d80753a1113d68b799fab6dc22/aiosignal-1.2.0-py3-none-any.whl
Collecting attrs>=17.3.0 (from aiohttp; extra == "http"->fsspec[http]>=2021.10.0->dvc)
  Using cached https://files.pythonhosted.org/packages/be/be/7abce643bfdf8ca01c48afa2ddf8308c2308b0c3b239a44e57d020afa0ef/attrs-21.4.0-py2.py3-none-any.whl
Collecting multidict<7.0,>=4.5 (from aiohttp; extra == "http"->fsspec[http]>=2021.10.0->dvc)
  Downloading https://files.pythonhosted.org/packages/82/43/81ddfbcfbdfaeaa0624f36dcb715dc8135562377b3292e93b0315a861e92/multidict-5.2.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (159kB)
    100% |████████████████████████████████| 163kB 2.0MB/s ta 0:00:01
[?25hCollecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->gitpython>3->dvc)
  Using cached https://files.pythonhosted.org/packages/6d/01/7caa71608bc29952ae09b0be63a539e50d2484bc37747797a66a60679856/smmap-5.0.0-py3-none-any.whl
Collecting text-unidecode>=1.3 (from python-slugify<7.0.0,>=6.0.1->python-benedict>=0.24.2->dvc)
  Downloading https://files.pythonhosted.org/packages/a6/a5/c0b6468d3824fe3fde30dbb5e1f687b291608f9473681bbf7dabbf5a87d7/text_unidecode-1.3-py2.py3-none-any.whl (78kB)
    100% |████████████████████████████████| 81kB 2.2MB/s ta 0:00:01
[?25hCollecting wcwidth (from ftfy<7.0.0,>=6.0.0->python-benedict>=0.24.2->dvc)
  Using cached https://files.pythonhosted.org/packages/59/7c/e39aca596badaf1b78e8f547c807b04dae603a433d3e7a7e04d67f2ef3e5/wcwidth-0.2.5-py2.py3-none-any.whl
Collecting pycparser (from cffi>=1.4.0->pygit2<1.7,>=1.5.0; python_version < "3.7"->dvc)
  Using cached https://files.pythonhosted.org/packages/62/d5/5f610ebe421e85889f2e55e33b7f9a6795bd982198517d912eb1c76e1a53/pycparser-2.21-py2.py3-none-any.whl
Building wheels for collected packages: nanotime, flufl.lock, psutil, configobj, pygtrie, dulwich, pygit2, contextvars, atpublic, mailchecker, ftfy, future, idna-ssl
  Running setup.py bdist_wheel for nanotime ... [?25ldone
[?25h  Stored in directory: /home/tomek/.cache/pip/wheels/41/99/17/7135f635215e1f61e906295afd11f4f791cfe4ab45f3bfdca2
  Running setup.py bdist_wheel for flufl.lock ... [?25ldone
[?25h  Stored in directory: /home/tomek/.cache/pip/wheels/4f/51/d7/f65a7b7f37da7594f7021b122fe677187667ad21f1171d2514
  Running setup.py bdist_wheel for psutil ... [?25ldone
[?25h  Stored in directory: /home/tomek/.cache/pip/wheels/6e/94/8f/ef906811f8dcf6824a9747df0381615be48d723073fb59a317
  Running setup.py bdist_wheel for configobj ... [?25ldone
[?25h  Stored in directory: /home/tomek/.cache/pip/wheels/f1/e4/16/4981ca97c2d65106b49861e0b35e2660695be7219a2d351ee0
  Running setup.py bdist_wheel for pygtrie ... [?25ldone
[?25h  Stored in directory: /home/tomek/.cache/pip/wheels/db/57/91/73782136379fe419036c5ec0e4070d8b3a35f2a36bd6a94ed8
  Running setup.py bdist_wheel for dulwich ... [?25ldone
[?25h  Stored in directory: /home/tomek/.cache/pip/wheels/63/28/8c/0bbff7d6e30f3fc523639b000b33aba9155152e9eb23689ba0
  Running setup.py bdist_wheel for pygit2 ... [?25lerror
  Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-hr8mtrcf/pygit2/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmp4kyicel3pip-wheel- --python-tag cp36:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.6
  creating build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/__init__.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/_build.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/_run.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/blame.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/callbacks.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/config.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/credentials.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/errors.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/ffi.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/index.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/packbuilder.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/refspec.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/remote.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/repository.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/settings.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/submodule.py -> build/lib.linux-x86_64-3.6/pygit2
  copying pygit2/utils.py -> build/lib.linux-x86_64-3.6/pygit2
  creating build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/attr.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/blame.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/buffer.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/callbacks.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/checkout.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/clone.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/common.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/config.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/describe.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/diff.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/errors.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/graph.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/index.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/indexer.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/merge.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/net.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/oid.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/pack.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/proxy.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/refspec.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/remote.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/repository.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/revert.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/stash.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/strarray.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/submodule.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/transport.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  copying pygit2/decl/types.h -> build/lib.linux-x86_64-3.6/pygit2/decl
  running build_ext
  generating cffi module 'build/temp.linux-x86_64-3.6/pygit2._libgit2.c'
  creating build/temp.linux-x86_64-3.6
  building 'pygit2._pygit2' extension
  creating build/temp.linux-x86_64-3.6/src
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/local/include -I/usr/include/python3.6m -c src/blob.c -o build/temp.linux-x86_64-3.6/src/blob.o
  In file included from src/blob.c:30:0:
  src/blob.h:33:10: fatal error: git2.h: No such file or directory
   #include <git2.h>
            ^~~~~~~~
  compilation terminated.
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
  
  ----------------------------------------
  Failed building wheel for pygit2
[?25h  Running setup.py clean for pygit2
  Running setup.py bdist_wheel for contextvars ... [?25ldone
[?25h  Stored in directory: /home/tomek/.cache/pip/wheels/a5/7d/68/1ebae2668bda2228686e3c1cf16f2c2384cea6e9334ad5f6de
  Running setup.py bdist_wheel for atpublic ... [?25ldone
[?25h  Stored in directory: /home/tomek/.cache/pip/wheels/33/25/82/57d46b60a048f8e30b31f10497539498a3b826c78e2433c2d4
  Running setup.py bdist_wheel for mailchecker ... [?25ldone
[?25h  Stored in directory: /home/tomek/.cache/pip/wheels/fd/e1/e7/804e77a70eac7103bdba2f4b3e1eba36840b38554a4b8152c8
  Running setup.py bdist_wheel for ftfy ... [?25ldone
[?25h  Stored in directory: /home/tomek/.cache/pip/wheels/99/2c/e6/109c8a28fef7a443f67ba58df21fe1d0067ac3322e75e6b0b7
  Running setup.py bdist_wheel for future ... [?25ldone
[?25h  Stored in directory: /home/tomek/.cache/pip/wheels/8b/99/a0/81daf51dcd359a9377b110a8a886b3895921802d2fc1b2397e
  Running setup.py bdist_wheel for idna-ssl ... [?25ldone
[?25h  Stored in directory: /home/tomek/.cache/pip/wheels/d3/00/b3/32d613e19e08a739751dd6bf998cfed277728f8b2127ad4eb7
Successfully built nanotime flufl.lock psutil configobj pygtrie dulwich contextvars atpublic mailchecker ftfy future idna-ssl
Failed to build pygit2
Installing collected packages: setuptools, colorama, dictdiffer, ply, nanotime, asynctest, idna, multidict, typing-extensions, yarl, idna-ssl, frozenlist, async-timeout, aiosignal, charset-normalizer, attrs, aiohttp, certifi, urllib3, requests, fsspec, decorator, networkx, atpublic, flufl.lock, shtab, shortuuid, smmap, gitdb, gitpython, zipp, importlib-resources, tqdm, zc.lockfile, dataclasses, commonmark, pygments, rich, aiohttp-retry, voluptuous, psutil, six, importlib-metadata, flatten-dict, configobj, pygtrie, tabulate, dulwich, appdirs, ruamel.yaml.clib, ruamel.yaml, text-unidecode, python-slugify, python-fsutil, mailchecker, pyyaml, toml, xmltodict, phonenumbers, wcwidth, ftfy, python-dateutil, python-benedict, future, pyparsing, grandalf, dpath, pycparser, cffi, cached-property, pygit2, pyasn1, pathspec, immutables, contextvars, distro, packaging, diskcache, pydot, funcy, dvc
  Running setup.py install for pygit2 ... [?25lerror
    Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-hr8mtrcf/pygit2/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-6tml5js1-record/install-record.txt --single-version-externally-managed --compile --user --prefix=:
    running install
    /home/tomek/.local/lib/python3.6/site-packages/setuptools/command/install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
      setuptools.SetuptoolsDeprecationWarning,
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.6
    creating build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/__init__.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/_build.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/_run.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/blame.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/callbacks.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/config.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/credentials.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/errors.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/ffi.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/index.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/packbuilder.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/refspec.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/remote.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/repository.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/settings.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/submodule.py -> build/lib.linux-x86_64-3.6/pygit2
    copying pygit2/utils.py -> build/lib.linux-x86_64-3.6/pygit2
    creating build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/attr.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/blame.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/buffer.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/callbacks.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/checkout.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/clone.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/common.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/config.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/describe.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/diff.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/errors.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/graph.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/index.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/indexer.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/merge.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/net.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/oid.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/pack.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/proxy.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/refspec.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/remote.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/repository.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/revert.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/stash.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/strarray.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/submodule.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/transport.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    copying pygit2/decl/types.h -> build/lib.linux-x86_64-3.6/pygit2/decl
    running build_ext
    generating cffi module 'build/temp.linux-x86_64-3.6/pygit2._libgit2.c'
    creating build/temp.linux-x86_64-3.6
    building 'pygit2._pygit2' extension
    creating build/temp.linux-x86_64-3.6/src
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/local/include -I/usr/include/python3.6m -c src/blob.c -o build/temp.linux-x86_64-3.6/src/blob.o
    In file included from src/blob.c:30:0:
    src/blob.h:33:10: fatal error: git2.h: No such file or directory
     #include <git2.h>
              ^~~~~~~~
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    
    ----------------------------------------
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-hr8mtrcf/pygit2/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-6tml5js1-record/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /tmp/pip-build-hr8mtrcf/pygit2/
[?25h

Stwórzmy katalog, w którym będziemy przechowywać nasz projekt:

!mkdir -p IUM_10/sample-ml-project
#Jupyter notebook magic https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-cd
%cd "IUM_10/sample-ml-project"
/home/tomek/repos/aitech-ium/IUM_10/sample-ml-project

Inicjalizujemy puste repozytorium Git (możemy też pominąć ten krok i działać w istniejącym już repozytorium)

!git init
Reinitialized existing Git repository in /home/tomek/repos/aitech-ium/IUM_10/sample-ml-project/.git/

Teraz inicjalizujemy repozytorium DVC:

!dvc init
ERROR: failed to initiate DVC - '.dvc' exists. Use `-f` to force.


Zobaczmy jakie pliki dodał (również do repozytorium git) DVC. Ich opis znajdziemy tutaj: https://dvc.org/doc/user-guide/project-structure/internal-files

!git status
On branch master
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
	modified:   .dvc/config

Możemy teraz zacommitować zmiany w git:

!git commit -m "Initial commit"
[master (root-commit) d00d0ac] Initial commit
 3 files changed, 6 insertions(+)
 create mode 100644 .dvc/.gitignore
 create mode 100644 .dvc/config
 create mode 100644 .dvcignore

Śledzenie plików za pomocą DVC

  • dużymi plikami, takimi jak plikami z danymi wejściowymi czy plikami modeli, trudno zarządza się za pomocą gita, ze względu na problemy z:
  • Git posiada rozszerzenie lfs(Large File Storage), które stanowi pewne rozwiązanie tego problemu. Same pliki przechowywane są na specjalnym zdalnym serwerze, w repozytorium przechowywane są jedynie odnośniki do tych plików i pewne metadane
  • DVC proponuje podobne podejście, ale:
    • pliki mogą być przechowywane na niemal dowolnym serwerze, również lokalnie
    • brak limitu wielkości plików (w Git-LFS na Github limit 2GB)
    • DVC zapewnia dodatkowe narzędzie umożliwiające śledzenie plików i ich powiązań z wynikami eksperymentów
    • więcej, patrz tutaj

Przygotujmy przykładowe dane, pobierając je z Kaggle:

!kaggle datasets download -d uciml/iris
!unzip -o iris.zip
!rm database.sqlite iris.zip
!mkdir -p data
!mv Iris.csv data/
Downloading iris.zip to /home/tomek/repos/aitech-ium/IUM_10/sample-ml-project
  0%|                                               | 0.00/3.60k [00:00<?, ?B/s]
100%|██████████████████████████████████████| 3.60k/3.60k [00:00<00:00, 2.10MB/s]
Archive:  iris.zip
  inflating: Iris.csv                
  inflating: database.sqlite         

Teraz dodamy plik(i) z danymi do DVC:

!dvc add data/Iris.csv
⠋ Checking graph                                                   ⠋ Checking graph
Adding...                                                                       
!
  0% Checking cache in '/home/tomek/repos/aitech-ium/IUM_10/sample-ml-project/.d
                                                                                
!
  0%|          |Transferring                          0/1 [00:00<?,     ?file/s]
                                                                                
!
  0%|          |.ZfWsiEaSUu83Mc7QqhDoTS.tmp        0.00/? [00:00<?,        ?B/s]
  0%|          |.ZfWsiEaSUu83Mc7QqhDoTS.tmp     0.00/4.00 [00:00<?,        ?B/s]
                                                                                
!
  0%|          |7820ef0af287ff346c5cabfb4c612c     0.00/? [00:00<?,        ?B/s]
  0%|          |7820ef0af287ff346c5cabfb4c612c 0.00/4.99k [00:00<?,        ?B/s]
100% Adding...|████████████████████████████████████████|1/1 [00:00,  5.92file/s]

To track the changes with git, run:

    git add data/Iris.csv.dvc

To enable auto staging, run:

	dvc config core.autostage true

  • DVC utworzył plik data/Iris.csv.dvc i dadał oryginalny plik do .gitignore
  • W repozytorium będzie obecny tylko plik *.dvc, zawierający odnośnik do prawdziwego pliku
!git status -u
On branch master
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
	modified:   .dvc/config

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   data/Iris.csv.dvc

Dodajmy pliki data/Iris.csv.dvc data/.gitignore do repozytorium git, zgodnie z sugestią DVC:

!git add data/Iris.csv.dvc data/.gitignore
!git commit -m "Dodano dane IRIS (DVC)"
[master b37a8d8] Dodano dane IRIS (DVC)
 2 files changed, 2 insertions(+), 6 deletions(-)

Plik *.dvc zawiera m.in. hash pliku. Więcej o plikach *.dvc: link

# %load data/Iris.csv.dvc
outs:
- md5: 717820ef0af287ff346c5cabfb4c612c
  size: 5107
  path: Iris.csv

Oryginalny plik Iris.csv został przeniesiony do katalogu ./dvc/cache/{wartość hash pliku) i podlinkowany z powrotem do oryginalnej lokalizacji. Sposób tworzenia linków może być różny w zależności od systemu plików.

!ls -l .dvc/cache/71
total 8
-r--r--r-- 1 tomek tomek 5107 Sep 19  2019 7820ef0af287ff346c5cabfb4c612c
!ls -l ./data
total 8
-rw-r--r-- 1 tomek tomek 5107 May 29 09:19 Iris.csv
-rw-r--r-- 1 tomek tomek   76 May 29 09:19 Iris.csv.dvc

dvc remote

  • żeby wysłać właściwe pliki śledzone przez DVC do zdalnej lokalizacji (z której będą mogłby być pobrane np. przez system CI albo innych użytkowników) musimy mieć skonfigurowaną taką lokazliację
  • służy do tego polecenie dvc remote add
  • użyjemy lokalnego "remote". Tutaj będzie to po prostu utworzony wcześniej katalog /dvcstore. Taki katalog istnieje też na naszym Jenkinsie, oczywiście należy go podmontować w Dockerze
  • w realnych zastosowaniach podalibyśmy tutaj ścieżkę do jakiegoś zasobu dostępnego przez inernet jak np. serwer SFTP, ścieżka do AWS S3 itp.

Obsługiwane typy zdalnych lokalizacji (remotes): https://dvc.org/doc/command-reference/remote/add#supported-storage-types

  • Amazon S3
  • S3-compatible storage
  • Microsoft Azure Blob Storage
  • Google Drive
  • Google Cloud Storage
  • Aliyun OSS
  • SSH
  • HDFS
  • WebHDFS
  • HTTP
  • WebDAV
  • local remote
!dvc remote add -d my_local_remote /dvcstore
Setting 'my_local_remote' as a default remote.

!git status
On branch master
nothing to commit, working tree clean
!git add .dvc/config
!git commit -m "Added DVC remote"
On branch master
nothing to commit, working tree clean

dvc push

Kiedy mamy już skonfigurowany "remote" możemy wypchnąć do niego pliki korzystając z polecenia dvc push:

!dvc push
  0% Transferring|                                   |0/1 [00:00<?,     ?file/s]
!
  0%|          |7820ef0af287ff346c5cabfb4c612c     0.00/? [00:00<?,        ?B/s]
  0%|          |7820ef0af287ff346c5cabfb4c612c 0.00/4.99k [00:00<?,        ?B/s]
1 file pushed                                                                   

!tree /dvcstore
/dvcstore
└── 71
    └── 7820ef0af287ff346c5cabfb4c612c

1 directory, 1 file

dvc pull

Żeby pobrać dane z DVC (np. w innej lokalizacji, przez innego użytkownika), musimy:

  • sklonować repozytorium git (żeby m.in. pobrać pliki *.dvc
  • wykonać dvc pull

Dodawanie nowych plików i modyfikacja istniejących wygląda podobnie jak przy zwykłych plikach śledzonych przez git, tylko zamiast git używamy polecenia dvc a dodatkowo pamiętamy o zarządzaniu plikami *.dvc za pomocą gita:

!head -n -1 data/Iris.csv | sponge data/Iris.csv
!git status
On branch master
nothing to commit, working tree clean
!dvc status
data/Iris.csv.dvc:                                                    core>
	changed outs:
		modified:           data/Iris.csv

!dvc add data/Iris.csv
⠹ Checking graph                                                   ⠋ Checking graph
Adding...                                                                       
!
  0% Checking cache in '/home/tomek/repos/aitech-ium/IUM_10/sample-ml-project/.d
                                                                                
!
  0%|          |Transferring                          0/1 [00:00<?,     ?file/s]
                                                                                
!
  0%|          |.AquCc93WCb2aAJ98voTeFG.tmp        0.00/? [00:00<?,        ?B/s]
  0%|          |.AquCc93WCb2aAJ98voTeFG.tmp     0.00/4.00 [00:00<?,        ?B/s]
                                                                                
!
  0%|          |4705c4d470a4d9dd152808e5e9f56f     0.00/? [00:00<?,        ?B/s]
  0%|          |4705c4d470a4d9dd152808e5e9f56f 0.00/4.92k [00:00<?,        ?B/s]
100% Adding...|████████████████████████████████████████|1/1 [00:00, 11.09file/s]

To track the changes with git, run:

    git add data/Iris.csv.dvc

To enable auto staging, run:

	dvc config core.autostage true

!git add data/Iris.csv.dvc
!git commit -m "Removed last line from Iris dataset"
[master 5379e3b] Removed last line from Iris dataset
 1 file changed, 2 insertions(+), 2 deletions(-)

dvc checkout

  • Polecenia dvc checkout używamy razem z git checkout, żeby zmienić branch, na którym pracujemy.
  • DVC podmieni wersje plików śledzonych przez siebie na pochodzące z innego brancha (o ile pliki te się różnią i różnią się pliki *.dvc w odpowiednich branchach
  • zmiana brancha przez git powoduje (ewentualną) zmianę plików *.dvc a dvc checkout kopiuje/linkuje pliki z katalogu .dvc/cache o wartościach hash odpowiadających tym z plików *.dvc

Wymiana danych między projektami

  • za pomocą poleceń dvc import i dvc update możemy dodać i później aktualizować pliki śledzone przez DVC w innym repozytorium
!dvc import https://github.com/iterative/dataset-registry \
             get-started/data.xml -o data/data.xml
Importing 'get-started/data.xml (https://github.com/iterative/dataset-registry)' -> 'data/data.xml'
  0% Downloading|                                    |0/1 [00:00<?,     ?file/s]
!
  0%|          |get-started/data.xml           0.00/37.9M [00:00<?,       ?it/s]
  0%|          |get-started/data.xml      64.0k/36.1M [00:00<02:12,     286kB/s]
  0%|          |get-started/data.xml       128k/36.1M [00:00<01:33,     403kB/s]
  1%|          |get-started/data.xml       256k/36.1M [00:00<00:57,     658kB/s]
  1%|          |get-started/data.xml       384k/36.1M [00:00<00:45,     818kB/s]
  1%|▏         |get-started/data.xml       512k/36.1M [00:00<00:53,     693kB/s]
  2%|▏         |get-started/data.xml       640k/36.1M [00:01<00:57,     644kB/s]
  2%|▏         |get-started/data.xml       768k/36.1M [00:01<00:59,     619kB/s]
  2%|▏         |get-started/data.xml       896k/36.1M [00:01<00:51,     718kB/s]
  3%|▎         |get-started/data.xml      1.00M/36.1M [00:01<00:55,     666kB/s]
  3%|▎         |get-started/data.xml      1.12M/36.1M [00:01<00:57,     633kB/s]
  3%|▎         |get-started/data.xml      1.25M/36.1M [00:02<00:57,     638kB/s]
  4%|▍         |get-started/data.xml      1.38M/36.1M [00:02<00:52,     698kB/s]
  4%|▍         |get-started/data.xml      1.50M/36.1M [00:02<00:55,     656kB/s]
  4%|▍         |get-started/data.xml      1.62M/36.1M [00:02<00:57,     628kB/s]
  5%|▍         |get-started/data.xml      1.69M/36.1M [00:02<00:58,     618kB/s]
  5%|▌         |get-started/data.xml      1.81M/36.1M [00:02<00:53,     675kB/s]
  5%|▌         |get-started/data.xml      1.94M/36.1M [00:03<00:53,     672kB/s]
  6%|▌         |get-started/data.xml      2.06M/36.1M [00:03<00:55,     642kB/s]
  6%|▌         |get-started/data.xml      2.12M/36.1M [00:03<00:56,     628kB/s]
  6%|▌         |get-started/data.xml      2.19M/36.1M [00:03<00:57,     616kB/s]
  6%|▌         |get-started/data.xml      2.25M/36.1M [00:03<00:58,     606kB/s]
  7%|▋         |get-started/data.xml      2.38M/36.1M [00:03<00:48,     732kB/s]
  7%|▋         |get-started/data.xml      2.50M/36.1M [00:04<00:52,     666kB/s]
  7%|▋         |get-started/data.xml      2.62M/36.1M [00:04<00:55,     636kB/s]
  8%|▊         |get-started/data.xml      2.75M/36.1M [00:04<00:56,     614kB/s]
  8%|▊         |get-started/data.xml      2.88M/36.1M [00:04<00:49,     711kB/s]
  8%|▊         |get-started/data.xml      3.00M/36.1M [00:04<00:52,     663kB/s]
  9%|▊         |get-started/data.xml      3.12M/36.1M [00:05<00:54,     637kB/s]
  9%|▉         |get-started/data.xml      3.25M/36.1M [00:05<00:55,     623kB/s]
  9%|▉         |get-started/data.xml      3.38M/36.1M [00:05<00:48,     710kB/s]
 10%|▉         |get-started/data.xml      3.50M/36.1M [00:05<00:51,     664kB/s]
 10%|█         |get-started/data.xml      3.62M/36.1M [00:05<00:45,     751kB/s]
 10%|█         |get-started/data.xml      3.75M/36.1M [00:05<00:49,     691kB/s]
 11%|█         |get-started/data.xml      3.88M/36.1M [00:06<00:43,     777kB/s]
 11%|█         |get-started/data.xml      4.00M/36.1M [00:06<00:47,     705kB/s]
 11%|█▏        |get-started/data.xml      4.12M/36.1M [00:06<00:42,     790kB/s]
 12%|█▏        |get-started/data.xml      4.25M/36.1M [00:06<00:46,     716kB/s]
 12%|█▏        |get-started/data.xml      4.38M/36.1M [00:06<00:44,     749kB/s]
 12%|█▏        |get-started/data.xml      4.50M/36.1M [00:07<00:45,     734kB/s]
 13%|█▎        |get-started/data.xml      4.62M/36.1M [00:07<00:40,     810kB/s]
 13%|█▎        |get-started/data.xml      4.75M/36.1M [00:07<00:42,     773kB/s]
 13%|█▎        |get-started/data.xml      4.88M/36.1M [00:07<00:41,     795kB/s]
 14%|█▍        |get-started/data.xml      5.00M/36.1M [00:07<00:37,     870kB/s]
 14%|█▍        |get-started/data.xml      5.12M/36.1M [00:07<00:34,     932kB/s]
 15%|█▍        |get-started/data.xml      5.25M/36.1M [00:07<00:35,     916kB/s]
 15%|█▍        |get-started/data.xml      5.38M/36.1M [00:08<00:35,     898kB/s]
 15%|█▌        |get-started/data.xml      5.50M/36.1M [00:08<00:33,     962kB/s]
 16%|█▌        |get-started/data.xml      5.62M/36.1M [00:08<00:33,     949kB/s]
 16%|█▌        |get-started/data.xml      5.75M/36.1M [00:08<00:31,    1.00MB/s]
 16%|█▋        |get-started/data.xml      5.88M/36.1M [00:08<00:30,    1.04MB/s]
 17%|█▋        |get-started/data.xml      6.06M/36.1M [00:08<00:26,    1.19MB/s]
 17%|█▋        |get-started/data.xml      6.19M/36.1M [00:08<00:26,    1.19MB/s]
 17%|█▋        |get-started/data.xml      6.31M/36.1M [00:08<00:26,    1.19MB/s]
 18%|█▊        |get-started/data.xml      6.50M/36.1M [00:08<00:23,    1.31MB/s]
 18%|█▊        |get-started/data.xml      6.62M/36.1M [00:09<00:23,    1.30MB/s]
 19%|█▉        |get-started/data.xml      6.81M/36.1M [00:09<00:21,    1.41MB/s]
 19%|█▉        |get-started/data.xml      7.00M/36.1M [00:09<00:20,    1.48MB/s]
 20%|█▉        |get-started/data.xml      7.19M/36.1M [00:09<00:19,    1.54MB/s]
 20%|██        |get-started/data.xml      7.38M/36.1M [00:09<00:18,    1.60MB/s]
 21%|██        |get-started/data.xml      7.56M/36.1M [00:09<00:18,    1.62MB/s]
 21%|██▏       |get-started/data.xml      7.75M/36.1M [00:09<00:17,    1.68MB/s]
 22%|██▏       |get-started/data.xml      7.94M/36.1M [00:09<00:17,    1.70MB/s]
 22%|██▏       |get-started/data.xml      8.12M/36.1M [00:10<00:17,    1.72MB/s]
 23%|██▎       |get-started/data.xml      8.38M/36.1M [00:10<00:15,    1.88MB/s]
 24%|██▎       |get-started/data.xml      8.56M/36.1M [00:10<00:15,    1.84MB/s]
 24%|██▍       |get-started/data.xml      8.81M/36.1M [00:10<00:14,    1.96MB/s]
 25%|██▌       |get-started/data.xml      9.06M/36.1M [00:10<00:13,    2.06MB/s]
 26%|██▌       |get-started/data.xml      9.31M/36.1M [00:10<00:13,    2.14MB/s]
 27%|██▋       |get-started/data.xml      9.62M/36.1M [00:10<00:11,    2.32MB/s]
 27%|██▋       |get-started/data.xml      9.88M/36.1M [00:10<00:11,    2.33MB/s]
 28%|██▊       |get-started/data.xml      10.2M/36.1M [00:10<00:11,    2.46MB/s]
 29%|██▉       |get-started/data.xml      10.4M/36.1M [00:11<00:10,    2.45MB/s]
 30%|██▉       |get-started/data.xml      10.8M/36.1M [00:11<00:10,    2.57MB/s]
 31%|███       |get-started/data.xml      11.1M/36.1M [00:11<00:09,    2.67MB/s]
 32%|███▏      |get-started/data.xml      11.4M/36.1M [00:11<00:09,    2.84MB/s]
 33%|███▎      |get-started/data.xml      11.8M/36.1M [00:11<00:08,    2.85MB/s]
 34%|███▎      |get-started/data.xml      12.1M/36.1M [00:11<00:08,    3.01MB/s]
 35%|███▍      |get-started/data.xml      12.5M/36.1M [00:11<00:07,    3.12MB/s]
 36%|███▌      |get-started/data.xml      12.9M/36.1M [00:11<00:07,    3.22MB/s]
 37%|███▋      |get-started/data.xml      13.2M/36.1M [00:11<00:07,    3.31MB/s]
 38%|███▊      |get-started/data.xml      13.7M/36.1M [00:12<00:06,    3.49MB/s]
 39%|███▉      |get-started/data.xml      14.1M/36.1M [00:12<00:06,    3.62MB/s]
 40%|████      |get-started/data.xml      14.6M/36.1M [00:12<00:06,    3.74MB/s]
 42%|████▏     |get-started/data.xml      15.0M/36.1M [00:12<00:05,    3.82MB/s]
 43%|████▎     |get-started/data.xml      15.4M/36.1M [00:12<00:05,    3.97MB/s]
 44%|████▍     |get-started/data.xml      15.9M/36.1M [00:12<00:05,    4.08MB/s]
 45%|████▌     |get-started/data.xml      16.4M/36.1M [00:12<00:04,    4.23MB/s]
 47%|████▋     |get-started/data.xml      17.0M/36.1M [00:12<00:04,    4.44MB/s]
 48%|████▊     |get-started/data.xml      17.5M/36.1M [00:12<00:04,    4.52MB/s]
 50%|████▉     |get-started/data.xml      18.1M/36.1M [00:13<00:04,    4.69MB/s]
 52%|█████▏    |get-started/data.xml      18.6M/36.1M [00:13<00:03,    4.84MB/s]
 53%|█████▎    |get-started/data.xml      19.2M/36.1M [00:13<00:03,    5.05MB/s]
 55%|█████▍    |get-started/data.xml      19.8M/36.1M [00:13<00:03,    5.16MB/s]
 57%|█████▋    |get-started/data.xml      20.4M/36.1M [00:13<00:03,    5.35MB/s]
 58%|█████▊    |get-started/data.xml      21.1M/36.1M [00:13<00:02,    5.49MB/s]
 60%|██████    |get-started/data.xml      21.8M/36.1M [00:13<00:02,    5.66MB/s]
 62%|██████▏   |get-started/data.xml      22.4M/36.1M [00:13<00:02,    5.83MB/s]
 64%|██████▍   |get-started/data.xml      23.2M/36.1M [00:14<00:02,    6.05MB/s]
 66%|██████▌   |get-started/data.xml      23.9M/36.1M [00:14<00:02,    6.20MB/s]
 68%|██████▊   |get-started/data.xml      24.6M/36.1M [00:14<00:01,    6.40MB/s]
 70%|███████   |get-started/data.xml      25.4M/36.1M [00:14<00:01,    6.51MB/s]
 72%|███████▏  |get-started/data.xml      26.0M/36.1M [00:14<00:01,    5.75MB/s]
 74%|███████▎  |get-started/data.xml      26.6M/36.1M [00:14<00:02,    4.26MB/s]
 75%|███████▍  |get-started/data.xml      27.1M/36.1M [00:14<00:02,    3.53MB/s]
 76%|███████▌  |get-started/data.xml      27.5M/36.1M [00:15<00:02,    3.26MB/s]
 77%|███████▋  |get-started/data.xml      27.9M/36.1M [00:15<00:02,    3.00MB/s]
 78%|███████▊  |get-started/data.xml      28.2M/36.1M [00:15<00:02,    2.95MB/s]
 79%|███████▉  |get-started/data.xml      28.5M/36.1M [00:15<00:02,    2.91MB/s]
 80%|███████▉  |get-started/data.xml      28.8M/36.1M [00:15<00:02,    2.88MB/s]
 81%|████████  |get-started/data.xml      29.1M/36.1M [00:15<00:02,    2.86MB/s]
 81%|████████▏ |get-started/data.xml      29.4M/36.1M [00:15<00:02,    2.84MB/s]
 82%|████████▏ |get-started/data.xml      29.8M/36.1M [00:16<00:02,    2.83MB/s]
 83%|████████▎ |get-started/data.xml      30.1M/36.1M [00:16<00:02,    2.83MB/s]
 84%|████████▍ |get-started/data.xml      30.4M/36.1M [00:16<00:02,    2.83MB/s]
 85%|████████▍ |get-started/data.xml      30.7M/36.1M [00:16<00:02,    2.83MB/s]
 86%|████████▌ |get-started/data.xml      31.0M/36.1M [00:16<00:01,    2.83MB/s]
 87%|████████▋ |get-started/data.xml      31.3M/36.1M [00:16<00:01,    2.83MB/s]
 88%|████████▊ |get-started/data.xml      31.6M/36.1M [00:16<00:01,    2.83MB/s]
 88%|████████▊ |get-started/data.xml      31.9M/36.1M [00:16<00:01,    2.84MB/s]
 89%|████████▉ |get-started/data.xml      32.2M/36.1M [00:16<00:01,    2.85MB/s]
 90%|█████████ |get-started/data.xml      32.6M/36.1M [00:17<00:01,    2.85MB/s]
 91%|█████████ |get-started/data.xml      32.9M/36.1M [00:17<00:01,    2.86MB/s]
 92%|█████████▏|get-started/data.xml      33.2M/36.1M [00:17<00:01,    2.86MB/s]
 93%|█████████▎|get-started/data.xml      33.5M/36.1M [00:17<00:00,    2.87MB/s]
 94%|█████████▎|get-started/data.xml      33.8M/36.1M [00:17<00:00,    2.87MB/s]
 94%|█████████▍|get-started/data.xml      34.1M/36.1M [00:17<00:00,    2.87MB/s]
 95%|█████████▌|get-started/data.xml      34.4M/36.1M [00:17<00:00,    2.87MB/s]
 96%|█████████▌|get-started/data.xml      34.8M/36.1M [00:17<00:00,    2.87MB/s]
 97%|█████████▋|get-started/data.xml      35.1M/36.1M [00:17<00:00,    2.87MB/s]
 98%|█████████▊|get-started/data.xml      35.4M/36.1M [00:18<00:00,    2.87MB/s]
 99%|█████████▉|get-started/data.xml      35.7M/36.1M [00:18<00:00,    2.88MB/s]
100%|█████████▉|get-started/data.xml      36.0M/36.1M [00:18<00:00,    2.87MB/s]
                                                                                
To track the changes with git, run:

	git add data/.gitignore data/data.xml.dvc

!dvc status
Data and pipelines are up to date.                                              

ls -l data
total 37020
-rw-rw-r-- 1 tomek tomek 37891850 maj 31 11:10 data.xml
-rw-rw-r-- 1 tomek tomek      284 maj 31 11:10 data.xml.dvc
-rw-rw-r-- 1 tomek tomek     5072 maj 31 11:01 Iris.csv
-rw-rw-r-- 1 tomek tomek       76 maj 31 11:01 Iris.csv.dvc
# %load data/data.xml.dvc
md5: a7cd139231cc35ed63541ce3829b96db
frozen: true
deps:
- path: get-started/data.xml
  repo:
    url: https://github.com/iterative/dataset-registry
    rev_lock: ba014f40e29670421a67cb1c47543f402348aa13
outs:
- md5: a304afb96060aad90176268345e10355
  size: 37891850
  path: data.xml

DVC pipelines

  • wprowadzenie: https://youtu.be/71IGzyH95UY
  • Getting started: https://dvc.org/doc/start/data-pipelines
  • dvc pipelines pozwala nam zbudować (za pomocą polecenie dvc run) lub zdefiniować (edytując plik dvc.yaml) graf zależności między krokami wykonywanymi w naszym projekcie (takimi jak "przygotowanie danych", "trenowanie", "ewaluacja")
  • tak zdefiniowany pipeline można potem uruchomić za pomocą polecenia dvc reproduce

Zadania [15pkt]

  1. Zainicjalizuj repozytorium DVC wewnątrz Twojego repozytorium z projektem [1pkt]
  2. Dodaj plik(i) z danymi w Twoim projekcie do DVC [1pkt]
  3. Skonfiguruj remote (dane do konfiguracji podane poniżej) [3pkt]
  4. Stwórz/zdefiniuj i dodaj do repozytorium plik dvc.yaml opisujący kroki wykonywane w Twoim projekcie. Wydziel przynajmniej 2 kroki (np. przygotowanie danych/trenowanie) powiązane ze sobą za pomocą zależności (skorzystaj z materiałów "Getting started", link powyżej) [6pkt]
  5. Stwórz projekt na Jenkinsie (s1233456-dvc), w którym sklonujesz repozytorium, ściągniesz pliki dvc (za pomocą dvc pull) i uruchomisz pipeline (za pomocą dvc reproduce) [4pkt]

SSH remote

Jednym z remote obsługiwanych przez DVC jest SFTP/SSH. W celu jego wykorzystania na serwerze tzietkiewicz.vm.wmi.amu.edu.pl utworzony został użytkownik ium-sftp i skonfigurowany serwer SFTP. Został też dla niego wygenerowany klucz ssh, który został dodany jako "Jenkins credential" (patrz opis konfiguracji na Jenkins poniżej)

Lokalnie

Będziemy potrzebować zależności (szczegóły)

conda install dvc-ssh

albo

pip install dvc[ssh] paramiko

conda install -c conda-forge dvc-ssh
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/tomek/miniconda3

  added / updated specs:
    - dvc-ssh


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    bcrypt-3.2.0               |   py39h3811e60_1          44 KB  conda-forge
    ca-certificates-2021.5.30  |       ha878542_0         136 KB  conda-forge
    certifi-2021.5.30          |   py39hf3d152e_0         141 KB  conda-forge
    dvc-2.3.0                  |   py39hf3d152e_0         542 KB  conda-forge
    dvc-ssh-2.3.0              |   py39hf3d152e_0           9 KB  conda-forge
    fsspec-2021.5.0            |     pyhd8ed1ab_0          77 KB  conda-forge
    invoke-1.5.0               |     pyhd3deb0d_0         137 KB  conda-forge
    paramiko-2.7.2             |     pyh9f0ad1d_0         135 KB  conda-forge
    pynacl-1.4.0               |   py39h3811e60_2         1.3 MB  conda-forge
    ------------------------------------------------------------
                                           Total:         2.5 MB

The following NEW packages will be INSTALLED:

  bcrypt             conda-forge/linux-64::bcrypt-3.2.0-py39h3811e60_1
  dvc-ssh            conda-forge/linux-64::dvc-ssh-2.3.0-py39hf3d152e_0
  invoke             conda-forge/noarch::invoke-1.5.0-pyhd3deb0d_0
  paramiko           conda-forge/noarch::paramiko-2.7.2-pyh9f0ad1d_0
  pynacl             conda-forge/linux-64::pynacl-1.4.0-py39h3811e60_2

The following packages will be UPDATED:

  ca-certificates                      2020.12.5-ha878542_0 --> 2021.5.30-ha878542_0
  certifi                          2020.12.5-py39hf3d152e_1 --> 2021.5.30-py39hf3d152e_0
  dvc                                  2.1.0-py39hf3d152e_0 --> 2.3.0-py39hf3d152e_0
  fsspec                                 0.9.0-pyhd8ed1ab_2 --> 2021.5.0-pyhd8ed1ab_0



Downloading and Extracting Packages
certifi-2021.5.30    | 141 KB    | ##################################### | 100% 
fsspec-2021.5.0      | 77 KB     | ##################################### | 100% 
dvc-2.3.0            | 542 KB    | ##################################### | 100% 
invoke-1.5.0         | 137 KB    | ##################################### | 100% 
paramiko-2.7.2       | 135 KB    | ##################################### | 100% 
bcrypt-3.2.0         | 44 KB     | ##################################### | 100% 
pynacl-1.4.0         | 1.3 MB    | ##################################### | 100% 
dvc-ssh-2.3.0        | 9 KB      | ##################################### | 100% 
ca-certificates-2021 | 136 KB    | ##################################### | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

Note: you may need to restart the kernel to use updated packages.
!dvc remote add -f -d ium_ssh_remote ssh://ium-sftp@tzietkiewicz.vm.wmi.amu.edu.pl
Setting 'ium_ssh_remote' as a default remote.

!dvc remote list
ium_ssh_remote	ssh://ium-sftp@tzietkiewicz.vm.wmi.amu.edu.pl


Zapisujemy hasło:

!dvc remote modify --local ium_ssh_remote password IUM@2021

!dvc push
  0% Transferring|                                   |0/1 [00:00<?,     ?file/s]
!
  0%|          |4705c4d470a4d9dd152808e5e9f56f     0.00/? [00:00<?,        ?B/s]
  0%|          |4705c4d470a4d9dd152808e5e9f56f 0.00/4.92k [00:00<?,        ?B/s]
1 file pushed                                                                   


Jenkins

W Jenkins można użyć mechanizmu "Credentials", żeby w bezpieczny sposób przekazać hasło albo klucz prywatny.

Takie dane dla użytkownika ium-sftp zostały stworzone na Jenkinsie:

Opis używania "Credentials" w Jenkinsfile: https://www.jenkins.io/doc/book/pipeline/jenkinsfile/#for-other-credential-types

Klucza ssh można użyć tak:

withCredentials(
    [sshUserPrivateKey(credentialsId: '48ac7004-216e-4260-abba-1fe5db753e18', keyFileVariable: 'IUM_SFTP_KEY', passphraseVariable: '', usernameVariable: '')]) {
                sh 'dvc remote add -d ium_ssh_remote ssh://ium-sftp@tzietkiewicz.vm.wmi.amu.edu.pl/ium-sftp'
                sh 'dvc remote modify --local ium_ssh_remote keyfile $IUM_SFTP_KEY'
                sh 'dvc pull'}

Secret text tak:

    withCredentials([string(credentialsId: 'ium-sftp-password', variable: 'IUM_SFTP_PASS')]) {
                sh 'dvc remote add -d ium_ssh_remote ssh://ium-sftp@tzietkiewicz.vm.wmi.amu.edu.pl/ium-sftp'
                sh 'dvc remote modify --local ium_ssh_remote password $IUM_SFTP_KEY'
                sh 'dvc pull'
    }

Przykład kongiguracji:

!dvc pull
Everything is up to date.                                                       

!git remote add origin git@git.wmi.amu.edu.pl:tzietkiewicz/sample-ml-project.git
!git push --set-upstream origin master
error: remote origin already exists.
Enumerating objects: 24, done.
Counting objects: 100% (24/24), done.
Delta compression using up to 4 threads
Compressing objects: 100% (21/21), done.
Writing objects: 100% (24/24), 2.10 KiB | 715.00 KiB/s, done.
Total 24 (delta 4), reused 0 (delta 0), pack-reused 0
remote: . Processing 1 references
remote: Processed 1 references in total
To git.wmi.amu.edu.pl:tzietkiewicz/sample-ml-project.git
 * [new branch]      master -> master
Branch 'master' set up to track remote branch 'master' from 'origin'.